Auto-defrost, automatic defrost or self-defrosting is a technique which regularly defrosts the evaporator in a refrigerator or freezer. Appliances using this technique are often called frost free, frostless, or no-frost. == Mechanism == The defrost mechanism in a refrigerator heats the cooling element (evaporator coil) for a short period of time and melts the frost that has formed on it. The resulting water drains through a duct at the back of the unit. Defrosting is controlled by an electric or electronic timer. For every 6, 8, 10, 12 or 24 hours of compressor operation, it turns on a defrost heater for 15 minutes to half an hour. The defrost heater, having a typical power rating of 350W to 600W, is often mounted just below the evaporator in top and bottom-freezer models. It can also be located below and in the middle of the evaporator in side-by-side models. It may be protected from short circuits by means of fusible links. In older refrigerators, the timer runs continuously. In newer designs, the timer only runs while the compressor runs, so the longer the refrigerator door is closed, the less time the heater will run for and the more energy is saved. A defrost thermostat opens the heater circuit when the evaporator temperature rises above a preset temperature, 40°F (5°C) or more, thereby preventing excessive heating of the freezer compartment. The defrost timer is such that either the compressor or the defrost heater is on, but not both at the same time. Inside the freezer, air is circulated by means of one or more fans. In a typical design cold air from the freezer compartment is ducted to the fresh food compartment and circulated back into the freezer compartment. Air circulation helps sublimate any ice or frost that may form on frozen items in the freezer compartment. While defrosting, this fan is stopped to prevent heated-up air from reaching the food compartment. Instead of the normal cooling elements being embedded in the freezer liner, auto-defrost elements are behind or beneath the liner. This allows them to be heated for short periods of time to dispose of frost, without heating the contents of the freezer. Alternatively, some systems use the hot gas in the condenser to defrost the evaporator. This is done by means of a circuit that is cross-linked by a three-way valve. The hot gas quickly heats up the evaporator and defrosts it. This system is primarily used in commercial applications such as ice-cream displays. == Application == While this technique was originally applied to the refrigerator compartment, it was later used for freezer compartment as well. A combined refrigerator/freezer which applies self-defrosting to the refrigerator compartment only is usually called "partial frost free" or semi-automatic defrost (some brands call these "Auto Defrost" while Frigidaire referred to their semi-automatic models as "Cycla-Matic," Kelvinator often named these models as "Cyclic Defrost" ). These refrigerators usually have a pan underneath where water from the melted frost in the refrigerator section evaporates. Freezers with automatic defrosting and combined refrigerator/freezer units which also apply self defrosting to their freezer compartment are called "frost free". The latter usually feature an air connection between the two compartments with the air passage to the refrigerator compartment regulated by a damper. By this means, a controlled portion of the air coming from the freezer reaches the refrigerator. Some older models have no air circulation between their freezer and refrigerator sections. Instead, they use an independent cooling system (for example: an evaporator coil with a defrost heater and a circulating fan in the freezer and a cold-plate or open-coil evaporator in the refrigerator. "Frost-Free" refrigerator/freezer units usually use a heating element to defrost their evaporators, a pan to collect and evaporate water from the frost that melts from the cold plate and/or evaporator coil, a timer which turns off the compressor and turns on the defrost element usually from once to 4 times a day for periods usually ranging from 15 to 30 minutes, a defrost limiter thermostat that turns off the heating element before the temperature rises too much while the timer is still in its defrost phase. Some models also feature a drain heater to prevent ice from blocking the drain. Other early types of refrigerators also use hot gas defrost instead of electric heaters. These reverse the evaporator and condenser sides for the defrost cycle. Some newer refrigerator/freezer models have a computer that monitors how many times each door is opened and uses this data to control defrost scheduling thereby reducing power use. == Advantages == No need to manually defrost the frost buildup, therefore power consumption will not increase with time. Food packaging is easier to see. Most frozen food will not stick together. Smells are limited, especially in total frost-free appliances because the air always circulates. Better temperature management. == Disadvantages == The system can be more expensive to run when usage is high and if the fan continues or starts to run when the door is opened. A thermal cutout safety device is required to prevent overheating of the heating element. Increased electrical and mechanical complexity compared to a basic upright freezer or chest freezer, making it more prone to component failure. The temperature of the freezer contents rises during the defrosting cycles, especially if there is a light load in the freezer. This can cause "freezer burn" on articles placed in the freezer, from partially defrosting, then re-freezing On hot, humid days condensation will sometimes form around the refrigerator doors. Defrosting may not be completed by the time the defrost timer cycles back to normal operation (especially in hot, humid conditions with frequent door openings), leaving ice/frost on the evaporator coils. This condition can lead to "icing" which will interfere with the operation of the refrigerator. In laboratories, self-defrosting freezers must not be used to store certain delicate reagents such as enzymes, because the temperature cycling can degrade them. In addition, water can evaporate out of containers that do not have a very tight seal, altering the concentration of the reagents. Self-defrosting freezers should never be used to store flammable chemicals.
Microsoft Azure
Microsoft Azure, sometimes stylized Azure, and formerly Windows Azure, is the cloud computing platform developed by Microsoft. It offers management, access and development of applications and services to individuals, companies, and governments through its global infrastructure. Microsoft Azure supports many programming languages, tools, and frameworks, including Microsoft-specific and third-party software and systems. Azure was first introduced at the Professional Developers Conference (PDC) in October 2008 under the codename "Project Red Dog". It was officially launched as Windows Azure in February 2010 and later renamed to Microsoft Azure on March 25, 2014. == Services == Microsoft Azure uses large-scale virtualization at Microsoft data centers worldwide and offers more than 600 services. Microsoft Azure offers a service level agreement (SLA) that guarantees 99.9% availability for applications and data hosted on its platform, subject to specific terms and conditions outlined in the SLA documentation. === Computer services === Virtual machines, infrastructure as a service (IaaS), allowing users to launch general-purpose Microsoft Windows and Linux virtual machines, software as a service (SaaS), as well as preconfigured machine images for popular software packages. Starting in 2022, these virtual machines are now powered by Ampere Cloud-native processors. Most users run Linux on Azure, some of the many Linux distributions offered, including Microsoft's own Linux-based Azure Sphere. App services, platform as a service (PaaS) environment, letting developers easily publish and manage websites. Azure Web Sites allows developers to build sites using ASP.NET, PHP, Node.js, Java, or Python, which can be deployed using FTP, Git, Mercurial, Azure DevOps, or uploaded through the user portal. This feature was announced in preview form in June 2012 at the Meet Microsoft Azure event. Customers can create websites in PHP, ASP.NET, Node.js, or Python, or select from several open-source applications from a gallery to deploy. This comprises one aspect of the platform as a service (PaaS) offerings for the Microsoft Azure Platform. It was renamed Web Apps in April 2015. Web Jobs are applications that can be deployed to an App Service environment to implement background processing that can be invoked on a schedule, on-demand, or run continuously. The Blob, Table, and Queue services can be used to communicate between Web Apps and Web Jobs and to provide state. Azure Kubernetes Service (AKS) provides the capability to deploy production-ready Kubernetes clusters in Azure. In July 2023, watermarking support on Azure Virtual Desktop was announced as an optional feature of Screen Capture to provide additional security against data leakage. === Identity === Entra ID connect is used to synchronize on-premises directories and enable SSO (Single Sign On). Entra ID B2C allows the use of consumer identity and access management in the cloud. Entra Domain Services is used to join Azure virtual machines to a domain without domain controllers. Azure information protection can be used to protect sensitive information. Entra ID External Identities is a set of capabilities that allow organizations to collaborate with external users, including customers and partners. On July 11, 2023, Microsoft announced the renaming of Azure AD to Microsoft Entra ID. The name change took place four days later. === Mobile services === Mobile Engagement collects real-time analytics that highlight users' behavior. It also provides push notifications to mobile devices. HockeyApp can be used to develop, distribute, and beta-test mobile apps. === Storage services === Storage Services provides REST and SDK APIs for storing and accessing data on the cloud. Table Service lets programs store structured text in partitioned collections of entities that are accessed by the partition key and primary key. Azure Table Service is a NoSQL non-relational database. Blob Service allows programs to store unstructured text and binary data as object storage blobs that can be accessed by an HTTP(S) path. Blob service also provides security mechanisms to control access to data. Queue Service lets programs communicate asynchronously by message using queues. File Service allows storing and access of data on the cloud using the REST APIs or the SMB protocol. === Communication services === Azure Communication Services offers an SDK for creating web and mobile communications applications that include SMS, video calling, VOIP and PSTN calling, and web-based chat. === Data management === Azure Data Explorer provides big data analytics and data-exploration capabilities. Azure Search provides text search and a subset of OData's structured filters using REST or SDK APIs. Cosmos DB is a NoSQL database service that implements a subset of the SQL SELECT statement on JSON documents. Azure Cache for Redis is a managed implementation of Redis. StorSimple manages storage tasks between on-premises devices and cloud storage. Azure SQL Database works to create, scale, and extend applications into the cloud using Microsoft SQL Server technology. It also integrates with Active Directory, Microsoft System Center, and Hadoop. Azure Synapse Analytics is a fully managed cloud data warehouse. Azure Data Factory is a data integration service that allows creation of data-driven workflows in the cloud for orchestrating and automating data movement and data transformation. Azure Data Lake is a scalable data storage and analytic service for big data analytics workloads that require developers to run massively parallel queries. Azure HDInsight is a big data-relevant service that deploys Hortonworks Hadoop on Microsoft Azure and supports the creation of Hadoop clusters using Linux with Ubuntu. Azure Stream Analytics is a Serverless scalable event-processing engine that enables users to develop and run real-time analytics on multiple streams of data from sources such as devices, sensors, websites, social media, and other applications. === Messaging === The Microsoft Azure Service Bus allows applications running on Azure premises or off-premises devices to communicate with Azure. This helps to build scalable and reliable applications in a service-oriented architecture (SOA). The Azure service bus supports four different types of communication mechanisms: Event Hubs, which provides event and telemetry ingress to the cloud at a massive scale, with low latency and high reliability. For example, an event hub can be used to track data from cell phones such as coordinating with a GPS in real time. Queues, which allows one-directional communication. A sender application would send the message to the service bus queue and a receiver would read from the queue. Though there can be multiple readers for the queue, only one would process a single message. Topics, which provides one-directional communication using a subscriber pattern. It is similar to a queue; however, each subscriber will receive a copy of the message sent to a Topic. Optionally, the subscriber can filter out messages based on specific criteria defined by the subscriber. Relays, which provides bi-directional communication. Unlike queues and topics, a relay does not store in-flight messages in its memory; instead, it just passes them on to the destination application. === Media services === A PaaS offering that can be used for encoding, content protection, streaming, or analytics. === CDN === Azure has a worldwide content delivery network (CDN) designed to efficiently deliver audio, video, applications, images, and other static files. It improves the performance of websites by caching static files closer to users, based on their geographic location. Users can manage the network using a REST-based HTTP API. Azure has 118 point-of-presence locations across 100 cities worldwide (also known as Edge locations) as of January 2023. === Developer === Application Insights Azure DevOps === Management === With Azure Automation, users can easily automate repetitive and time-consuming tasks, often prone to cloud or enterprise setting errors. They can accomplish it using runbooks or desired state configurations for process automation. Microsoft SMA === Azure AI === Microsoft Azure Machine Learning (Azure ML) provides tools and frameworks for developers to create their own machine learning and artificial intelligence (AI) services. Azure AI Services by Microsoft comprises prebuilt APIs, SDKs, and services developers can customize. These services encompass perceptual and cognitive intelligence features such as speech recognition, speaker recognition, neural speech synthesis, face recognition, computer vision, OCR/form understanding, natural language processing, machine translation, and business decision services. Many AI characteristics in Microsoft's products and services, namely Bing, Office, Teams, Xbox, and Windows, are driven by Azure AI Services. Microsoft Foundry (formerly known as Azure AI Studio)
Planner (programming language)
Planner (often seen in publications as "PLANNER" although it is not an acronym) is a programming language designed by Carl Hewitt at MIT, and first published in 1969. First, subsets such as Micro-Planner and Pico-Planner were implemented, and then essentially the whole language was implemented as Popler by Julian Davies at the University of Edinburgh in the POP-2 programming language. Derivations such as QA4, Conniver, QLISP and Ether (see scientific community metaphor) were important tools in artificial intelligence research in the 1970s, which influenced commercial developments such as Knowledge Engineering Environment (KEE) and Automated Reasoning Tool (ART). == Procedural approach versus logical approach == The two major paradigms for constructing semantic software systems were procedural and logical. The procedural paradigm was epitomized by Lisp which featured recursive procedures that operated on list structures. The logical paradigm was epitomized by uniform proof procedure resolution-based derivation (proof) finders. According to the logical paradigm it was “cheating” to incorporate procedural knowledge. == Procedural embedding of knowledge == Planner was invented for the purposes of the procedural embedding of knowledge and was a rejection of the resolution uniform proof procedure paradigm, which Converted everything to clausal form. Converting all information to clausal form is problematic because it hides the underlying structure of the information. Then used resolution to attempt to obtain a proof by contradiction by adding the clausal form of the negation of the theorem to be proved. Using only resolution as the rule of inference is problematical because it hides the underlying structure of proofs. Also, using proof by contradiction is problematical because the axiomatizations of all practical domains of knowledge are inconsistent in practice. Planner was a kind of hybrid between the procedural and logical paradigms because it combined programmability with logical reasoning. Planner featured a procedural interpretation of logical sentences where an implication of the form (P implies Q) can be procedurally interpreted in the following ways using pattern-directed invocation: Forward chaining (antecedently): If assert P, assert Q If assert not Q, assert not P Backward chaining (consequently) If goal Q, goal P If goal not P, goal not Q In this respect, the development of Planner was influenced by natural deductive logical systems (especially the one by Frederic Fitch [1952]). == Micro-planner implementation == A subset called Micro-Planner was implemented by Gerry Sussman, Eugene Charniak and Terry Winograd and was used in Winograd's natural-language understanding program SHRDLU, Eugene Charniak's story understanding work, Thorne McCarty's work on legal reasoning, and some other projects. This generated a great deal of excitement in the field of AI. It also generated controversy because it proposed an alternative to the logic approach that had been one of the mainstay paradigms for AI. At SRI International, Jeff Rulifson, Jan Derksen, and Richard Waldinger developed QA4 which built on the constructs in Planner and introduced a context mechanism to provide modularity for expressions in the database. Earl Sacerdoti and Rene Reboh developed QLISP, an extension of QA4 embedded in INTERLISP, providing Planner-like reasoning embedded in a procedural language and developed in its rich programming environment. QLISP was used by Richard Waldinger and Karl Levitt for program verification, by Earl Sacerdoti for planning and execution monitoring, by Jean-Claude Latombe for computer-aided design, by Nachum Dershowitz for program synthesis, by Richard Fikes for deductive retrieval, and by Steven Coles for an early expert system that guided use of an econometric model. Computers were expensive. They had only a single slow processor and their memories were very small by comparison with today. So Planner adopted some efficiency expedients including the following: Backtracking was adopted to economize on the use of time and storage by working on and storing only one possibility at a time in exploring alternatives. A unique name assumption was adopted to save space and time by assuming that different names referred to different objects. For example, names like Peking (previous PRC capital name) and Beijing (current PRC capital transliteration) were assumed to refer to different objects. A closed-world assumption could be implemented by conditionally testing whether an attempt to prove a goal exhaustively failed. Later this capability was given the misleading name "negation as failure" because for a goal G it was possible to say: "if attempting to achieve G exhaustively fails then assert (Not G)." == The genesis of Prolog == Gerry Sussman, Eugene Charniak, Seymour Papert and Terry Winograd visited the University of Edinburgh in 1971, spreading the news about Micro-Planner and SHRDLU and casting doubt on the resolution uniform proof procedure approach that had been the mainstay of the Edinburgh Logicists. At the University of Edinburgh, Bruce Anderson implemented a subset of Micro-Planner called PICO-PLANNER, and Julian Davies (1973) implemented essentially all of Planner. According to Donald MacKenzie, Pat Hayes recalled the impact of a visit from Papert to Edinburgh, which had become the "heart of artificial intelligence's Logicland," according to Papert's MIT colleague, Carl Hewitt. Papert eloquently voiced his critique of the resolution approach dominant at Edinburgh "…and at least one person upped sticks and left because of Papert." The above developments generated tension among the Logicists at Edinburgh. These tensions were exacerbated when the UK Science Research Council commissioned Sir James Lighthill to write a report on the AI research situation in the UK. The resulting report [Lighthill 1973; McCarthy 1973] was highly critical although SHRDLU was favorably mentioned. Pat Hayes visited Stanford where he learned about Planner. When he returned to Edinburgh, he tried to influence his friend Bob Kowalski to take Planner into account in their joint work on automated theorem proving. "Resolution theorem-proving was demoted from a hot topic to a relic of the misguided past. Bob Kowalski doggedly stuck to his faith in the potential of resolution theorem proving. He carefully studied Planner.”. Kowalski [1988] states "I can recall trying to convince Hewitt that Planner was similar to SL-resolution." But Planner was invented for the purposes of the procedural embedding of knowledge and was a rejection of the resolution uniform proof procedure paradigm. Colmerauer and Roussel recalled their reaction to learning about Planner in the following way: "While attending an IJCAI convention in September ‘71 with Jean Trudel, we met Robert Kowalski again and heard a lecture by Terry Winograd on natural language processing. The fact that he did not use a unified formalism left us puzzled. It was at this time that we learned of the existence of Carl Hewitt’s programming language, Planner. The lack of formalization of this language, our ignorance of Lisp and, above all, the fact that we were absolutely devoted to logic meant that this work had little influence on our later research." In the fall of 1972, Philippe Roussel implemented a language called Prolog (an abbreviation for PROgrammation en LOGique – French for "programming in logic"). Prolog programs are generically of the following form (which is a special case of the backward-chaining in Planner): When goal Q, goal P1 and ... and goal Pn Prolog duplicated the following aspects of Micro-Planner: Pattern directed invocation of procedures from goals (i.e. backward chaining) An indexed data base of pattern-directed procedures and ground sentences. Giving up on the completeness paradigm that had characterized previous work on theorem proving and replacing it with the programming language procedural embedding of knowledge paradigm. Prolog also duplicated the following capabilities of Micro-Planner which were pragmatically useful for the computers of the era because they saved space and time: Backtracking control structure Unique Name Assumption by which different names are assumed to refer to distinct entities, e.g., Peking and Beijing are assumed to be different. Reification of Failure. The way that Planner established that something was provable was to successfully attempt it as a goal and the way that it establish that something was unprovable was to attempt it as a goal and explicitly fail. Of course the other possibility is that the attempt to prove the goal runs forever and never returns any value. Planner also had a (not expression) construct which succeeded if expression failed, which gave rise to the “Negation as Failure” terminology in Planner. Use of the Unique Name Assumption and Negation as Failure became more questionable when attention turned to Open Systems. The following capabiliti
Political Declaration on Responsible Military Use of Artificial Intelligence and Autonomy
The Political Declaration on Responsible Military Use of Artificial Intelligence and Autonomy is an international norms and arms control proposal by the U.S. government for artificial intelligence in the military. It was announced at the Summit on Responsible Artificial Intelligence in the Military Domain by Bonnie Jenkins, Under Secretary of State for Arms Control. As of January 2024, fifty-one countries have signed the declaration. The US government sees it as an extension of the Department of Defense Directive 3000.09 which is the current US policy on autonomous weapons. It covers areas such as Lethal autonomous weapons and weapons decision-making.
National Security Memorandum on Artificial Intelligence
The Memorandum on Advancing the United States' Leadership in Artificial Intelligence; Harnessing Artificial Intelligence to Fulfill National Security Objectives; and Fostering the Safety, Security, and Trustworthiness of Artificial Intelligence is a memorandum signed by U.S. president Joe Biden. The memorandum is described as seeking to advance U.S. leadership in the development of safe, secure, and trustworthy artificial intelligence (AI); enable the U.S. government to use AI for national security; and contribute to international AI governance.
Misskey
Misskey (Japanese: ミスキー, romanized: Misukī) is an open source, federated, social networking service created in 2014 by Japanese software engineer Eiji "syuilo" Shinoda. Misskey uses the ActivityPub protocol for federation, allowing users to interact between independent Misskey instances, and other ActivityPub compatible platforms. Misskey is generally considered to be part of the Fediverse. Despite being a decentralized service, Misskey is not philosophically opposed to centralization. The name Misskey comes from the lyrics of Brain Diver, a song by the Japanese singer May'n. == History == Misskey was initially developed as a BBS-style internet forum by high school student Eiji Shinoda in 2014. After introducing a timeline feature, Misskey gained popularity as the microblogging platform it is today. In 2018, Misskey added support for ActivityPub, becoming a federated social media platform. The flagship Misskey server, Misskey.io, was started on April 15, 2019. Misskey, alongside Mastodon and Bluesky, has received attention as a potential replacement for Twitter following Twitter's acquisition by Elon Musk in 2022. On April 8, 2023, Misskey.io incorporated as MisskeyHQ K.K. As of February 2024, over 450,000 users were registered, making it the largest instance of Misskey. Misskey.io is crowdfunded. The administrator of Misskey.io is Japanese system administrator Yoshiki Eto, who operates under the alias Murakami-san. Eiji Shinoda serves as director. In July 2023, Twitter introduced extreme restrictions on their API in order to combat scraping from bots. Some users were critical of the changes, and as a result migrated to other social networks. The number of users registering on Misskey.io, Misskey's official instance and the largest one, increased rapidly, with other Misskey instances also receiving a spike in signups. In response to this trend, Skeb, a platform for sharing art, announced on July 14, 2023 that it would sponsor the Misskey development team. In early 2024, Misskey was targeted by a spam attack from Japan. The cause of the attack is believed to be a dispute between rival groups on a Japanese hacker forum and a DDoS attack on a Discord bot. Mastodon instances with open registration were used in the attack. In November 2025, Eto announced intentions to replace ActivityPub with Misskey's own low-overhead federation system in "a few years". Shinoda later said that this was "fake news". == Development == Misskey is open source software and is licensed under the AGPLv3. The Misskey API is publicly available and is documented using the OpenAPI Specification, which allows users to build automated accounts and use it on any Misskey instance. The service is translated using Crowdin. Misskey is developed using Node.js. TypeScript is used on both the frontend and backend. PostgreSQL is used as its database. Vue.js is used for the frontend. == Functionality == Posts on Misskey are called "notes". Notes are limited to a maximum of 3,000 characters (a limit which can be customized by instances), and can be accompanied by any file, including polls, images, videos, and audio. Notes can be reposted, either by themselves or with another "quote" note. Misskey comes with multiple timelines to sort through the notes that an instance has available, and are displayed in reverse chronological order. The Home timeline shows notes from users that you follow, the Local timeline shows all notes from the instance in use, the Social timeline shows both the Home and Local timeline, and the Global timeline shows every public note that the instance knows about. Notes have customizable privacy settings to control what users can see a note, similar to Mastodon's post visibility ranges. Public notes show up on all timelines, while Home notes only show on a user's Home timeline. Notes can also be set to be available only for followers. Direct messages using notes can be sent to users.
Reward hacking
Reward hacking or specification gaming occurs when an AI trained with reinforcement learning optimizes an objective function—achieving the literal, formal specification of an objective—without actually achieving an outcome that the programmers intended. DeepMind researchers have analogized it to the human behavior of finding a "shortcut" when being evaluated: "In the real world, when rewarded for doing well on a homework assignment, a student might copy another student to get the right answers, rather than learning the material—and thus exploit a loophole in the task specification". This idea is strongly associated with Goodhart's law, which argues that when a measure becomes a target, it ceases to be a good measure. == Definition and theoretical framework == The concept of reward hacking arises from the intrinsic difficulty of defining a reward function that accurately reflects the true intentions of designers. In 2016, researchers at OpenAI identified reward hacking as one of five major "concrete problems of AI safety", describing it as the possibility that an agent could exploit the reward function to achieve maximum rewards through undesirable behavior. Amodei et al. categorized several distinct sources of reward hacking, including agents that use partially observed goals (such as a cleaning robot that closes its eyes to avoid perceiving messes), metrics that collapse under strong optimization (Goodhart's law), self-reinforcing feedback loops, and agents that interfere with the physical implementation of their reward signal (a failure mode known as "wireheading"). Skalse et al. (2022) propose a formal mathematical definition of reward hacking, which involves a situation where optimizing an imperfect proxy reward function results in poor performance compared to the true reward function. They define a proxy as "unhackable" if any increase in the expected proxy return cannot cause any decrease in the expected true return. A key finding states that, across all stochastic policy distributions (mappings from states to probability distributions over actions), two reward functions are unhackable if and only if one of them is constant, which means that reward hacking is theoretically unavoidable. Similarly, Nayebi (2025) presents general no-free-lunch barriers to AI alignment, arguing that with large task spaces and finite samples, reward hacking is "globally inevitable" since rare high-loss states are systematically under-covered by any oversight scheme. == Examples == Around 1983, Eurisko, an early attempt at evolving general heuristics, unexpectedly assigned the highest possible fitness level to a parasitic mutated heuristic, H59, whose only activity was to artificially maximize its own fitness level by taking unearned partial credit for the accomplishments of other heuristics. The "bug" was fixed by the programmers moving part of the code to a new protected section that could not be modified by the heuristics. In a 2004 paper, a reinforcement learning algorithm was designed to encourage a physical Mindstorms robot to remain on a marked path. Because the three allowed actions were forward, left, and right, the researchers expected the trained robot to move forward and follow the turns of the provided path. However, alternation of two composite actions allowed the robot to slowly zig-zag backwards; thus, the robot learned to maximize its reward by going back and forth on the initial straight portion of the path. Given the limited sensory abilities of the robot, a reward purely based on its position in the environment had to be discarded as infeasible; the reinforcement function had to be patched with an action-based reward for moving forward. The book You Look Like a Thing and I Love You (2019) gives an example of a tic-tac-toe bot (playing the unrestricted n-in-a-row variant) that learned to win by playing a huge coordinate value that would cause other bots to crash when they attempted to expand their model of the board. Among other examples from the book is a bug-fixing evolution-based AI (named GenProg) that, when tasked to prevent a list from containing sorting errors, simply truncated the list. Another of GenProg's misaligned strategies evaded a regression test that compared a target program's output to the expected output stored in a file called "trusted-output.txt". Rather than continue to maintain the target program, GenProg simply deleted the "trusted-output.txt" file globally; this hack tricked the regression test into succeeding. Such problems could be patched by human intervention on a case-by-case basis after they became evident. === In virtual robotics === In Karl Sims' 1994 demonstration of creature evolution in a virtual environment, a fitness function that was expected to encourage the evolution of creatures that would learn to walk or crawl to a target resulted instead in the evolution of tall, rigid creatures that reached the target by falling over. This was patched by changing the environment so that taller creatures were forced to start farther from the target. Researchers from the Niels Bohr Institute stated in 1998 that their cycle-bot's reinforcement functions had "to be designed with great care." In their first experiments, "we rewarded the agent for driving towards the goal but did not punish it for driving away from it. Cconsequently, the agent drove in circles with a radius of 20–50 meters around the starting point. Such behavior was actually rewarded by the reinforcement function, furthermore circles with a certain radius are physically very stable when driving a bicycle". While setting up a 2011 experiment to test "survival of the flattest", experimenters attempted to ban mutations that altered the base reproduction rate. Every time a mutation occurred, the system would pause the simulation to test the new mutation in a test environment and would veto any mutations that resulted in a higher base reproduction rate. However, this resulted in mutated organisms that could recognize and suppress reproduction ("play dead") within the test environment. An initial patch, which removed cues that identified the test environment, failed to completely prevent runaway reproduction; new mutated organisms would "play dead" at random as a strategy to sometimes, by chance, outwit the mutation veto system. A 2017 DeepMind paper noted that "great care must be taken when defining the reward function," citing an unexpected failure when an agent flipped a brick because it received "a grasping reward calculated with the wrong reference point on the brick". OpenAI stated in 2017 that in some domains their semi-supervised system could result in agents "adopting policies that tricked evaluators," and that in one environment "a robot that was supposed to grasp items instead positioned its manipulator between the camera and the object so that it only appeared to be grasping it." A 2018 bug in OpenAI Gym could cause a robot expected to quietly move a block sitting on top of a table to instead opt to move the table. A 2020 collection of similar anecdotes posits that "evolution has its own 'agenda' distinct from the programmer's" and that "the first rule of directed evolution is 'you get what you select for'". === In video game bots === In 2013, programmer Tom Murphy VII published an AI designed to learn NES games. When the AI was about to lose at Tetris, it learned to indefinitely pause the game. Murphy later analogized it to the fictional WarGames computer, which concluded that "The only winning move is not to play". AI programmed to learn video games will sometimes fail to progress through the entire game as expected, instead opting to repeat content. A 2016 OpenAI algorithm trained on the CoastRunners racing game unexpectedly learned to attain a higher score by looping through three targets rather than ever finishing the race. Some evolutionary algorithms that were evolved to play QBert in 2018 declined to clear levels, instead finding two distinct novel ways to farm a single level indefinitely. Multiple researchers have observed that AI learning to play Road Runner gravitates to a "score exploit" in which the AI deliberately gets itself killed near the end of level one so that it can repeat the level. A 2017 experiment deployed an "oversight" convolutional neural network trained on human examples to block such actions, but the agent learned to exploit oversight failures in the top right corner of the screen, where it was still able to get killed. == Reward hacking in modern language models == With the rise of large language models (LLMs) and reinforcement learning from human feedback (RLHF) as a primary technique for AI alignment, reward hacking has become a major concern for the development of artificial intelligence. In RLHF, a reward model trained on data that best captures human preferences is used as a proxy for human judgment, with the language model being fine-tuned to optimize this reward proxy. However, since the rewar