ParkMobile is a mobile and web app providing parking payments in North America. Headquartered in Atlanta, Georgia, users can pay for on-street and off-street parking via app on their smartphone, web browser, or through calling a phone number. ParkMobile also offers parking reservations at stadiums or venues for concerts and sporting events, and in metro area garages. == History == ParkMobile was founded in the United States in 2008 by Albert Bogaard after originally starting in the Netherlands. The initial product served only zone (on-demand) parkers and payment for the parking spot was made via a phone call through an IVR system. In 2009, the ParkMobile app was released and the product launched in its first city, Grand Rapids, Michigan. Parking payments have since been accepted through a user's account by connecting a credit card. ParkMobile deployed in Washington, D.C., in 2011. As of 2023, ParkMobile now has over 50 million users. Parking reservations were introduced in 2017, allowing users to reserve parking in advance. In 2018, the company recapitalized with BMW as the shareholder. ParkMobile was then acquired by a joint venture with BMW and Daimler. Under this joint venture, ParkMobile parking payment functionality was available and integrated with BMW's navigation system in many of its 2018 models. EasyPark Group, the Swedish-based parking solutions company, acquired ParkMobile in 2021 and is the current owner rebranded as Arrive. In 2022, ParkMobile launched in the City of Boston with a city-wide parking app, ParkBoston, powered by ParkMobile. == Operations == === Products === ParkMobile's product offerings include zone (on-demand) parking payments, parking reservations, and a self-service reporting engine. Zone parking is the company's most widely used service. Users can use the app on their smartphone to pay parking fees. In 2017, ParkMobile began offering parking reservations. The service is provided in addition to on-demand parking options at stadiums and venues, as well as metro area parking garages. After launching the reservations feature, ParkMobile became the first mobile parking app provider in North America to have a consolidated app with both on-demand and reservations parking in one. ParkMobile 360, the company's self-service management and reporting platform for operators, launched in 2018. It is a web-based application for parking operators to manage parking inventory, adjust rates, create special parking events, and track analytics. In 2020, ParkMobile began offering an option to pay for parking with Google through integrating the ParkMobile experience with Google Maps In 2021, ParkMobile launched its web application, allowing users to complete their parking transactions directly from the mobile website without having to download the app or have an account. ParkMobile integrates with parking gate equipment so customers can use their app to pay for parking and scan to enter and exit the garage. === Locations === ParkMobile has over 50 million users across the United States, Canada, and Puerto Rico. The app is available in over 550 cities in the U.S. and over 150 colleges and universities. == Controversies == === Predatory towing and excessive ticketing === Since all paid parking sessions from a single supplier are able to be viewed together, the ease of viewing and enforcing parking violations has caused controversy. Parking Enforcement Services in Birmingham, Alabama, has been the subject complaints by users of the ParkMobile app who had paid for a parking session and still had their vehicle towed. Customers often use old or expired license plates and forget to update to the correct number, or mistype when entering their information into the ParkMobile app. The complaints are that the towing companies offer no lenience for these mistakes. They return to their car as the session expires, and find their car has been towed. Additionally, other municipality across the country have received complaints about excessive parking ticket issuing when inputting their information incorrectly in the ParkMobile app. In Stone Harbor, New Jersey, parking ticket violations increased by over 1,600% from the previous year since launching with the ParkMobile app. Police officers refute complaints of being "too strict" on writing tickets by admitting the ParkMobile system allows officers to "more seamlessly enforce" the city's parking laws. === Data security breach === In March 2021, ParkMobile suffered a cybersecurity incident "linked to a vulnerability in a third-party software," potentially exposing users' email addresses, phone numbers, and license plate numbers. ParkMobile responded by launching an investigation and notifying law enforcement authorities and affected municipalities. The investigation concluded "no sensitive data or Payment Card Information was affected" but ParkMobile confirmed that basic account information, such as license plate numbers and possibly email addresses or phone numbers, was accessed.
Cognition Network Technology
Cognition Network Technology (CNT), also known as Definiens Cognition Network Technology, is an object-based image analysis method developed by Nobel laureate Gerd Binnig together with a team of researchers at Definiens AG in Munich, Germany. It serves for extracting information from images using a hierarchy of image objects (groups of pixels), as opposed to traditional pixel processing methods. To emulate the human mind's cognitive powers, Definiens used patented image segmentation and classification processes, and developed a method to render knowledge in a semantic network. CNT examines pixels not in isolation, but in context. It builds up a picture iteratively, recognizing groups of pixels as objects. It uses the color, shape, texture and size of objects as well as their context and relationships to draw conclusions and inferences, similar to human analysis. == History == In 1994 Professor Gerd Binnig founded Definiens. CNT was first available with the launch of the eCognition software in May 2000. In June 2010, Trimble Navigation Ltd (NASDAQ: TRMB) acquired Definiens business asset in earth sciences markets, including eCognition software, and also licensed Definiens' patented CNT. In 2014, Definiens was acquired by MedImmune, the global biologics research and development arm of AstraZeneca, for an initial consideration of $150 million. == Software == Definiens Tissue Studio Definiens Tissue Studio is a digital pathology image analysis software application based on CNT. The intended use of Definiens Tissue Studio is for biomarker translational research in formalin-fixed, paraffin-embedded tissue samples which have been treated with immunohistochemical staining assays, or hematoxylin and eosin (H&E). The central concept behind Definiens Tissue Studio is a user interface that facilitates machine learning from example digital histopathology images to derive an image analysis solution suitable for the measurement of biomarkers and/or histological features within pre-defined regions of interest on a cell-by-cell basis, and within sub-cellular compartments. The derived image analysis solution is then automatically applied to subsequent digital images to objectively measure defined sets of multiparametric image features. These data sets are used for further understanding the underlying biological processes that drive cancer and other diseases. Image processing and data analysis are performed either on a local desktop computer workstation, or on a server grid. eCognition The eCognition suite offers three components that can be used stand-alone or in combination to solve image analysis tasks. eCognition Developer is a development environment for object-based image analysis. It is used in earth sciences to develop rule sets (or applications) for the analysis of remote sensing data. eCognition Architect enables non-technical users to configure, calibrate and execute image analysis workflows created in eCognition Developer. eCognition Server software provides a processing environment for batch execution of image analysis jobs. eCognition software is utilized in numerous remote sensing and geospatial application scenarios and environments, using a variety of data types: Generic: Rapid Mapping, Change Detection, Object Recognition By environment: Diverse Landcover Mapping, Urban Analysis (i.e. impervious surface area analysis for taxation, property assessment for insurance, inventory of green infrastructure), Forestry (i.e. biomass measurement, species identification, firescar measurement), Agriculture (i.e. regional planning, precision farming, crisis response), Marine and Riparian (i.e. ecosystem evaluation, disaster management, harbor monitoring). Other: Defense, security, atmosphere and climate The online eCognition community was launched in July 2009 and had 2813 members as of July 9, 2010. Membership is distributed globally and user conferences are held regularly, the last having taken place in November 2009 in Munich, Germany. The bi-annual GEOBIA (Geographic Object-Based Image Analysis) conference is heavily attended by eCognition users, with the majority of presentations based on eCognition software.
Semantic network
A semantic network, or frame network is a knowledge base that represents semantic relations between concepts in a network. This is often used as a form of knowledge representation. It is a directed or undirected graph consisting of vertices, which represent concepts, and edges, which represent semantic relations between concepts, mapping or connecting semantic fields. A semantic network may be instantiated as, for example, a graph database or a concept map. Typical standardized semantic networks are expressed as semantic triples. Semantic networks are used in natural language processing applications such as semantic parsing and word-sense disambiguation. Semantic networks can also be used as a method to analyze large texts and identify the main themes and topics (e.g., of social media posts), to reveal biases (e.g., in news coverage), or even to map an entire research field. == History == Examples of the use of semantic networks in logic, directed acyclic graphs as a mnemonic tool, dates back centuries. The earliest documented use being the Greek philosopher Porphyry's commentary on Aristotle's categories in the third century AD. In computing history, "Semantic Nets" for the propositional calculus were first implemented for computers by Richard H. Richens of the Cambridge Language Research Unit in 1956 as an "interlingua" for machine translation of natural languages. Although the importance of this work and the CLRU was only belatedly realized. Semantic networks were also independently implemented by Robert F. Simmons and Sheldon Klein, using the first order predicate calculus as a base, after being inspired by a demonstration of Victor Yngve. The "line of research was originated by the first President of the Association [Association for Computational Linguistics], Victor Yngve, who in 1960 had published descriptions of algorithms for using a phrase structure grammar to generate syntactically well-formed nonsense sentences. Sheldon Klein and I about 1962-1964 were fascinated by the technique and generalized it to a method for controlling the sense of what was generated by respecting the semantic dependencies of words as they occurred in text." Other researchers, most notably M. Ross Quillian and others at System Development Corporation helped contribute to their work in the early 1960s as part of the SYNTHEX project. It's from these publications at SDC that most modern derivatives of the term "semantic network" cite as their background. Later prominent works were done by Allan M. Collins and Quillian (e.g., Collins and Quillian; Collins and Loftus Quillian). Still later in 2006, Hermann Helbig fully described MultiNet. In the late 1980s, two Netherlands universities, Groningen and Twente, jointly began a project called Knowledge Graphs, which are semantic networks but with the added constraint that edges are restricted to be from a limited set of possible relations, to facilitate algebras on the graph. In the subsequent decades, the distinction between semantic networks and knowledge graphs was blurred. In 2012, Google gave their knowledge graph the name Knowledge Graph. The Semantic Link Network was systematically studied as a social semantics networking method. Its basic model consists of semantic nodes, semantic links between nodes, and a semantic space that defines the semantics of nodes and links and reasoning rules on semantic links. The systematic theory and model was published in 2004. This research direction can trace to the definition of inheritance rules for efficient model retrieval in 1998 and the Active Document Framework ADF. Since 2003, research has developed toward social semantic networking. This work is a systematic innovation at the age of the World Wide Web and global social networking rather than an application or simple extension of the Semantic Net (Network). Its purpose and scope are different from that of the Semantic Net (or network). The rules for reasoning and evolution and automatic discovery of implicit links play an important role in the Semantic Link Network. Recently it has been developed to support Cyber-Physical-Social Intelligence. It was used for creating a general summarization method. The self-organised Semantic Link Network was integrated with a multi-dimensional category space to form a semantic space to support advanced applications with multi-dimensional abstractions and self-organised semantic links It has been verified that Semantic Link Network play an important role in understanding and representation through text summarisation applications. Semantic Link Network has been extended from cyberspace to cyber-physical-social space. Competition relation and symbiosis relation as well as their roles in evolving society were studied in the emerging topic: Cyber-Physical-Social Intelligence More specialized forms of semantic networks has been created for specific use. For example, in 2008, Fawsy Bendeck's PhD thesis formalized the Semantic Similarity Network (SSN) that contains specialized relationships and propagation algorithms to simplify the semantic similarity representation and calculations. == Basics of semantic networks == A semantic network is used when one has knowledge that is best understood as a set of concepts that are related to one another. Most semantic networks are cognitively based. They also consist of arcs and nodes which can be organized into a taxonomic hierarchy. Semantic networks contributed ideas of spreading activation, inheritance, and nodes as proto-objects. == Examples == === In Lisp === The following code shows an example of a semantic network in the Lisp programming language using an association list. To extract all the information about the "canary" type, one would use the assoc function with a key of "canary". === WordNet === An example of a semantic network is WordNet, a lexical database of English. It groups English words into sets of synonyms called synsets, provides short, general definitions, and records the various semantic relations between these synonym sets. Some of the most common semantic relations defined are meronymy (A is a meronym of B if A is part of B), holonymy (B is a holonym of A if B contains A), hyponymy (or troponymy) (A is subordinate of B; A is kind of B), hypernymy (A is superordinate of B), synonymy (A denotes the same as B) and antonymy (A denotes the opposite of B). WordNet properties have been studied from a network theory perspective and compared to other semantic networks created from Roget's Thesaurus and word association tasks. From this perspective the three of them are a small world structure. === Other examples === It is also possible to represent logical descriptions using semantic networks such as the existential graphs of Charles Sanders Peirce or the related conceptual graphs of John F. Sowa. These have expressive power equal to or exceeding standard first-order predicate logic. Unlike WordNet or other lexical or browsing networks, semantic networks using these representations can be used for reliable automated logical deduction. Some automated reasoners exploit the graph-theoretic features of the networks during processing. Other examples of semantic networks are Gellish models. Gellish English with its Gellish English dictionary, is a formal language that is defined as a network of relations between concepts and names of concepts. Gellish English is a formal subset of natural English, just as Gellish Dutch is a formal subset of Dutch, whereas multiple languages share the same concepts. Other Gellish networks consist of knowledge models and information models that are expressed in the Gellish language. A Gellish network is a network of (binary) relations between things. Each relation in the network is an expression of a fact that is classified by a relation type. Each relation type itself is a concept that is defined in the Gellish language dictionary. Each related thing is either a concept or an individual thing that is classified by a concept. The definitions of concepts are created in the form of definition models (definition networks) that together form a Gellish Dictionary. A Gellish network can be documented in a Gellish database and is computer interpretable. SciCrunch is a collaboratively edited knowledge base for scientific resources. It provides unambiguous identifiers (Research Resource IDentifiers or RRIDs) for software, lab tools etc. and it also provides options to create links between RRIDs and from communities. Another example of semantic networks, based on category theory, is ologs. Here each type is an object, representing a set of things, and each arrow is a morphism, representing a function. Commutative diagrams also are prescribed to constrain the semantics. In the social sciences people sometimes use the term semantic network to refer to co-occurrence networks. == Software tools == There are also elaborate types of semantic networks connected with corresponding sets of software tools used for
Lukas Biewald
Lukas Biewald (born 1981) is an American entrepreneur and a prominent figure in artificial intelligence. He is recognized for his contributions to machine learning and as the CEO and co-founder of Weights & Biases, a company that builds developer tools for AI, that sold to CoreWeave in 2025 for $1.7B. He previously founded and was CEO of Figure Eight, a human-in-the-loop machine learning platform. He has co-authored 26 AI research papers from 2004 through 2018. == Early life and education == Biewald was born in Boston, Massachusetts in 1981. He attended Cambridge Rindge and Latin School and later earned both a Bachelor's and Master's degree in Computer science from Stanford University. == Early Career and Founding Figure Eight == After graduation, Biewald joined Yahoo! as an engineer, working on machine translations to improve search results, and eventually led the Search Relevance Team for Yahoo! Japan. He later joined Powerset, a natural language search technology company, as their Senior Scientist, which was acquired by Microsoft in 2008 for an estimated $100M. In 2007, Biewald co-founded Figure Eight (formerly CrowdFlower), a data labeling and crowdsourcing company that created datasets for training machine learning models. Figure Eight was acquired by Appen in 2019 for $300 million. == Weights and Biases == In 2017, Biewald co-founded Weights & Biases with Chris Van Pelt and Shawn Lewis. The company provides tools for tracking machine learning experiments, model management, and collaborative AI and LLM app development. The platform has been adopted by organizations such as OpenAI, Salesforce, and Microsoft. In March 2025 Coreweave acquired Weights and Biases at $1.7 billion, with the transaction closing on May 5, 2025. == Gradient Dissent == Biewald hosts the bi-weekly podcast Gradient Dissent. Guest have included: Anthony Goldbloom – Co-founder & CEO of Kaggle. “How to Win Kaggle Competitions” (podcast, Sep. 9, 2020). Shared tips on data-science competitions from the founder of the largest ML community. Richard Socher – Founder & CEO of You.com; former Chief Scientist at Salesforce. “The Challenges of Making ML Work in the Real World” (podcast, September 28, 2020). A leading NLP researcher, he spoke on multimodal search engines powered by large language models. Jensen Huang – Founder & CEO of NVIDIA. “NVIDIA’s CEO on the Next Generation of AI and MLOps” (podcast, March 3, 2022). Huang’s GPUs power modern ML research and production. Emad Mostaque – Co-founder & CEO of Stability AI. “Stable Diffusion, Stability AI, and What’s Next” (podcast, Nov. 15, 2022). Leads the company behind Stable Diffusion, which helped spark the generative-AI imaging boom. Drago Anguelov – Head of Research at Waymo. “Robustness, Safety, and Scalability at Waymo” (podcast, July 14, 2022). Covered Waymo’s self-driving AI advances and deployment challenges. Jeremy Howard – Co-founder of fast.ai. “The Simple but Profound Insight Behind Diffusion” (podcast, Jan. 5, 2023). Known for democratizing deep-learning education; discussed diffusion models and accessible AI tooling. Aidan Gomez – Co-founder & CEO of Cohere. “Scaling LLMs and Accelerating Adoption” (podcast, April 20, 2023). Co-author of “Attention Is All You Need,” he shared how Cohere delivers large-scale NLP models as a service. Chelsea Finn – Stanford Assistant Professor (AI & Robotics). “Shaping the World of Robotics with Chelsea Finn” (podcast, February 15, 2024). A pioneer in meta-learning and robotics, she detailed robots learning complex tasks like cooking. Andrew Feldman – Co-founder & CEO of Cerebras Systems. "Launching the Fastest AI Inference Solution" (podcast, August 27, 2024). Described wafer-scale AI chips achieving new training performance records. Thomas Dohmke – CEO of GitHub. “GitHub CEO on Copilot and the Future of Software Development” (podcast, June 10, 2025). Discussed building Copilot and the future of AI-assisted coding. Martin Shkreli – Founder of Godel Terminal. “From Pharma to AGI Hype, and Developing AI in Finance: Martin Shkreli’s Journey” (podcast, May 20, 2025). Shkreli reflects on his pharma controversies, prison experience, and his new AI-driven trading platform. Jarek Kutylowski – Founder & CEO of DeepL. “How DeepL Built a Translation Powerhouse with AI” (podcast, July 8, 2025). Shared how DeepL’s neural-MT rivals Google Translate through model and infrastructure innovation. == Awards and recognition == In 2010, Lukas Biewald won the Netexplorateur Award for creating the GiveWork iPhone app, which allows users to perform small tasks that assist refugees and people in developing countries. In 2010, Inc Magazine included Biewald and Van Pelt on its list of the Top 30 Entrepreneurs Under 30. == Publications == Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution by John Le, Andy Edmonds, Vaughn Hester, Lukas Biewald. SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation, July 2010. Superficial Data Analysis: Exploring Millions of Social Stereotypes by Lukas Biewald, Brendan O’Connor. O’Reilly July 2009 Biewald has co-authored 26 AI research papers from 2004 through 2018.
Leading the Future
Leading the Future is an American super PAC network focused on lobbying for policies friendly to the artificial intelligence industry. It was launched in 2025 with over $100 million from industry stakeholders including Andreessen Horowitz, OpenAI President Greg Brockman and Palantir co-founder Joe Lonsdale. The launch was preceded by talks between Collin McCune, head of government affairs at Andreessen Horowitz, and Chris Lehane, chief global affairs officer at OpenAI. Among the members of the network are the American Mission PAC, which supported Chris Gober, and the Think Big PAC, which targeted Alex Bores. Leading the Future is affiliated with the nonprofit Build American AI, which Axios describes as a dark money advocacy "offshoot" operating alongside the super PAC. NBC News states that the network’s efforts are modeled after the pro-cryptocurrency group Fairshake. Leading the Future is led by Zac Moffatt and Josh Vlasto, the latter of whom previously served as an advisor to Fairshake. In response to the creation of Leading the Future, former members of Congress Brad Carson and Chris Stewart co-founded the super PAC network Public First, aiming to counter the group’s influence. In April 2026, an investigation by Model Republic linked Leading the Future to The Wire By Acutus, an automated news website that allegedly used AI agents posing as human journalists to solicit interviews. The site's content was found to closely mirror the PAC's deregulatory policy goals while targeting researchers and advocates skeptical of rapid AI development. In May 2026, Wired revealed that Build American AI used a "dark money" campaign to pay TikTok and Instagram influencers $5,000 per video to promote scripted narratives framing Chinese AI as a "national security threat." According to internal documents and staff at the marketing agency managing the project, the campaign's explicit goal was to "subtly shift public debate" toward the deregulation of AI industries while intentionally avoiding technical discussions regarding AI quality or safety. During the 2026 primary season Leading the Future went on to endorse several candidates in both Democratic and Republican races with several of them going on to win.
Stochastic parrot
In machine learning, the term stochastic parrot is a metaphor that frames large language models as systems that statistically mimic text without real understanding. The word "stochastic" – from the ancient Greek "στοχαστικός" (stokhastikos, 'based on guesswork') – is a term from probability theory meaning "randomly determined". The word "parrot" refers to parrots' ability to mimic human speech. The term was introduced in a 2021 paper on AI ethics titled "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜" and authored by Timnit Gebru, Emily M. Bender, Angelina McMillan-Major, and Margaret Mitchell. The paper outlined possible risks associated with large language models (LLMs). In December 2020, it was the subject of a workplace dispute between Gebru (then co-leader of Google's Ethical Artificial Intelligence Team) and Google, which had requested the retraction of the paper. The incident culminated in Gebru's controversial departure from the company. The paper was later presented at the 2021 ACM Conference, and the term "stochastic parrot" has seen widespread use in academic research concerning generative AI and LLMs. The term has been interpreted negatively as an insult towards AI. == Background == Timnit Gebru is an AI ethics researcher, Emily M. Bender is a linguist specializing in computational linguistics, and Margaret Mitchell is a computer scientist specializing in algorithmic bias. Gebru had joined Google in 2018, where she co-led a team on the ethics of artificial intelligence with Mitchell. In late 2020, the paper "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜" was co-written by Gebru and five other researchers, four of whom were Google employees. The paper argues that large language models (LLMs) present significant risks such as environmental and financial costs, inscrutability leading to unknown dangerous biases, and potential for deception as LLMs do not understand the concepts underlying what they learn. The paper states that LLMs are "stitching together sequences of linguistic forms ... observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning." Therefore, they are labeled "stochastic parrots". === Dismissal of Gebru by Google === After the paper was submitted for consideration to the 2021 ACM Conference, Google requested that Gebru either retract the paper from the conference or remove the names of Google employees from it. Gebru refused to do so without further discussion, and emailed Google Research vice president Megan Kacholia that if the company could not explain the request for retraction and address other concerns regarding similar projects, she would plan to resign after a transition period, stating that they could "work on a last date". The following day, on December 2, 2020, Gebru received an email saying that Google was "accepting her resignation". Her abrupt firing sparked protests by Google employees and negative publicity for the company. == Usage == The phrase has been used by AI skeptics to signify that LLMs lack understanding of the meaning of their outputs. Sam Altman, CEO of OpenAI, used the term shortly after the release of ChatGPT in December 2022, tweeting "i am a stochastic parrot, and so r u". The term was nominated as the 2023 AI-related Word of the Year by the American Dialect Society. == Debate == Some LLMs, such as ChatGPT, have become capable of interacting with users in convincingly human-like conversations. The development of these new systems has deepened the discussion of the extent to which LLMs understand or are simply "parroting". According to machine learning researchers Lindholm, Wahlström, Lindsten, and Schön, the term "stochastic parrot" highlights two vital limitations of LLMs: LLMs are limited by the data they are trained on and are simply stochastically repeating contents of datasets. Because they are just making up outputs based on training data, LLMs do not understand if they are saying something incorrect or inappropriate. Lindholm et al. noted that, with poor quality datasets and other limitations, a learning machine might produce results that are "dangerously wrong". === Subjective experience === In the mind of a human being, words and language correspond to things one has experienced. For LLMs, according to proponents of the theory, words correspond only to other words and patterns of usage fed into their training data. Proponents of the idea of stochastic parrots thus conclude that statements about LLMs are due to "the human tendency to attribute meaning to text", and claim this occurs despite the LLMs not actually understanding language. === Fine-tuning === Kelsey Piper argued that the claim that LLMs are stochastic parrots or mere "next-token predictors" focuses on pre-training, ignoring that modern LLMs are also fine-tuned to follow instructions and to prefer accurate answers. === Hallucinations and mistakes === The tendency of LLMs to pass off false information as fact is held as support. Called hallucinations or confabulations, LLMs will occasionally synthesize information that matches some pattern. LLMs may fail to distinguish fact and fiction, which leads to the claim that they can't connect words to a comprehension of the world, as humans do. Furthermore, LLMs may fail to decipher complex or ambiguous grammar cases that rely on understanding the meaning of language. For example: The wet newspaper that fell down off the table is my favorite newspaper. But now that my favorite newspaper fired the editor I might not like reading it anymore. Can I replace 'my favorite newspaper' by 'the wet newspaper that fell down off the table' in the second sentence? GPT-4, an LLM released in March 2023, responded yes, not understanding that the meaning of "newspaper" is different in these two contexts; it is first an object and second an institution. === Benchmarks and experiments === One argument against the hypothesis that LLMs are stochastic parrot is their results on benchmarks for reasoning, common sense and language understanding. In 2023, some LLMs have shown good results on many language understanding tests, such as the Super General Language Understanding Evaluation (SuperGLUE). GPT-4 scored in the >90th-percentile on the Uniform Bar Examination and achieved 93% accuracy on the MATH benchmark of high-school Olympiad problems, results that exceed rote pattern-matching expectations. Such tests, and the smoothness of many LLM responses, help as many as 51% of AI professionals believe they can truly understand language with enough data, according to a 2022 survey. === Expert rebuttals === Some AI researchers dispute the notion that LLMs merely "parrot" their training data. Geoffrey Hinton, a pioneering figure in neural networks, counters that the metaphor misunderstands the prerequisite for accurate language prediction. He argues that "to predict the next word accurately, you have to understand the sentence", a view he presented on 60 Minutes in 2023. From this perspective, understanding is not an alternative to statistical prediction, but rather an emergent property required to perform it effectively at scale. Hinton also uses logical puzzles to demonstrate that LLMs actually understand language. A 2024 Scientific American investigation described a closed Berkeley workshop where state-of-the-art models solved novel tier-4 mathematics problems and produced coherent proofs, indicating reasoning abilities beyond memorization. The GPT-4 Technical Report showed human-level results on professional and academic exams (e.g., the Uniform Bar Exam and USMLE), challenging the "parrot" characterization. Anthropic conducted mechanistic interpretability research on Claude, using attribution graphs to identify circuits. The research showed how the LLM processes information via chains of fuzzy logical inference, and indicated an ability to plan ahead. They found that Claude 3.5 Haiku "employs remarkably general abstractions", forms "internally generated plans for its future outputs" and "works backwards from its longer-term goals". They noted that "The mechanisms of the model can apparently only be faithfully described using an overwhelmingly large causal graph." They also found that the model includes "mechanisms that could underlie a simple form of metacognition", in that it "thinks about" the level of its own knowledge before reaching its answer. === Interpretability === Another line of evidence against the 'stochastic parrot' claim comes from mechanistic interpretability, a research field dedicated to reverse-engineering LLMs to understand their internal workings. Rather than only observing the model's input-output behavior, these techniques probe the model's internal activations, which can be used to determine if they contain structured representations of the world. The goal is to investigate whether LLMs are merely manipulating surface statistics or if t
Capsule neural network
A capsule neural network (CapsNet) is a machine learning system that is a type of artificial neural network (ANN) that can be used to better model hierarchical relationships. The approach is an attempt to more closely mimic biological neural organization. The idea is to add structures called "capsules" to a convolutional neural network (CNN), and to reuse output from several of those capsules to form more stable (with respect to various perturbations) representations for higher capsules. The output is a vector consisting of the probability of an observation, and a pose for that observation. This vector is similar to what is done for example when doing classification with localization in CNNs. Among other benefits, capsnets address the "Picasso problem" in image recognition: images that have all the right parts but that are not in the correct spatial relationship (e.g., in a "face", the positions of the mouth and one eye are switched). For image recognition, capsnets exploit the fact that while viewpoint changes have nonlinear effects at the pixel level, they have linear effects at the part/object level. This can be compared to inverting the rendering of an object of multiple parts. == History == In 2000, Geoffrey Hinton et al. described an imaging system that combined segmentation and recognition into a single inference process using parse trees. So-called credibility networks described the joint distribution over the latent variables and over the possible parse trees. That system proved useful on the MNIST handwritten digit database. A dynamic routing mechanism for capsule networks was introduced by Hinton and his team in 2017. The approach was claimed to reduce error rates on MNIST and to reduce training set sizes. Results were claimed to be considerably better than a CNN on highly overlapped digits. In Hinton's original idea one minicolumn would represent and detect one multidimensional entity. == Transformations == An invariant is an object property that does not change as a result of some transformation. For example, the area of a circle does not change if the circle is shifted to the left. Informally, an equivariant is a property that changes predictably under transformation. For example, the center of a circle moves by the same amount as the circle when shifted. A nonequivariant is a property whose value does not change predictably under a transformation. For example, transforming a circle into an ellipse means that its perimeter can no longer be computed as π times the diameter. In computer vision, the class of an object is expected to be an invariant over many transformations. I.e., a cat is still a cat if it is shifted, turned upside down or shrunken in size. However, many other properties are instead equivariant. The volume of a cat changes when it is scaled. Equivariant properties such as a spatial relationship are captured in a pose, data that describes an object's translation, rotation, scale and reflection. Translation is a change in location in one or more dimensions. Rotation is a change in orientation. Scale is a change in size. Reflection is a mirror image. Unsupervised capsnets learn a global linear manifold between an object and its pose as a matrix of weights. In other words, capsnets can identify an object independent of its pose, rather than having to learn to recognize the object while including its spatial relationships as part of the object. In capsnets, the pose can incorporate properties other than spatial relationships, e.g., color (cats can be of various colors). Multiplying the object by the manifold poses the object (for an object, in space). == Pooling == Capsnets reject the pooling layer strategy of conventional CNNs that reduces the amount of detail to be processed at the next higher layer. Pooling allows a degree of translational invariance (it can recognize the same object in a somewhat different location) and allows a larger number of feature types to be represented. Capsnet proponents argue that pooling: violates biological shape perception in that it has no intrinsic coordinate frame; provides invariance (discarding positional information) instead of equivariance (disentangling that information); ignores the linear manifold that underlies many variations among images; routes statically instead of communicating a potential "find" to the feature that can appreciate it; damages nearby feature detectors, by deleting the information they rely upon. == Capsules == A capsule is a set of neurons that individually activate for various properties of a type of object, such as position, size and hue. Formally, a capsule is a set of neurons that collectively produce an activity vector with one element for each neuron to hold that neuron's instantiation value (e.g., hue). Graphics programs use instantiation value to draw an object. Capsnets attempt to derive these from their input. The probability of the entity's presence in a specific input is the vector's length, while the vector's orientation quantifies the capsule's properties. Artificial neurons traditionally output a scalar, real-valued activation that loosely represents the probability of an observation. Capsnets replace scalar-output feature detectors with vector-output capsules and max-pooling with routing-by-agreement. Because capsules are independent, when multiple capsules agree, the probability of correct detection is much higher. A minimal cluster of two capsules considering a six-dimensional entity would agree within 10% by chance only once in a million trials. As the number of dimensions increase, the likelihood of a chance agreement across a larger cluster with higher dimensions decreases exponentially. Capsules in higher layers take outputs from capsules at lower layers, and accept those whose outputs cluster. A cluster causes the higher capsule to output a high probability of observation that an entity is present and also output a high-dimensional (20-50+) pose. Higher-level capsules ignore outliers, concentrating on clusters. This is similar to the Hough transform, the RHT and RANSAC from classic digital image processing. == Routing by agreement == The outputs from one capsule (child) are routed to capsules in the next layer (parent) according to the child's ability to predict the parents' outputs. Over the course of a few iterations, each parents' outputs may converge with the predictions of some children and diverge from those of others, meaning that that parent is present or absent from the scene. For each possible parent, each child computes a prediction vector by multiplying its output by a weight matrix (trained by backpropagation). Next the output of the parent is computed as the scalar product of a prediction with a coefficient representing the probability that this child belongs to that parent. A child whose predictions are relatively close to the resulting output successively increases the coefficient between that parent and child and decreases it for parents that it matches less well. This increases the contribution that that child makes to that parent, thus increasing the scalar product of the capsule's prediction with the parent's output. After a few iterations, the coefficients strongly connect a parent to its most likely children, indicating that the presence of the children imply the presence of the parent in the scene. The more children whose predictions are close to a parent's output, the more quickly the coefficients grow, driving convergence. The pose of the parent (reflected in its output) progressively becomes compatible with that of its children. The coefficients' initial logits are the log prior probabilities that a child belongs to a parent. The priors can be trained discriminatively along with the weights. The priors depend on the location and type of the child and parent capsules, but not on the current input. At each iteration, the coefficients are adjusted via a "routing" softmax so that they continue to sum to 1 (to express the probability that a given capsule is the parent of a given child.) Softmax amplifies larger values and diminishes smaller values beyond their proportion of the total. Similarly, the probability that a feature is present in the input is exaggerated by a nonlinear "squashing" function that reduces values (smaller ones drastically and larger ones such that they are less than 1). This dynamic routing mechanism provides the necessary deprecation of alternatives ("explaining away") that is needed for segmenting overlapped objects. This learned routing of signals has no clear biological equivalent. Some operations can be found in cortical layers, but they do not seem to relate this technique. === Math/code === The pose vector u i {\textstyle \mathbf {u} _{i}} is rotated and translated by a matrix W i j {\textstyle \mathbf {W} _{ij}} into a vector u ^ j | i {\textstyle \mathbf {\hat {u}} _{j|i}} that predicts the output of the parent capsule. u ^ j | i = W i j u i {\displaystyle \mathbf {