A facial recognition system is a technology potentially capable of matching a human face from a digital image or a video frame against a database of faces. Such a system is typically employed to authenticate users through ID verification services, and works by pinpointing and measuring facial features from a given image. Development on similar systems began in the 1960s as a form of computer application. Since their inception, facial recognition systems have seen wider uses in recent times on smartphones and in other forms of technology, such as robotics. Because computerized facial recognition involves the measurement of a human's physiological characteristics, facial recognition systems are categorized as biometrics. Although the accuracy of facial recognition systems as a biometric technology is lower than iris recognition, fingerprint image acquisition, palm recognition or voice recognition, it is widely adopted due to its contactless process. Facial recognition systems have been deployed in advanced human–computer interaction, video surveillance, law enforcement, passenger screening, decisions on employment and housing, and automatic indexing of images. Facial recognition systems are employed throughout the world today by governments and private companies. Their effectiveness varies, and some systems have previously been scrapped because of their ineffectiveness. The use of facial recognition systems has also raised controversy, with claims that the systems violate citizens' privacy, commonly make incorrect identifications, encourage gender norms and racial profiling, and do not protect important biometric data. The appearance of synthetic media such as deepfakes has also raised concerns about its security. These claims have led to the ban of facial recognition systems in several cities in the United States. Growing societal concerns led social networking company Meta Platforms to shut down its Facebook facial recognition system in 2021, deleting the face-scan data of more than one billion users. The change represented one of the largest shifts in facial recognition usage in the technology's history. IBM also stopped offering facial recognition technology due to similar concerns. == History of facial recognition technology == Automated facial recognition was pioneered in the 1960s by Woody Bledsoe, Helen Chan Wolf, and Charles Bisson, whose work focused on teaching computers to recognize human faces. Their early facial recognition project was dubbed "man-machine" because a human first needed to establish the coordinates of facial features in a photograph before they could be used by a computer for recognition. Using a graphics tablet, a human would pinpoint facial features coordinates, such as the pupil centers, the inside and outside corners of eyes, and the widows peak in the hairline. The coordinates were used to calculate 20 individual distances, including the width of the mouth and of the eyes. A human could process about 40 pictures an hour, building a database of these computed distances. A computer would then automatically compare the distances for each photograph, calculate the difference between the distances, and return the closed records as a possible match. In 1970, Takeo Kanade publicly demonstrated a face-matching system that located anatomical features such as the chin and calculated the distance ratio between facial features without human intervention. Later tests revealed that the system could not always reliably identify facial features. Nonetheless, interest in the subject grew and in 1977 Kanade published the first detailed book on facial recognition technology. In 1993, the Defense Advanced Research Project Agency (DARPA) and the Army Research Laboratory (ARL) established the face recognition technology program FERET to develop "automatic face recognition capabilities" that could be employed in a productive real life environment "to assist security, intelligence, and law enforcement personnel in the performance of their duties." Face recognition systems that had been trialled in research labs were evaluated. The FERET tests found that while the performance of existing automated facial recognition systems varied, a handful of existing methods could viably be used to recognize faces in still images taken in a controlled environment. The FERET tests spawned three US companies that sold automated facial recognition systems. Vision Corporation and Miros Inc were founded in 1994, by researchers who used the results of the FERET tests as a selling point. Viisage Technology was established by an identification card defense contractor in 1996 to commercially exploit the rights to the facial recognition algorithm developed by Alex Pentland at MIT. Following the 1993 FERET face-recognition vendor test, the Department of Motor Vehicles (DMV) offices in West Virginia and New Mexico became the first DMV offices to use automated facial recognition systems to prevent people from obtaining multiple driving licenses using different names. Driver's licenses in the United States were at that point a commonly accepted form of photo identification. DMV offices across the United States were undergoing a technological upgrade and were in the process of establishing databases of digital ID photographs. This enabled DMV offices to deploy the facial recognition systems on the market to search photographs for new driving licenses against the existing DMV database. DMV offices became one of the first major markets for automated facial recognition technology and introduced US citizens to facial recognition as a standard method of identification. The increase of the US prison population in the 1990s prompted U.S. states to established connected and automated identification systems that incorporated digital biometric databases, in some instances this included facial recognition. In 1999, Minnesota incorporated the facial recognition system FaceIT by Visionics into a mug shot booking system that allowed police, judges and court officers to track criminals across the state. Until the 1990s, facial recognition systems were developed primarily by using photographic portraits of human faces. Research on face recognition to reliably locate a face in an image that contains other objects gained traction in the early 1990s with the principal component analysis (PCA). The PCA method of face detection is also known as Eigenface and was developed by Matthew Turk and Alex Pentland. Turk and Pentland combined the conceptual approach of the Karhunen–Loève theorem and factor analysis, to develop a linear model. Eigenfaces are determined based on global and orthogonal features in human faces. A human face is calculated as a weighted combination of a number of Eigenfaces. Because few Eigenfaces were used to encode human faces of a given population, Turk and Pentland's PCA face detection method greatly reduced the amount of data that had to be processed to detect a face. Pentland in 1994 defined Eigenface features, including eigen eyes, eigen mouths and eigen noses, to advance the use of PCA in facial recognition. In 1997, the PCA Eigenface method of face recognition was improved upon using linear discriminant analysis (LDA) to produce Fisherfaces. LDA Fisherfaces became dominantly used in PCA feature based face recognition. While Eigenfaces were also used for face reconstruction. In these approaches no global structure of the face is calculated which links the facial features or parts. Purely feature based approaches to facial recognition were overtaken in the late 1990s by the Bochum system, which used Gabor filter to record the face features and computed a grid of the face structure to link the features. Christoph von der Malsburg and his research team at the University of Bochum developed Elastic Bunch Graph Matching in the mid-1990s to extract a face out of an image using skin segmentation. By 1997, the face detection method developed by Malsburg outperformed most other facial detection systems on the market. The so-called "Bochum system" of face detection was sold commercially on the market as ZN-Face to operators of airports and other busy locations. The software was "robust enough to make identifications from less-than-perfect face views. It can also often see through such impediments to identification as mustaches, beards, changed hairstyles and glasses—even sunglasses". Real-time face detection in video footage became possible in 2001 with the Viola–Jones object detection framework for faces. Paul Viola and Michael Jones combined their face detection method with the Haar-like feature approach to object recognition in digital images to launch AdaBoost, the first real-time frontal-view face detector. By 2015, the Viola–Jones algorithm had been implemented using small low power detectors on handheld devices and embedded systems. Therefore, the Viola–Jones algorithm has not only broadened the practical application of face recognition systems but
Springpad
Springpad was a free online application and web service that allowed its registered users to save, organize and share collected ideas and information. As users added content to their Springpad accounts, the application automatically identified and categorized it, then generated additional snippets based on the types of objects added—for example, listing price comparisons for products and showtimes for movies. Springpad was also available as apps on the iPad, iPhone and Android that synchronized with the Web interface. Springpad was bundled on new Toshiba notebook computers through a Web application subscription service. On May 23, 2014, Springpad announced that it would cease operations on June 25, 2014. The company then allowed users to export their data (as JSON and read-only HTML formats), or to automatically migrate it to Evernote accounts before the expiration date. == Features == Springpad users could use the main site interface which uses HTML5 from most browsers or use the smartphone app to capture notes, tasks, or lists which were then added to the user's "My Stuff", the user's personal database or collection. Additionally Springpad let users look up items of interest which were then automatically categorized based on type or manually categorized by the user. Category types included recipes, movies, products, restaurants and wine. Events could also be added to Springpad, and if the user used Google Calendar, they could opt to sync the event to it. In addition to the smartphone app and site, Springpad could be used via browser extension for Google Chrome, or the Springpad Clipper, a bookmarklet to analyze webpages and clip relevant information from them—for example, the ingredients needed for a recipe—or to add the site as a normal bookmark. Another way users could add content to their Springpad "My Stuff" was by emailing entries to an email address specified on Springpad registration. Springpad's smartphone apps could be used to scan barcodes to identify products, save them to the user's "My Stuff", and automatically generate additional product information and links. The mobile app could also save images taken with the phone's camera, and locate nearby businesses. With most of the content added to a user's "My Stuff", relevant news, useful links and other helpful information could be viewed. Users could also attach additional notes and images to content they had already saved, and could add reminders and alerts which could be emailed to the user or texted to their phone. Springpad also added alerts to its own Alerts section for relevant news, deals or coupons for specific products users added. For additional organization, anything added to Springpad could also be tagged. Users could also add entries to "Notebooks" to separate content by projects, or any other way they wished. Each Notebook included a section called a "Board", which acted as a pin board where users could "pin" content they'd added to the Notebook, allowing them to visually lay out items. If the user added a map to the Board and had entries that included an address, Springpad could automatically point out entries on the map. By default, everything added to Springpad was private. However users could change the privacy settings for each of the types of items added, decide to make specific items public and shareable on Facebook and Twitter, add them to their public page, or keep them private but links to them with specific people.
OCR Systems
OCR Systems, Inc., was an American computer hardware manufacturer and software publisher dedicated to optical character recognition technologies. The company's first product, the System 1000 in 1970, was used by numerous large corporations for bill processing and mail sorting. Following a series of pitfalls in the 1970s and early 1980s, founder Theodor Herzl Levine put the company in the hands of Gregory Boleslavsky and Vadim Brikman, the company's vice presidents and recent immigrants from the Soviet Ukraine, who were able to turn OCR System's fortunes around and expand its employee base. The company released the software-based OCR application ReadRight for DOS, later ported to Windows, in the late 1980s. Adobe Inc. bought the company in 1992. == History == OCR Systems was co-founded by Theodor Herzl Levine (c. 1923 – May 30, 2005). Levine served in the U.S. Army Signal Corps during World War II in the Solomon Islands, where he helped develop a sonar to find ejected pilots in the ocean. After the war, Levine spent 22 years at the University of Pennsylvania, earning his bachelor's degree in 1951, his master's degree in electrical engineering in 1957, and his doctorate in 1968. Alongside his studies, Levine taught statistics and calculus at Temple University, Rutgers University, La Salle University and Penn State Abington. Sometime in the 1960s, Levine was hired at Philco. He and two of his co-workers decided to form their own company dedicated to optical character recognition, founding OCR Systems in 1969 in Bensalem, Pennsylvania. OCR Systems's first product, the System 1000, was announced in 1970. OCR Systems entered a partnership with 3M to resell the System 1000 throughout the United States in March 1973. This was 3M's entry into the data entry field, managed by the company's Microfilm Products Division and accompanying 3M's suite of data retrieval systems. It soon found use among Texas Instruments, AT&T, Ricoh, Panasonic and Canon for bill processing and mail sorting. Later in the mid-1970s an unspecified Fortune 500 company reneged on a contract to distribute the System 1000; later still a Canadian company distributing the System 1000 in Canada went defunct. Both incidents led OCR Systems to go nearly bankrupt, although it eventually recovered. By the early 1980s, however, the company was almost insolvent. In 1983 Levine had only $8,000 in his savings and became bedridden with an illness. He left the company in the hands of Gregory Boleslavsky and Vadim Brikman, two Soviet Ukraine expats whom Levine had hired earlier in the 1980s. Boleslavsky was hired as a wire wrapper for the System 1000 and as a programmer and beta tester for ReadRight—a software package developed by Levine implementing patents from Nonlinear Technology, another OCR-centric company from Greenbelt, Maryland. Boleslavsky in turn recommended Brikman to Levine. The two soon became vice presidents of the company while Levine was bedridden; in Boleslavsky's case, he worked 14-hour work days for over half a year in pursuit of the title. The two presented OCR Systems' products to the National Computer Conference in Chicago, where they were massively popular. The company soon gained such clients as Allegheny Energy in Pennsylvania and the postal service of Belgium and received an influx of employees—mostly expats from Russia but also Poland and South Korea, as well as American-born workers. To accommodate the company's employee base, which had grown to over 30 in 1988, Levine moved OCR System's headquarters from Bensalem to the Masons Mill Business Park in Bryn Athyn. Chinon Industries of Japan signed an agreement with OCR Systems in 1987 to distribute OCR's ReadRight 1.0 software with Chinon's scanners, starting with their N-205 overhead scanner. In 1988, OCR opened their agreement to distribute ReadRight to other scanner manufacturers, including Canon, Hewlett-Packard, Skyworld, Taxan, Diamond Flower and Abaton. That year, the company posted a revenue of $3 million. OCR Systems extended their agreement with Chinon in 1989 and introduced version 2.0 of ReadRight. OCR Systems faced stiff competition in the software OCR market in the turn of the 1990s. The Toronto-based software firm Delrina signed a letter of intent to purchase the company in November 1991, expecting the deal to close in December and have OCR software available by Christmas. OCR was to receive $3 million worth of Delrina shares in a stock swap, but the deal collapsed in January 1992. Delrine later marketed its own Extended Character Recognition, or XCR, software package to compete with ReadRight. In July 1992, OCR Systems was purchased by Adobe Inc. for an undisclosed sum. == Products == === System 1000 === The System 1000 was based on the 16-bit Varian Data 620/i minicomputer with 4 KB of core memory. The system used the 620/i for controlling the paper feed, interpreting the format of the documents, the optical character recognition process itself, error detection, sequencing and output. The System was initially programmed to recognize 1428 OCR (used by Selectrics); IBM 407 print; and the full character sets of OCR-A, OCR-B and Farrington 7B; as well as optical marks and handwritten numbers. OCR Systems promised added compatibility with more fonts available down the line—per request—in 1970. The number of fonts supported was limited by the amount of core memory, which was expandable in 4 KB increments up to 32 KB. The System 1000 later supported generalized typewriter and photocopier fonts. The rest of the System 1000 comprised the document transport, one or more scanner elements, a CRT display and a Teletype Model 33 or 35. Pages are fed via friction with a rubber belt. Up to three lines could be scanned per document, while the rest of the scanned document could be laid out in any manner granted there was enough space around the fields to be read. The reader initially supported pages as small as 3.25 in by 3.5 in dimension (later supporting 2.6 in by 3.5 in utility cash stubs) all the way to the standard ANSI letter size (8.5 in by 11 in; later 8.5 in by 12 in as used in stock certificates). The initial System 1000 had a maximum throughput of 420 documents per minute per transport (later 500 documents per minute), contingent on document size and content. A feature unique to the System 1000 over other optical character recognition systems of the time was its ability to alert the operator when a field was unreadable or otherwise invalid. This feature, called Document Referral, placed the document in front of the operator and displayed a blank field on the screen of the included CRT monitor for manual re-entry via keyboard. Once input, data could be output to 7- or 9-track tape, paper tape, punched cards and other mass storage media or to System/360 mainframes for further processing. The complete System 1000 could be purchased for US$69,000. Options for renting were $1,800 per month on a three-year lease or $1,600 per month for five years. Computerworld wrote that it was less than half the cost of its competitors while more capable and user-friendly. Competing systems included the Recognition Equipment Retina, the Scan-Optics IC/20 and the Scan-Data 250/350. === ReadRight === ReadRight processes individual letters topographically: it breaks down the scanned letter into parts—strokes, curves, angles, ascenders and descenders—and follows a tree structure of letters broken down into these parts to determine the corresponding character code. ReadRight was entirely software-based, requiring no expansion card to work. Version 2.01, the last version released for DOS, runs in real mode in under 640 KB of RAM. OCR Systems released the Windows-only version 3.0 in 1991 while offering version 2.01 alongside it. The company unveiled a sister product, ReadRight Personal, dedicated to handheld scanners and for Windows only in October 1991. This version adds real-time scanning—each word is updated to the screen while lines are being scanned. ReadRight proper was later made a Windows-only product with version 3.1 in 1992. The inclusion of ReadRight 2.0 with Canon's IX-12F flatbed scanner led PC Magazine to award it an Editor's Choice rating in 1989. Despite this, reviewer Robert Kendall found qualification with ReadRight's ability to parse proportional typefaces such as Helvetica and Times New Roman. Mitt Jones of the same publication found version 2.01 to have improved its ability to read such typefaces and praised its ease of use and low resource intensiveness. Jones disliked the inability to handle uneven page paragraph column widths and graphics, noting that the manual recommended the user block out graphics with a Post-it Note. Version 3.1 for Windows received mixed reviews. Mike Heck of InfoWorld wrote that its "low cost and rich collection of features are hard to ignore" but rated its speed and accuracy average. Barry Simon of PC Magazine called it economical but inaccurate, unable to correct errors it did
Powerset construction
In the theory of computation and automata theory, the powerset construction or subset construction is a standard method for converting a nondeterministic finite automaton (NFA) into a deterministic finite automaton (DFA) that recognizes the same formal language. It is important in theory because it establishes that NFAs, despite their additional flexibility, are unable to recognize any language that cannot be recognized by some DFA. It is also important in practice for converting easier-to-construct NFAs into more efficiently executable DFAs. However, if the NFA has n states, the resulting DFA may have up to 2n states, an exponentially larger number, which sometimes makes the construction impractical for large NFAs. The construction, sometimes called the Rabin–Scott powerset construction (or subset construction) to distinguish it from similar constructions for other types of automata, was first published by Michael O. Rabin and Dana Scott in 1959. == Intuition == To simulate the operation of a DFA on a given input string, one needs to keep track of a single state at any time: the state that the automaton will reach after seeing a prefix of the input. In contrast, to simulate an NFA, one needs to keep track of a set of states: all of the states that the automaton could reach after seeing the same prefix of the input, according to the nondeterministic choices made by the automaton. If, after a certain prefix of the input, a set S of states can be reached, then after the next input symbol x the set of reachable states is a deterministic function of S and x. Therefore, the sets of reachable NFA states play the same role in the NFA simulation as single DFA states play in the DFA simulation, and in fact the sets of NFA states appearing in this simulation may be re-interpreted as being states of a DFA. == Construction == The powerset construction applies most directly to an NFA that does not allow state transformations without consuming input symbols (aka: "ε-moves"). Such an automaton may be defined as a 5-tuple (Q, Σ, T, q0, F), in which Q is the set of states, Σ is the set of input symbols, T is the transition function (mapping a state and an input symbol to a set of states), q0 is the initial state, and F is the set of accepting states. The corresponding DFA has states corresponding to subsets of Q. The initial state of the DFA is {q0}, the (one-element) set of initial states. The transition function of the DFA maps a state S (representing a subset of Q) and an input symbol x to the set T(S,x) = ∪{T(q,x) | q ∈ S}, the set of all states that can be reached by an x-transition from a state in S. A state S of the DFA is an accepting state if and only if at least one member of S is an accepting state of the NFA. In the simplest version of the powerset construction, the set of all states of the DFA is the powerset of Q, the set of all possible subsets of Q. However, many states of the resulting DFA may be useless as they may be unreachable from the initial state. An alternative version of the construction creates only the states that are actually reachable. === NFA with ε-moves === For an NFA with ε-moves (also called an ε-NFA), the construction must be modified to deal with these by computing the ε-closure of states: the set of all states reachable from some given state using only ε-moves. Van Noord recognizes three possible ways of incorporating this closure computation in the powerset construction: Compute the ε-closure of the entire automaton as a preprocessing step, producing an equivalent NFA without ε-moves, then apply the regular powerset construction. This version, also discussed by Hopcroft and Ullman, is straightforward to implement, but impractical for automata with large numbers of ε-moves, as commonly arise in natural language processing application. During the powerset computation, compute the ε-closure { q ′ | q → ε ∗ q ′ } {\displaystyle \{q'~|~q\to _{\varepsilon }^{}q'\}} of each state q that is considered by the algorithm (and cache the result). During the powerset computation, compute the ε-closure { q ′ | ∃ q ∈ Q ′ , q → ε ∗ q ′ } {\displaystyle \{q'~|~\exists q\in Q',q\to _{\varepsilon }^{}q'\}} of each subset of states Q' that is considered by the algorithm, and add its elements to Q'. === Multiple initial states === If NFAs are defined to allow for multiple initial states, the initial state of the corresponding DFA is the set of all initial states of the NFA, or (if the NFA also has ε-moves) the set of all states reachable from initial states by ε-moves. == Example == The NFA below has four states; state 1 is initial, and states 3 and 4 are accepting. Its alphabet consists of the two symbols 0 and 1, and it has ε-moves. The initial state of the DFA constructed from this NFA is the set of all NFA states that are reachable from state 1 by ε-moves; that is, it is the set {1,2,3}. A transition from {1,2,3} by input symbol 0 must follow either the arrow from state 1 to state 2, or the arrow from state 3 to state 4. Additionally, neither state 2 nor state 4 have outgoing ε-moves. Therefore, T({1,2,3},0) = {2,4}, and by the same reasoning the full DFA constructed from the NFA is as shown below. As can be seen in this example, there are five states reachable from the start state of the DFA; the remaining 11 sets in the powerset of the set of NFA states are not reachable. == Complexity == Because the DFA states consist of sets of NFA states, an n-state NFA may be converted to a DFA with at most 2n states. For every n, there exist n-state NFAs such that every subset of states is reachable from the initial subset, so that the converted DFA has exactly 2n states, giving Θ(2n) worst-case time complexity. A simple example requiring nearly this many states is the language of strings over the alphabet {0,1} in which there are at least n characters, the nth from last of which is 1. It can be represented by an (n + 1)-state NFA, but it requires 2n DFA states, one for each n-character suffix of the input; cf. picture for n=4. == Applications == Brzozowski's algorithm for DFA minimization uses the powerset construction, twice. It converts the input DFA into an NFA for the reverse language, by reversing all its arrows and exchanging the roles of initial and accepting states, converts the NFA back into a DFA using the powerset construction, and then repeats its process. Its worst-case complexity is exponential, unlike some other known DFA minimization algorithms, but in many examples it performs more quickly than its worst-case complexity would suggest. Safra's construction, which converts a non-deterministic Büchi automaton with n states into a deterministic Muller automaton or into a deterministic Rabin automaton with 2O(n log n) states, uses the powerset construction as part of its machinery.
Marcus Hutter
Marcus Hutter (born 14 April 1967 in Munich) is a German computer scientist, professor and artificial intelligence researcher. As a senior researcher at DeepMind, he studies the mathematical foundations of artificial general intelligence. Hutter studied physics and computer science at the Technical University of Munich. In 2000, he joined Jürgen Schmidhuber's group at the Dalle Molle Institute for Artificial Intelligence Research in Manno, Switzerland. He developed a mathematical formalism of artificial general intelligence named AIXI. He has served as a professor at the College of Engineering, Computing and Cybernetics of the Australian National University in Canberra, Australia. == Research == Starting in 2000, Hutter developed and published a mathematical theory of artificial general intelligence, AIXI, based on idealised intelligent agents and reward-motivated reinforcement learning. His first book Universal Artificial Intelligence: Sequential Decisions Based on Algorithmic Probability was published in 2005 by Springer. Also in 2005, Hutter published with his doctoral student Shane Legg an intelligence test for artificial intelligence devices. In 2009, Hutter developed and published the theory of feature reinforcement learning. In 2014, Lattimore and Hutter published an asymptotically optimal extension of the AIXI agent. An accessible podcast with Lex Fridman about his theory of Universal AI appeared in 2021 and a more technical follow-up with Tim Nguyen in 2024 in the Cartesian Cafe. His new (2024) book also gives a more accessible introduction to Universal AI and progress in the 20 years since his first book, including a chapter on ASI safety, which featured as a keynote at the inaugural workshop on AI safety in Sydney. == Hutter Prize == In 2006, Hutter announced the Hutter Prize for Lossless Compression of Human Knowledge, with a total of €50,000 in prize money. In 2020, Hutter raised the prize money for the Hutter Prize to €500,000.
Automated medical scribe
Automated medical scribes (also called artificial intelligence scribes, AI scribes, digital scribes, virtual scribes, ambient AI scribes, AI documentation assistants, and digital/virtual/smart clinical assistants) are tools for transcribing medical speech, such as patient consultations and dictated medical notes. Many also produce summaries of consultations. Automated medical scribes based on large language models (LLMs, commonly called "AI", short for "artificial intelligence") increased drastically in popularity in 2024. There are privacy and antitrust concerns. Accuracy concerns also exist, and intensify in situations in which tools try to go beyond transcribing and summarizing, and are asked to format information by its meaning, since LLMs do not deal well with meaning (see weak artificial intelligence). Medics using these scribes are generally expected to understand the ethical and legal considerations, and supervise the outputs. The privacy protections of automated medical scribes vary widely. While it is possible to do all the transcription and summarizing locally, with no connection to the internet, most closed-source providers require that data be sent to their own servers over the internet, processed there, and the results sent back (as with digital voice assistants). Some retailers say their tools use zero-knowledge encryption (meaning that the service provider can't access the data). Others explicitly say that they use patient data to train their AIs, or rent or resell it to third parties; the nature of privacy protections used in such situations is unclear, and they are likely not to be fully effective. Most providers have not published any safety or utility data in academic journals, and are not responsive to requests from medical researchers studying their products. == Privacy == Some providers unclear about what happens to user data. Some may sell data to third parties. Some explicitly send user data to for-profit tech companies for secondary purposes, which may not be specified. Some require users to sign consents to such reuse of their data. Some ingest user data to train the software, promising to anonymize it; however, deanonymization may be possible (that is, it may become obvious who the patient is). It is intrinsically impossible to prevent an LLM from correlating its inputs; they work by finding similar patterns across very large data sets. Some information on the patient will be known from other sources (for instance, information that they were injured in an incident on a certain day might be available from the news media; information that they attended specific appointment locations at specific times is probably available to their cellphone provider/apps/data brokers; information about when they had a baby is probably implied by their online shopping records; and they might mention lifestyle changes to their doctor and on a forum or blog). The software may correlate such information with the "anonymized" clinical consultation record, and, asked about the named patient, provide information which they only told their doctor privately. Because a patient's record is all about the same patient, it is all unavoidably linked; in very many cases, medical histories are intrinsically identifiable. Depending on how common a condition and what other data is available, K-anonymity may be useless. Differential privacy could theoretically preserve privacy. Data broker companies like Google, Amazon, Apple and Microsoft have produced or bought up medical scribes, some of which use user data for secondary purposes, which has led to antitrust concerns. Transfer of patient records for AI training has, in the past, prompted legal action. Open-source programs typically do all the transcription locally, on the doctor's own computer. Open-source software is widely used in healthcare, with some national public healthcare bodies holding hack days. === Data resale and commercialization === Several AI medical scribe providers include terms in their service agreements that allow the reuse, sale, or commercialization of de-identified or user-submitted data. Although such data are generally described as anonymized or aggregated, these practices have raised ethical concerns among clinicians and privacy advocates regarding secondary uses of medical information beyond clinical documentation. Freed, an AI transcription and scribe platform, states in its Terms of Use that it may "collect, use, publish, disseminate, sell, transfer, and otherwise exploit" de-identified and aggregated data derived from user inputs. OpenEvidence similarly states that it may "collect, use, transfer, sell, and disclose non-personal information and customer usage data for any purpose including commercial uses." Doximity, which offers an AI-enabled medical scribe as part of its physician platform, grants itself a "nonexclusive, irrevocable, worldwide, perpetual, unlimited, assignable, sublicensable, royalty-free" license to "copy, prepare derivative works from, improve, distribute, publish, ... analyze, index, tag, [and] commercialize" content submitted by users, subject to its privacy policy. Because these terms allow broad secondary use—including sale, licensing, model-training, derivative works, and commercial exploitation of de-identified or user-submitted data—some commentators have recommended that clinicians review data-handling provisions carefully when adopting AI-scribe tools, particularly in clinical environments where patient privacy and regulatory compliance are critical. === Encryption === Multifactor authentication for access to the data is expected practice. Typically, Diffie–Hellman key exchange is used for encryption; this is the standard method commonly used for things like online banking. This encryption is expensive but not impossible to break; it is not generally considered safe against eavesdroppers with the resources of a nation-state. If content is encrypted between the client and the service provider's remote server (transport cryptography), then the server has an unencrypted copy. This is necessary if the data is used by the service provider (for instance, to train the software). Zero-knowledge encryption implies that the only unencrypted copy is at the client, and the server cannot decrypt the data any more easily than a monster-in-the-middle attacker. == Platforms == Scribes may operate on desktops, laptop, or mobile computers, under a variety of operating systems. These vary in their risks; for instance, mobiles can be lost. The underlying mobile or desktop operating systems are also part of the trusted computing base, and if they are not secure, the software relying on them cannot be secure either. Some AI medical scribe platforms are designed to operate as cloud-based applications that generate structured clinical documentation from clinician–patient conversations. These systems may offer features such as real-time transcription, document generation, and integration with electronic health record (EHR) systems. == Confabulation, omissions, and other errors == Like other LLMs, medical-scribe LLMs are prone to hallucinations, where they make up content based on statistically associations between their training data and the transcription audio. LLMs do not distinguish between trying to transcribe the audio and guessing what words will come next, but perform both processes mixed together. They are especially likely to take short silences or non-speech noises and invent some sort of speech to transcribe them as. LLM medical scribes have been known to confabulate racist and otherwise prejudiced content; this is partly because the training datasets of many LLMs contain pseudoscientific texts about medical racism. They may misgender patients. A survey found that most doctors preferred, in principle, that scribes be trained on data reviewed by medical subject experts. Relevant, accurate training data increases the probability of an accurate transcription, but does not guarantee accuracy. Software trained on thousands of real clinical conversations generated transcripts with lower word error rates. Software trained on manually-transcribed training data did better than software trained with automatically transcribed training data such as YouTube captions. Autoscribes omit parts of the conversation classes as irrelevant. The may wrongly classify pertinent information as irrelevant and omit it. They may also confuse historic and current symptoms, or otherwise misclassify information. They may also simply wrongly transcribe the speech, writing something incorrect instead. If clinicians do not carefully check the recording, such mistakes could make their way into their medical records and cause patient harms. == Patient consent == Professional organizations generally require that scribes be used only with patient consent; some bodies may require written consent. Medics must also abide by local surveillance laws, which may criminalize recording pri
Diane Litman
Diane Litman is an American professor of computer science at the University of Pittsburgh. She also jointly holds the positions of senior scientist with the Learning Research and Development Center and faculty with the Intelligent Systems department. Litman is noted for her work in the areas of artificial intelligence, computational linguistics, knowledge representation and reasoning, natural language processing, and user modeling. == Education == Litman did her undergraduate studies at the College of William and Mary and her master's and PhD degrees at the University of Rochester. == Career == Before joining the University of Pittsburgh, she was an assistant professor at Columbia University. She additionally held the position of a research scientist in the Artificial Intelligence Principles Research Department Laboratory at AT&T Labs. Litman has held the position of Chair of the North American Chapter of the Association for Computational Linguistics two times, elected twice for the position, whose tenure lasts four years. She is also a distinguished member of the executive committee of the Association for Computational Linguistics, and a member of the editorial boards of Computational Linguistics and User Modeling and User-Adapted Interaction. She has also held the position of Leverhulme Professor at the University of Edinburgh. Litman was the keynote speaker at the Speech and Language Technology in Education 2013 symposium, the 2006 SIGdial Meeting on Discourse and Dialogue, and at the 2008 Symposium of the Annual Meeting of the Society for the Study of Artificial Intelligence and Simulation of Behaviour. She also sits on the board of the several interest groups, including the International Speech Communication Association's Special Interest Group on Speech and Language Technology in Education. Litman has served as chair, organizer, and a senior member of numerous committees of peer-reviewed scientific journals. == Awards and recognition == She has also co-authored numerous award-winning papers and was awarded senior member status by the Association for the Advancement of Artificial Intelligence in 2011, an award designed to honor those who have "achieved significant accomplishments within the field of artificial intelligence."