A text-to-video model is a form of generative artificial intelligence that uses a natural language description as input to produce a video relevant to the input text. Advancements during the 2020s in the generation of high-quality, text-conditioned videos have largely been driven by the development of video diffusion models. == Models == There are different models, including open source models. Chinese-language input CogVideo is the earliest text-to-video model "of 9.4 billion parameters" to be developed, with its demo version of open source codes first presented on GitHub in 2022. That year, Meta Platforms released a partial text-to-video model called "Make-A-Video", and Google's Brain (later Google DeepMind) introduced Imagen Video, a text-to-video model with 3D U-Net. === 2023 === In February 2023, Runway released Gen-1 and Gen-2, among the first commercially available text-to-video and video-to-video models accessible to the public through a web interface. Gen-1, initially released as a video-to-video model, allowed users to transform existing video footage using text or image prompts. Gen-2, introduced in March 2023 and made publicly available in June 2023, added text-to-video capabilities, enabling users to generate videos from text prompts alone. In March 2023, a research paper titled "VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation" was published, presenting a novel approach to video generation. The VideoFusion model decomposes the diffusion process into two components: base noise and residual noise, which are shared across frames to ensure temporal coherence. By utilizing a pre-trained image diffusion model as a base generator, the model efficiently generated high-quality and coherent videos. Fine-tuning the pre-trained model on video data addressed the domain gap between image and video data, enhancing the model's ability to produce realistic and consistent video sequences. In the same month, Adobe introduced Firefly AI as part of its features. === 2024 === In January 2024, Google announced development of a text-to-video model named Lumiere which is anticipated to integrate advanced video editing capabilities. Matthias Niessner and Lourdes Agapito at AI company Synthesia work on developing 3D neural rendering techniques that can synthesise realistic video by using 2D and 3D neural representations of shape, appearances, and motion for controllable video synthesis of avatars. In June 2024, Luma Labs launched its Dream Machine video tool. That same month, Kuaishou extended its Kling AI text-to-video model to international users. In July 2024, TikTok owner ByteDance released Jimeng AI in China, through its subsidiary, Faceu Technology. By September 2024, the Chinese AI company MiniMax debuted its video-01 model, joining other established AI model companies like Zhipu AI, Baichuan, and Moonshot AI, which contribute to China's involvement in AI technology. In December 2024 Lightricks launched LTX Video as an open source model. === 2025 === Alternative approaches to text-to-video models include Google's Phenaki, Hour One, Colossyan, Runway's Gen-3 Alpha, and OpenAI's Sora, Several additional text-to-video models, such as Plug-and-Play, Text2LIVE, and TuneAVideo, have emerged. FLUX.1 developer Black Forest Labs has announced its text-to-video model SOTA. Google was preparing to launch a video generation tool named Veo for YouTube Shorts in 2025. In May 2025, Google launched the Veo 3 iteration of the model. It was noted for its impressive audio generation capabilities, which were a previous limitation for text-to-video models. In July 2025 Lightricks released an update to LTX Video capable of generating clips reaching 60 seconds, and in October 2025 it released LTX-2, with audio capabilities built in. === 2026 === In February 2026, ByteDance released Seedance 2.0, it was noted for its impressive realistic generation, motion and camera control and 15 second generation, however the model faced huge critiscism from Motion Picture Association for copyright infringement. After viewing a viral clip of a fight between actors Brad Pitt and Tom Cruise, Rhett Reese, who is the co-writer of Deadpool & Wolverine and Zombieland announced that on social media "I hate to say it. It’s likely over for us," further stating that "In next to no time, one person is going to be able to sit at a computer and create a movie indistinguishable from what Hollywood now releases." == Architecture and training == There are several architectures that have been used to create text-to-video models. Similar to text-to-image models, these models can be trained using Recurrent Neural Networks (RNNs) such as long short-term memory (LSTM) networks, which has been used for Pixel Transformation Models and Stochastic Video Generation Models, which aid in consistency and realism respectively. An alternative for these include transformer models. Generative adversarial networks (GANs), Variational autoencoders (VAEs), — which can aid in the prediction of human motion — and diffusion models have also been used to develop the image generation aspects of the model. Text-video datasets used to train models include, but are not limited to, WebVid-10M, HDVILA-100M, CCV, ActivityNet, and Panda-70M. These datasets contain millions of original videos of interest, generated videos, captioned-videos, and textual information that help train models for accuracy. Text-video datasets used to train models include, but are not limited to PromptSource, DiffusionDB, and VidProM. These datasets provide the range of text inputs needed to teach models how to interpret a variety of textual prompts. The video generation process involves synchronizing the text inputs with video frames, ensuring alignment and consistency throughout the sequence. This predictive process is subject to decline in quality as the length of the video increases due to resource limitations. The Will Smith Eating Spaghetti test is a benchmark for models. == Limitations == Despite the rapid evolution of text-to-video models in their performance, a primary limitation is that they are very computationally heavy which limits its capacity to provide high quality and lengthy outputs. Additionally, these models require a large amount of specific training data to be able to generate high quality and coherent outputs, which brings about the issue of accessibility. Moreover, models may misinterpret textual prompts, resulting in video outputs that deviate from the intended meaning. This can occur due to limitations in capturing semantic context embedded in text, which affects the model's ability to align generated video with the user's intended message. Various models, including Make-A-Video, Imagen Video, Phenaki, CogVideo, GODIVA, and NUWA, are currently being tested and refined to enhance their alignment capabilities and overall performance in text-to-video generation. Another issue with the outputs is that text or fine details in AI-generated videos often appear garbled, a problem that stable diffusion models also struggle with. Examples include distorted hands and unreadable text. == Ethics == The deployment of text-to-video models raises ethical considerations related to content generation. These models have the potential to create inappropriate or unauthorized content, including explicit material, graphic violence, misinformation, and likenesses of real individuals without consent. Ensuring that AI-generated content complies with established standards for safe and ethical usage is essential, as content generated by these models may not always be easily identified as harmful or misleading. The ability of AI to recognize and filter out NSFW or copyrighted content remains an ongoing challenge, with implications for both creators and audiences. == Impacts and applications == Text-to-video models offer a broad range of applications that may benefit various fields, from educational and promotional to creative industries. These models can streamline content creation for training videos, movie previews, gaming assets, and visualizations, making it easier to generate content. During the Russo-Ukrainian war, fake videos made with artificial intelligence were created as part of a propaganda war against Ukraine and shared in social media. These included depictions of children in the Ukrainian Armed Forces, fake ads targeting children encouraging them to denounce critics of the Ukrainian government, or fictitious statements by Ukrainian President Volodymyr Zelenskyy about the country's surrender, among others. === Movies === Kaur vs Kore is the first Indian feature film made using generative AI which features dual role for the AI character of Sunny Leone, set to release in 2026. Chiranjeevi Hanuman – The Eternal is an Indian movie made entirely using Generative AI created by Vijay Subramaniam which is set for theatrical release in 2026. The movie was widely criticised by the Film makers in the Bollywood industr
List of security hacking incidents
This list of security hacking incidents covers important or noteworthy events in the history of security hacking and cracking. == 1900 == === 1903 === Magician and inventor Nevil Maskelyne disrupts John Ambrose Fleming's public demonstration of Guglielmo Marconi's purportedly secure wireless telegraphy technology, sending insulting Morse code messages through the auditorium's projector. == 1930s == === 1932 === Polish cryptologists Marian Rejewski, Henryk Zygalski and Jerzy Różycki broke the Enigma machine code. === 1939 === Alan Turing, Gordon Welchman and Harold Keen worked together to develop the codebreaking device Bombe (based off of Rejewski's work on Bomba). The Enigma machine's use of a reliably small key space makes it vulnerable to brute force attacks. == 1940s == === 1943 === René Carmille, comptroller general of the Vichy French Army, hacked the punch card system used by the Nazis to locate Jews. === 1949 === The theory that underlies computer viruses was first made public in 1949, when computer pioneer John von Neumann presented a paper titled "Theory and Organization of Complicated Automata". In the paper, von Neumann speculated that computer programs could reproduce themselves. == 1950s == === 1955 === At MIT, "hack" first came to mean playing with machines. An April 1955 meeting of the Tech Model Railroad Club has one say that "Mr. Eccles requests that anyone working or hacking on the electrical system turn the power off to avoid fuse blowing." === 1957 === Joe "Joybubbles" Engressia, a blind seven-year-old boy with perfect pitch, discovered that whistling the fourth E above middle C (a frequency of 2600 Hz) would interfere with AT&T's automated telephone systems, thereby inadvertently opening the door for phreaking. == 1960s == Various phreaking boxes are used to interact with automated telephone systems. === 1963 === The first ever reference to malicious hacking is 'phreaking' in MIT's student newspaper, The Tech, containing hackers tying up the lines with Harvard, configuring the PDP-1 to make free calls, war dialing and accumulating large phone bills. === 1965 === William D. Mathews from MIT finds a vulnerability in a CTSS running on an IBM 7094. The standard text editor on the system was designed to be used by one user at a time, working in one directory, and so it created a temporary file with a constant name for all instances of the editor. The flaw was discovered when two system programmers were editing at the same time and the temporary files for the message of the day and the password file became swapped, causing the contents of the system CTSS password file to display to any user logging into the system. === 1967 === The first known incidence of network penetration hacking took place when members of a computer club at a suburban Chicago high school were provided access to IBM's APL network. In the fall of 1967, IBM (through Science Research Associates) approached Evanston Township High School with the offer of four 2741 Selectric teletypewriter-based terminals with dial-up modem connectivity to an experimental computer system which implemented an early version of the APL programming language. The APL network system was structured into workspaces which were assigned to various clients using the system. Working independently, the students quickly learned the language and the system. They were free to explore the system, often using existing code available in public workspaces as models for their own creations. Eventually, curiosity drove the students to explore the system's wider context. This first informal network penetration effort was later acknowledged as helping harden the security of one of the first publicly accessible networks:Science Research Associates undertook to write a full APL system for the IBM 1500. They modeled their system after APL/360, which had by that time been developed and seen substantial use inside of IBM, using code borrowed from MAT/1500 where possible. In their documentation, they acknowledge their gratitude to "a number of high school students for their compulsion to bomb the system". This was an early example of a kind of sportive, but very effective, debugging that was often repeated in the evolution of APL systems. == 1970s == === 1971 === John T. Draper (later nicknamed Captain Crunch), his friend Joe Engressia (also known as Joybubbles), and blue box phone phreaking hit the news with an Esquire magazine feature story. === 1979 === Kevin Mitnick breaks into his first major computer system, the Ark, which was the computer system Digital Equipment Corporation (DEC) used for developing their RSTS/E operating system software. == 1980s == === 1980 === The FBI investigates a breach of security at National CSS (NCSS). The New York Times, reporting on the incident in 1981, describes hackers as: Technical experts, skilled, often young, computer programmers who almost whimsically probe the defenses of a computer system, searching out the limits and the possibilities of the machine. Despite their seemingly subversive role, hackers are a recognized asset in the computer industry, often highly prized. The newspaper describes white hat activities as part of a "mischievous but perversely positive 'hacker' tradition". When a National CSS employee revealed the existence of his password cracker, which he had used on customer accounts, the company chastised him not for writing the software but for not disclosing it sooner. The letter of reprimand stated that "The Company realizes the benefit to NCSS and in fact encourages the efforts of employees to identify security weaknesses to the VP, the directory, and other sensitive software in files". === 1981 === Chaos Computer Club forms in Germany. Ian Murphy, aka Captain Zap, was the first cracker to be tried and convicted as a felon. Murphy broke into AT&T's computers in 1981 and changed the internal clocks that metered billing rates. People were getting late-night discount rates when they called at midday. Of course, the bargain-seekers who waited until midnight to call long distance were hit with high bills. === 1983 === The 414s break into 60 computer systems at institutions ranging from the Los Alamos National Laboratory to Manhattan's Memorial Sloan-Kettering Cancer Center. The incident appeared as the cover story of Newsweek with the title "Beware: Hackers at play". As a result, the U.S. House of Representatives held hearings on computer security and passed several laws. The group KILOBAUD is formed in February, kicking off a series of other hacker groups that formed soon after. The movie WarGames introduces the wider public to the phenomenon of hacking and creates a degree of mass paranoia about hackers and their supposed abilities to bring the world to a screeching halt by launching nuclear ICBMs. The U.S. House of Representatives begins hearings on computer security hacking. In his Turing Award lecture, Ken Thompson mentions "hacking" and describes a security exploit that he calls a "Trojan horse". === 1984 === Someone calling himself Lex Luthor founds the Legion of Doom. Named after a Saturday morning cartoon, the LOD had the reputation of attracting "the best of the best"—until one of the most talented members called Phiber Optik feuded with Legion of Doomer Erik Bloodaxe and got 'tossed out of the clubhouse'. Phiber's friends formed a rival group, the Masters of Deception. The Comprehensive Crime Control Act gives the Secret Service jurisdiction over computer fraud. The Cult of the Dead Cow forms in Lubbock, Texas, and begins publishing its underground ezine. The hacker magazine 2600 begins regular publication, right when TAP was putting out its final issue. The editor of 2600, "Emmanuel Goldstein" (whose real name is Eric Corley), takes his handle from the leader of the resistance in George Orwell's Nineteen Eighty-Four. The publication provides tips for would-be hackers and phone phreaks, as well as commentary on the hacker issues of the day. Today, copies of 2600 are sold at most large retail bookstores. The Chaos Communication Congress, the annual European hacker conference organized by the Chaos Computer Club, is held in Hamburg, Germany. William Gibson's groundbreaking science fiction novel Neuromancer, about "Case", a futuristic computer hacker, is published. Considered the first major cyberpunk novel, it brought into hacker jargon such terms as "cyberspace", "the matrix", "simstim", and "ICE". === 1985 === KILOBAUD is re-organized into P.H.I.R.M. and begins sysopping hundreds of bulletin board systems (BBSs) throughout the United States, Canada, and Europe. The online 'zine Phrack is established. The Hacker's Handbook is published in the UK. The FBI, Secret Service, Middlesex County NJ Prosecutor's Office and various local law enforcement agencies execute seven search warrants concurrently across New Jersey on July 12, 1985, seizing equipment from BBS operators and users alike for "complicity in computer theft", under a n
Futel
Futel is a public arts organization in Portland, Oregon dedicated to preserving and maintaining public telephone hardware and offering free phone and basic information services. Futel was founded by Karl Anderson, a former software engineer, and Elijah St. Clair. == Technology == Karl Anderson stated that one motivation for the project was to explore the idea of urban furniture. Other reasons were to preserve an important part of hacker history, and to salvage and re-use manufactured items at the end of their lifecycle. The original Futel phones were set up in Portland, Oregon. The organization cleans and repurposes old public payphones which are often salvaged from Craigslist or scrappers. Using interface boxes, they are converted into VoIP phones which are made available publicly, with no cost for phone calls. Anderson has said the service runs on "Asterisk and OpenVPN and a lot of scripts." The payphones operate using publicly-available internet connections. The phones have automated phone trees and users can make a call to local social services, to a weather forecast line, or access local transit information. Volunteers act as telephone operators, offering information about the Futel service, or are available for conversation. Users using Futel's phones may also access voicemail boxes. The system has a "wildcard line" where people can listen to samples of audio left on the main voicemail line along with commentary from Anderson and others. == Network == In February 2021, there were 10 Futel phones in Portland and 3 in other cities. Phones were set up in Detroit and Ypsilanti, Michigan, and Long Beach, Washington. The organization has provided free phone service for a Portland-area homeless encampment after receiving funding from the Awesome Foundation. In 2019 the organization reported their phones being used to make 12,000 phone calls. Futel also said their usage went up and not down during the first year of the COVID-19 pandemic when they outfitted their phone kiosks with handwashing stations and used volunteers to keep the phones clean. The project is funded is primarily through grants and is staffed with volunteers. The project has inspired others such as the PhilTel project in Philadelphia and the RandTel project in Randolph, Vermont. Futel publishes a zine called Party Line.
Librem
Librem is a line of computers manufactured by Purism, SPC featuring free (libre) software. The laptop line is designed to protect privacy and freedom by omitting non-free (proprietary) software in their operating system and kernel, avoiding the Intel Active Management Technology, and gradually freeing and securing firmware. Librem laptops feature hardware kill switches for the microphone, webcam, Bluetooth and Wi-Fi. == Models == === Laptops === ==== Librem 13, Librem 15 and Librem 14 ==== In 2014, Purism launched a crowdfunding campaign on Crowd Supply to fund the creation and production of the Librem 15 laptop, conceived as a modern alternative to existing open-source hardware laptops, all of which used older hardware. The 15 in the name refers to its 15-inch screen size. The campaign succeeded after extending the original campaign, and the laptops were shipped to backers. In a second revision of the laptop, hardware kill switches for the camera, microphone, Wi-Fi, and Bluetooth were added. After the successful launch of the Librem 15, Purism created another campaign on Crowd Supply for a 13-inch laptop named Librem 13, which also came with hardware kill switches similar to those on the Librem 15v2. The campaign was again successful and the laptops were shipped to customers. Purism announced in December 2016 that it would start shipping from inventory rather than building to order with the new batches of Librem 15 and 13. As of January 2023, Purism has one laptop model in production, the Librem 14. ==== Comparison of laptops ==== === Librem Mini === The Librem Mini is a small form factor desktop computer, which began shipping in June 2020. === Librem 5 === On August 24, 2017, Purism began a crowdfunding campaign for the Librem 5, a smartphone aimed to run 100% free software, which would "[focus] on security by design and privacy protection by default". Purism claimed that the phone would become "the world's first ever IP-native mobile handset, using end-to-end encrypted decentralized communication." Purism cooperated with KDE and GNOME in its development of Librem 5. Security features of the Librem 5 include separation of the CPU from the baseband processor, which, according to Linux Magazine, makes the Librem 5 unique in comparison to other mobile phones. The Librem 5 also features hardware kill switches for Wi-Fi and Bluetooth communication and the phone's camera, microphone, and baseband processor. The default operating system for the Librem 5 is Purism's PureOS, a Debian derivative. The operating system uses a new user interface named Phosh, based on Wayland, wlroots, GTK and GNOME middleware. It is planned that Phosh/Plasma Mobile, Ubuntu Touch, and postmarketOS can also be installed on the phone. The release of the Librem 5 has been postponed several times. In September 2018, Purism announced that the launch date of Librem 5 would be moved from January to April 2019, because of two hardware bugs and the holiday season in Europe and North America. The Librem 5's DevKits for software developers were shipped in December 2018. The launch date was later postponed to the third quarter because of the necessity of further CPU tests. On September 24, 2019, Purism announced that the first batch of Librem 5 phones had begun shipping. The finished version of the Librem 5, known as "Evergreen", was finally shipped on November 18, 2020. === Librem Server === The Librem server is a rack mounted server, released to the public in December 2019. === Librem Key === Announced on 20 September 2018, the Librem Key is a hardware USB security token with multiple features, including integration with a tamper-evident Heads BIOS, which ensures that the Librem laptop Basic Input/Output System (BIOS) was not maliciously altered since the last laptop launch. The Librem Key also features one-time password storage with 3x HMAC-based One-time Password algorithm (HOTP) (RFC 4226) and 15 x Time-based One-time Password algorithm (TOTP) (RFC 6238) and an integrated password manager (16 entries), 40 kbit/s true random number generator, and a tamper-resistant smart card. The key supports type A USB 2.0, has dimensions of 48 x 19 x 7 mm, and weighs 6 g. == Operating system == Initially planning to preload its Librem laptops with the Trisquel operating system, Purism eventually moved off the Trisquel platform to Debian for the 2.0 release of its PureOS Linux operating system. As an alternative to PureOS, Librem laptops are purchasable with Qubes OS preinstalled. In December 2017, the Free Software Foundation added PureOS to its list of endorsed GNU/Linux distributions. == BIOS == In 2015, Purism began research to port the Librem 13 to coreboot but the effort was initially stalled. By the end of the year, a coreboot developer completed an initial port of the Librem 13 and submitted it for review. In December 2016, hardware enablement developer Youness Alaoui joined Purism and was tasked to complete the coreboot port for the original Librem 13 and prepare a port for the second revision of the device. Since summer 2017, new Librem laptops are shipped with coreboot as their standard BIOS, and updates are available for all older models. Purism calls a collection of these six components, involved in the boot process, as PureBoot: Neutralized and disabled Intel Management Engine coreboot A Trusted Platform Module (TPM) chip Heads, which has tamper-evident features to detect if the BIOS or important boot files have been modified Librem Key, Purism's USB security token Multi-factor authentication that unlocks disk encryption using the Librem Key PureBoot protects the users from various attacks like theft, BIOS malware and kernel rootkits, vulnerabilities and malicious code in the Intel Management Engine, and interdiction.
Acquisition of DirecTV by AT&T
AT&T Inc. announced an agreement with the DirecTV Group on May 18, 2014, to acquire the company for $48.5 billion in a joint cash-stock transaction and assumed debts of $18.6 billion for a total offer of $67.1 billion. Due to stalling growth in the wireless sector, AT&T began diversifying into mass media to expand its consumer offerings. After regulatory agencies approved the purchase on July 24, 2015, AT&T briefly became the largest Pay-TV provider. DirecTV was brought under AT&T's communication segment and DirecTV Now was launched on November 30, 2016, as an alternative to cord-cutting. In the years following the purchase, DirecTV lost millions of subscribers across its satellite and streaming services and by 2019, calls grew for AT&T to divest itself off the business. Initially, AT&T rejected these calls and defended the acquisition, but by February 2021, it reached a deal with TPG Inc. to transfer ownership of DirecTV. Under the terms of the agreement, AT&T would retain a 70% majority stake in DirecTV but would no longer oversee its daily operations. The deal was finalized by August 2, 2021, with AT&T receiving $7.1 billion. By July 3, 2025, AT&T sold its majority stake to TPG, ending any ties of involvement. == Background and Development == === AT&T's history === The company to bear the name "AT&T" was founded on March 3, 1885, as American Telephone and Telegraph Company (or AT&T Corporation) by Theodore Newton Vail as a long-distance subsidiary of the Bell Telephone Company. By December 1899, the Bell Telephone's assets were transferred to AT&T, with the latter gaining control of the Bell System, a regional network of local telecom companies. Theodore Vail became AT&T's President in 1907 and under his leadership, AT&T gained a monopoly over the telephone sector in the United States. This near century dominance earned AT&T the nickname of "Ma Bell." In 1974, the U.S. Department of Justice sued AT&T on accounts of antitrust violations. AT&T challenged the lawsuit, but in 1982, it reached a settlement with the DOJ to break apart its Bell System monopoly into seven regional companies. On January 1, 1984, the Bell System came to an end and led to a reshaped telecom industry. One of these regional companies, Southwestern Bell, emerged as the smallest, but after the passage of the 1996 Telecom Act, deregulated telecom rules allowed SBC to become a major telecom company. AT&T briefly became the largest cable and broadband company by the end of the 20th Century, but later deconsolidated to exit those industries. In 2005, SBC acquired its former parent, AT&T, and took on its branding as AT&T Inc, while retaining its previous business history. The newly reincorporated AT&T acquired BellSouth in 2006 and reconstituted much of its former Bell System. === DirecTV's history === == Acquisition Timeline == == Managing DirecTV == == Divestment and Spinoff ==
ELMo
ELMo (embeddings from language model) is a word embedding method for representing a sequence of words as a corresponding sequence of vectors. It was created by researchers at the Allen Institute for Artificial Intelligence, and University of Washington and first released in February 2018. It is a bidirectional LSTM which takes character-level as inputs and produces word-level embeddings, trained on a corpus of about 30 million sentences and 1 billion words. The architecture of ELMo accomplishes a contextual understanding of tokens. Deep contextualized word representation is useful for many natural language processing tasks, such as coreference resolution and polysemy resolution. ELMo was historically important as a pioneer of self-supervised generative pretraining followed by fine-tuning, where a large model is trained to reproduce a large corpus, then the large model is augmented with additional task-specific weights and fine-tuned on supervised task data. It was an instrumental step in the evolution towards transformer-based language modelling. == Architecture == ELMo is a multilayered bidirectional LSTM on top of a token embedding layer. The output of all LSTMs concatenated together consists of the token embedding. The input text sequence is first mapped by an embedding layer into a sequence of vectors. Then two parts are run in parallel over it. The forward part is a 2-layered LSTM with 4096 units and 512 dimension projections, and a residual connection from the first to second layer. The backward part has the same architecture, but processes the sequence back-to-front. The outputs from all 5 components (embedding layer, two forward LSTM layers, and two backward LSTM layers) are concatenated and multiplied by a linear matrix ("projection matrix") to produce a 512-dimensional representation per input token. ELMo was pretrained on a text corpus of 1 billion words. The forward part is trained by repeatedly predicting the next token, and the backward part is trained by repeatedly predicting the previous token. After the ELMo model is pretrained, its parameters are frozen, except for the projection matrix, which can be fine-tuned to minimize loss on specific language tasks. This is an early example of the pretraining-fine-tune paradigm. The original paper demonstrated this by improving state of the art on six benchmark NLP tasks. === Contextual word representation === The architecture of ELMo accomplishes a contextual understanding of tokens. For example, the first forward LSTM of ELMo would process each input token in the context of all previous tokens, and the first backward LSTM would process each token in the context of all subsequent tokens. The second forward LSTM would then incorporate those to further contextualize each token. Deep contextualized word representation is useful for many natural language processing tasks, such as coreference resolution and polysemy resolution. For example, consider the sentenceShe went to the bank to withdraw money.In order to represent the token "bank", the model must resolve its polysemy in context. The first forward LSTM would process "bank" in the context of "She went to the", which would allow it to represent the word to be a location that the subject is going towards. The first backward LSTM would process "bank" in the context of "to withdraw money", which would allow it to disambiguate the word as referring to a financial institution. The second forward LSTM can then process "bank" using the representation vector provided by the first backward LSTM, thus allowing it to represent it to be a financial institution that the subject is going towards. == Historical context == ELMo is one link in a historical evolution of language modelling. Consider a simple problem of document classification, where we want to assign a label (e.g., "spam", "not spam", "politics", "sports") to a given piece of text. The simplest approach is the "bag of words" approach, where each word in the document is treated independently, and its frequency is used as a feature for classification. This was computationally cheap but ignored the order of words and their context within the sentence. GloVe and Word2Vec built upon this by learning fixed vector representations (embeddings) for words based on their co-occurrence patterns in large text corpora. Like BERT (but unlike "bag of words" such as Word2Vec and GloVe), ELMo word embeddings are context-sensitive, producing different representations for words that share the same spelling. It was trained on a corpus of about 30 million sentences and 1 billion words. Previously, bidirectional LSTM was used for contextualized word representation. ELMo applied the idea to a large scale, achieving state of the art performance. After the 2017 publication of Transformer architecture, the architecture of ELMo was changed from a multilayered bidirectional LSTM to a Transformer encoder, giving rise to BERT. BERT has a similar pretrain-fine-tune workflow, but uses a Transformer with implications for more parallelizable training.
Digital classics
Digital classics is the application of the tools of digital humanities to the field of classics, or more broadly to the study of the ancient world. == History == Classics was one of the first of the humanities disciplines to adopt computing approaches; the first references to the use of computing in the classical humanities date to the early 1960s, which might be surprising considering the reputation of the discipline as old-fashioned and stuffily traditionalist. Major projects such as the Thesaurus Linguae Graecae, founded in 1972, and the text collections of the Packard Humanities Institute set the trend, and there are still a significantly large number of ancient world projects among Humanities Computing projects today. Also, the success of traditional scholarly publications in digital guises, such as seen in the Bryn Mawr Classical Review, and the early adoption of hypertext in high profile projects like the Perseus Digital Library helped to legitimize computing in the study of classics in ways that has not always been the case in other areas of the humanities. This apparent paradox may be as a result of the many methodologies and different sources of evidence that classicists have always had to embrace, from literary sources and linguistics, to art history and archaeology, history, philosophy, religious theory, ancient documents such as inscriptions and papyri, and so forth. The fragmentary nature of many of the texts and languages of the ancient world, the scattered evidence from the material culture of ancient Greece and Rome, and the necessity to evaluate all these varieties of evidence in context are particularly likely to benefit from digital approaches such as databases, text markup, image manipulation and machine learning. == Digital classics projects == There are currently several major projects that aim to encourage and develop digital approaches to classical scholarship. The Stoa Consortium at the University of Kentucky distributes news of the discipline, and serves as a peer-reviewed electronic publication venue, and encourages open source approaches to digital classics. The Perseus Project is a digital library that also provides a collection of digital texts and analysis tools to the public; principally (but not exclusively) classical. Digital Classicist is another project and community which shares information and advice about the digital humanities applied to the field of classics. Epigraphy.info is an international open community pursuing a collaborative environment for digital epigraphy. The Liverpool Classics Mailing List is a project which can be subscribed to in which one receives email regarding Classics events around the world, as well as call for papers, studentships and public lectures.