TeeChart is a charting library for programmers, developed and managed by Steema Software of Girona, Catalonia, Spain. It is available as commercial and non-commercial software. TeeChart has been included in most Delphi and C++Builder products since 1997, and TeeChart Standard currently is part of Embarcadero RAD Studio 13 Florence. TeeChart Pro version is a commercial product that offers shareware releases for all of its formats. The TeeChart Charting Library offers charts, maps and gauges in versions for Delphi VCL/FMX, ActiveX, C# for Microsoft Visual Studio .NET. Full source code has always been available for all versions except the ActiveX version. TeeChart's user interface is translated into 38 languages. == History == The first version of TeeChart was authored in 1995 by David Berneda, co-founder of Steema, using the Borland Delphi Visual Component Library programming environment and TeeChart was first released as a shareware version and made available via Compuserve in the same year. It was written in the first version of Delphi VCL, as a 16-bit Charting Library named TeeChart version 1. The next version of TeeChart was released as a 32-bit library (Delphi 2 supported 32-bit compilation) but was badged as TeeChart VCL v3 to coincide with Borland's naming convention for inclusion on the toolbox palette of Borland Delphi v3 in 1997 and with C++ Builder v3 in 1998. It has been on the Delphi/C++ Builder toolbox palette ever since. The current version is Embarcadero RAD Studio 13 Florence. TeeChart's first ActiveX version named "version 3" too, to match the VCL version's nomenclature, was released in 1998. The version was optimised to work with Microsoft's Visual Studio v97 and v6.0 developer suites that include Visual Basic and Microsoft Visual C++ programming languages. Support for new programming environments followed with TeeChart's first native C# version for Microsoft Visual Studio .NET released in 2002 and TeeChart.Lite for .NET, a free charting component, released for Visual Studio.NET in 2003 and supporting too, Mono (programming). Steema Software released the first native TeeChart Java (programming language) version in 2006 and TeeChart's first native PHP version was released in 2009 and published as open-source in June 2010. Mobile versions of TeeChart, for Android (operating system) devices and Windows Phone 7 devices were released during the first half of 2011. In 2012 TeeChart extended functionality to iPhone/iPad and BlackBerry OS devices and a new JavaScript version was released in the same year to support HTML5 Canvas. In 2013 Steema launched TeeChart for .NET Chart for Windows Store applications and included support for Microsoft's Windows Phone 8 mobile platform. TeeChart for Xamarin.Forms written with 100% C# code and cross-platform support for .NET desktops, Windows Phone, iOS and Android was released in 2014. Also since 2014 Webforms charts now offers HTML5 interactivity. Steema launched TeeChart for Avalonia (software framework) in 2022 and in 2023 .NET_MAUI support was added to the TeeChart for .NET. == Usage == TeeChart is a general purpose charting component designed for use in differing ambits, offering a wide range of aesthetics to chart data. Generally TeeCharts published in the field, in areas where large amounts of data must be interpreted regularly, remain by designer choice in their simplest form to maximize the "data-ink ratio". Sloan Digital Sky Survey, SDSS Web Services' use for charting "Scientific .. plotting of online data" at The Virtual Observatory Spectrum Services reflects that approach. The SDSS chart authors choose to represent data using TeeChart's standard 2D line display. Speed is also a factor when choosing how to most effectively plot data. Realtime data, at frequencies of up to tens or hundreds of data points or more per second, require the most processor economic approach to charting. Computer processing time dedicated to the plotting of data needs to be as lightweight as possible, freeing-up computer tasks "to achieve real-time data acquisition, display and analysis". A critical and stated aspect of many data visualisation applications is the ability to offer interactivity to the user; NASA's document, the Orbital Debris Engineering Model Model ORDEM 3.0 - User's Guide, 2014, states that "The user may manipulate the graphs to zoom, pan, and copy to the clipboard and export to various file types" and Computer and Computing Technologies in Agriculture II, Volume 1, Daoliang, Li; Chunjiang, Zhao (2009), also using TeeChart, states "the properties at any point in the chart can be viewed moving the mouse over it". Writing about control education, Juha Lindfors states "The desired charting functionality (such as zooming and scaling) is achieved..". Charting applications have become increasingly 'onlined', made available either to a wider public or to a territorially remote userbase via networked applications. The World Wide Web (the Web) has become "by far, the most popular Internet protocol" to disseminate online applications. Most major IDEs now offer environments for web application developede aimed at browser hosted applications. Charting components, TeeChart among them, have adapted to provide models that work within a browser environment, often using static images and scripted layering techniques such as Ajax (programming) to offer a level of interactivity, improve response times and hide apparent delay from the user. Options to enrich client, browser-side processing flexibility are exploited by TeeChart libraries via modules that offer 'micro-environments' within the browser, such as the long established ActiveX technology, Adobe Flash, Microsoft Silverlight or Java Applets. Serverside environments offer too, a means to interact with browser based script to dynamically respond to charting requests. Joomla and CodeIgniter are host environments for TeeChart PHP and an example of an Embarcadero IntraWeb VCL designed application using TeeChart, is documented here. == Programmer reference == The Code Project includes a demo that uses TeeChart.Lite, called 'Self-Organizing Feature Maps (Kohonen maps)' written by Bashir Magomedovl and SourceForge includes a Database Stress and Monitor that also uses TeeChart.Lite. Books and information sources that include substantial sections about working with the Delphi version of TeeChart include "Mastering Delphi 6" by Marco Cantù, "C++ Builder 5 developer's guide", a video Delphi Tutorial on charting JPEG compression and support forums and reference pages at TeeChart Support Forums. Non-English language document sources include, in Czech "Myslíme v jazyku Delphi 7: knihovna zkušeného programátora" by Marco Cantù, and Chinese, Delphi 6, Delphi, and Delphi 5.
List of COBOL software and tools
This is a list of software and programming tools for the COBOL programming language, which includes compilers, IDEs, build tools, testing, frameworks, and related projects. == Compilers and runtimes == Fujitsu NetCOBOL — COBOL compiler for Windows, Linux, and mainframes GnuCOBOL — open-source COBOL compiler translating COBOL to C and then compiling with GCC IBM COBOL — mainframe COBOL compiler for IBM z/OS and IBM i platforms Micro Focus COBOL — commercial COBOL compiler and runtime for enterprise systems FairCom RTG – A commercial real-time database and runtime solution developed by FairCom Corporation. It provides integration with COBOL applications for transaction processing and modernization projects, and is used in enterprise environments requiring high-performance data management. == Integrated development environments == Eclipse IDE — with COBOL plugin support, Micro Focus or Bitlang extensions. IBM Developer for z/OS — IDE for COBOL and PL/I mainframe development Micro Focus Visual COBOL — IDE integration for Visual Studio, Visual Studio Code, and Eclipse OpenCOBOLIDE — open-source lightweight IDE for GnuCOBOL Visual Studio Code — with COBOL extensions via Bitlang COBOL and GnuCOBOL Language Server == Frameworks, libraries, and APIs == ACUCOBOL-GT — runtime and API library suite from Micro Focus CICS — IBM middleware for transaction processing in COBOL applications DB2 and IMS APIs — database access libraries commonly used with COBOL applications == Build tools and package managers == Apache Ant — scripting and build automation for COBOL/Java hybrid systems GNU Make — common build tool for compiling COBOL via GnuCOBOL Jenkins — used for CI/CD automation with COBOL builds == Testing and quality assurance == COBOL Check — open-source unit testing framework for COBOL IBM Rational Performance Tester — automated performance testing of web and server-based applications from the Rational Software division of IBM Micro Focus Unit Testing Framework — integrated COBOL unit testing tool == Debugging and profiling tools == GnuCOBOL debug mode — command-line debugging integrated in GnuCOBOL compiler IBM Debug Tool for z/OS — mainframe debugging for COBOL and PL/I Micro Focus Animator — step-through debugger for COBOL code
Leela Chess Zero
Leela Chess Zero (abbreviated as LCZero, lc0) is a free, open-source chess engine and volunteer computing project based on Google's AlphaZero engine. It was spearheaded by Gary Linscott, a developer for the Stockfish chess engine, and adapted from the Leela Zero Go engine. Like Leela Zero and AlphaGo Zero, early iterations of Leela Chess Zero started with no intrinsic chess-specific knowledge other than the basic rules of the game. It learned how to play chess through reinforcement learning from repeated self-play, using a distributed computing network coordinated at the Leela Chess Zero website. However, as of November 2024 most models used by the engine are trained through supervised learning on data generated by previous reinforcement learning runs. As of June 2025, Leela Chess Zero has played over 2.5 billion games against itself, playing around 1 million games every day, and is capable of play at a level that is comparable with Stockfish, the leading conventional chess program. == History == The Leela Chess Zero project was first announced on TalkChess.com on January 9, 2018, as an open-source, self-learning chess engine attempting to recreate the success of AlphaZero. Within the first few months of training, Leela Chess Zero had already reached the Grandmaster level, surpassing the strength of early releases of Rybka, Stockfish, and Komodo, despite evaluating orders of magnitude fewer positions due to the size of the deep neural network it uses as its evaluation function. In December 2018, the AlphaZero team published a paper in Science magazine revealing previously undisclosed details of the architecture and training parameters used for AlphaZero. These changes were soon incorporated into Leela Chess Zero and increased both its strength and training efficiency. Work on Leela Chess Zero has informed the AobaZero project for shogi. The engine has been rewritten and carefully iterated upon since its inception, and since 2019 has run on multiple backends, allowing it to run on both CPU and GPU. The engine can be configured to use different weights, including even different architectures. This same mechanism of substitutable weights can also be used for alternative chess rules, such as for the Fischer Random Chess variant, which was done in 2019. == Neural network == Like AlphaZero, Leela Chess Zero employs neural networks which output both a policy vector, a distribution over subsequent moves used to guide search, and a position evaluation. These neural networks are designed to run on GPU, unlike traditional engines. It originally used residual neural networks, but in 2022 switched to using a transformer-based architecture designed by Daniel Monroe and Philip Chalmers. These models represent a chessboard as a sequence of 64 tokens and apply a trunk consisting of a stack of Post-LN encoder layers, outputting a sequence of 64 encoded tokens which is used to generate a position evaluation and a distribution over subsequent moves. They use a custom domain-specific position encoding called smolgen to improve the self-attention layer. As of November 2024, the models used by the engine are significantly larger and more efficient than the residual network used by AlphaZero, reportedly achieving grandmaster-level strength at one position evaluation per move. These models are able to detect and exploit positional features like trapped pieces and fortresses to outmaneuver traditional engines, giving Leela a unique playstyle. There is also evidence that they are able to perform look-ahead. == Program and use == Like AlphaZero, Leela Chess Zero learns through reinforcement learning, continually training on data generated through self-play. However, unlike AlphaZero, Leela Chess Zero decentralizes its data generation through distributed computing, with volunteers generating self-play data on local hardware which is fed to the reinforcement algorithm. In order to contribute training games, volunteers must download the latest non-release candidate (non-rc) version of the engine and the client. The client connects to the Leela Chess Zero server and iteratively receives the latest neural network version and produces self-play games which are sent back to the server and use to train the network. In order to run the Leela Chess Zero engine, two components are needed: the engine binary used to perform search, and a network used to evaluate positions. The client, which is used to contribute training data to the project, is not needed for this purpose. Older networks can also be downloaded and used by placing those networks in the folder with the Lc0 binary. == Spinoffs == In season 15 of the Top Chess Engine Championship, the engine AllieStein competed alongside Leela. AllieStein is a combination of two different spinoffs from Leela: Allie, which uses the same neural network as Leela, but has a unique search algorithm for exploring different lines of play, and Stein, a network which was trained using supervised learning on existing game data from games between other engines. While neither of these projects were admitted to TCEC separately due to their similarity to Leela, the combination of Allie's search algorithm with the Stein network, called AllieStein, was deemed unique enough to warrant its inclusion in the competition. In early 2021, the LcZero blog announced Ceres, a transliteration of the engine to C# which introduced several algorithmic improvements. The engine has performed competitively in tournaments, achieving third place in the TCEC Swiss 7 and fourth place in the TCEC Cup 14. In 2024, the CeresTrain framework was announced to support training deep neural networks for chess in PyTorch. == Competition results == In April 2018, Leela Chess Zero became the first engine using a deep neural network to enter the Top Chess Engine Championship (TCEC), during Season 12 in the lowest division, Division 4. Out of 28 games, it won one, drew two, and lost the remainder; its sole victory came from a position in which its opponent, Scorpio 2.82, crashed in three moves. However, it improved quickly. In July 2018, Leela placed seventh out of eight competitors at the 2018 World Computer Chess Championship. In August 2018, it won division 4 of TCEC season 13 with a record of 14 wins, 12 draws, and 2 losses. In Division 3, Leela scored 16/28 points, finishing third behind Ethereal, which scored 22.5/28 points, and Arasan on tiebreak. By September 2018, Leela had become competitive with the strongest engines in the world. In the 2018 Chess.com Computer Chess Championship (CCCC), Leela placed fifth out of 24 entrants. The top eight engines advanced to round 2, where Leela placed fourth. Leela then won the 30-game match against Komodo to secure third place in the tournament. Leela participated in the "TCEC Cup", an event in which engines from different TCEC divisions can play matches against one another. Leela defeated higher-division engines Laser, Ethereal and Fire before finally being eliminated by Stockfish in the semi-finals. In December 2018, Leela participated in Season 14 of the Top Chess Engine Championship. Leela dominated divisions 3, 2, and 1, easily finishing first in all of them. In the premier division, Stockfish dominated while Houdini, Komodo and Leela competed for second place. It came down to a final-round game where Leela needed to hold Stockfish to a draw with black to finish second ahead of Komodo. Leela managed this and therefore met Stockfish in the superfinal. In a back and forth match, first Stockfish and then Leela took three game leads before Stockfish won by the narrow margin of 50.5–49.5. In February 2019, Leela scored its first major tournament win when it defeated Houdini in the final of the second TCEC cup. Leela did not lose a game the entire tournament. In April 2019, Leela won the Chess.com Computer Chess Championship 7: Blitz Bonanza, becoming the first neural-network project to take the title. In the season 15 of the Top Chess Engine Championship (May 2019), Leela defended its TCEC Cup title, this time defeating Stockfish with a score of 5.5–4.5 (+2 =7 −1) in the final after Stockfish blundered a seven-man tablebase draw. Leela also won the Superfinal for the first time, scoring 53.5–46.5 (+14 −7 =79) versus Stockfish, including winning as both white and black in the same predetermined opening in games 61 and 62. Season 16 of TCEC saw Leela finish in third place in premier division, missing qualification for the Superfinal to Stockfish and the new deep neural network engine AllieStein. Leela was the only engine not to suffer any losses in the Premier division, and defeated Stockfish in one of the six games they played. However, Leela only managed to score nine wins, while AllieStein and Stockfish both scored 14 wins. This inability to defeat weaker engines led to Leela finishing third, half a point behind AllieStein and a point behind Stockfish. In the fourth TCEC Cup, Leela was seeded first as the defending champion,
Oracle Database
Oracle AI Database (commonly referred to as Oracle Database, Oracle DBMS, Oracle Autonomous Database, or simply as Oracle) is a proprietary multi-model database management system produced and marketed by Oracle Corporation. It is a database commonly used for running online transaction processing (OLTP), data warehousing (DW) and mixed (OLTP & DW) database workloads. Oracle AI Database uses SQL for database updating and retrieval. Oracle Database runs on-premises, on Oracle engineered systems such as Oracle Exadata, on Oracle Cloud Infrastructure, and as a managed Autonomous Database service. It is also offered inside Microsoft Azure, Google Cloud, and Amazon Web Services data centers through Oracle's multicloud offerings. The current long-term support release, Oracle AI Database 26ai, became available in the cloud and on Oracle engineered systems in October 2025 and on-premises for Linux x86-64 in January 2026. == History == Larry Ellison and his two friends and former co-workers, Bob Miner and Ed Oates, started a consultancy called Software Development Laboratories (SDL) in 1977, later Oracle Corporation. SDL developed the original version of the Oracle software. The name Oracle comes from the code-name of a Central Intelligence Agency-funded project Ellison had worked on while formerly employed by Ampex; the CIA was Oracle's first customer, and allowed the company to use the code name for the new product. Ellison wanted his database to be compatible with IBM System R, but that company's Don Chamberlin declined to release its error codes. By 1985 Oracle advertised, however, that "Programs written for SQL/DS or DB2 will run unmodified" on the many non-IBM mainframes, minicomputers, and microcomputers its database supported "Because all versions of ORACLE are identical". Later releases introduced capabilities associated with successive eras of the product, including PL/SQL stored procedures and triggers in Oracle7 (1992), Real Application Clusters in Oracle9i (2001), grid infrastructure and automatic management in Oracle 10g (2003), the multitenant architecture and In-Memory Column Store in Oracle Database 12c (2013), and AI Vector Search and JSON Relational Duality in Oracle Database 23ai (2024). In October 2025 Oracle rebranded the 23ai line as Oracle AI Database 26ai. (see Release History) == Architecture == An Oracle Database system consists of an instance and a database. The instance is a set of memory structures and background processes; the database is the set of files that store data. The instance exists only in memory, and a single instance is associated with one multitenant container database. The principal memory structures are the System Global Area, which is shared, and the Program Global Areas, which are private to individual processes. The shared pool, database buffer cache, and redo log buffer are components of the System Global Area, and the optional In-Memory Column Store also resides there. Background processes operate on the database files and use these memory structures; they include the database writer, the log writer, the checkpoint process, and the system and process monitor processes. Server processes handle connections from client programs and run their SQL statements. Storage is organized logically and physically. Logically, data is held in tablespaces composed of segments, extents, and data blocks. Physically, the database comprises datafiles, control files, and online redo log files, with archived redo logs supporting media recovery. == High Availability and Scalability == Oracle Database includes several technologies for high availability, disaster recovery, and scale. Oracle Real Application Clusters allows multiple instances on separate servers to access one shared database concurrently; it was introduced with Oracle9i in 2001. Oracle Data Guard maintains standby databases synchronized with a primary database, and Active Data Guard additionally allows read-only workloads on a standby while it applies changes. Oracle GoldenGate performs logical replication and data integration across heterogeneous systems. Native sharding, introduced in Oracle Database 12c Release 2, distributes one logical database across independent shards. Oracle Exadata is an engineered system that pairs database servers with storage servers and offloads operations such as filtering to the storage tier; it is available on-premises, in Oracle Cloud Infrastructure, and through Cloud@Customer. == Notable Features == AI Vector Search adds a vector data type, vector indexes, and vector distance operators to the database. These allow similarity search over machine-learning embeddings to be expressed in SQL and combined with queries over relational, JSON, spatial, and graph data. It became generally available in Oracle Database 23ai. JSON Relational Duality exposes the same data both as relational tables and as JSON documents through duality views, so that an application can read and write either representation of the data. It became generally available in Oracle Database 23ai. In-Memory Column Store maintains a column-oriented copy of selected tables in memory in addition to the row-oriented format, and the optimizer can use the columnar copy for analytic queries. It was introduced in Oracle Database 12c Release 1.Partitioning divides large tables and indexes into independently managed pieces. Advanced Compression and Hybrid Columnar Compression are compression features for transactional and warehouse data respectively. == Data Types == Oracle AI Database supports a variety of data types and data models within a single system. These include traditional relational data types as well as semi-structured, unstructured, and specialized data formats, enabling different types of data to be stored and queried together. == Releases and versions == Oracle products follow a custom release-numbering and -naming convention. The "ai" in the current release, Oracle AI Database 26ai, stands for "Artificial Intelligence". Previous releases (e.g. Oracle Database 19c, 10g, and Oracle9i Database) have used suffixes of "c", "g", and "i" which stand for "Cloud", "Grid", and "Internet" respectively. Prior to the release of Oracle8i Database, no suffixes featured in Oracle AI Database naming conventions. There was no v1 of Oracle AI Database, as Ellison "knew no one would want to buy version 1". For some database releases, Oracle also provides an Express Edition (XE) that is free to use. Oracle AI Database release numbering has used the following codes: The Introduction to Oracle AI Database includes a brief history on some of the key innovations introduced with each major release of Oracle AI Database. See My Oracle Support (MOS) note Release Schedule of Current Database Releases (Doc ID 742060.1) for the current Oracle AI Database releases and their patching end dates. == Patch updates and security alerts == Prior to Oracle Database 18c, Oracle Corporation released Critical Patch Updates (CPUs) and Security Patch Updates (SPUs) and Security Alerts to close security vulnerabilities. These releases are issued quarterly; some of these releases have updates issued prior to the next quarterly release. Starting with Oracle Database 18c, Oracle Corporation releases Release Updates (RUs) and Release Update Revisions (RURs). RUs usually contain security, regression (bug), optimizer, and functional fixes which may include feature extensions as well. RURs include all fixes from their corresponding RU but only add new security and regression fixes. However, no new optimizer or functional fixes are included. == Competition == In the market for relational databases, Oracle AI Database competes against commercial products such as IBM Db2 and Microsoft SQL Server. Oracle and IBM tend to battle for the mid-range database market on Unix and Linux platforms, while Microsoft dominates the mid-range database market on Microsoft Windows platforms. However, since they share many of the same customers, Oracle and IBM tend to support each other's products in many middleware and application categories (for example: WebSphere, PeopleSoft, and Siebel Systems CRM), and IBM's hardware divisions work closely with Oracle on performance-optimizing server-technologies (for example, Linux on IBM Z). Niche commercial competitors include Teradata (in data warehousing and business intelligence), Software AG's ADABAS, Sybase, and IBM's Informix, among many others. In the cloud, Oracle AI Database competes against the database services of AWS, Microsoft Azure, and Google Cloud Platform. Increasingly, the Oracle AI Database products compete against open-source software relational and non-relational database systems such as PostgreSQL, MongoDB, Couchbase, Neo4j, ArangoDB and others. Oracle acquired Innobase, supplier of the InnoDB codebase to MySQL, in part to compete better against open source alternatives, and acquired Sun Microsystems, owner of MySQL, in 2010. Database products licensed as open
Agent Communications Language
Agent Communication Language (ACL) consists of computer communication protocols that are intended for AI agents to communicate with each other. In 2007, protocols of this nature were proposed which include: FIPA-ACL (by the Foundation for Intelligent Physical Agents, a standardization consortium) KQML (Knowledge Query and Manipulation Language) After the surge in Generative AI with the use of Transformers and Large language models, the definition of agent has shifted away from physical agents to signify software systems built using the principles of Agentic AI. A new protocol to emerge in this area is Natural Language Interaction Protocol (NLIP). NLIP is an application-level communication protocol defined between AI Agents or between a human and an AI agent. Ecma International; a standards body which develops and publishes international standards for the information and communication industry; published on 10 December 2025 five new standards and one technical report defining the Natural Language Interaction Protocol (NLIP). As a result, we can define agent communication protocols into two categories: ontology based agent communication protocols and generative AI based agent communication protocols. Ontology based agent communication protocols use a common ontology to be used between agents. An ontology is a part of the agent's knowledge base that describes what kind of things an agent can deal with and how they are related to each other. FIPA-ACL and KQML are examples of such protocols. These protocols rely on speech act theory developed by Searle in the 1960s and enhanced by Winograd and Flores in the 1970s. They define a set of performatives, also called Communicative Acts, and their meaning (e.g. ask-one). The content of the performative is not standardized, but varies from system to system. Implementation support of FIPA-ACL is included in FIPA-OS and Jade. Generative AI based agent communication protocols such as NLIP do not require a shared ontology among communicating agents. In its stead, they use generative AI models to translate natural language text, images, videos or other modalities of data into a local ontology. This provides for hot-extensibility where the same protocol can be used for multiple communication needs, and simplifies version control since different agents can use different versions of a shared ontology. NLIP has been designed with security considerations in mind. The specification and standards comprising NLIP are developed and maintained by Ecma Technical Community 56.
Contrastive Language-Image Pre-training
Contrastive Language-Image Pre-training (CLIP) is a technique for training a pair of neural network models, one for image understanding and one for text understanding, using a contrastive objective. This method has enabled broad applications across multiple domains, including cross-modal retrieval, text-to-image generation, and aesthetic ranking. == Algorithm == The CLIP method trains a pair of models contrastively. One model takes in a piece of text as input and outputs a single vector representing its semantic content. The other model takes in an image and similarly outputs a single vector representing its visual content. The models are trained so that the vectors corresponding to semantically similar text-image pairs are close together in the shared vector space, while those corresponding to dissimilar pairs are far apart. To train a pair of CLIP models, one would start by preparing a large dataset of image-caption pairs. During training, the models are presented with batches of N {\displaystyle N} image-caption pairs. Let the outputs from the text and image models be respectively v 1 , . . . , v N , w 1 , . . . , w N {\displaystyle v_{1},...,v_{N},w_{1},...,w_{N}} . Two vectors are considered "similar" if their dot product is large. The loss incurred on this batch is the multi-class N-pair loss, which is a symmetric cross-entropy loss over similarity scores: − 1 N ∑ i ln e v i ⋅ w i / T ∑ j e v i ⋅ w j / T − 1 N ∑ j ln e v j ⋅ w j / T ∑ i e v i ⋅ w j / T {\displaystyle -{\frac {1}{N}}\sum _{i}\ln {\frac {e^{v_{i}\cdot w_{i}/T}}{\sum _{j}e^{v_{i}\cdot w_{j}/T}}}-{\frac {1}{N}}\sum _{j}\ln {\frac {e^{v_{j}\cdot w_{j}/T}}{\sum _{i}e^{v_{i}\cdot w_{j}/T}}}} In essence, this loss function encourages the dot product between matching image and text vectors ( v i ⋅ w i {\displaystyle v_{i}\cdot w_{i}} ) to be high, while discouraging high dot products between non-matching pairs. The parameter T > 0 {\displaystyle T>0} is the temperature, which is parameterized in the original CLIP model as T = e − τ {\displaystyle T=e^{-\tau }} where τ ∈ R {\displaystyle \tau \in \mathbb {R} } is a learned parameter. Other loss functions are possible. For example, Sigmoid CLIP (SigLIP) proposes the following loss function: L = 1 N ∑ i , j ∈ 1 : N f ( ( 2 δ i , j − 1 ) ( e τ w i ⋅ v j + b ) ) {\displaystyle L={\frac {1}{N}}\sum _{i,j\in 1:N}f((2\delta _{i,j}-1)(e^{\tau }w_{i}\cdot v_{j}+b))} where f ( x ) = ln ( 1 + e − x ) {\displaystyle f(x)=\ln(1+e^{-x})} is the negative log sigmoid loss, and the Dirac delta symbol δ i , j {\displaystyle \delta _{i,j}} is 1 if i = j {\displaystyle i=j} else 0. == CLIP models == While the original model was developed by OpenAI, subsequent models have been trained by other organizations as well. === Image model === The image encoding models used in CLIP are typically vision transformers (ViT). The naming convention for these models often reflects the specific ViT architecture used. For instance, "ViT-L/14" means a "vision transformer large" (compared to other models in the same series) with a patch size of 14, meaning that the image is divided into 14-by-14 pixel patches before being processed by the transformer. The size indicator ranges from B, L, H, G (base, large, huge, giant), in that order. Other than ViT, the image model is typically a convolutional neural network, such as ResNet (in the original series by OpenAI), or ConvNeXt (in the OpenCLIP model series by LAION). Since the output vectors of the image model and the text model must have exactly the same length, both the image model and the text model have fixed-length vector outputs, which in the original report is called "embedding dimension". For example, in the original OpenAI model, the ResNet models have embedding dimensions ranging from 512 to 1024, and for the ViTs, from 512 to 768. Its implementation of ViT was the same as the original one, with one modification: after position embeddings are added to the initial patch embeddings, there is a LayerNorm. Its implementation of ResNet was the same as the original one, with 3 modifications: In the start of the CNN (the "stem"), they used three stacked 3x3 convolutions instead of a single 7x7 convolution, as suggested by. There is an average pooling of stride 2 at the start of each downsampling convolutional layer (they called it rect-2 blur pooling according to the terminology of ). This has the effect of blurring images before downsampling, for antialiasing. The final convolutional layer is followed by a multiheaded attention pooling. ALIGN a model with similar capabilities, trained by researchers from Google used EfficientNet, a kind of convolutional neural network. === Text model === The text encoding models used in CLIP are typically Transformers. In the original OpenAI report, they reported using a Transformer (63M-parameter, 12-layer, 512-wide, 8 attention heads) with lower-cased byte pair encoding (BPE) with 49152 vocabulary size. Context length was capped at 76 for efficiency. Like GPT, it was decoder-only, with only causally-masked self-attention. Its architecture is the same as GPT-2. Like BERT, the text sequence is bracketed by two special tokens [SOS] and [EOS] ("start of sequence" and "end of sequence"). Take the activations of the highest layer of the transformer on the [EOS], apply LayerNorm, then a final linear map. This is the text encoding of the input sequence. The final linear map has output dimension equal to the embedding dimension of whatever image encoder it is paired with. These models all had context length 77 and vocabulary size 49408. ALIGN used BERT of various sizes. == Dataset == === WebImageText === The CLIP models released by OpenAI were trained on a dataset called "WebImageText" (WIT) containing 400 million pairs of images and their corresponding captions scraped from the internet. The total number of words in this dataset is similar in scale to the WebText dataset used for training GPT-2, which contains about 40 gigabytes of text data. The dataset contains 500,000 text-queries, with up to 20,000 (image, text) pairs per query. The text-queries were generated by starting with all words occurring at least 100 times in English Wikipedia, then extended by bigrams with high mutual information, names of all Wikipedia articles above a certain search volume, and WordNet synsets. The dataset is private and has not been released to the public, and there is no further information on it. ==== Data preprocessing ==== For the CLIP image models, the input images are preprocessed by first dividing each of the R, G, B values of an image by the maximum possible value, so that these values fall between 0 and 1, then subtracting by [0.48145466, 0.4578275, 0.40821073], and dividing by [0.26862954, 0.26130258, 0.27577711]. The rationale was that these are the mean and standard deviations of the images in the WebImageText dataset, so this preprocessing step roughly whitens the image tensor. These numbers slightly differ from the standard preprocessing for ImageNet, which uses [0.485, 0.456, 0.406] and [0.229, 0.224, 0.225]. If the input image does not have the same resolution as the native resolution (224×224 for all except ViT-L/14@336px, which has 336×336 resolution), then the input image is first scaled by bicubic interpolation, so that its shorter side is the same as the native resolution, then the central square of the image is cropped out. === Others === ALIGN used over one billion image-text pairs, obtained by extracting images and their alt-tags from online crawling. The method was described as similar to how the Conceptual Captions dataset was constructed, but instead of complex filtering, they only applied a frequency-based filtering. Later models trained by other organizations had published datasets. For example, LAION trained OpenCLIP with published datasets LAION-400M, LAION-2B, and DataComp-1B. == Training == In the original OpenAI CLIP report, they reported training 5 ResNet and 3 ViT (ViT-B/32, ViT-B/16, ViT-L/14). Each was trained for 32 epochs. The largest ResNet model took 18 days to train on 592 V100 GPUs. The largest ViT model took 12 days on 256 V100 GPUs. All ViT models were trained on 224×224 image resolution. The ViT-L/14 was then boosted to 336×336 resolution by FixRes, resulting in a model. They found this was the best-performing model. In the OpenCLIP series, the ViT-L/14 model was trained on 384 A100 GPUs on the LAION-2B dataset, for 160 epochs for a total of 32B samples seen. == Applications == === Cross-modal retrieval === CLIP's cross-modal retrieval enables the alignment of visual and textual data in a shared latent space, allowing users to retrieve images based on text descriptions and vice versa, without the need for explicit image annotations. In text-to-image retrieval, users input descriptive text, and CLIP retrieves images with matching embeddings. In image-to-text retrieval, images are used to find related text content. CLIP’s ability to connect vis
Horovod (machine learning)
Horovod is a free and open-source distributed deep learning training framework for TensorFlow, Keras, PyTorch and Apache MXNet. It is designed to scale existing single-GPU training scripts to efficiently run on multiple GPUs and computer nodes with minimal code changes, using synchronous data-parallel training based on the ring-allreduce communication pattern. Horovod was initially developed at Uber and released as an open-source project in 2017, and is now hosted by the LF AI & Data Foundation, a project of the Linux Foundation. == History == Horovod was created at Uber as part of the company's internal machine learning platform Michelangelo to simplify scaling TensorFlow models across many GPUs. The first public release of the library, version 0.9.0, was tagged on GitHub in August 2017 under the Apache 2.0 licence. In October 2017, Uber Engineering publicly introduced Horovod as an open-source component of its deep learning toolkit. In February 2018 Alexander Sergeev and Mike Del Balso published a technical paper describing Horovod's design and benchmarking its performance on up to 512 GPUs, showing near-linear scaling for several image-classification models when compared with single-GPU baselines. In December 2018 Uber contributed Horovod to the LF Deep Learning Foundation (later LF AI & Data), making it a Linux Foundation project. Horovod entered incubation under LF AI & Data and graduated as a full foundation project in 2020. Since its initial release the project has expanded beyond TensorFlow to provide APIs for PyTorch, Keras and Apache MXNet, as well as integrations with frameworks such as Apache Spark and Ray, support for elastic training, and tooling for automated performance tuning and profiling. == Design and features == Horovod core principles are based on the MPI concepts size, rank, local rank, allreduce, allgather, broadcast, and alltoall. Horovod implements synchronous data-parallel training, in which each worker process maintains a replica of the model and computes gradients on different mini-batches of data. The gradients are aggregated across workers using the ring-allreduce communication pattern rather than a central parameter server, which reduces communication bottlenecks and can improve scaling on multi-GPU clusters. Communication is built on top of collective-communication libraries such as MPI, NCCL, Gloo and Intel oneCCL, and supports both GPU and CPU training. In the benchmark experiments reported in the original paper, Horovod achieved around 90% scaling efficiency on 512 GPUs for the ResNet-101 and Inception v3 convolutional neural networks, and around 68% scaling efficiency for the VGG-16 model. Horovod can be deployed on-premises or in cloud environments and is distributed as a Python package with optional GPU support via CUDA. The official documentation provides guides for running Horovod with Docker, Kubernetes (including via Kubeflow and the MPI Operator), commercial platforms such as Databricks, and cluster schedulers such as LSF. == Adoption and use cases == Within Uber, Horovod has been used for applications including autonomous driving research, fraud detection and trip forecasting. Major cloud providers have integrated Horovod into their managed machine learning offerings. Amazon Web Services supports distributed training with Horovod in services such as Amazon SageMaker and AWS Deep Learning Containers, while Microsoft Azure documents Horovod-based training workflows for Azure Synapse Analytics. Technical guides from academic and research computing centres, including Purdue University and the NASA Advanced Supercomputing programme, describe Horovod-based workflows for multi-GPU training on supercomputers and clusters. Horovod is also used in conjunction with Apache Spark and dedicated storage systems as part of end-to-end data processing and model-training pipelines. Industry blogs and technical tutorials describe deployments of Horovod on Kubernetes, on-premises clusters and cloud-managed Kubernetes services such as Amazon EKS.