Contrastive Language-Image Pre-training (CLIP) is a technique for training a pair of neural network models, one for image understanding and one for text understanding, using a contrastive objective. This method has enabled broad applications across multiple domains, including cross-modal retrieval, text-to-image generation, and aesthetic ranking. == Algorithm == The CLIP method trains a pair of models contrastively. One model takes in a piece of text as input and outputs a single vector representing its semantic content. The other model takes in an image and similarly outputs a single vector representing its visual content. The models are trained so that the vectors corresponding to semantically similar text-image pairs are close together in the shared vector space, while those corresponding to dissimilar pairs are far apart. To train a pair of CLIP models, one would start by preparing a large dataset of image-caption pairs. During training, the models are presented with batches of N {\displaystyle N} image-caption pairs. Let the outputs from the text and image models be respectively v 1 , . . . , v N , w 1 , . . . , w N {\displaystyle v_{1},...,v_{N},w_{1},...,w_{N}} . Two vectors are considered "similar" if their dot product is large. The loss incurred on this batch is the multi-class N-pair loss, which is a symmetric cross-entropy loss over similarity scores: − 1 N ∑ i ln e v i ⋅ w i / T ∑ j e v i ⋅ w j / T − 1 N ∑ j ln e v j ⋅ w j / T ∑ i e v i ⋅ w j / T {\displaystyle -{\frac {1}{N}}\sum _{i}\ln {\frac {e^{v_{i}\cdot w_{i}/T}}{\sum _{j}e^{v_{i}\cdot w_{j}/T}}}-{\frac {1}{N}}\sum _{j}\ln {\frac {e^{v_{j}\cdot w_{j}/T}}{\sum _{i}e^{v_{i}\cdot w_{j}/T}}}} In essence, this loss function encourages the dot product between matching image and text vectors ( v i ⋅ w i {\displaystyle v_{i}\cdot w_{i}} ) to be high, while discouraging high dot products between non-matching pairs. The parameter T > 0 {\displaystyle T>0} is the temperature, which is parameterized in the original CLIP model as T = e − τ {\displaystyle T=e^{-\tau }} where τ ∈ R {\displaystyle \tau \in \mathbb {R} } is a learned parameter. Other loss functions are possible. For example, Sigmoid CLIP (SigLIP) proposes the following loss function: L = 1 N ∑ i , j ∈ 1 : N f ( ( 2 δ i , j − 1 ) ( e τ w i ⋅ v j + b ) ) {\displaystyle L={\frac {1}{N}}\sum _{i,j\in 1:N}f((2\delta _{i,j}-1)(e^{\tau }w_{i}\cdot v_{j}+b))} where f ( x ) = ln ( 1 + e − x ) {\displaystyle f(x)=\ln(1+e^{-x})} is the negative log sigmoid loss, and the Dirac delta symbol δ i , j {\displaystyle \delta _{i,j}} is 1 if i = j {\displaystyle i=j} else 0. == CLIP models == While the original model was developed by OpenAI, subsequent models have been trained by other organizations as well. === Image model === The image encoding models used in CLIP are typically vision transformers (ViT). The naming convention for these models often reflects the specific ViT architecture used. For instance, "ViT-L/14" means a "vision transformer large" (compared to other models in the same series) with a patch size of 14, meaning that the image is divided into 14-by-14 pixel patches before being processed by the transformer. The size indicator ranges from B, L, H, G (base, large, huge, giant), in that order. Other than ViT, the image model is typically a convolutional neural network, such as ResNet (in the original series by OpenAI), or ConvNeXt (in the OpenCLIP model series by LAION). Since the output vectors of the image model and the text model must have exactly the same length, both the image model and the text model have fixed-length vector outputs, which in the original report is called "embedding dimension". For example, in the original OpenAI model, the ResNet models have embedding dimensions ranging from 512 to 1024, and for the ViTs, from 512 to 768. Its implementation of ViT was the same as the original one, with one modification: after position embeddings are added to the initial patch embeddings, there is a LayerNorm. Its implementation of ResNet was the same as the original one, with 3 modifications: In the start of the CNN (the "stem"), they used three stacked 3x3 convolutions instead of a single 7x7 convolution, as suggested by. There is an average pooling of stride 2 at the start of each downsampling convolutional layer (they called it rect-2 blur pooling according to the terminology of ). This has the effect of blurring images before downsampling, for antialiasing. The final convolutional layer is followed by a multiheaded attention pooling. ALIGN a model with similar capabilities, trained by researchers from Google used EfficientNet, a kind of convolutional neural network. === Text model === The text encoding models used in CLIP are typically Transformers. In the original OpenAI report, they reported using a Transformer (63M-parameter, 12-layer, 512-wide, 8 attention heads) with lower-cased byte pair encoding (BPE) with 49152 vocabulary size. Context length was capped at 76 for efficiency. Like GPT, it was decoder-only, with only causally-masked self-attention. Its architecture is the same as GPT-2. Like BERT, the text sequence is bracketed by two special tokens [SOS] and [EOS] ("start of sequence" and "end of sequence"). Take the activations of the highest layer of the transformer on the [EOS], apply LayerNorm, then a final linear map. This is the text encoding of the input sequence. The final linear map has output dimension equal to the embedding dimension of whatever image encoder it is paired with. These models all had context length 77 and vocabulary size 49408. ALIGN used BERT of various sizes. == Dataset == === WebImageText === The CLIP models released by OpenAI were trained on a dataset called "WebImageText" (WIT) containing 400 million pairs of images and their corresponding captions scraped from the internet. The total number of words in this dataset is similar in scale to the WebText dataset used for training GPT-2, which contains about 40 gigabytes of text data. The dataset contains 500,000 text-queries, with up to 20,000 (image, text) pairs per query. The text-queries were generated by starting with all words occurring at least 100 times in English Wikipedia, then extended by bigrams with high mutual information, names of all Wikipedia articles above a certain search volume, and WordNet synsets. The dataset is private and has not been released to the public, and there is no further information on it. ==== Data preprocessing ==== For the CLIP image models, the input images are preprocessed by first dividing each of the R, G, B values of an image by the maximum possible value, so that these values fall between 0 and 1, then subtracting by [0.48145466, 0.4578275, 0.40821073], and dividing by [0.26862954, 0.26130258, 0.27577711]. The rationale was that these are the mean and standard deviations of the images in the WebImageText dataset, so this preprocessing step roughly whitens the image tensor. These numbers slightly differ from the standard preprocessing for ImageNet, which uses [0.485, 0.456, 0.406] and [0.229, 0.224, 0.225]. If the input image does not have the same resolution as the native resolution (224×224 for all except ViT-L/14@336px, which has 336×336 resolution), then the input image is first scaled by bicubic interpolation, so that its shorter side is the same as the native resolution, then the central square of the image is cropped out. === Others === ALIGN used over one billion image-text pairs, obtained by extracting images and their alt-tags from online crawling. The method was described as similar to how the Conceptual Captions dataset was constructed, but instead of complex filtering, they only applied a frequency-based filtering. Later models trained by other organizations had published datasets. For example, LAION trained OpenCLIP with published datasets LAION-400M, LAION-2B, and DataComp-1B. == Training == In the original OpenAI CLIP report, they reported training 5 ResNet and 3 ViT (ViT-B/32, ViT-B/16, ViT-L/14). Each was trained for 32 epochs. The largest ResNet model took 18 days to train on 592 V100 GPUs. The largest ViT model took 12 days on 256 V100 GPUs. All ViT models were trained on 224×224 image resolution. The ViT-L/14 was then boosted to 336×336 resolution by FixRes, resulting in a model. They found this was the best-performing model. In the OpenCLIP series, the ViT-L/14 model was trained on 384 A100 GPUs on the LAION-2B dataset, for 160 epochs for a total of 32B samples seen. == Applications == === Cross-modal retrieval === CLIP's cross-modal retrieval enables the alignment of visual and textual data in a shared latent space, allowing users to retrieve images based on text descriptions and vice versa, without the need for explicit image annotations. In text-to-image retrieval, users input descriptive text, and CLIP retrieves images with matching embeddings. In image-to-text retrieval, images are used to find related text content. CLIP’s ability to connect vis
List of 3D rendering software
3D rendering software products are the dedicated engines used for rendering computer-generated imagery. This is not the same as 3D modeling software, which involves the creation of 3D models, for which the software listed below can produce realistically rendered visualisations.General-purpose packages which can have their own built-in rendering capabilities are not listed here; these can be found in the list of 3D computer graphics software and list of 3D animation software. See 3D computer graphics software for more discussion about the distinctions.
CloudMinds
CloudMinds is an operator of cloud-based systems for cognitive robotics. == History == CloudMinds was founded in 2015 and is backed by SoftBank, Foxconn, Walden Venture Investments, and Keytone Ventures. CloudMinds has developed research in smart devices, robot control, high-speed security networks, and cloud intelligence integration. CloudMinds developed the Mobile Intranet Cloud Services (MCS) based on these technologies in order to increase the information security of the cloud robot remote control. The technology has been applied in the fields of finance, medicine, the military, public safety, and large-scale manufacturing. == U.S. sanctions == In May 2020, CloudMinds was added to the Bureau of Industry and Security's Entity List due to U.S. national security concerns.
Umple
Umple is a language for both object-oriented programming and modelling with class diagrams and state diagrams. The name Umple is a portmanteau of "UML", "ample" and "Simple", indicating that it is designed to provide ample features to extend programming languages with UML capabilities. == History and philosophy == The design of Umple started in 2008 at the University of Ottawa. Umple was open-sourced and its development was moved to Google Code in early 2011 and to GitHub in 2015. Umple was developed, in part, to address certain problems observed in the modelling community. Most specifically, it was designed to bring modelling and programming into alignment, It was intended to help overcome inhibitions against modelling common in the programmer community. It was also intended to reduce some of the difficulties of model-driven development that arise from the need to use large, expensive or incomplete tools. One design objective is to enable programmers to model in a way they see as natural, by adding modelling constructs to programming languages. == Features and capabilities == Umple can be used to represent in a textual manner many UML modelling entities found in class diagrams and state diagrams. Umple can generate code for these in various programming languages. Currently Umple fully supports Java, C++ and PHP as target programming languages and has functional, but somewhat incomplete support for Ruby. Umple also incorporates various features not related to UML, such as the singleton pattern, keys, immutability, mixins and aspect-oriented code injection. The class diagram notations Umple supports includes classes, interfaces, attributes, associations, generalizations and operations. The code Umple generates for attributes include code in the constructor, 'get' methods and 'set' methods. The generated code differs considerably depending on whether the attribute has properties such as immutability, has a default value, or is part of a key. Umple generates many methods for manipulating, querying and navigating associations. It supports all combinations of UML multiplicity and enforces referential integrity. Umple supports the vast majority of UML state machine notation, including arbitrarily deep nested states, concurrent regions, actions on entry, exit and transition, plus long-lasting activities while in a state. A state machine is treated as an enumerated attribute where the value is controlled by events. Events encoded in the state machine can be methods written by the user, or else generated by the Umple compiler. Events are triggered by calling the method. An event can trigger transitions (subject to guards) in several different state machines. Since a program can be entirely written around one or more state machines, Umple enables automata-based programming. The bodies of methods are written in one of the target programming languages. The same is true for other imperative code such as state machine actions and guards, and code to be injected in an aspect-oriented manner. Such code can be injected before many of the methods in the code Umple generates, for example before or after setting or getting attributes and associations. The Umple notation for UML constructs can be embedded in any of its supported target programming languages. When this is done, Umple can be seen as a pre-processor: The Umple compiler expands the UML constructs into code of the target language. Code in a target language can be passed to the Umple compiler directly; if no Umple-specific notation is found, then the target-language code is emitted unchanged by the Umple compiler. Umple, combined with one of its target languages for imperative code, can be seen and used as a complete programming language. Umple plus Java can therefore be seen as an extension of Java. Alternatively, if imperative code and Umple-specific concepts are left out, Umple can be seen as a way of expressing a large subset of UML in a purely textual manner. Code in one of the supported programming languages can be added in the same manner as UML envisions adding action language code. == License == Umple is licensed under an MIT-style license. == Examples == Here is the classic Hello world program written in Umple (extending Java): This example looks just like Java, because Umple extends other programming languages. With the program saved in a file named HelloWorld.ump, it can be compiled from the command line: $ java -jar umple.jar HelloWorld.ump To run it: $ java HelloWorld The following is a fully executable example showing embedded Java methods and declaration of an association. The following example describes a state machine called status, with states Open, Closing, Closed, Opening and HalfOpen, and with various events that cause transitions from one state to another. class GarageDoor { status { Open { buttonOrObstacle -> Closing; } Closing { buttonOrObstacle -> Opening; reachBottom -> Closed; } Closed { buttonOrObstacle -> Opening; } Opening { buttonOrObstacle -> HalfOpen; reachTop -> Open; } HalfOpen { buttonOrObstacle -> Opening; } } } == Umple use in practice == The first version of the Umple compiler was written in Java, Antlr and Jet (Java Emitter Templates), but in a bootstrapping process, the Java code was converted to Umple following a technique called Umplification. The Antlr and Jet were also later converted to native Umple. Umple is therefore now written entirely in itself, in other words it is self-hosted and serves as its own largest test case. Umple and UmpleOnline have been used in the classroom by several instructors to teach UML and modelling. In one study it was found to help speed up the process of teaching UML, and was also found to improve the grades of students. == Tools == Umple is available as a Jar file so it can be run from the command line, and as an Eclipse plugin. There is also an online tool for Umple called UmpleOnline , which allows a developer to create an Umple system by drawing a UML class diagram, editing Umple code or both. Umple models created with UmpleOnline are stored in the cloud. Currently UmpleOnline only supports Umple programs consisting of a single input file. In addition to code, Umple's tools can generate a variety of other types of output, including user interfaces based on the Umple model.
Magisto
Magisto provided an online video editing tool (both as a web application and a mobile app) for automated video editing and production. In 2019, the company was acquired by Vimeo for an estimated US$200 million. The Magisto app contained a library of music. The music, largely by independent artists, was sorted by mood and is licensed for in-app use. Magisto had a freemium business model where users can create basic video clips for free. In addition, advanced business, professional and personal service tiers are available via various subscription plans, unlocking more features; such as longer videos, HD, premium themes, customization, and control features. == History == Magisto was founded in 2009 as SightEra (LTD) by Oren Boiman (CEO) and Alex Rav-Acha (CTO). Boiman, frustrated with the amount of time it took editing together videos of his daughter, wanted an easier to use application to capture and share videos. Boiman, a computer scientist that graduated from Tel Aviv University, followed with graduate work in computer vision at the Weizmann Institute of Science. Boiman developed several patent-pending image analysis technologies that analyze unedited videos to identify the most interesting parts. The system recognized faces, animals, landscapes, action sequences, movements and other important content within the video, as well as analyzing speech and audio. These scenes are then edited together, along with music and effects. Magisto was launched publicly on September 20, 2011, as a video editing software web application through which users could upload unedited video footage, choose a title and soundtrack and have their video edited for them automatically. On the following day, Magisto was added to YouTube Create's collection of video production applications. The Magisto iPhone app was launched publicly at the 2012 International Consumer Electronics Show (CES) in Las Vegas. At CES, the company was also awarded first place in the 2012 CES Mobile App Showdown. In August 2012, Magisto launched the Android app on Google Play. In September 2012, Magisto launched a Google Chrome App and announced Google Drive integration. In March 2013, Magisto claimed it had 5 million users. Google listed Magisto as an "Editors’ Choice" on its list of "Best Apps of 2013". In September 2013, the company claimed that 10 million users had downloaded the App. In February 2014, Magisto claimed that they had 20 million users, with 2 million new users per month. The company also confirmed investment from Mail.Ru. In September 2014, Magisto rolled out a feature called 'Instagram Ready' which allowed users to upload 15 second clips that are automatically formatted for Instagram. In the same month, Magisto launched a feature for iOS and Android users, called 'Surprise Me', which created video from still photography on users’ smartphones. In October 2014, Magisto was placed 9th on the 2014 Deloitte Israel Technology Fast 50 list and named as a finalist in the Red Herring's Top 100 Europe award. In July 2015, Magisto released an editing theme dedicated to Jerry Garcia. In April 2019, the company was acquired by Vimeo, the IAC-owned platform for hosting, sharing and monetizing streamed video, for an estimated $200 million. === Financing === In 2011, the company received more than $5.5 million in a Series B venture round funding from Magma Venture Partners and Horizons Ventures. In September 2011, at the same time as the public launch of their web application, Magisto announced a $5.5 million Series B funding round led by Li Ka-shing’s Horizons Ventures. Li Ka-Shing is known for making early-stage investments in companies like Facebook, Spotify, SecondMarket and Siri. In October 2013, the company received $13 million in funding from Qualcomm and Sandisk. In 2014, the company received $2 million in Venture Funding from Magma Venture Partners, Qualcomm Ventures, Horizons Ventures and the Mail.Ru Group. == Awards == Magisto won first place at Technonomy3, an annual Internet Technology start-up competition in Israel. Judges of the competition included Jeff Pulver, TechCrunch editor Mike Butcher, investor Yaron Samid, Bessemer Venture Partners Israel partner Adam Fisher and Brad McCarty of The Next Web. Magisto won first place at CES 2012 Mobile app competition, during the launch of Magisto iOS mobile app. Magisto was awarded twice the Google Play Editor's Choice and was part of iPhone App Store Best App awards for 2013 and 2014, and Wired Essential iPad Apps. Magisto was declared by Deloitte as the 7th fastest growing company in Europe, the Middle East, and Africa in 2016.
Sample (graphics)
In computer graphics, a sample is an intersection of a channel and a pixel. The diagram below depicts a 24-bit pixel, consisting of 3 samples for Red, Green, and Blue. In this particular diagram, the Red sample occupies 9 bits, the Green sample occupies 7 bits and the Blue sample occupies 8 bits, totaling 24 bits per pixel. Note that the samples do not have to be equal size and not all samples are mandatory in a pixel. Also, a pixel can consist of more than 3 samples (e.g. 4 samples of the RGBA color space). A sample is related to a subpixel on a physical display.
Transportation Economic Development Impact System
Transportation Economic Development Impact System (TREDIS) is an economic analysis system sold by consulting firm Economic Development Research Group that is used in planning major transportation investments in the US and Canada. The role of economic impact analysis and TREDIS in the transportation planning process is explained in guidebooks of the US Department of Transportation and the American Association of State Highway and Transportation Officials. TREDIS has been most commonly used for assessing the expected economic impacts of statewide highway programs, regional multi-modal plans and public transport investment. Its history and theoretical foundation are explained in peer reviewed journal articles. == How It Works == TREDIS has a series of modules that calculate different forms of impacts and benefits. One module is an accounting framework that calculates user benefits, including impacts on cargo transportation and commuting costs, based on transportation forecasting results. A second module calculates wider economic development benefits, including impacts on business productivity, economic development and multiplier effects from the input-output analysis. It applies an economic model to estimate impacts on jobs, income, gross regional product and business output, by sector of the economy. A third module applies cost-benefit analysis from alternative perspectives.