Zero-shot learning (ZSL) is a problem setup in deep learning where, at test time, a learner observes samples from classes which were not observed during training, and needs to predict the class that they belong to. The name is a play on words based on the earlier concept of one-shot learning, in which classification can be learned from only one, or a few, examples. Zero-shot methods generally work by associating observed and non-observed classes through some form of auxiliary information, which encodes observable distinguishing properties of objects. For example, given a set of images of animals to be classified, along with auxiliary textual descriptions of what animals look like, an artificial intelligence model which has been trained to recognize horses, but has never been given a zebra, can still recognize a zebra when it also knows that zebras look like striped horses. This problem is widely studied in computer vision, natural language processing, and machine perception. == Background and history == The first paper on zero-shot learning in natural language processing appeared in a 2008 paper by Chang, Ratinov, Roth, and Srikumar, at the AAAI'08, but the name given to the learning paradigm there was dataless classification. The first paper on zero-shot learning in computer vision appeared at the same conference, under the name zero-data learning. The term zero-shot learning itself first appeared in the literature in a 2009 paper from Palatucci, Hinton, Pomerleau, and Mitchell at NIPS'09. This terminology was repeated later in another computer vision paper and the term zero-shot learning caught on, as a take-off on one-shot learning that was introduced in computer vision years earlier. In computer vision, zero-shot learning models learned parameters for seen classes along with their class representations and rely on representational similarity among class labels so that, during inference, instances can be classified into new classes. In natural language processing, the key technical direction developed builds on the ability to "understand the labels"—represent the labels in the same semantic space as that of the documents to be classified. This supports the classification of a single example without observing any annotated data, the purest form of zero-shot classification. The original paper made use of the Explicit Semantic Analysis (ESA) representation but later papers made use of other representations, including dense representations. This approach was also extended to multilingual domains, fine entity typing and other problems. Moreover, beyond relying solely on representations, the computational approach has been extended to depend on transfer from other tasks, such as textual entailment and question answering. The original paper also points out that, beyond the ability to classify a single example, when a collection of examples is given, with the assumption that they come from the same distribution, it is possible to bootstrap the performance in a semi-supervised like manner (or transductive learning). Unlike standard generalization in machine learning, where classifiers are expected to correctly classify new samples to classes they have already observed during training, in ZSL, no samples from the classes have been given during training the classifier. It can therefore be viewed as an extreme case of domain adaptation. == Prerequisite information for zero-shot classes == Naturally, some form of auxiliary information has to be given about these zero-shot classes, and this type of information can be of several types. Learning with attributes: classes are accompanied by pre-defined structured description. For example, for bird descriptions, this could include "red head", "long beak". These attributes are often organized in a structured compositional way, and taking that structure into account improves learning. While this approach was used mostly in computer vision, there are some examples for it also in natural language processing. Learning from textual description. As pointed out above, this has been the key direction pursued in natural language processing. Here class labels are taken to have a meaning and are often augmented with definitions or free-text natural-language description. This could include for example a wikipedia description of the class. Class-class similarity. Here, classes are embedded in a continuous space. A zero-shot classifier can predict that a sample corresponds to some position in that space, and the nearest embedded class is used as a predicted class, even if no such samples were observed during training. == Generalized zero-shot learning == The above ZSL setup assumes that at test time, only zero-shot samples are given, namely, samples from new unseen classes. In generalized zero-shot learning, samples from both new and known classes, may appear at test time. This poses new challenges for classifiers at test time, because it is very challenging to estimate if a given sample is new or known. Some approaches to handle this include: a gating module, which is first trained to decide if a given sample comes from a new class or from an old one, and then, at inference time, outputs either a hard decision, or a soft probabilistic decision a generative module, which is trained to generate feature representation of the unseen classes—a standard classifier can then be trained on samples from all classes, seen and unseen. == Domains of application == Zero shot learning has been applied to the following fields: image classification semantic segmentation image generation object detection natural language processing computational biology abstract reasoning
Information space analysis
Within the field of information science, information space analysis is a deterministic method, enhanced by machine intelligence, for locating and assessing resources for team-centric efforts. Organizations need to be able to quickly assemble teams backed by the support services, information, and material to do the job. To do so, these teams need to find and assess sources of services that are potential participants in the team effort. To support this initial team and resource development, information needs to be developed via analysis tools that help make sense of sets of data sources in an Intranet or Internet. Part of the process is to characterize them, partition them, and sort and filter them. These tools focus on three key issues in forming a collaborative team: Help individuals responsible for forming the team understand what is available. Assist team members in identifying the structure and categorize the information available to them in a manner specifically suited to the task at hand. Aid team members to understand the mappings of their information between their organization and that used by others who might participate. Information space analysis tools combine multiple methods to assist in this task. This causes the tools to be particularly well-suited to integrating additional technologies in order to create specialized systems.
DreamBooth
DreamBooth is a deep learning generation model used to personalize existing text-to-image models by fine-tuning. It was developed by researchers from Google Research and Boston University in 2022. Originally developed using Google's own Imagen text-to-image model, DreamBooth implementations can be applied to other text-to-image models, where it can allow the model to generate more fine-tuned and personalized outputs after training on three to five images of a subject. == Technology == Pretrained text-to-image diffusion models, while often capable of offering a diverse range of different image output types, lack the specificity required to generate images of lesser-known subjects, and are limited in their ability to render known subjects in different situations and contexts. The methodology used to run implementations of DreamBooth involves the fine-tuning the full UNet component of the diffusion model using a few images (usually 3--5) depicting a specific subject. Images are paired with text prompts that contain the name of the class the subject belongs to, plus a unique identifier. As an example, a photograph of a [Nissan R34 GTR] car, with car being the class); a class-specific prior preservation loss is applied to encourage the model to generate diverse instances of the subject based on what the model is already trained on for the original class. Pairs of low-resolution and high-resolution images taken from the set of input images are used to fine-tune the super-resolution components, allowing the minute details of the subject to be maintained. == Usage == DreamBooth can be used to fine-tune models such as Stable Diffusion, where it may alleviate a common shortcoming of Stable Diffusion not being able to adequately generate images of specific individual people. Such a use case is quite VRAM intensive, however, and thus cost-prohibitive for hobbyist users. The Stable Diffusion adaptation of DreamBooth in particular is released as a free and open-source project based on the technology outlined by the original paper published by Ruiz et. al. in 2022. Concerns have been raised regarding the ability for bad actors to utilise DreamBooth to generate misleading images for malicious purposes, and that its open-source nature allows anyone to utilise or even make improvements to the technology. In addition, artists have expressed their apprehension regarding the ethics of using DreamBooth to train model checkpoints that are specifically aimed at imitating specific art styles associated with human artists; one such critic is Hollie Mengert, an illustrator for Disney and Penguin Random House who has had her art style trained into a checkpoint model via DreamBooth and shared online, without her consent.
Historical Thesaurus of English
The Historical Thesaurus of English (HTE) is the largest thesaurus in the world. It is called a historical thesaurus as it arranges the whole vocabulary of English, from the earliest written records in Old English to the present, according to the first documented occurrence of a word in the entire history of the English language. The HTE was conceived and begun in 1965 by the English Language & Linguistics department of the University of Glasgow, who have ever since continued to compile the thesaurus. From the 1980s onwards the project was moved from paper-based records to a computer database. Today, the HTE is available to the public online, but a print version, the Historical Thesaurus of the Oxford English Dictionary (HTOED), was published in 2009. == Main project: The Historical Thesaurus of English (HTE) == The Historical Thesaurus of English (HTE) is a complete database of all the words in the Oxford English Dictionary and other dictionaries (including Old English), arranged by semantic field and date. In this way, the HTE arranges the whole vocabulary of English, from the earliest written records in Old English to the present, alongside dates of use. It is the first historical thesaurus to be compiled for any of the world's languages and contains 800,000 meanings for 600,000 words, within 230,000 categories. As the HTE website states, "in addition to providing hitherto unavailable information for linguistic and textual scholars, the Historical Thesaurus online is a rich resource for students of social and cultural history, showing how concepts developed through the words that refer to them." === Structure === The work is divided into three main sections: the External World, the Mind, and Society. These are broken down into successively narrower domains. The text eventually discriminates more than 236,000 categories. The second order categories are: === History === The ambitious project was announced at a 1965 meeting of the Philological Society by its originator, Michael Samuels. Work on the HTE started in the same year. In 2017, the University of Glasgow was awarded the Queen's Anniversary Prize for Higher Education for the HTE. A second edition of the online HTE is currently in progress and is expected to be launched in late 2020. Work is released on the freely-available HTE website when available. == Print edition: Historical Thesaurus of the Oxford English Dictionary (HTOED) == On 22 October 2009, after 44 years of work, version 1.0 of the HTE was published by Oxford University Press in a two-volume slipcased set as the Historical Thesaurus of the Oxford English Dictionary (HTOED). The two hardcover volumes together total nearly 4,500 pages.
Learning rule
An artificial neural network's learning rule or learning process is a method, mathematical logic or algorithm which improves the network's performance and/or training time. Usually, this rule is applied repeatedly over the network. It is done by updating the weight and bias levels of a network when it is simulated in a specific data environment. A learning rule may accept existing conditions (weights and biases) of the network, and will compare the expected result and actual result of the network to give new and improved values for the weights and biases. Depending on the complexity of the model being simulated, the learning rule of the network can be as simple as an XOR gate or mean squared error, or as complex as the result of a system of differential equations. The learning rule is one of the factors which decides how fast or how accurately the neural network can be developed. Depending on the process to develop the network, there are three main paradigms of machine learning: supervised learning, unsupervised learning, and reinforcement learning. == Background == A lot of the learning methods in machine learning work similar to each other, and are based on each other, which makes it difficult to classify them in clear categories. But they can be broadly understood in 4 categories of learning methods, though these categories don't have clear boundaries and they tend to belong to multiple categories of learning methods - Hebbian - Neocognitron, Brain-state-in-a-box Gradient Descent - ADALINE, Hopfield Network, Recurrent Neural Network Competitive - Learning Vector Quantisation, Self-Organising Feature Map, Adaptive Resonance Theory Stochastic - Boltzmann Machine, Cauchy Machine Though these learning rules might appear to be based on similar ideas, they do have subtle differences, as they are a generalisation or application over the previous rule, and hence it makes sense to study them separately based on their origins and intents. === Hebbian Learning === Developed by Donald Hebb in 1949 to describe biological neuron firing. In the mid-1950s it was also applied to computer simulations of neural networks. Δ w i = η x i y {\displaystyle \Delta w_{i}=\eta x_{i}y} Where η {\displaystyle \eta } represents the learning rate, x i {\displaystyle x_{i}} represents the input of neuron i, and y is the output of the neuron. It has been shown that Hebb's rule in its basic form is unstable. Oja's Rule, BCM Theory are other learning rules built on top of or alongside Hebb's Rule in the study of biological neurons. ==== Perceptron Learning Rule (PLR) ==== The perceptron learning rule originates from the Hebbian assumption, and was used by Frank Rosenblatt in his perceptron in 1958. The net is passed to the activation (transfer) function and the function's output is used for adjusting the weights. The learning signal is the difference between the desired response and the actual response of a neuron. The step function is often used as an activation function, and the outputs are generally restricted to -1, 0, or 1. The weights are updated with w new = w old + η ( t − o ) x i {\displaystyle w_{\text{new}}=w_{\text{old}}+\eta (t-o)x_{i}} where "t" is the target value and "o" is the output of the perceptron, and η {\displaystyle \eta } is called the learning rate. The algorithm converges to the correct classification if: the training data is linearly separable η {\displaystyle \eta } is sufficiently small (though smaller η {\displaystyle \eta } generally means a longer learning time and more epochs) It should also be noted that a single layer perceptron with this learning rule is incapable of working on linearly non-separable inputs, and hence the XOR problem cannot be solved using this rule alone === Backpropagation === Seppo Linnainmaa in 1970 is said to have developed the Backpropagation Algorithm but the origins of the algorithm go back to the 1960s with many contributors. It is a generalisation of the least mean squares algorithm in the linear perceptron and the Delta Learning Rule. It implements gradient descent search through the space possible network weights, iteratively reducing the error, between the target values and the network outputs. ==== Widrow-Hoff Learning (Delta Learning Rule) ==== Similar to the perceptron learning rule but with different origin. It was developed for use in the ADALINE network, which differs from the Perceptron mainly in terms of the training. The weights are adjusted according to the weighted sum of the inputs (the net), whereas in perceptron the sign of the weighted sum was useful for determining the output as the threshold was set to 0, -1, or +1. This makes ADALINE different from the normal perceptron. Delta rule (DR) is similar to the Perceptron Learning Rule (PLR), with some differences: Error (δ) in DR is not restricted to having values of 0, 1, or -1 (as in PLR), but may have any value DR can be derived for any differentiable output/activation function f, whereas in PLR only works for threshold output function Sometimes only when the Widrow-Hoff is applied to binary targets specifically, it is referred to as Delta Rule, but the terms seem to be used often interchangeably. The delta rule is considered to a special case of the back-propagation algorithm. Delta rule also closely resembles the Rescorla-Wagner model under which Pavlovian conditioning occurs. === Competitive Learning === Competitive learning is considered a variant of Hebbian learning, but it is special enough to be discussed separately. Competitive learning works by increasing the specialization of each node in the network. It is well suited to finding clusters within data. Models and algorithms based on the principle of competitive learning include vector quantization and self-organizing maps (Kohonen maps).
Logogen model
The logogen model of 1969 is a model of speech recognition that uses units called "logogens" to explain how humans comprehend spoken or written words. Logogens are a vast number of specialized recognition units, each able to recognize one specific word. This model provides for the effects of context on word recognition. == Overview == The word logogen can be traced back to the Greek-language word logos, which means "word", and genus, which means "birth". British scientist John Morton's logogen model was designed to explain word recognition using a new type of unit known as a logogen. A critical element of this theory is the involvement of lexicons, or specialized aspects of memory that include semantic and phonemic information about each item that is contained in memory. A given lexicon consists of many smaller, abstract items known as logogens. Logogens contain a variety of properties about given word such as their appearance, sound, and meaning. Logogens do not store words within themselves, but rather they store information that is specifically necessary for retrieval of whatever word is being searched for. A given logogen will become activated by psychological stimuli or contextual information (words) that is consistent with the properties of that specific logogen and when the logogen's activation level rises to or above its threshold level, the pronunciation of the given word is sent to the output system. Certain stimuli can affect the activation levels of more than one word at a time, usually involving words that are similar to one another. When this occurs, whichever of the words' activation levels reaches the threshold level, it is that word that is then sent to the output system with the subject remaining unaware of any partially excited logogens. This assumption was made by Marslen-Wilson and Welch (1978), who added to the model some assumptions of their own in order to account for their experimental results. They also assumed that the analysis of phonetic input can only become available to other parts of the system by process of how the input affects the logogen system. Finally, Marslen-Wilson and Welch assume that the first syllable of a given word will increase the activation level of a given logogen more than those of the latter syllables, which supported the data found at the time. == Analysis == The logogen model can be used to help linguists explain particular occurrences in the human language. The most-helpful application of the model is to show how one accesses words and their meanings in the lexicon. The word-frequency effect is best explained by the logogen model in that words (or logogens) that have a higher frequency (or are more common) have a lower threshold. This means that they require less perceptual power in the brain to be recognized and decoded from the lexicon and are recognized faster than those words that are less common. Also, with high-frequency words, the recovery from lowering the item's threshold is less fulfilled compared to low-frequency words so less sensory information is needed for that particular item's recognition. There are ways to lower thresholds, such as repetition and semantic priming. Also, each time a word is encountered through these methods, the threshold for that word is temporarily lowered partially because of its recovering ability. This model also conveys that specific concrete words are recalled better because they use images and logogens, whereas abstract words are not as easily recalled well because they only use logogens, hence showing the difference in thresholds between these two types of words. At the time of its conception, Morton's logogen model was one of the most influential models in springing up other parallel word access models and served as the essential basis for these subsequent models. Morton's model also strongly influenced other contemporary theories on lexical access. However, despite the advantages that the logogen theory presents, it also displays some negative facets. First and foremost, the logogen model does not explain all occurrences in language, such as the introduction of new words or non-words into a person's lexicon. Also, because of the distinctive model application, it may vary in its effectiveness in different languages. == Criticisms == While this model does a reasonable job of understanding the underlying semantics of many aspects in psycholinguistics, there are some flaws that have been pointed out in the logogen model. It has been argued that the prior stimulus patterns that have been seen in the logogen theory are not centrally localized in the logogen itself but are actually distributed throughout the different pathways over which the stimulus is being processed. What this directs at is that the notion and proliferation of logogens was due to modality. In essence, the logogen is unnecessary in the idea of attaining the title of being a recognition unit because of the variety of pathways that it is open to, not just logogens. Another criticism has been that this model essentially ignores larger and more critical structures in language and phonetics such as the different syntactic rules or grammatical construction that innately exists in language. Since this model overtly limits itself to the scope of lexical access then this model is seen as biased and misunderstood. To many psychologists, the logogen model does not meet the functional or representational adequacy that a theory should include to sufficiently comprehend language. Also, another criticism is that the logogen theory was supposed to predict that stimulus degradation should affect priming and word frequency in humans. However, many psychologists have conducted studies and researched the model to show that only priming and not word frequency is interacted with stimulus degradation. Priming is supposed to deteriorate a stimulus because it postulates that the semantic characteristics of previously known words are fed back into the detector of a person which in turn raises the threshold of related items. In word frequency, stimulus degradation is supposed to occur because it postulates that familiar words have lower thresholds than their low-frequency counterparts. However, in studies, priming is the only structure that does show observable and notable stimulus decadence. Even though the logogen theory has many unfilled holes, Morton was a revolutionary of his field whose speculation and research has opened up a remarkable era of psycholinguistics. == Other models to consider == cohort model – This model was proposed by Marslen-Wilson and was designed specifically to account for auditory word recognition. It works by breaking the word down and states that when a word is heard all words that begin with the first sound of the target word are activated. This set of words is considered the cohort. Once the first cohort has been activated, the other information, or sounds in the word narrow down the choices. The person recognizes the word when you are left with a single choice; this is considered the "recognition point". checking model – This model was developed by Norris in 1986. In this particular model, he took the approach that any word that partially matches the input is analyzed and checked to see if it fits with the context of the situation. interactive-activation model – This model is considered a connectionist model. Proposed by McClelland and Rumelhart in the 1981 to 1982 period, it is based around nodes, which are visual features, and positions of letters within a given word. They also act as word detectors which have inhibitory and excitatory connections between them. This model starts with first letter and suggests that all the words with that first letter are activated at first and then going through the word one can determine what the word is they are looking at. The main principle is that mental phenomena can be described by interconnected networks of simple units. verification model – The model was developed by Curtis Becker in 1970. The main idea is that a small number of candidates that are activated in parallel are subject to a serial-verification process. This model starts the word-recognition process with a basic representation of the stimulus. Then, sensory trace, consisting of line features is used to activate word detectors. When an acceptable number of detectors are activated these are used to generate a search set. These items are drawn from the lexicon on the basis of similarity to the sensory trace, which help with the identity of the stimulus. Then, in a serial process the candidates are compared to the representation of the sensory-trace input. == Related concepts == word frequency – This is the belief that the speed and accuracy with which a word is recognized is related to how frequently the word occurs in our language. Each logogen has a threshold (for identification) and words with higher frequencies have lower thresholds. Words with higher freq
RevoScaleR
RevoScaleR is a machine learning package in R created by Microsoft. It is available as part of Machine Learning Server, Microsoft R Client, and Machine Learning Services in Microsoft SQL Server 2016. The package contains functions for creating linear model, logistic regression, random forest, decision tree and boosted decision tree, and K-means, in addition to some summary functions for inspecting and visualizing data. It has a Python package counterpart called revoscalepy. Another closely related package is MicrosoftML, which contains machine learning algorithms that RevoScaleR does not have, such as neural network and SVM. In June 2021, Microsoft announced to open source the RevoScaleR and revoscalepy packages, making them freely available under the MIT License. == Concepts == Many R packages are designed to analyze data that can fit in the memory of the machine and usually do not make use of parallel processing. RevoScaleR was designed to address these limitations. The functions in RevoScaleR orientate around three main abstraction concepts that users can specify to process large amount of data that might not fit in memory and exploit parallel resources to speed up the analysis. === Compute Contexts === A compute context refers to the location where the computation on the data happens. It could be "local" (on the client machine) or "remote" (on a data platform such as a SQL server, or Spark). Pushing the computation to a remote server allows people to take advantage of the greater compute resources that a remote machine may have. If the data being analyzed reside on the same machine, using a remote compute context also removes the need to pull data across the network onto the client machine. === Data source === Data source defines where the data comes from. There are various data sources available in RevoScaleR, such as text data, Xdf data, in-SQL data, and a spark dataframe. People can wrap their data in a data source object and use that as run analytics in different compute context. Different data sources are available in different compute context. For example, if the compute context is set to SQL server, then the only data source one can use would be an in-SQL data source. === Analytics === Analytic functions in RevoScaleR takes in data source object, a compute context, and the other parameters needed to build the specific model, such as formula for the logistic regression or the number of trees in a decision tree. In addition to those parameters, one can also specify the level of parallelism, such as the size of the data chunk for each process or number of processes to build the model. However, parallelism is only available in non-express edition. == Limitations == The package is mostly meant to be used with a SQL server or other remote machines. To fully leverage the abstractions it uses to process a large dataset, one needs a remote server and non-Express free edition of the package. It cannot be easily installed such as by running "install.packages("RevoScaleR")" like most open source R packages. It's available only through Microsoft R Client, a distribution of R for data science, or Microsoft Machine Learning Server (stand-alone with no SQL server attached), or Microsoft Machine Learning Services (a SQL server services). However, one can still use the analytics functions in an Express, free version of the package.