These datasets are used in machine learning (ML) research and have been cited in peer-reviewed academic journals. Datasets are an integral part of the field of machine learning. Major advances in this field can result from advances in learning algorithms (such as deep learning), computer hardware, and, less intuitively, the availability of high-quality training datasets. High-quality labeled training datasets for supervised and semi-supervised machine-learning algorithms are usually difficult and expensive to produce because of the large amount of time needed to label the data. Although they do not need to be labeled, high-quality unlabeled datasets for unsupervised learning can also be difficult and costly to produce. Many organizations, including governments, publish and share their datasets, often using common metadata formats (such as Croissant). The datasets are classified, based on the licenses, into two groups: open data and non-open data. The datasets from various governmental-bodies are presented in List of open government data sites. The datasets are ported on open data portals. They are made available for searching, depositing and accessing through interfaces like Open API. The datasets are made available as various sorted types and subtypes. == List of sorting used for datasets == The data portal is classified based on its type of license. The open source license based data portals are known as open data portals which are used by many government organizations and academic institutions. == List of open data portals == == List of portals suitable for multiple types of applications == The data portal sometimes lists a wide variety of subtypes of datasets pertaining to many machine learning applications. == List of portals suitable for a specific subtype of applications == The data portals which are suitable for a specific subtype of machine learning application are listed in the subsequent sections. == Image data == == Text data == These datasets consist primarily of text for tasks such as natural language processing, sentiment analysis, translation, and cluster analysis. === Reviews === === News articles === === Messages === === Twitter and tweets === === Dialogues === === Legal === === Other text === == Sound data == These datasets consist of sounds and sound features used for tasks such as speech recognition and speech synthesis. === Speech === === Music === === Other sounds === == Signal data == Datasets containing electric signal information requiring some sort of signal processing for further analysis. === Electrical === === Motion-tracking === === Other signals === == Chemical data == Datasets from physical systems. === Chemical Reactions with transition states (TS) === === OpenReACT-CHON-EFH === OpenReACT-CHON-EFH (Open Reaction Dataset of Atomic ConfiguraTions comprising C, H, O and N with Energies, Forces and Hessians) is a 2025 open-access benchmark for machine-learning interatomic potentials. RTP set – 35,087 stationary-point geometries (reactant, transition state and product) drawn from 11,961 elementary reactions, each labeled with density-functional energies, atomic forces and full Hessian matrices at the ωB97X-D/6-31G(d) level. IRC set – 34,248 structures along 600 minimum-energy reaction paths, used to test extrapolation beyond trained stationary points. NMS set – 62,527 off-equilibrium geometries generated by normal-mode sampling to probe model robustness under thermal perturbations. The collection underpins the study Does Hessian Data Improve the Performance of Machine Learning Potentials? and was used to train and benchmark the machine-learning interatomic potentials reported therein. The dataset itself is distributed under a CC licence via Figshare. == Physical data == Datasets from physical systems. === High-energy physics === === Systems === === Astronomy === === Earth science === === Other physical === == Biological data == Datasets from biological systems. === Human === === Animal === === Fungi === === Plant === === Microbe === === Drug discovery === == Anomaly data == == Question answering data == This section includes datasets that deals with structured data. == Dialog or instruction prompted data == This section includes datasets that contains multi-turn text with at least two actors, a "user" and an "agent". The user makes requests for the agent, which performs the request. == Cybersecurity == == Climate and sustainability == == Code data == == Multivariate data == === Financial === === Weather === === Census === === Transit === === Internet === === Games === === Other multivariate === == Curated repositories of datasets == As datasets come in myriad formats and can sometimes be difficult to use, there has been considerable work put into curating and standardizing the format of datasets to make them easier to use for machine learning research. OpenML: Web platform with Python, R, Java, and other APIs for downloading hundreds of machine learning datasets, evaluating algorithms on datasets, and benchmarking algorithm performance against dozens of other algorithms. PMLB: A large, curated repository of benchmark datasets for evaluating supervised machine learning algorithms. Provides classification and regression datasets in a standardized format that are accessible through a Python API. Metatext NLP: https://metatext.io/datasets web repository maintained by community, containing nearly 1000 benchmark datasets, and counting. Provides many tasks from classification to QA, and various languages from English, Portuguese to Arabic. Appen: Off The Shelf and Open Source Datasets hosted and maintained by the company. These biological, image, physical, question answering, signal, sound, text, and video resources number over 250 and can be applied to over 25 different use cases.
Sentence embedding
In natural language processing, a sentence embedding is a representation of a sentence as a vector of numbers which encodes meaningful semantic information. State of the art embeddings are based on the learned hidden layer representation of dedicated sentence transformer models. BERT pioneered an approach involving the use of a dedicated [CLS] token prepended to the beginning of each sentence inputted into the model; the final hidden state vector of this token encodes information about the sentence and can be fine-tuned for use in sentence classification tasks. In practice however, BERT's sentence embedding with the [CLS] token achieves poor performance, often worse than simply averaging non-contextual word embeddings. SBERT later achieved superior sentence embedding performance by fine tuning BERT's [CLS] token embeddings through the usage of a siamese neural network architecture on the SNLI dataset. Other approaches are loosely based on the idea of distributional semantics applied to sentences. Skip-Thought trains an encoder-decoder structure for the task of neighboring sentences predictions; this has been shown to achieve worse performance than approaches such as InferSent or SBERT. An alternative direction is to aggregate word embeddings, such as those returned by Word2vec, into sentence embeddings. The most straightforward approach is to simply compute the average of word vectors, known as continuous bag-of-words (CBOW). However, more elaborate solutions based on word vector quantization have also been proposed. One such approach is the vector of locally aggregated word embeddings (VLAWE), which demonstrated performance improvements in downstream text classification tasks. == Applications == In recent years, sentence embedding has seen a growing level of interest due to its applications in natural language queryable knowledge bases through the usage of vector indexing for semantic search. LangChain for instance utilizes sentence transformers for purposes of indexing documents. In particular, an indexing is generated by generating embeddings for chunks of documents and storing (document chunk, embedding) tuples. Then given a query in natural language, the embedding for the query can be generated. A top k similarity search algorithm is then used between the query embedding and the document chunk embeddings to retrieve the most relevant document chunks as context information for question answering tasks. This approach is also known formally as retrieval-augmented generation. Though not as predominant as BERTScore, sentence embeddings are commonly used for sentence similarity evaluation which sees common use for the task of optimizing a Large language model's generation parameters is often performed via comparing candidate sentences against reference sentences. By using the cosine-similarity of the sentence embeddings of candidate and reference sentences as the evaluation function, a grid-search algorithm can be utilized to automate hyperparameter optimization. == Evaluation == A way of testing sentence encodings is to apply them on Sentences Involving Compositional Knowledge (SICK) corpus for both entailment (SICK-E) and relatedness (SICK-R). In the best results are obtained using a BiLSTM network trained on the Stanford Natural Language Inference (SNLI) Corpus. The Pearson correlation coefficient for SICK-R is 0.885 and the result for SICK-E is 86.3. A slight improvement over previous scores is presented in: SICK-R: 0.888 and SICK-E: 87.8 using a concatenation of bidirectional Gated recurrent unit.
AI Customer-support Bots: Free vs Paid (2026)
Curious about the best AI customer-support bot? An AI customer-support bot is software that uses machine learning to help you get more done — it combines speed, accuracy, and an interface that just works. Hands-on testing shows real-world results vary, so a short free trial is the smartest way to decide. Whether you are a beginner or a pro, the right AI customer-support bot slots into your workflow and pays for itself fast. This guide breaks down the top picks, their pros and cons, and who each one is best for.
Tree transducer
In theoretical computer science and formal language theory, a tree transducer (TT) is an abstract machine taking as input a tree, and generating output – generally other trees, but models producing words or other structures exist. Roughly speaking, tree transducers extend tree automata in the same way that word transducers extend word automata. Manipulating tree structures instead of words enable TT to model syntax-directed transformations of formal or natural languages. However, TT are not as well-behaved as their word counterparts in terms of algorithmic complexity, closure properties, etcetera. In particular, most of the main classes are not closed under composition. The main classes of tree transducers are: == Top-Down Tree Transducers (TOP) == A TOP T is a tuple (Q, Σ, Γ, I, δ) such that: Q is a finite set, the set of states; Σ is a finite ranked alphabet, called the input alphabet; Γ is a finite ranked alphabet, called the output alphabet; I is a subset of Q, the set of initial states; and δ is a set of rules of the form q ( f ( x 1 , … , x n ) ) → u {\displaystyle q(f(x_{1},\dots ,x_{n}))\to u} , where f is a symbol of Σ, n is the arity of f, q is a state, and u is a tree on Γ and Q × 1.. n {\displaystyle Q\times 1..n} , such pairs being nullary. === Examples of rules and intuitions on semantics === For instance, q ( f ( x 1 , … , x 3 ) ) → g ( a , q ′ ( x 1 ) , h ( q ″ ( x 3 ) ) ) {\displaystyle q(f(x_{1},\dots ,x_{3}))\to g(a,q'(x_{1}),h(q''(x_{3})))} is a rule – one customarily writes q ( x i ) {\displaystyle q(x_{i})} instead of the pair ( q , x i ) {\displaystyle (q,x_{i})} – and its intuitive semantics is that, under the action of q, a tree with f at the root and three children is transformed into g ( a , q ′ ( x 1 ) , h ( q ″ ( x 3 ) ) ) {\displaystyle g(a,q'(x_{1}),h(q''(x_{3})))} where, recursively, q ′ ( x 1 ) {\displaystyle q'(x_{1})} and q ″ ( x 3 ) {\displaystyle q''(x_{3})} are replaced, respectively, with the application of q ′ {\displaystyle q'} on the first child and with the application of q ″ {\displaystyle q''} on the third. === Semantics as term rewriting === The semantics of each state of the transducer T, and of T itself, is a binary relation between input trees (on Σ) and output trees (on Γ). A way of defining the semantics formally is to see δ {\displaystyle \delta } as a term rewriting system, provided that in the right-hand sides the calls are written in the form q ( x i ) {\displaystyle q(x_{i})} , where states q are unary symbols. Then the semantics [ [ q ] ] {\displaystyle [\![q]\!]} of a state q is given by [ [ q ] ] = { u ↦ v ∣ u is a tree on Σ , v is a tree on Γ , and q ( u ) → δ ∗ v } . {\displaystyle [\![q]\!]=\{u\mapsto v\mid u{\text{ is a tree on }}\Sigma ,\ v{\text{ is a tree on }}\Gamma {\text{, and }}q(u)\to _{\delta }^{}v\}.} The semantics of T is then defined as the union of the semantics of its initial states: [ [ T ] ] = ⋃ q ∈ I [ [ q ] ] . {\displaystyle [\![T]\!]=\bigcup _{q\in I}[\![q]\!].} === Determinism and domain === As with tree automata, a TOP is said to be deterministic (abbreviated DTOP) if no two rules of δ share the same left-hand side, and there is at most one initial state. In that case, the semantics of the DTOP is a partial function from input trees (on Σ) to output trees (on Γ), as are the semantics of each of the DTOP's states. The domain of a transducer is the domain of its semantics. Likewise, the image of a transducer is the image of its semantics. === Properties of DTOP === DTOP are not closed under union: this is already the case for deterministic word transducers. The domain of a DTOP is a regular tree language. Furthermore, the domain is recognisable by a deterministic top-down tree automaton (DTTA) of size at most exponential in that of the initial DTOP. That the domain is DTTA-recognizable is not surprising, considering that the left-hand sides of DTOP rules are the same as for DTTA. As for the reason for the exponential explosion in the worst case (that does not exist in the word case), consider the rule q ( f ( x 1 , x 2 ) ) → g ( p 1 ( x 1 ) , p 2 ( x 1 ) , p 3 ( x 2 ) ) {\displaystyle q(f(x_{1},x_{2}))\to g(p_{1}(x_{1}),p_{2}(x_{1}),p_{3}(x_{2}))} . In order for the computation to succeed, it must succeed for both children. That means that the right child must be in the domain of p 3 {\displaystyle p_{3}} . As for the left child, it must be in the domain of both p 1 {\displaystyle p_{1}} and p 2 {\displaystyle p_{2}} . Generally, since subtrees can be copied, a single subtree can be evaluated by multiple states during a run, despite the determinism, and unlike DTTA. Thus the construction of the DTTA recognising the domain of a DTOP must account for sets of states and compute the intersections of their domains, hence the exponential. In the special case of linear DTOP, that is to say DTOP where each x i {\displaystyle x_{i}} appears at most once in the right-hand side of each rule, the construction is linear in time and space. The image of a DTOP is not a regular tree language. Consider the transducer coding the transformation f ( x ) → g ( x , x ) {\displaystyle f(x)\to g(x,x)} ; that is, duplicate the child of the input. This is easily done by a rule q ( f ( x 1 ) ) → g ( p ( x 1 ) , p ( x 1 ) ) {\displaystyle q(f(x_{1}))\to g(p(x_{1}),p(x_{1}))} , where p encodes the identity. Then, absent any restrictions on the first child of the input, the image is a classical non-regular tree language. However, the domain of a DTOP cannot be restricted to a regular tree language. That is to say, given a DTOP T and a language L, one cannot in general build a DTOP T ′ {\displaystyle T'} such that the semantics of T ′ {\displaystyle T'} is that of T, restricted to L. This property is linked to the reason deterministic top-down tree automata are less expressive than bottom-up automata: once you go down a given path, information from other paths is inaccessible. Consider the transducer coding the transformation f ( x , y ) → y {\displaystyle f(x,y)\to y} ; that is, output the right child of the input. This is easily done by a rule q ( f ( x 1 , x 2 ) ) → p ( x 2 ) {\displaystyle q(f(x_{1},x_{2}))\to p(x_{2})} , where p encodes the identity. Now let's say we want to restrict this transducer to the finite (and thus, in particular, regular) domain { f ( c , a ) , f ( c , b ) } {\displaystyle \{f(c,a),\ f(c,b)\}} . We must use the rules q ( f ( x 1 , x 2 ) ) → p ( x 2 ) , p ( a ) → a , p ( b ) → b {\displaystyle q(f(x_{1},x_{2}))\to p(x_{2}),\ p(a)\to a,\ p(b)\to b} . But in the first rule, x 1 {\displaystyle x_{1}} does not appear at all, since nothing is produced from the left child. Thus, it is not possible to test that the left child is c. In contrast, since we produce from the right child, we can test that it is a or b. In general, the criterion is that DTOP cannot test properties of subtrees from which they do not produce output. DTOP are not closed under composition. However this problem can be solved by the addition of a lookahead: a tree automaton, coupled to the transducer, that can perform tests on the domain which the transducer is incapable of. This follows from the point about domain restriction: composing the DTOP encoding identity on { f ( c , a ) , f ( c , b ) } {\displaystyle \{f(c,a),\ f(c,b)\}} with the one encoding f ( x , y ) → y {\displaystyle f(x,y)\to y} must yield a transducer with the semantics { f ( c , a ) ↦ a , f ( c , b ) ↦ b } {\displaystyle \{f(c,a)\mapsto a,\ f(c,b)\mapsto b\}} , which we know is not expressible by a DTOP. The typechecking problem—testing whether the image of a regular tree language is included in another regular tree language—is decidable. The equivalence problem—testing whether two DTOP define the same functions—is decidable. == Bottom-Up Tree Transducers (BOT) == As in the simpler case of tree automata, bottom-up tree transducers are defined similarly to their top-down counterparts, but proceed from the leaves of the tree to the root, instead of from the root to the leaves. Thus the main difference is in the form of the rules, which are of the form f ( q 1 ( x 1 ) , … , q n ( x n ) ) → q ( u ) {\displaystyle f(q_{1}(x_{1}),\dots ,q_{n}(x_{n}))\to q(u)} .
Ofer Dekel (researcher)
Ofer Dekel (Hebrew: עופר דקל) is a computer science researcher in the Machine Learning Department of Microsoft Research. He obtained his PhD in computer science from the Hebrew University of Jerusalem and is an affiliate faculty at the Computer Science & Engineering department at the University of Washington. == Areas of research == Dekel's research topics include machine learning, online prediction, statistical learning theory, and stochastic optimization. He is currently engaged in the application of machine learning techniques in the development of the Bing search engine.
Super-resolution optical fluctuation imaging
Super-resolution optical fluctuation imaging (SOFI) is a post-processing method for the calculation of super-resolved images from recorded image time series that is based on the temporal correlations of independently fluctuating fluorescent emitters. SOFI has been developed for super-resolution of biological specimen that are labelled with independently fluctuating fluorescent emitters (organic dyes, fluorescent proteins). In comparison to other super-resolution microscopy techniques such as STORM or PALM that rely on single-molecule localization and hence only allow one active molecule per diffraction-limited area (DLA) and timepoint, SOFI does not necessitate a controlled photoswitching and/ or photoactivation as well as long imaging times. Nevertheless, it still requires fluorophores that are cycling through two distinguishable states, either real on-/off-states or states with different fluorescence intensities. In mathematical terms SOFI-imaging relies on the calculation of cumulants, for what two distinguishable ways exist. For one thing an image can be calculated via auto-cumulants that by definition only rely on the information of each pixel itself, and for another thing an improved method utilizes the information of different pixels via the calculation of cross-cumulants. Both methods can increase the final image resolution significantly although the cumulant calculation has its limitations. Actually SOFI is able to increase the resolution in all three dimensions. == Principle == Likewise to other super-resolution methods SOFI is based on recording an image time series on a CCD- or CMOS camera. In contrary to other methods the recorded time series can be substantially shorter, since a precise localization of emitters is not required and therefore a larger quantity of activated fluorophores per diffraction-limited area is allowed. The pixel values of a SOFI-image of the n-th order are calculated from the values of the pixel time series in the form of a n-th order cumulant, whereas the final value assigned to a pixel can be imagined as the integral over a correlation function. The finally assigned pixel value intensities are a measure of the brightness and correlation of the fluorescence signal. Mathematically, the n-th order cumulant is related to the n-th order correlation function, but exhibits some advantages concerning the resulting resolution of the image. Since in SOFI several emitters per DLA are allowed, the photon count at each pixel results from the superposition of the signals of all activated nearby emitters. The cumulant calculation now filters the signal and leaves only highly correlated fluctuations. This provides a contrast enhancement and therefore a background reduction for good measure. As it is implied in the figure on the left the fluorescence source distribution: ∑ k = 1 N δ ( r → − r → k ) ⋅ ε k ⋅ s k ( t ) {\displaystyle \sum _{k=1}^{N}\delta ({\vec {r}}-{\vec {r}}_{k})\cdot \varepsilon _{k}\cdot s_{k}(t)} is convolved with the system's point spread function (PSF) U(r). Hence the fluorescence signal at time t and position r → {\displaystyle {\vec {r}}} is given by F ( r → , t ) = ∑ k = 1 N U ( r → − r → k ) ⋅ ε k ⋅ s k ( t ) . {\displaystyle F({\vec {r}},t)=\sum _{k=1}^{N}U({\vec {r}}-{\vec {r}}_{k})\cdot \varepsilon _{k}\cdot s_{k}(t).} Within the above equations N is the amount of emitters, located at the positions r → k {\displaystyle {\vec {r}}_{k}} with a time-dependent molecular brightness ε k ⋅ s k {\displaystyle \varepsilon _{k}\cdot s_{k}} where ε k {\displaystyle \varepsilon _{k}} is a variable for the constant molecular brightness and s k ( t ) {\displaystyle s_{k}(t)} is a time-dependent fluctuation function. The molecular brightness is just the average fluorescence count-rate divided by the number of molecules within a specific region. For simplification it has to be assumed that the sample is in a stationary equilibrium and therefore the fluorescence signal can be expressed as a zero-mean fluctuation: δ F ( r → , t ) = F ( r → , t ) − ⟨ F ( r → , t ) ⟩ t {\displaystyle \delta F({\vec {r}},t)=F({\vec {r}},t)-\langle F({\vec {r}},t)\rangle _{t}} where ⟨ ⋯ ⟩ t {\displaystyle \langle \cdots \rangle _{t}} denotes time-averaging. The auto-correlation here e.g. the second-order can then be described deductively as follows for a certain time-lag τ {\displaystyle \tau } : δ F ( r → , t ) = ⟨ δ F ( r → , t + τ ) ⋅ δ F ( r → , t ) ⟩ t {\displaystyle \delta F({\vec {r}},t)=\langle \delta F({\vec {r}},t+\tau )\cdot \delta F({\vec {r}},t)\rangle _{t}} From these equations it follows that the PSF of the optical system has to be taken to the power of the order of the correlation. Thus in a second-order correlation the PSF would be reduced along all dimensions by a factor of 2 {\displaystyle {\sqrt {2}}} . As a result, the resolution of the SOFI-images increases according to this factor. === Cumulants versus correlations === Using only the simple correlation function for a reassignment of pixel values, would ascribe to the independency of fluctuations of the emitters in time in a way that no cross-correlation terms would contribute to the new pixel value. Calculations of higher-order correlation functions would suffer from lower-order correlations for what reason it is superior to calculate cumulants, since all lower-order correlation terms vanish. == Cumulant-calculation == === Auto-cumulants === For computational reasons it is convenient to set all time-lags in higher-order cumulants to zero so that a general expression for the n-th order auto-cumulant can be found: A C n ( r → , τ 1 … n − 1 = 0 ) = ∑ k = 1 N U n ( r → − r → k ) ε k n w k ( 0 ) {\displaystyle AC_{n}({\vec {r}},\tau _{1\ldots n-1}=0)=\sum _{k=1}^{N}U^{n}({\vec {r}}-{\vec {r}}_{k})\varepsilon _{k}^{n}w_{k}(0)} w k {\displaystyle w_{k}} is a specific correlation based weighting function influenced by the order of the cumulant and mainly depending on the fluctuation properties of the emitters. Albeit there is no fundamental limitation in calculating very high orders of cumulants and thereby shrinking the FWHM of the PSF there are practical limitations according to the weighting of the values assigned to the final image. Emitters with a higher molecular brightness will show a strong increase in terms of the pixel cumulant value assigned at higher-orders as well as this performance can be expected from a diverse appearance of fluctuations of different emitters. A wide intensity range of the resulting image can therefore be expected and as a result dim emitters can get masked by bright emitters in higher-order images:. The calculation of auto-cumulants can be realized in a very attractive way in a mathematical sense. The n-th order cumulant can be calculated with a basic recursion from moments K n ( r → ) = μ n ( r → ) − ∑ i = 1 n − 1 ( n − 1 i ) K n − i ( r → ) μ i ( r → ) {\displaystyle K_{n}({\vec {r}})=\mu _{n}({\vec {r}})-\sum _{i=1}^{n-1}{\begin{pmatrix}n-1\\i\end{pmatrix}}K_{n-i}({\vec {r}})\mu _{i}({\vec {r}})} where K is a cumulant of the index's order, likewise μ {\displaystyle \mu } represents the moments. The term within the brackets indicates a binomial coefficient. This way of computation is straightforward in comparison with calculating cumulants with standard formulas. It allows for the calculation of cumulants with only little time of computing and is, as it is well implemented, even suitable for the calculation of high-order cumulants on large images. === Cross-cumulants === In a more advanced approach cross-cumulants are calculated by taking the information of several pixels into account. Cross-cumulants can be described as follows: C C n ( r → , τ 1 … n − 1 = 0 ) = ∏ j < l n U ( r → j − r → l n ) ⋅ ∑ i = 1 N U n ( r → i − ∑ k n r → k n ) ε i n w i ( 0 ) {\displaystyle CC_{n}({\vec {r}},\tau _{1\ldots n-1}=0)=\prod _{j Curious about the best AI art generator? An AI art generator is software that uses machine learning to help you get more done — it combines speed, accuracy, and an interface that just works. Hands-on testing shows real-world results vary, so a short free trial is the smartest way to decide. Whether you are a beginner or a pro, the right AI art generator slots into your workflow and pays for itself fast. Read on for hands-on impressions, pricing tiers, and the standout features that matter.Is an AI Art Generator Worth It in 2026?