AI Data Trainer

AI Data Trainer — independent reviews, comparisons, pricing and step-by-step guides on Aizhi.

• 

  Gemini Enterprise Agent Platform

  Gemini Enterprise Agent Platform (formerly known as Vertex AI) is a managed machine learning (ML) and artificial intelligence (AI) platform developed by Google Cloud. It provides a unified environment for building, training, deploying, and scaling ML models and generative AI applications. The platform integrates tools for the full ML lifecycle, including data preparation, model training, evaluation, deployment, and monitoring, under a single API and user interface. Vertex AI was announced at Google I/O and released as a generally available product on May 18, 2021. At launch, Google described Vertex AI as unifying its AutoML offerings with its prior Cloud AI Platform capabilities, and as adding operational features intended to help teams move models from experimentation into production use. On April 22, 2026, Google announced Gemini Enterprise Agent Platform as the replacement evolution of Vertex AI. == History == Google Cloud announced the general availability of Vertex AI on May 18, 2021, at the Google I/O developer conference. The platform was designed to consolidate Google Cloud's previously separate ML offerings, including AutoML and the legacy AI Platform, into a single system. At launch, Google claimed that Vertex AI required roughly 80% fewer lines of code to train a model compared to competing platforms. In June 2023, Google made generative AI support in Vertex AI generally available, giving developers access to foundation models including PaLM 2, Imagen, and Codey through the platform's Model Garden and the newly launched Generative AI Studio. At the time of this launch, Model Garden included over 60 models from Google and its partners. In August 2023, at the Google Cloud Next conference, Google announced further updates to Vertex AI, including the addition of third-party models such as Claude 2 from Anthropic and Llama 2 from Meta to the Model Garden, as well as new tools called Vertex AI Extensions for connecting models to APIs for real-time data retrieval. At the same event, Vertex AI Search and Conversation were made generally available, providing enterprise search and chatbot capabilities powered by foundation models. In April 2024, at Google Cloud Next, the company introduced Vertex AI Agent Builder, a no-code tool for creating AI-powered conversational agents built on top of Gemini large language models. This brought together the existing Vertex AI Search and Conversation products with new developer tools for building generative AI experiences. == Features == === Model training === Vertex AI supports both AutoML, which enables code-free model training on tabular, image, text, or video data, and custom training, which gives users full control over the ML framework, training code, and hyperparameter tuning. The platform provides serverless training as well as dedicated training clusters with GPU and TPU accelerators. Vertex AI Vizier handles automatic hyperparameter tuning, and Vertex AI Experiments allows comparison and tracking of training runs. === Model Garden === The Vertex AI Model Garden is a curated catalog of over 200 enterprise-ready models, including Google's own foundation models (such as Gemini, Imagen, and Veo), third-party models (such as Anthropic's Claude and Mistral AI models), and popular open-source models (such as Llama and Gemma). Models are accessible as fully managed model-as-a-service APIs. === Pipelines (workflow orchestration) === Vertex AI Pipelines provides managed orchestration of ML workflows and supports pipelines built with the Kubeflow Pipelines SDK, among other options described in Google Cloud documentation. === Vertex AI Studio === Vertex AI Studio provides tools for prompt design, testing, and model management, allowing developers to prototype and build generative AI applications using natural language, code, images, or video. === Agent Builder and Agent Engine === Vertex AI Agent Builder is a suite of products for building, deploying, and governing AI agents in production environments. It supports development with the open-source Agent Development Kit (ADK) and other frameworks. Vertex AI Agent Engine provides the underlying infrastructure for deploying and scaling agents, with support for enterprise security features including HIPAA compliance, customer-managed encryption keys (CMEK), and VPC Service Controls. === Generative AI tooling and model access === Google markets Vertex AI as providing access to Google foundation models (including the Gemini family) and developer tools such as Vertex AI Studio, along with a model catalog that includes Google and selected open source models (marketed as "Model Garden"). Google has also offered products within Vertex AI aimed at building generative search and conversational applications, including offerings named "Vertex AI Search" and "Vertex AI Conversation" as reported in 2023 coverage of platform updates. === MLOps tools === The platform includes a range of MLOps capabilities: Vertex AI Pipelines for orchestrating and automating ML workflows as reusable pipelines. Vertex AI Feature Store for serving, sharing, and reusing ML features across projects. Vertex AI Model Registry for storing, versioning, and managing trained models. Vertex AI Model Monitoring for detecting training-serving skew and inference drift in deployed models. Vertex Explainable AI for interpreting model predictions. Vertex AI Workbench for managed JupyterLab notebook environments integrated with Google Cloud Storage and BigQuery. == Industry recognition == Google was named a Leader for the fifth consecutive year in the 2024 Gartner Magic Quadrant for Cloud AI Developer Services, a recognition that encompasses Vertex AI and its related offerings. Google was also recognized as a Leader in the 2024 Gartner Magic Quadrant for Data Science and Machine Learning Platforms and was named a Leader in the Forrester Wave for AI/ML Platforms, Q3 2024. In October 2025, Google was also named a Leader in the 2025 IDC (International Data Corporation) MarketScape for Worldwide GenAI Life-Cycle Foundation Model Software. == Pricing == Vertex AI uses a pay-as-you-go pricing model, with costs determined by the specific services consumed, including model training, prediction serving, and data storage. For generative AI tasks, pricing is based on a per-token model, with rates varying depending on the specific model used and whether tokens are input or output. Google offers a free tier for new users, which includes limited custom training hours and online prediction usage, along with an introductory US$300 in Google Cloud credits valid for 90 days. == Adoption == In the year following its 2021 launch, Google reported that usage of Vertex AI and BigQuery had driven 2.5 times more machine learning predictions compared to the prior year, and that active customers of Vertex AI Workbench had grown 25-fold over a six-month period. Early enterprise adopters included Ford, Wayfair, and Seagate, among others. Wayfair reported that it was able to run large model training jobs 5 to 10 times faster using the platform.

  Read more →
• 

  Technical data management system

  A technical data management system (TDMS) is a document management system (DMS) pertaining to the management of technical and engineering drawings and documents. Often the data are contained in 'records' of various forms, such as on paper, microfilms or digital media. Hence technical data management is also concerned with record management involving technical data. Technical document management systems are used within large organisations with large scale projects involving engineering. For example, a TDMS can be used for integrated steel plants (ISP), automobile factories, aero-space facilities, infrastructure companies, city corporations, research organisations, etc. In such organisations, technical archives or technical documentation centres are created as central facilities for effective management of technical data and records. TDMS functions are similar to that of conventional archive functions in concepts, except that the archived materials in this case are essentially engineering drawings, survey maps, technical specifications, plant and equipment data sheets, feasibility reports, project reports, operation and maintenance manuals, standards, etc. Document registration, indexing, repository management, reprography, etc. are parts of TDMS. Various kinds of sophisticated technologies such as document scanners, microfilming and digitization camera units, wide format printers, digital plotters, software, etc. are available, making TDMS functions an easier process than previous times. == Constituents of a technical data management system == Technical data refers to both scientific and technical information recorded and presented in any form or manner (excluding financial and management information). A Technical Data Management System is created within an organisation for archiving and sharing information such as technical specifications, datasheets and drawings. Similar to other types of data management system, a Technical Data Management System consists of the 4 crucial constituents mentioned below. === Data planning === Data plans (long-term or short-term) are constructed as the first essential step of a proper and complete TDMS. It is created to ultimately help with the 3 other constituents, data acquisition, data management and data sharing. A proper data plan should not exceed 2 pages and should address the following basics: Types of data (samples, experiment results, reports, drawings, etc.) and metadata (data that summarizes and describes other data. In this case, it refers to details such as sample sizes, experiment conditions and procedures, dates of reports, explanations of drawings, etc.) Means of researches and collections of data (field works, experiments in production lines, etc.) Costs of researches Policies for access, sharing (re-use within the organisation and re-distribution to the public) Proposals for archiving data and maintaining access to it === Data acquisition === Raw data is collected from primary sites of the organisations through the use of modern technologies. Please reference the table below for examples. The data collected is then transferred to technical data centres for data management. === Data management === After data acquisition, data is sorted out, whilst useful data is archived, unwanted data is disposed. When managing and archiving data, the features below of the data are considered. Names, labels, values and descriptions for variables and records. (In the case of TDMS, one example is names of equipments on an equipment datasheet) Derived data from the original data, with code, algorithm or command file used to create them. (In the case of TDMS, one example is an expectation report derived from the analysis of an equipment datasheet) Metadata associates with the data being archived === Data sharing === Archived and managed data are accessible to rightful entities. A proper and complete TDMS should share data to a suitable extent, under suitable security, in order to achieve optimal usage of data within the organisation. It aims for easy access when reused by other researchers and hence it enhances other research processes. Data is often referred in other tests and technical specifications, where new analysis is generated, managed and archived again. As a result, data is flowing within the organisation under effective management through the use of TDMS. == Advantages and disadvantages of usage of technical data management systems == There are strengths and weakness when using technical data management systems (TDMS) to archive data. Some of the advantages and disadvantages are listed below. === Advantages === ==== 1. Faster and easier data management ==== Since TDMS is integrated into the organisation's systems, whenever workers develop data files (SolidWorks, AutoCAD, Microsoft Word, etc.), they can also archive and manage data, linking what they need to their current work, at the same time they can also update the archives with useful data. This speeds up working processes and makes them more efficient. ==== 2. Increased security ==== All data files are centralized, hence internal and external data leakages are less likely to happen, and the data flow is more closely monitored. As a result, data in the organisation is more secured. ==== 3. Increased collaboration within the organisation ==== Since the data files are centralized and the data flow within the organisation increases, researchers and workers within the organisation are able to work on joint projects. More complex tasks can be performed for higher yields. ==== 4. Compatible to various formats of data ==== TDMS is compatible to many formats of data, from basic data like Microsoft Words to complex data like voice data. This enhances the quality of the management of data archived. === Disadvantages === ==== 1. Higher financial costs ==== Implementing TDMS into the organisation's systems involves monetary costs. Maintenance costs certain amount of human resources and money as well. These resources involve opportunity costs as they can be utilized in other aspects. ==== 2. Lower stability ==== Since TDMS manages and centralizes all the data the organisation processes, it links the working processes within the whole organisation together. It also increases the vulnerability of the organisation data network. If TDMS is not stable enough or when it is exposed to hacker and virus attacks, the organisation's data flow might shut down completely, affecting the work in an organisation-wide scale and leading to a lower stability as results. == Comparison between traditional data management approaches and technical data management systems == Test engineers and researchers are facing great challenges in turning complex test results and simulation data into usable information for higher yields of firms. These challenges are listed below. Increase in complication of designs Reduced in time and budgets available Higher quality is demanded === Traditional data management approaches === Many organisations are still applying the conventional file management systems, due to the difficulty in building a proper and complete archives for data management. The first approach is the simple file-folder system. This costs the problem of ineffectiveness as workers and researchers have to manually go through numerous layers of systems and files for the target data. Moreover, the target data may contain files with different formats and these files may not be stored in the same machine. These files are also easily lost if renamed or moved to another location. The second approach is conventional databases such as Oracle. These databases are capable of enabling easy search and access of data. However, a great drawback is that huge effort for preparing and modeling the data is required. For large-scale projects, huge monetary costs are induced, and extra IT human resources must be employed for constant handling, expanding and maintaining the inflexible system, which is custom for specific tasks, instead of all tasks. In the long-term, it is not cost-effective. === Technical data management systems (TDMS) === TDMS is developed based on 3 principles, flexible and organized file storage, self-scaling hybrid data index, and an interactive post-processing environment. The system in practical, mainly consists of 3 components, data files with essential and relevant Metadata, data finders for organizing and managing data regardless of files formats, and, a software of searching, analyzing and reporting. With metadata attached to original data files, the data finder can identify different related data files during searches, even if they are in different file formats. TDMS hence allows researchers to search for data like browsing the Internet. Last but not least, it can adapt to changes and update itself according to the changes, unlike databases. == Comparison between strong information systems and weak information systems == Complex organizations may need large amounts

  Read more →
• 

  Adaptive algorithm

  An adaptive algorithm is an algorithm that changes its behavior at the time it is run, based on information available and on a priori defined reward mechanism (or criterion). Such information could be the story of recently received data, information on the available computational resources, or other run-time acquired (or a priori known) information related to the environment in which it operates. Among the most used adaptive algorithms is the Widrow-Hoff’s least mean squares (LMS), which represents a class of stochastic gradient-descent algorithms used in adaptive filtering and machine learning. In adaptive filtering the LMS is used to mimic a desired filter by finding the filter coefficients that relate to producing the least mean square of the error signal (difference between the desired and the actual signal). For example, stable partition, using no additional memory is O(n lg n) but given O(n) memory, it can be O(n) in time. As implemented by the C++ Standard Library, stable_partition is adaptive and so it acquires as much memory as it can get (up to what it would need at most) and applies the algorithm using that available memory. Another example is adaptive sort, whose behavior changes upon the presortedness of its input. An example of an adaptive algorithm in radar systems is the constant false alarm rate (CFAR) detector. In machine learning and optimization, many algorithms are adaptive or have adaptive variants, which usually means that the algorithm parameters such as learning rate are automatically adjusted according to statistics about the optimisation thus far (e.g. the rate of convergence). Examples include adaptive simulated annealing, adaptive coordinate descent, adaptive quadrature, AdaBoost, Adagrad, Adadelta, RMSprop, and Adam. In data compression, adaptive coding algorithms such as Adaptive Huffman coding or Prediction by partial matching can take a stream of data as input, and adapt their compression technique based on the symbols that they have already encountered. In signal processing, the Adaptive Transform Acoustic Coding (ATRAC) codec used in MiniDisc recorders is called "adaptive" because the window length (the size of an audio "chunk") can change according to the nature of the sound being compressed, to try to achieve the best-sounding compression strategy.

  Read more →
• 

  Query language

  A query language, also known as data query language or database query language (DQL), is a computer language used to make queries in databases and information systems. In database systems, query languages rely on strict theory to retrieve information. A well known example is the Structured Query Language (SQL). == Types == Broadly, query languages can be classified according to whether they are database query languages or information retrieval query languages. The difference is that a database query language attempts to give factual answers to factual questions, while an information retrieval query language attempts to find documents containing information that is relevant to an area of inquiry. Other types of query languages include: Full-text. The simplest query language is treating all terms as bag of words that are to be matched with the postings in the inverted index and where subsequently ranking models are applied to retrieve the most relevant documents. Only tokens are defined in the CFG. Web search engines often use this approach. Boolean. A query language that also supports the use of the Boolean operators AND, OR, NOT. Structured. A language that supports searching within (a combination of) fields when a document is structured and has been indexed using its document structure. Natural language. A query language that supports natural language by parsing the natural language query to a form that can be best used to retrieve relevant documents, for example with Question answering systems or conversational search. == Examples == Attempto Controlled English is a query language that is also a controlled natural language. AQL is a query language for the ArangoDB native multi-model database system. .QL is a proprietary object-oriented query language for querying relational databases; successor of Datalog. CodeQL is the analysis engine used by developers to automate security checks, and by security researchers to perform variant analysis on GitHub. Contextual Query Language (CQL) a formal language for representing queries to information retrieval systems such as web indexes or bibliographic catalogues. Cypher is a query language for the Neo4j graph database. DMX is a query language for data mining models. Datalog is a query language for deductive databases. F-logic is a declarative object-oriented language for deductive databases and knowledge representation. FQL enables you to use a SQL-style interface to query the data exposed by the Graph API. It provides advanced features not available in the Graph API. Gellish English is a language that can be used for queries in Gellish English Databases, for dialogues (requests and responses) as well as for information modeling and knowledge modeling. Gremlin is an Apache Software Foundation graph traversal language for OLTP and OLAP graph systems. GraphQL is a data query language developed by Facebook as an alternate to REST and ad-hoc webservice architectures. HTSQL is a query language that translates HTTP queries to SQL. ISBL is a query language for PRTV, one of the earliest relational database management systems. Jaql is a functional data processing and query language most commonly used for JSON query processing. JPQL is a query language defined as part of Jakarta Persistence (used in Java applications to make queries to a relational DB using entity objects instead of DB tables). jq is a functional programming language often used for processing queries against one or more JSON documents, including very large ones. JSONiq is a declarative query language designed for collections of JSON documents. KQL (Kusto Query Language), a query language by Microsoft used in Azure Data Explorer LDAP is an application protocol for querying and modifying directory services running over TCP/IP. LogiQL is a variant of Datalog and is the query language for the LogicBlox system. M Formula language, a mashup query language used in Microsoft's Power Query. MQL is a cheminformatics query language for a substructure search allowing beside nominal properties also numerical properties. MDX is a query language for OLAP databases. N1QL is a Couchbase's query language finding data in Couchbase Servers. Object Query Language OCL (Object Constraint Language). Despite its name, OCL is also an object query language and an OMG standard. OPath, intended for use in querying WinFS Stores. Poliqarp Query Language is a special query language designed to analyze annotated text. Used in the Poliqarp search engine. PQL is a special-purpose programming language for managing process models based on information about scenarios that these models describe. PRQL PRQL (Pipelined Relational Query Language) is a modern language for transforming data. Consists of a curated set of orthogonal transformations, which are combined together to form a pipeline. PTQL based on relational queries over program traces, allowing programmers to write expressive, declarative queries about program behavior. QUEL is a relational database access language, similar in most ways to SQL. RDQL is a RDF query language. SMARTS is the cheminformatics standard for a substructure search. SPARQL is a query language for RDF graphs. SQL is a well-known query language and data manipulation language for relational databases. XQuery is a query language for XML data sources. XPath is a declarative language for navigating XML documents. YQL is an SQL-like query language created by Yahoo!. Search engine query languages, e.g., as used by Google. or Bing

  Read more →
• 

  Conditional random field

  Conditional random fields (CRFs) are a class of statistical modeling methods often applied in pattern recognition and machine learning and used for structured prediction. Whereas a classifier predicts a label for a single sample without considering "neighbouring" samples, a CRF can take context into account. To do so, the predictions are modelled as a graphical model, which represents the presence of dependencies between the predictions. The kind of graph used depends on the application. For example, in natural language processing, "linear chain" CRFs are popular, for which each prediction is dependent only on its immediate neighbours. In image processing, the graph typically connects locations to nearby and/or similar locations to enforce that they receive similar predictions. Other examples where CRFs are used are: labeling or parsing of sequential data for natural language processing or biological sequences, part-of-speech tagging, shallow parsing, named entity recognition, gene finding, peptide critical functional region finding, and object recognition and image segmentation in computer vision. == Description == CRFs are a type of discriminative undirected probabilistic graphical model. Lafferty, McCallum and Pereira define a CRF on observations X {\displaystyle {\boldsymbol {X}}} and random variables Y {\displaystyle {\boldsymbol {Y}}} as follows: Let G = ( V , E ) {\displaystyle G=(V,E)} be a graph such that Y = ( Y v ) v ∈ V {\displaystyle {\boldsymbol {Y}}=({\boldsymbol {Y}}_{v})_{v\in V}} , so that Y {\displaystyle {\boldsymbol {Y}}} is indexed by the vertices of G {\displaystyle G} . Then ( X , Y ) {\displaystyle ({\boldsymbol {X}},{\boldsymbol {Y}})} is a conditional random field when each random variable Y v {\displaystyle {\boldsymbol {Y}}_{v}} , conditioned on X {\displaystyle {\boldsymbol {X}}} , obeys the Markov property with respect to the graph; that is, its probability is dependent only on its neighbours in G and not its past states: P ( Y v | X , { Y w : w ≠ v } ) = P ( Y v | X , { Y w : w ∼ v } ) {\displaystyle P({\boldsymbol {Y}}_{v}|{\boldsymbol {X}},\{{\boldsymbol {Y}}_{w}:w\neq v\})=P({\boldsymbol {Y}}_{v}|{\boldsymbol {X}},\{{\boldsymbol {Y}}_{w}:w\sim v\})} , where w ∼ v {\displaystyle {\mathit {w}}\sim v} means that w {\displaystyle w} and v {\displaystyle v} are neighbors in G {\displaystyle G} . What this means is that a CRF is an undirected graphical model whose nodes can be divided into exactly two disjoint sets X {\displaystyle {\boldsymbol {X}}} and Y {\displaystyle {\boldsymbol {Y}}} , the observed and output variables, respectively; the conditional distribution p ( Y | X ) {\displaystyle p({\boldsymbol {Y}}|{\boldsymbol {X}})} is then modeled. === Inference === For general graphs, the problem of exact inference in CRFs is intractable. The inference problem for a CRF is basically the same as for an MRF and the same arguments hold. However, there exist special cases for which exact inference is feasible: If the graph is a chain or a tree, message passing algorithms yield exact solutions. The algorithms used in these cases are analogous to the forward-backward and Viterbi algorithm for the case of HMMs. If the CRF only contains pair-wise potentials and the energy is submodular, combinatorial min cut/max flow algorithms yield exact solutions. If exact inference is impossible, several algorithms can be used to obtain approximate solutions. These include: Loopy belief propagation Alpha expansion Mean field inference Linear programming relaxations === Parameter learning === Learning the parameters θ {\displaystyle \theta } is usually done by maximum likelihood learning for p ( Y i | X i ; θ ) {\displaystyle p(Y_{i}|X_{i};\theta )} . If all nodes have exponential family distributions and all nodes are observed during training, this optimization is convex. It can be solved for example using gradient descent algorithms, or Quasi-Newton methods such as the L-BFGS algorithm. On the other hand, if some variables are unobserved, the inference problem has to be solved for these variables. Exact inference is intractable in general graphs, so approximations have to be used. === Examples === In sequence modeling, the graph of interest is usually a chain graph. An input sequence of observed variables X {\displaystyle X} represents a sequence of observations and Y {\displaystyle Y} represents a hidden (or unknown) state variable that needs to be inferred given the observations. The Y i {\displaystyle Y_{i}} are structured to form a chain, with an edge between each Y i − 1 {\displaystyle Y_{i-1}} and Y i {\displaystyle Y_{i}} . As well as having a simple interpretation of the Y i {\displaystyle Y_{i}} as "labels" for each element in the input sequence, this layout admits efficient algorithms for: model training, learning the conditional distributions between the Y i {\displaystyle Y_{i}} and feature functions from some corpus of training data. decoding, determining the probability of a given label sequence Y {\displaystyle Y} given X {\displaystyle X} . inference, determining the most likely label sequence Y {\displaystyle Y} given X {\displaystyle X} . The conditional dependency of each Y i {\displaystyle Y_{i}} on X {\displaystyle X} is defined through a fixed set of feature functions of the form f ( i , Y i − 1 , Y i , X ) {\displaystyle f(i,Y_{i-1},Y_{i},X)} , which can be thought of as measurements on the input sequence that partially determine the likelihood of each possible value for Y i {\displaystyle Y_{i}} . The model assigns each feature a numerical weight and combines them to determine the probability of a certain value for Y i {\displaystyle Y_{i}} . Linear-chain CRFs have many of the same applications as conceptually simpler hidden Markov models (HMMs), but relax certain assumptions about the input and output sequence distributions. An HMM can loosely be understood as a CRF with very specific feature functions that use constant probabilities to model state transitions and emissions. Conversely, a CRF can loosely be understood as a generalization of an HMM that makes the constant transition probabilities into arbitrary functions that vary across the positions in the sequence of hidden states, depending on the input sequence. Notably, in contrast to HMMs, CRFs can contain any number of feature functions, the feature functions can inspect the entire input sequence X {\displaystyle X} at any point during inference, and the range of the feature functions need not have a probabilistic interpretation. == Variants == === Higher-order CRFs and semi-Markov CRFs === CRFs can be extended into higher order models by making each Y i {\displaystyle Y_{i}} dependent on a fixed number k {\displaystyle k} of previous variables Y i − k , . . . , Y i − 1 {\displaystyle Y_{i-k},...,Y_{i-1}} . In conventional formulations of higher order CRFs, training and inference are only practical for small values of k {\displaystyle k} (such as k ≤ 5), since their computational cost increases exponentially with k {\displaystyle k} . However, another recent advance has managed to ameliorate these issues by leveraging concepts and tools from the field of Bayesian nonparametrics. Specifically, the CRF-infinity approach constitutes a CRF-type model that is capable of learning infinitely-long temporal dynamics in a scalable fashion. This is effected by introducing a novel potential function for CRFs that is based on the Sequence Memoizer (SM), a nonparametric Bayesian model for learning infinitely-long dynamics in sequential observations. To render such a model computationally tractable, CRF-infinity employs a mean-field approximation of the postulated novel potential functions (which are driven by an SM). This allows for devising efficient approximate training and inference algorithms for the model, without undermining its capability to capture and model temporal dependencies of arbitrary length. There exists another generalization of CRFs, the semi-Markov conditional random field (semi-CRF), which models variable-length segmentations of the label sequence Y {\displaystyle Y} . This provides much of the power of higher-order CRFs to model long-range dependencies of the Y i {\displaystyle Y_{i}} , at a reasonable computational cost. Finally, large-margin models for structured prediction, such as the structured Support Vector Machine can be seen as an alternative training procedure to CRFs. === Latent-dynamic conditional random field === Latent-dynamic conditional random fields (LDCRF) or discriminative probabilistic latent variable models (DPLVM) are a type of CRFs for sequence tagging tasks. They are latent variable models that are trained discriminatively. In an LDCRF, like in any sequence tagging task, given a sequence of observations x = x 1 , … , x n {\displaystyle x_{1},\dots ,x_{n}} , the main problem the model must solve is how to assign a sequence of labels y = y 1 , … , y n {\displaystyle y_{1},\dots ,y_{n}} from one finite set

  Read more →
• 

  Pol.is

  Polis (or Pol.is) is wiki survey software designed for large group collaborations. As a civic technology, Polis allows people to share their opinions and ideas, and its algorithm is intended to elevate ideas that can facilitate better decision-making, especially when there are lots of participants. Polis has been credited for assisting the passage of legislation in Taiwan. Pol.is has been used by governments in the United States, Canada, Singapore, Philippines, Finland, Spain and elsewhere. == History == Pol.is was founded by Colin Megill, Christopher Small, and Michael Bjorkegren after the Occupy Wall Street and Arab Spring movements. In Taiwan, pol.is has been "one of the key parts" of vTaiwan's suite of open-source tools for its citizen engagement efforts arising out of the Sunflower Student Movement. vTaiwan claims that of the 26 national issues related to technology discussed on the platform, 80% led to government action. Pol.is is also utilized by "Join," a national platform for online deliberation run by the Taiwanese government. In 2022, Wired reported that Polis was an influence on the Community Notes project at Twitter. In 2023, Megill advised OpenAI on how to facilitate deliberation at scale in a way that was more efficient than Polis, which still required significant human labor and analysis at the time. He helped to award $1 million in grants to teams working on solving the problem of deliberation at scale. In 2023, Anthropic was also exploring steering model behavior using Polis. In 2025, it helped the county that includes Bowling Green, Kentucky make a 25 year plan by facilitating the collection and review of ideas from thousands of residents, representing 10% of the county. 2,370 of 3,940 unique ideas were agreed-upon by over 80% of survey respondents. Ideas were screened by volunteers if they were redundant to an existing idea, off-topic or obscene. == How it works == Pol.is participants are anonymous and cannot reply directly to others posts, in an effort to avoid personal attacks for users. Its algorithms are designed not for engagement and scrolling, but to find areas of agreement to better understand the nuances of a wide range of opinions. Participants are prompted for ideas and vote on other participants' ideas. == Reception == Andrew Leonard, The Financial Times, and VentureBeat describe Pol.is as a possible antidote to the divisiveness of traditional internet discourse by gamifying consensus. Audrey Tang agreed saying, "Polis is quite well known in that it's a kind of social media that instead of polarizing people to drive so called engagement or addiction or attention, it automatically drives bridge making narratives and statements. So only the ideas that speak to both sides or to multiple sides will gain prominence in Polis." Niall Ferguson argues that the approach to utilize tools like Pol.is and Join in Taiwan empowers ordinary people instead of the elite and protects individual freedoms, providing a contrast to the AI-enhanced panopticon model seen in China. Carl Miller praised the technology as having "gamified finding consensus." Darshana Narayanan, in an op-ed in the Economist, argues that open-source machine-learning-based tools like Polis can help to bypass the influence of special interests or experts. Jamie Susskind cited polis and vTaiwan as a model for democracies, particularly around digital policy issues.

  Read more →
• 

  In-place algorithm

  In computer science, an in-place algorithm is an algorithm that operates directly on the input data structure without requiring extra space proportional to the input size. In other words, it modifies the input in place, without creating a separate copy of the data structure. An algorithm which is not in-place is sometimes called not-in-place or out-of-place. In-place can have slightly different meanings. In its strictest form, the algorithm can only have a constant amount of extra space, counting everything including function calls and pointers. However, this form is very limited as simply having an index to a length n array requires O(log n) bits. More broadly, in-place means that the algorithm does not use extra space for manipulating the input but may require a small though non-constant extra space for its operation. Usually, this space is O(log n), though sometimes anything in o(n) is allowed. Note that space complexity also has varied choices in whether or not to count the index lengths as part of the space used. Often, the space complexity is given in terms of the number of indices or pointers needed, ignoring their length. In this article, we refer to total space complexity (DSPACE), counting pointer lengths. Therefore, the space requirements here have an extra log n factor compared to an analysis that ignores the lengths of indices and pointers. An algorithm may or may not count the output as part of its space usage. Since in-place algorithms usually overwrite their input with output, no additional space is needed. When writing the output to write-only memory or a stream, it may be more appropriate to only consider the working space of the algorithm. In theoretical applications such as log-space reductions, it is more typical to always ignore output space (in these cases it is more essential that the output is write-only). == Examples == Given an array a of n items, suppose we want an array that holds the same elements in reversed order and to dispose of the original. One seemingly simple way to do this is to create a new array of equal size, fill it with copies from a in the appropriate order and then delete a. function reverse(a[0..n - 1]) allocate b[0..n - 1] for i from 0 to n - 1 b[n − 1 − i] := a[i] return b Unfortunately, this requires O(n) extra space for having the arrays a and b available simultaneously. Also, allocation and deallocation are often slow operations. Since we no longer need a, we can instead overwrite it with its own reversal using this in-place algorithm which will only need constant number (2) of integers for the auxiliary variables i and tmp, no matter how large the array is. function reverse_in_place(a[0..n-1]) for i from 0 to floor((n-2)/2) tmp := a[i] a[i] := a[n − 1 − i] a[n − 1 − i] := tmp As another example, many sorting algorithms rearrange arrays into sorted order in-place, including: bubble sort, comb sort, selection sort, insertion sort, heapsort, and Shell sort. These algorithms require only a few pointers, so their space complexity is O(log n). Quicksort operates in-place on the data to be sorted. However, quicksort requires O(log n) stack space pointers to keep track of the subarrays in its divide and conquer strategy. Consequently, quicksort needs O(log2 n) additional space. Although this non-constant space technically takes quicksort out of the in-place category, quicksort and other algorithms needing only O(log n) additional pointers are usually considered in-place algorithms. Most selection algorithms are also in-place, although some considerably rearrange the input array in the process of finding the final, constant-sized result. Some text manipulation algorithms such as trim and reverse may be done in-place. == In computational complexity == In computational complexity theory, the strict definition of in-place algorithms includes all algorithms with O(1) space complexity, the class DSPACE(1). This class is very limited; it equals the regular languages. In fact, it does not even include any of the examples listed above. Algorithms are usually considered in L, the class of problems requiring O(log n) additional space, to be in-place. This class is more in line with the practical definition, as it allows numbers of size n as pointers or indices. This expanded definition still excludes quicksort, however, because of its recursive calls. Identifying the in-place algorithms with L has some interesting implications; for example, it means that there is a (rather complex) in-place algorithm to determine whether a path exists between two nodes in an undirected graph, a problem that requires O(n) extra space using typical algorithms such as depth-first search (a visited bit for each node). This in turn yields in-place algorithms for problems such as determining if a graph is bipartite or testing whether two graphs have the same number of connected components. == Role of randomness == In many cases, the space requirements of an algorithm can be drastically cut by using a randomized algorithm. For example, if one wishes to know if two vertices in a graph of n vertices are in the same connected component of the graph, there is no known simple, deterministic, in-place algorithm to determine this. However, if we simply start at one vertex and perform a random walk of about 20n3 steps, the chance that we will stumble across the other vertex provided that it is in the same component is very high. Similarly, there are simple randomized in-place algorithms for primality testing such as the Miller–Rabin primality test, and there are also simple in-place randomized factoring algorithms such as Pollard's rho algorithm. == In functional programming == Functional programming languages often discourage or do not support explicit in-place algorithms that overwrite data, since this is a type of side effect; instead, they only allow new data to be constructed. However, good functional language compilers will often recognize when an object very similar to an existing one is created and then the old one is thrown away, and will optimize this into a simple mutation "under the hood". Note that it is possible in principle to carefully construct in-place algorithms that do not modify data (unless the data is no longer being used), but this is rarely done in practice.

  Read more →
• 

  Archetype (information science)

  In the field of informatics, an archetype is a formal re-usable model of a domain concept. Traditionally, the term archetype is used in psychology to mean an idealized model of a person, personality or behaviour (see Archetype). The usage of the term in informatics is derived from this traditional meaning, but applied to domain modelling instead. An archetype is defined by the OpenEHR Foundation (for health informatics) as follows: An archetype is a computable expression of a domain content model in the form of structured constraint statements, based on some reference model. openEHR archetypes are based on the openEHR reference model. Archetypes are all expressed in the same formalism. In general, they are defined for wide re-use, however, they can be specialized to include local particularities. They can accommodate any number of natural languages and terminologies. == Formal specifications == The modern archetype formalism is specified and maintained by the openEHR Foundation, and although originally developed for the health IT domain, is completely domain-independent, and has been used in geospatial modelling, telecommunications, and defence. The archetype formalism consists of a number of specifications including: 'ADL 1.4': original release of Archetype Definition Language (ADL) and Archetype Object Model (AOM); widely implemented in health IT domain; 'ADL 2': modern release of Archetype Definition Language (ADL), Archetype Object Model (AOM), Archetype Identification specification and Archetype Technology Overview. The Archetype Technology Overview provides a short technical overview of the archetype formalism useful for new users. The ADL/AOM 1.4 specifications were provided to ISO TC 215 in 2008 by the openEHR Foundation and became the ISO 13606-2 standard, extant until 2019. ISO TC 215 accepted the AOM 2 specification as the basis for a revision of this standard, which was issued in 2019. In late 2015, the Object Management Group (OMG) accepted an RfP entitled 'Archetype Modeling Language (AML)' as a new candidate standard. This specification is a form of ADL re-engineered as a UML profile so as to enable archetype modelling to be supported within UML tools. == Tools == A number of tools area available for working with archetypes. Most are listed on the openEHR modelling tools page. They include: ADL Designer, a modern AOM2-based web editing application Archetype Editor, an original desktop clinical modelling tool Template Designer, an original desktop clinical templating tool LinkEHR, an archetype and data integration tool ADL Workbench, reference compiler and visualiser tool == Example ==

  Read more →
• 

  Stop Motion Studio

  Stop Motion Studio is a stop motion animation software developed by Cateater LLC. It is available as both an app for iOS and Android and as a software for Windows and Mac. Two versions of the software exist, the standard Stop Motion Studio for free, and the paid Stop Motion Studio Pro, which contains extra, more advanced features. The software is commonly used in brickfilming.

  Read more →
• 

  Species distribution modelling

  Species distribution modelling (SDM), also known as environmental (or ecological) niche modelling (ENM), habitat suitability modelling, predictive habitat distribution modelling, and range mapping uses ecological models to predict the distribution of a species across geographic space and time using environmental data. The environmental data are most often climate data (e.g. temperature, precipitation), but can include other variables such as soil type, water depth, and land cover. SDMs are used in several research areas in conservation biology, ecology and evolution. These models can be used to understand how environmental conditions influence the occurrence or abundance of a species, and for predictive purposes (ecological forecasting). Predictions from an SDM may be of a species' future distribution under climate change, a species' past distribution in order to assess evolutionary relationships, or the potential future distribution of an invasive species. Predictions of current and/or future habitat suitability can be useful for management applications (e.g. reintroduction or translocation of vulnerable species, reserve placement in anticipation of climate change). There are two main types of SDMs. Correlative SDMs, also known as climate envelope models, bioclimatic models, or resource selection function models, model the observed distribution of a species as a function of environmental conditions. Mechanistic SDMs, also known as process-based models or biophysical models, use independently derived information about a species' physiology to develop a model of the environmental conditions under which the species can exist. The extent to which such modelled data reflect real-world species distributions will depend on a number of factors, including the nature, complexity, and accuracy of the models used and the quality of the available environmental data layers; the availability of sufficient and reliable species distribution data as model input; and the influence of various factors such as barriers to dispersal, geologic history, or biotic interactions, that increase the difference between the realized niche and the fundamental niche. Environmental niche modelling may be considered a part of the discipline of biodiversity informatics. == History == A. F. W. Schimper used geographical and environmental factors to explain plant distributions in his 1898 Pflanzengeographie auf physiologischer Grundlage (Plant Geography Upon a Physiological Basis) and his 1908 work of the same name. Andrew Murray used the environment to explain the distribution of mammals in his 1866 The Geographical Distribution of Mammals. Robert Whittaker's work with plants and Robert MacArthur's work with birds strongly established the role the environment plays in species distributions. Elgene O. Box constructed environmental envelope models to predict the range of tree species. His computer simulations were among the earliest uses of species distribution modelling. The adoption of more sophisticated generalised linear models (GLMs) made it possible to create more sophisticated and realistic species distribution models. The expansion of remote sensing and the development of GIS-based environmental modelling increase the amount of environmental information available for model-building and made it easier to use. == Correlative vs mechanistic models == === Correlative SDMs === SDMs originated as correlative models. Correlative SDMs model the observed distribution of a species as a function of geographically referenced climatic predictor variables using multiple regression approaches. Given a set of geographically referred observed presences of a species and a set of climate maps, a model defines the most likely environmental ranges within which a species lives. Correlative SDMs assume that species are at equilibrium with their environment and that the relevant environmental variables have been adequately sampled. The models allow for interpolation between a limited number of species occurrences. For these models to be effective, it is required to gather observations not only of species presences, but also of absences, that is, where the species does not live. Records of species absences are typically not as common as records of presences, thus often "random background" or "pseudo-absence" data are used to fit these models. If there are incomplete records of species occurrences, pseudo-absences can introduce bias. Since correlative SDMs are models of a species' observed distribution, they are models of the realized niche (the environments where a species is found), as opposed to the fundamental niche (the environments where a species can be found, or where the abiotic environment is appropriate for the survival). For a given species, the realized and fundamental niches might be the same, but if a species is geographically confined due to dispersal limitation or species interactions, the realized niche will be smaller than the fundamental niche. Correlative SDMs are easier and faster to implement than mechanistic SDMs, and can make ready use of available data. Since they are correlative however, they do not provide much information about causal mechanisms and are not good for extrapolation. They will also be inaccurate if the observed species range is not at equilibrium (e.g. if a species has been recently introduced and is actively expanding its range). In standard SDMs, the distribution of a single species is often modeled, with unique parameters describing how environmental (abiotic) factors influence its occurrence probability. This allows for differentiated responses to environmental drivers among species, but can be problematic for data-deficient species. In contrast, similarities in environmental responses can be accounted for in multi-species SDMs, which model several species jointly using shared or hierarchically related parameters. However, neither approach explicitly accounts for community-level biotic interactions, which can be important in explaining species diversity patterns. Joint species distribution models (joint SDMs or J-SDMs) address this by modeling species co-occurrence patterns directly. The occurrence probability of a given species is thus influenced not only by abiotic drivers but also by inferred biotic associations with other species. This can improve accuracy for rarer taxa and provide insights into community ecology. Both standard SDMs and J-SDMs can be used to generate community-level metrics, such as species richness, by aggregating outputs across multiple species. These can be important for decision-making such as conservation planning. === Mechanistic SDMs === Mechanistic SDMs are more recently developed. In contrast to correlative models, mechanistic SDMs use physiological information about a species (taken from controlled field or laboratory studies) to determine the range of environmental conditions within which the species can persist. These models aim to directly characterize the fundamental niche, and to project it onto the landscape. A simple model may simply identify threshold values outside of which a species can't survive. A more complex model may consist of several sub-models, e.g. micro-climate conditions given macro-climate conditions, body temperature given micro-climate conditions, fitness or other biological rates (e.g. survival, fecundity) given body temperature (thermal performance curves), resource or energy requirements, and population dynamics. Geographically referenced environmental data are used as model inputs. Because the species distribution predictions are independent of the species' known range, these models are especially useful for species whose range is actively shifting and not at equilibrium, such as invasive species. Mechanistic SDMs incorporate causal mechanisms and are better for extrapolation and non-equilibrium situations. However, they are more labor-intensive to create than correlational models and require the collection and validation of a lot of physiological data, which may not be readily available. The models require many assumptions and parameter estimates, and they can become very complicated. Dispersal, biotic interactions, and evolutionary processes present challenges, as they aren't usually incorporated into either correlative or mechanistic models. Correlational and mechanistic models can be used in combination to gain additional insights. For example, a mechanistic model could be used to identify areas that are clearly outside the species' fundamental niche, and these areas can be marked as absences or excluded from analysis. See for a comparison between mechanistic and correlative models. == Niche models (correlative) == There are a variety of mathematical methods that can be used for fitting, selecting, and evaluating correlative SDMs. Models include "profile" methods, which are simple statistical techniques that use e.g. environmental distance to known sites of occurrence such as

  Read more →
• 

  Rendezvous hashing

  Rendezvous or highest random weight (HRW) hashing is an algorithm that allows clients to achieve distributed agreement on a set of k {\displaystyle k} options out of a possible set of n {\displaystyle n} options. A typical application is when clients need to agree on which sites (or proxies) objects are assigned to. Consistent hashing addresses the special case k = 1 {\displaystyle k=1} using a different method. Rendezvous hashing is both much simpler and more general than consistent hashing (see below). == History == Rendezvous hashing was invented by David Thaler and Chinya Ravishankar at the University of Michigan in 1996. Consistent hashing appeared a year later in the literature. Given its simplicity and generality, rendezvous hashing is now being preferred to consistent hashing in real-world applications. Rendezvous hashing was used very early on in many applications including mobile caching, router design, secure key establishment, and sharding and distributed databases. Other examples of real-world systems that use Rendezvous Hashing include the GitHub load balancer, the Apache Ignite distributed database, the Tahoe-LAFS file store, the CoBlitz large-file distribution service, Apache Druid, IBM's Cloud Object Store, the Arvados Data Management System, Apache Kafka, and the Twitter EventBus pub/sub platform. One of the first applications of rendezvous hashing was to enable multicast clients on the Internet (in contexts such as the MBONE) to identify multicast rendezvous points in a distributed fashion. It was used in 1998 by Microsoft's Cache Array Routing Protocol (CARP) for distributed cache coordination and routing. Some Protocol Independent Multicast routing protocols use rendezvous hashing to pick a rendezvous point. == Problem definition and approach == === Algorithm === Rendezvous hashing solves a general version of the distributed hash table problem: We are given a set of n {\displaystyle n} sites (servers or proxies, say). How can any set of clients, given an object O {\displaystyle O} , agree on a k-subset of sites to assign to O {\displaystyle O} ? The standard version of the problem uses k = 1. Each client is to make its selection independently, but all clients must end up picking the same subset of sites. This is non-trivial if we add a minimal disruption constraint, and require that when a site fails or is removed, only objects mapping to that site need be reassigned to other sites. The basic idea is to give each site S j {\displaystyle S_{j}} a score (a weight) for each object O i {\displaystyle O_{i}} , and assign the object to the highest scoring site. All clients first agree on a hash function h ( ⋅ ) {\displaystyle h(\cdot )} . For object O i {\displaystyle O_{i}} , the site S j {\displaystyle S_{j}} is defined to have weight w i , j = h ( O i , S j ) {\displaystyle w_{i,j}=h(O_{i},S_{j})} . Each client independently computes these weights w i , 1 , w i , 2 … w i , n {\displaystyle w_{i,1},w_{i,2}\dots w_{i,n}} and picks the k sites that yield the k largest hash values. The clients have thereby achieved distributed k {\displaystyle k} -agreement. If a site S {\displaystyle S} is added or removed, only the objects mapping to S {\displaystyle S} are remapped to different sites, satisfying the minimal disruption constraint above. The HRW assignment can be computed independently by any client, since it depends only on the identifiers for the set of sites S 1 , S 2 … S n {\displaystyle S_{1},S_{2}\dots S_{n}} and the object being assigned. HRW easily accommodates different capacities among sites. If site S k {\displaystyle S_{k}} has twice the capacity of the other sites, we simply represent S k {\displaystyle S_{k}} twice in the list, say, as S k , 1 , S k , 2 {\displaystyle S_{k,1},S_{k,2}} . Clearly, twice as many objects will now map to S k {\displaystyle S_{k}} as to the other sites. === Properties === Consider the simple version of the problem, with k = 1, where all clients are to agree on a single site for an object O. Approaching the problem naively, it might appear sufficient to treat the n sites as buckets in a hash table and hash the object name O into this table. Unfortunately, if any of the sites fails or is unreachable, the hash table size changes, forcing all objects to be remapped. This massive disruption makes such direct hashing unworkable. Under rendezvous hashing, however, clients handle site failures by picking the site that yields the next largest weight. Remapping is required only for objects currently mapped to the failed site, and disruption is minimal. Rendezvous hashing has the following properties: Low overhead: The hash function used is efficient, so overhead at the clients is very low. Load balancing: Since the hash function is randomizing, each of the n sites is equally likely to receive the object O. Loads are uniform across the sites. Site capacity: Sites with different capacities can be represented in the site list with multiplicity in proportion to capacity. A site with twice the capacity of the other sites will be represented twice in the list, while every other site is represented once. High hit rate: Since all clients agree on placing an object O into the same site SO, each fetch or placement of O into SO yields the maximum utility in terms of hit rate. The object O will always be found unless it is evicted by some replacement algorithm at SO. Minimal disruption: When a site fails, only the objects mapped to that site need to be remapped. Disruption is at the minimal possible level. Distributed k-agreement: Clients can reach distributed agreement on k sites simply by selecting the top k sites in the ordering. == O(log n) running time via skeleton-based hierarchical rendezvous hashing == The standard version of Rendezvous Hashing described above works quite well for moderate n, but when n {\displaystyle n} is extremely large, the hierarchical use of Rendezvous Hashing achieves O ( log ⁡ n ) {\displaystyle O(\log n)} running time. This approach creates a virtual hierarchical structure (called a "skeleton"), and achieves O ( log ⁡ n ) {\displaystyle O(\log n)} running time by applying HRW at each level while descending the hierarchy. The idea is to first choose some constant m {\displaystyle m} and organize the n {\displaystyle n} sites into c = ⌈ n / m ⌉ {\displaystyle c=\lceil n/m\rceil } clusters C 1 = { S 1 , S 2 … S m } , C 2 = { S m + 1 , S m + 2 … S 2 m } … {\displaystyle C_{1}=\left\{S_{1},S_{2}\dots S_{m}\right\},C_{2}=\left\{S_{m+1},S_{m+2}\dots S_{2m}\right\}\dots } Next, build a virtual hierarchy by choosing a constant f {\displaystyle f} and imagining these c {\displaystyle c} clusters placed at the leaves of a tree T {\displaystyle T} of virtual nodes, each with fanout f {\displaystyle f} . In the accompanying diagram, the cluster size is m = 4 {\displaystyle m=4} , and the skeleton fanout is f = 3 {\displaystyle f=3} . Assuming 108 sites (real nodes) for convenience, we get a three-tier virtual hierarchy. Since f = 3 {\displaystyle f=3} , each virtual node has a natural numbering in octal. Thus, the 27 virtual nodes at the lowest tier would be numbered 000 , 001 , 002 , . . . , 221 , 222 {\displaystyle 000,001,002,...,221,222} in octal (we can, of course, vary the fanout at each level - in that case, each node will be identified with the corresponding mixed-radix number). The easiest way to understand the virtual hierarchy is by starting at the top, and descending the virtual hierarchy. We successively apply Rendezvous Hashing to the set of virtual nodes at each level of the hierarchy, and descend the branch defined by the winning virtual node. We can in fact start at any level in the virtual hierarchy. Starting lower in the hierarchy requires more hashes, but may improve load distribution in the case of failures. For example, instead of applying HRW to all 108 real nodes in the diagram, we can first apply HRW to the 27 lowest-tier virtual nodes, selecting one. We then apply HRW to the four real nodes in its cluster, and choose the winning site. We only need 27 + 4 = 31 {\displaystyle 27+4=31} hashes, rather than 108. If we apply this method starting one level higher in the hierarchy, we would need 9 + 3 + 4 = 16 {\displaystyle 9+3+4=16} hashes to get to the winning site. The figure shows how, if we proceed starting from the root of the skeleton, we may successively choose the virtual nodes ( 2 ) 3 {\displaystyle (2)_{3}} , ( 20 ) 3 {\displaystyle (20)_{3}} , and ( 200 ) 3 {\displaystyle (200)_{3}} , and finally end up with site 74. The virtual hierarchy need not be stored, but can be created on demand, since the virtual nodes names are simply prefixes of base- f {\displaystyle f} (or mixed-radix) representations. We can easily create appropriately sorted strings from the digits, as required. In the example, we would be working with the strings 0 , 1 , 2 {\displaystyle 0,1,2} (at tier 1), 20 , 21 , 22 {\displaystyle 20,21,22} (at tier 2), and 200 , 201 , 202

  Read more →
• 

  Causal AI

  Causal AI is a technique in artificial intelligence that builds a causal model and can thereby make inferences using causality rather than just correlation. One practical use for causal AI is for organisations to explain decision-making and the causes for a decision. Systems based on causal AI, by identifying the underlying web of causality for a behaviour or event, provide insights that solely predictive AI models might fail to extract from historical data. An analysis of causality may be used to supplement human decisions in situations where understanding the causes behind an outcome is necessary, such as quantifying the impact of different interventions, policy decisions or performing scenario planning. A 2024 paper from Google DeepMind demonstrated mathematically that "Any agent capable of adapting to a sufficiently large set of distributional shifts must have learned a causal model". The paper offers the interpretation that learning to generalise beyond the original training set requires learning a causal model, concluding that causal AI is necessary for artificial general intelligence. == History == The concept of causal AI and the limits of machine learning were raised by Judea Pearl, the Turing Award-winning computer scientist and philosopher, in 2018's The Book of Why: The New Science of Cause and Effect. Pearl asserted: “Machines' lack of understanding of causal relations is perhaps the biggest roadblock to giving them human-level intelligence.” In 2020, Columbia University established a Causal AI Lab under Director Elias Bareinboim. Professor Bareinboim's research focuses on causal and counterfactual inference and their applications to data-driven fields in the health and social sciences as well as artificial intelligence and machine learning. Technological research and consulting firm Gartner for the first time included causal AI in its 2022 Hype Cycle report, citing it as one of five critical technologies in accelerated AI automation. Causal AI is closely related to but distinct from fields such as causal inference, explainable AI and causal reasoning. While causal inference focuses on estimating cause-effect relationships (often from observational data), causal AI emphasises the integration of those causal models into AI systems for prediction, planning and adaptation.

  Read more →
• 

  Integrated test facility

  An integrated test facility (ITF) creates a fictitious entity in a database to process test transactions simultaneously with live input. ITF can be used to incorporate test transactions into a normal production run of a system. Its advantage is that periodic testing does not require separate test processes. However, careful planning is necessary, and test data must be isolated from production data. Moreover, ITF validates the correct operation of a transaction in an application, but it does not ensure that a system is being operated correctly. Integrated test facility is considered a useful audit tool during an IT audit because it uses the same programs to compare processing using independently calculated data. This involves setting up dummy entities on an application system and processing test or production data against the entity as a means of verifying processing accuracy.

  Read more →
• 

  WCF Data Services

  WCF Data Services (formerly ADO.NET Data Services, codename "Astoria") is a platform for what Microsoft calls Data Services. It is actually a combination of the runtime and a web service through which the services are exposed. It also includes the Data Services Toolkit which lets Astoria Data Services be created from within ASP.NET itself. The Astoria project was announced at MIX 2007, and the first developer preview was made available on April 30, 2007. The first CTP was made available as a part of the ASP.NET 3.5 Extensions Preview. The final version was released as part of Service Pack 1 of the .NET Framework 3.5 on August 11, 2008. The name change from ADO.NET Data Services to WCF data Services was announced at the 2009 PDC. == Overview == WCF Data Services exposes data, represented as Entity Data Model (EDM) objects, via web services accessed over HTTP. The data can be addressed using a REST-like URI. The data service, when accessed via the HTTP GET method with such a URI, will return the data. The web service can be configured to return the data in either plain XML, JSON or RDF+XML. In the initial release, formats like RSS and ATOM are not supported, though they may be in the future. In addition, using other HTTP methods like PUT, POST or DELETE, the data can be updated as well. POST can be used to create new entities, PUT for updating an entity, and DELETE for deleting an entity. == Description == Windows Communication Foundation (WCF) comes to the rescue when we find ourselves not able to achieve what we want to achieve using web services, i.e., other protocols support and even duplex communication. With WCF, we can define our service once and then configure it in such a way that it can be used via HTTP, TCP, IPC, and even Message Queues. We can consume Web Services using server side scripts (ASP.NET), JavaScript Object Notations (JSON), and even REST (Representational State Transfer). Understanding the basics When we say that a WCF service can be used to communicate using different protocols and from different kinds of applications, we will need to understand how we can achieve this. If we want to use a WCF service from an application, then we have three major questions: 1.Where is the WCF service located from a client's perspective? 2.How can a client access the service, i.e., protocols and message formats? 3.What is the functionality that a service is providing to the clients? Once we have the answer to these three questions, then creating and consuming the WCF service will be a lot easier for us. The WCF service has the concept of endpoints. A WCF service provides endpoints which client applications can use to communicate with the WCF service. The answer to these above questions is what is known as the ABC of WCF services and in fact are the main components of a WCF service. So let's tackle each question one by one. Address: Like a webservice, a WCF service also provides a URI which can be used by clients to get to the WCF service. This URI is called as the Address of the WCF service. This will solve the first problem of "where to locate the WCF service?" for us. Binding: Once we are able to locate the WCF service, one should think about how to communicate with the service (protocol wise). The binding is what defines how the WCF service handles the communication. It could also define other communication parameters like message encoding, etc. This will solve the second problem of "how to communicate with the WCF service?" for us. Contract: Now the only question one is left with is about the functionalities that a WCF service provides. The contract is what defines the public data and interfaces that WCF service provides to the clients. The URIs representing the data will contain the physical location of the service, as well as the service name. It will also need to specify an EDM Entity-Set or a specific entity instance, as in respectively http://dataserver/service.svc/MusicCollection or http://dataserver/service.svc/MusicCollection[SomeArtist] The former will list all entities in the Collection set whereas the latter will list only for the entity which is indexed by SomeArtist. The URIs can also specify a traversal of a relationship in the Entity Data Model. For example, http://dataserver/service.svc/MusicCollection[SomeSong]/Genre traverses the relationship Genre (in SQL parlance, joins with the Genre table) and retrieves all instances of Genre that are associated with the entity SomeSong. Simple predicates can also be specified in the URI, like http://dataserver/service.svc/MusicCollection[SomeArtist]/ReleaseDate[Year eq 2006] will fetch the items that are indexed by SomeArtist and had their release in 2006. Filtering and partition information can also be encoded in the URL as http://dataserver/service.svc/MusicCollection?$orderby=ReleaseDate&$skip=100&$top=50 Although the presence of skip and top keywords indicates paging support, in Data Services version 1 there is no method of determining the number of records available and thus impossible to determine how many pages there may be. The OData 2.0 spec adds support for the $count path segment (to return just a count of entities) and $inlineCount (to retrieve a page worth of entities and a total count without a separate round-trip....).

  Read more →
• 

  Artificial intelligence in marketing

  Artificial intelligence marketing (AI marketing) is a form of marketing that uses artificial intelligence concepts and models such as machine learning, natural language processing, and computer vision to achieve marketing goals. The main difference between AI marketing and traditional forms of marketing reside in the reasoning, which is performed through a computer algorithm rather than a human. Each form of marketing has a different technique to the core of the marketing theory. Traditional marketing directly focuses on the needs of consumers; meanwhile some believe the shift AI may cause will lead marketing agencies to manage consumer needs instead. AI is used in various digital marketing spaces, such as content marketing, email marketing, online advertisement (in combination with machine learning), social media marketing, affiliate marketing, and beyond. == Historical development == AI in marketing has a long history, which goes all the way back to the 1980s. At this time, AI research was focusing on expert systems and robotics. Despite the initial research and the studies that were carried out, AI adoption remained limited. Research on it came to a stop for a while, until research was revived two decades later with the advancement in technology, the rise of big data, and a significant increase in computational power. Eventually, AI became very popular in the marketing world, and caught the eyes of many researchers as well as professionals. A large‐scale bibliometric study covering 1,580 peer‑reviewed papers published between 1982 and 2020 confirms that scholarly output on AI in marketing has surged since 2017, with Expert Systems with Applications emerging as the most prolific outlet. Prior to the application of artificial Intelligence in marketing, there was something called "collaborative filtering". This was used as early as 1998 by Amazon, and one of the first ways companies predicted consumer behavior, which enabled millions of recommendations to different customers. Personalized recommender systems are now widely used, for example to suggest music on Spotify, or TV shows on Netflix. A big milestone in AI marketing happened in 2014, when programmatic ad buying gained much greater popularity. Marketing consists of numerous manual tasks such as researching target markets, insertion orders, and managing high budgets as well as prices. In order to cut costs, and remove the need for these tedious tasks, many companies started to automate the marketing process with AI. In 2015, Google introduced RankBrain, a machine learning component of its search algorithm designed to interpret the intent behind user queries. RankBrain was followed by further AI-based search updates, including BERT in 2019, which improved the understanding of conversational queries, and the Multitask Unified Model (MUM) in 2021, which is multimodal and processes information across 75 languages. These advances shifted search engine optimization practice away from keyword matching toward content that satisfies user intent. Artificial intelligence is increasingly used in marketing to personalize user experiences and automate decision-making. For example, Netflix uses AI algorithms to recommend content based on viewing history, while Sephora employs chatbots to assist customers with product selection and availability. Programmatic advertising platforms like Google Ads leverage machine learning to optimize bidding strategies and target audiences more effectively. These applications demonstrate how AI enhances efficiency, engagement, and conversion rates across digital channels. === Artificial neural networks === An artificial neural network is a form of computer program modeled on the brain and nervous system of humans. Neural networks are composed of a series of interconnected processing neurons that function in unison to achieve certain outcomes. Using “human-like trial and error learning methods neural networks detect patterns existing within a data set ignoring data that is not significant while emphasizing the data which is most influential”. From a marketing perspective, neural networks are a form of software tool used to assist in decision making. Neural networks are effective in gathering and extracting information from large data sources and have the ability to identify cause and effect within tha data. These neural nets through the process of learning, identify relationships and connections between databases. Once knowledge has been accumulated, neural networks can be relied on to provide generalizations and can apply past knowledge and learning to a variety of situations. Neural networks help fulfill the role of marketing companies through effectively aiding in market segmentation and measurement of performance while reducing costs and improving accuracy. Due to their learning ability, flexibility, adaption, and knowledge discovery, neural networks offer many advantages over traditional models. Neural networks can be used to assist in pattern classification, forecasting and marketing analysis. == Tools and uses == Classification of customers can be facilitated through the neural network approach allowing companies to make informed marketing decisions. An example of this was employed by Spiegel Inc., a firm dealing in direct-mail operations that used neural networks to improve efficiencies. Using software developed by NeuralWare Inc., Spiegel identified the demographics of customers who had made a single purchase and those customers who had made repeat purchases. Neural networks where then able to identify the key patterns and consequently identify the customers that were most likely to repeat purchase. Understanding this information allowed Spiegel to streamline marketing efforts, and reduced costs. Sales forecasting “is the process of estimating future events with the goal of providing benchmarks for monitoring actual performance and reducing uncertainty". Artificial intelligence techniques have emerged to facilitate the process of forecasting through increasing accuracy in the areas of demand for products, distribution, employee turnover, performance measurement, and inventory control. An example of forecasting using neural networks is the Airline Marketing Assistant/Tactician; an application developed by BehabHeuristics which allows for the forecasting of passenger demand and consequent seat allocation through neural networks. This system has been used by National air Canada and USAir. Neural networks provide a useful alternative to traditional statistical models due to their reliability, time-saving characteristics and ability to recognize patterns from incomplete or noisy data. Examples of marketing analysis systems includes the Target Marketing System developed by Churchull Systems for Veratex Corporation. This support system scans a market database to identify dormant customers allowing management to make decisions regarding which key customers to target. When performing marketing analysis, neural networks can assist in the gathering and processing of information ranging from consumer demographics and credit history to the purchase patterns of consumers. Predictive analytics is a form of analytics involving the use of historical data and artificial intelligence algorithms to predict future trends and outcomes. It serves as a tool for anticipating and understanding user behavior based on patterns found in data. Predictive analytics uses artificial intelligence machine learning algorithms to recognize and predict patterns within data. Machine learning algorithms analyze the data, recognize patterns, and make predictions through continuous learning and adaptation. Predictive analytics is widely used across businesses and industries as a way to identify opportunities, avoid risks, and anticipate customer needs based on information derived from the analysis of user data. By analyzing historical customer data, artificial intelligence algorithms can deliver relevant and targeted marketing content. Recent systematic reviews show that generative large‑language models such as GPT‑3 and GPT‑4 are now routinely embedded in predictive‑analytics pipelines to mine unstructured market data and anticipate customer intent with greater precision. Personalization engines use artificial intelligence and machine learning to provide content or advertisements that are relevant to the user. User data is gathered, which then gets processed with machine learning, and patterns and trends among the users are identified. Users with shared characteristics or behaviors are then segmented into groups, and the personalization engine adjusts content and advertisements to match each segment's preferences. By processing a large amount of data, personalization engines are able to match users to advertisements and recommendations that align with their interests or preferences. Field evidence from consumer‑goods and electronics firms indicates that AI‑driven personalization can raise

  Read more →