AI Data Analyst Zalando

AI Data Analyst Zalando — independent reviews, comparisons, pricing and step-by-step guides on Aizhi.

  • Stixel

    Stixel

    In computer vision, a stixel (portmanteau of "stick" and "pixel") is a superpixel representation of depth information in an image, in the form of a vertical stick that approximates the closest obstacles within a certain vertical slice of the scene. Introduced in 2009, stixels have applications in robotic navigation and advanced driver-assistance systems, where they can be used to define a representation of robotic environments and traffic scenes with a medium level of abstraction. == Definition == One of the problems of scene understanding in computer vision is to determine horizontal freespace around the camera, where the agent can move, and the vertical obstacles delimiting it. An image can be paired with depth information (produced e.g. from stereo disparity, lidar, or monocular depth estimation), allowing a dense tridimensional reconstruction of the observed scene. One drawback of dense reconstruction is the large amount of data involved, since each pixel in the image is mapped to an element of a point cloud. Vision problems characterised by planar freespace delimited by mostly vertical obstacles, such as traffic scenes or robotic navigation, can benefit from a condensed representation that allows to save memory and processing time. Stixels are thin vertical rectangles representing a slice of a vertical surface belonging to the closest obstacle in the observed scene. They allow to dramatically reduce the amount of information needed to represent a scene in such problems. A stixel is characterised by three parameters: vertical coordinate of the bottom, height of the stick, and depth. Stixels have fixed width, with each stixel spanning over a certain number of image columns, allowing downsampling of the horizontal image resolution. In the original formulation, each column of the image would contain at most one stixel, and later extensions were developed to allow multiple stixels on each column, allowing to represent multiple objects at different distances. == Stixel estimation == The input to stixel estimation is a dense depth map, that can be computed from stereo disparity or other means. The original approach computes an occupancy grid that can be segmented to estimate the freespace, with dynamic programming providing an efficient method to find an optimal segmentation. Alternative approaches can be used instead of occupancy grid mapping, such as manifold-based methods. The freespace boundary provides the base points of the obstacles at closest longitudinal distance, however multiple objects at different distances might appear in each column of the image. To fully define the obstacles, their height should be estimated, and this is accomplished by segmenting the depth of the object from the depth of the background. A membership function over the pixels can be defined based on the depth value, where the membership represents the confidence of a pixel belonging to the closest vertical obstacle or to the background, and a cut separating the obstacles from the background can again be computed effectively with dynamic programming. Once both the freespace and the obstacle height are known, the stixels can be estimated by fusing the information over the columns spanned by each stixel, and finally a refined depth of the stixel can be estimated via model fitting over the depth of the pixels covered by the stixel, possibly paired with confidence information (e.g. disparity confidence produced by methods such as semi-global matching).

    Read more →
  • Terminology extraction

    Terminology extraction

    Terminology extraction (also known as term extraction, glossary extraction, term recognition, or terminology mining) is a subtask of information extraction. The goal of terminology extraction is to automatically extract relevant terms from a given corpus. In the semantic web era, a growing number of communities and networked enterprises started to access and interoperate through the internet. Modeling these communities and their information needs is important for several web applications, like topic-driven web crawlers, web services, recommender systems, etc. The development of terminology extraction is also essential to the language industry. One of the first steps to model a knowledge domain is to collect a vocabulary of domain-relevant terms, constituting the linguistic surface manifestation of domain concepts. Several methods to automatically extract technical terms from domain-specific document warehouses have been described in the literature. Typically, approaches to automatic term extraction make use of linguistic processors (part of speech tagging, phrase chunking) to extract terminological candidates, i.e. syntactically plausible terminological noun phrases. Noun phrases include compounds (e.g. "credit card"), adjective noun phrases (e.g. "local tourist information office"), and prepositional noun phrases (e.g. "board of directors"). In English, the first two (compounds and adjective noun phrases) are the most frequent. Terminological entries are then filtered from the candidate list using statistical and machine learning methods. Once filtered, because of their low ambiguity and high specificity, these terms are particularly useful for conceptualizing a knowledge domain or for supporting the creation of a domain ontology or a terminology base. Furthermore, terminology extraction is a very useful starting point for semantic similarity, knowledge management, human translation and machine translation, etc. == Bilingual terminology extraction == The methods for terminology extraction can be applied to parallel corpora. Combined with e.g. co-occurrence statistics, candidates for term translations can be obtained. Bilingual terminology can be extracted also from comparable corpora (corpora containing texts within the same text type, domain but not translations of documents between each other).

    Read more →
  • Information professional

    Information professional

    The term information professional or information specialist refers to professionals responsible for the collection, documentation, organization, storage, preservation, retrieval, and dissemination of printed and digital information. The service delivered to the client is known as an information service. The term "information professional" is a versatile one, used to describe similar and sometimes overlapping professions, such as librarians, archivists, information managers, information systems specialists, information scientists, records managers, and information consultants. However, terminology differs among sources and organisations. Information professionals are employed in a variety of private, public, and academic institutions, as well as independently. == Skills == Since the term information professional is broad, the skills required for this profession are also varied. A Gartner report in 2011 pointed out that "Professional roles focused on information management will be different to that of established IT roles. An 'information professional' will not be one type of role or skill set, but will in fact have a number of specializations". Thus, an information professional can possess a variety of different skills, depending on the sector in which the person is employed. Some essential cross-sector skills are: IT skills, such as word-processing and spreadsheets, digitisation skills, and conducting Internet searches, together with skills loan systems, databases, content management systems, and specially designed programmes and packages. Customer service. An information professional should have the ability to address the information needs of customers. Language proficiency. This is essential in order to manage the information at hand and deal with customer needs. Soft skills. These include skills such as negotiating, conflict resolution, and time management. Management training. An information professional should be familiar with notions such as strategic planning and project management. Moreover, an information professional should be skilled in planning and using relevant systems, in capturing and securing information, and in accessing it to deliver service whenever the information is required. == Associations == Most countries have a professional association who oversee the professional and academic standards of librarians and other information professionals. There are also international associations related to LIS (library and information science), the most prominent of which is the International Federation of Library Associations and Institutions (IFLA). In many countries, LIS courses are accredited by the relevant professional association, as the American Library Association (ALA) in the USA, the Chartered Institute of Library and Information Professionals (CILIP) in the UK, and the Australian Library and Information Association (ALIA) in Australia. == Qualifications == Educational institutions around the world offer academic degrees, or degrees on related subjects such as Archival Studies, Information Systems, Information Management, and Records Management. Some of the institutions offering information science education refer to themselves as an iSchool, such as the CiSAP (Consortium of iSchools Asia Pacific, founded 2006) in Asia and the iSchool Caucus in the USA. There are also online e-learning resources, some of which offer certification for information professionals. === Africa === Information development in Africa started later than in other continents, mainly due to a lack of internet access, expertise and resources to manage digital infrastructure, and "opportunities for capacity development and knowledge-sharing". Nowadays, academic degrees in information studies are available at many universities of African countries, such as the University of Pretoria (South Africa), University of Nairobi (Kenya), Makerere University (Uganda), University of Botswana (Botswana), and University of Nigeria (Nigeria). === Asia === LIS-related studies are available in more than 30 Asian countries. Some examples listed by iSchools Inc. are the University of Hong Kong, University of Tsukuba, Japan, Yonsei University, South Korea, National Taiwan University and Wuhan University, China. Centre of Library and Information Management Science (CLIMS) at Tata Institute of Social Science in Mumbai, India. In Southeast Asia, the Congress of Southeast Asian Librarians (CONSAL) connects librarians and libraries in more than 10 countries with resources, networking opportunities, and support for growing library systems. === Australasia === The Australian Library and Information Association (ALIA) as of 2021 lists six schools offering undergraduate and postgraduate accredited university courses for "Librarian and Information Specialists" on their website. In New Zealand, the Open Polytechnic of New Zealand and the Victoria University of Wellington offer undergraduate and postgraduate degree courses for information professionals. === Europe === The majority of European countries have universities, colleges, or schools which offer bachelor's degrees in LIS studies. Over 40 universities offer master's degrees in LIS-related fields, and many institutions, such as the Swedish School of Library and Information Science at the University of Borås (Sweden), the University of Barcelona (Spain), Loughborough University (UK), and Aberystwyth University (Wales, UK) also offer PhD degrees. === North America === Information studies and degrees are available at numerous academic institutions throughout the U.S. and Canada. U.S. professional associations, together with their European counterparts, have undertaken many educational initiatives and pioneered many advances in the field of Information studies, such as increased interdisciplinarity and more effective delivery of distance learning. The Association for Intelligent Information Management, based in Silver Spring, Maryland, offers a qualification called Certified Information Professional (CIP), earned upon passing an examination, with certification remaining valid for three years. === South America === There are many schools and colleges in Latin America, which offer courses in Library Science, Archival Studies, and Information Studies, however these subjects are taught completely separately.

    Read more →
  • Kullback–Leibler Upper Confidence Bound

    Kullback–Leibler Upper Confidence Bound

    In multi-armed bandit problems, KL-UCB (for Kullback–Leibler Upper Confidence Bound) is a UCB-type algorithm that is asymptotically optimal, in the sense that its regret matches the problem-dependent Lai-Robbins lower bound. == Multi-armed bandit problem == The Multi-armed bandit problem is a sequential game where one player has to choose at each turn between K {\displaystyle K} actions (arms). Behind every arm a {\displaystyle a} there is an unknown distribution ν a {\displaystyle \nu _{a}} that lies in a set D {\displaystyle {\mathcal {D}}} known by the player (for example, D {\displaystyle {\mathcal {D}}} can be the set of Gaussian distributions or Bernoulli distributions). At each turn t {\displaystyle t} the player chooses (pulls) an arm a t {\displaystyle a_{t}} , he then gets an observation X t {\displaystyle X_{t}} of the distribution ν a t {\displaystyle \nu _{a_{t}}} . === Regret minimization === The goal is to minimize the regret at time T {\displaystyle T} that is defined as R T := ∑ a = 1 K Δ a E [ N a ( T ) ] {\displaystyle R_{T}:=\sum _{a=1}^{K}\Delta _{a}\mathbb {E} [N_{a}(T)]} where μ a := E [ ν a ] {\displaystyle \mu _{a}:=\mathbb {E} [\nu _{a}]} is the mean of arm a {\displaystyle a} μ ∗ := max a μ a {\displaystyle \mu ^{}:=\max _{a}\mu _{a}} is the highest mean Δ a := μ ∗ − μ a {\displaystyle \Delta _{a}:=\mu ^{}-\mu _{a}} N a ( t ) {\displaystyle N_{a}(t)} is the number of pulls of arm a {\displaystyle a} up to turn t {\displaystyle t} The player has to find an algorithm that chooses at each turn t {\displaystyle t} which arm to pull based on the previous actions and observations ( a s , X s ) s < t {\displaystyle (a_{s},X_{s})_{s μ } {\displaystyle {\mathcal {K}}_{inf}(\nu ,\mu ,{\mathcal {D}}):=\inf \left\{\mathrm {KL} (\nu ,{\tilde {\nu }})\ |\ {\tilde {\nu }}\in {\mathcal {D}},\ \mathbb {E} [{\tilde {\nu }}]>\mu \right\}} K L {\displaystyle \mathrm {KL} } is the Kullback–Leibler divergence ν ^ a ( t ) {\displaystyle {\hat {\nu }}_{a}(t)} is the empirical distribution of arm a {\displaystyle a} at turn t {\displaystyle t} δ t {\displaystyle \delta _{t}} is a well-chosen sequence of positive numbers, often equal to ln ⁡ t + c ln ⁡ ln ⁡ t {\displaystyle \ln t+c\ln \ln t} with c > 0 {\displaystyle c>0} . Then we choose the arm a t {\displaystyle a_{t}} with the highest index: a t := arg ⁡ max a U a ( t ) {\displaystyle a_{t}:=\arg \max _{a}U_{a}(t)} We note that the algorithm does not require knowledge of T {\displaystyle T} . === Example === In the special case of Gaussian distribution with fixed variance σ 2 {\displaystyle \sigma ^{2}} , we have: U a ( t ) = μ ^ a ( t ) + 2 σ 2 δ t N a ( t ) {\displaystyle U_{a}(t)={\hat {\mu }}_{a}(t)+{\sqrt {\frac {2\sigma ^{2}\delta _{t}}{N_{a}(t)}}}} with μ ^ a ( t ) {\displaystyle {\hat {\mu }}_{a}(t)} being the empirical mean of arm a {\displaystyle a} at turn t {\displaystyle t} . === Pseudocode === The player gets the set D for each arm i do: n[i] ← 1; nu[i] ← None; d ← ln(K) for t from 1 to K do: select arm t observe reward r n[t] ← n[t] + 1 nu[t] ← update empirical distribution for t from K+1 to T do: for each arm i do: index[i] ← compute_index(n[i], nu[i], D, d) select arm a with highest index[a] observe reward r n[a] ← n[a] + 1 nu[a] ← update empirical distribution d ← ln(t+1) == Theoretical results == In the multi-armed bandit problem we have the Lai–Robbins asymptotic lower bound on regret. The algorithm KL-UCB matches this lower bound for one-dimensional exponential families with δ t := ln ⁡ t + 3 ln ⁡ ln ⁡ t {\displaystyle \delta _{t}:=\ln t+3\ln \ln t} and for distributions bounded in [ 0 , 1 ] {\displaystyle [0,1]} with δ t := ln ⁡ t + ln ⁡ ln ⁡ t {\displaystyle \delta _{t}:=\ln t+\ln \ln t} . === Lai–Robbins lower bound === In 1985 Lai and Robbins proved an asymptotic, problem-dependent lower bound on regret. It states that for every consistent algorithm on the set D {\displaystyle {\mathcal {D}}} — that is, an algorithm for which, for every ( ν 1 , … , ν K ) ∈ D K {\displaystyle (\nu _{1},\dots ,\nu _{K})\in {\mathcal {D}}^{K}} , the regret R T {\displaystyle R_{T}} is subpolynomial (i.e. R T = o T → + ∞ ( T α ) {\displaystyle R_{T}=o_{T\to +\infty }(T^{\alpha })} for all α > 0 {\displaystyle \alpha >0} ) — we have: R T ≥ ( ∑ a : μ a < μ ∗ Δ a K inf ( ν a , μ ∗ , D ) ) ln ⁡ T + o T → + ∞ ( ln ⁡ T ) . {\displaystyle R_{T}\geq \left(\sum _{a:\mu _{a}<\mu ^{}}{\frac {\Delta _{a}}{{\mathcal {K}}_{\inf }(\nu _{a},\mu ^{},{\mathcal {D}})}}\right)\ln T+o_{T\to +\infty }(\ln T).} This bound is asymptotic (as T → + ∞ {\displaystyle T\to +\infty } ) and gives a first-order lower bound of order ln ⁡ T {\displaystyle \ln T} with the optimal constant in front of it. === Regret bound for KL-UCB === The algorithm matches the Lai–Robbins lower bound for one-dimensional exponential-family distributions and for distributions bounded in [ 0 , 1 ] {\displaystyle [0,1]} . ==== One-dimensional exponential family ==== For D {\displaystyle {\mathcal {D}}} being the set of one-dimensional exponential families, with δ t := ln ⁡ t + 3 ln ⁡ ln ⁡ t {\displaystyle \delta _{t}:=\ln t+3\ln \ln t} we have the following upper bound on the regret of KL-UCB: R T ≤ ( ∑ a : μ a < μ ∗ Δ a K inf ( ν a , μ ∗ , D ) ) ln ⁡ T + O T ( ln ⁡ T ) . {\displaystyle R_{T}\leq \left(\sum _{a:\mu _{a}<\mu ^{}}{\frac {\Delta _{a}}{{\mathcal {K}}_{\inf }(\nu _{a},\mu ^{},{\mathcal {D}})}}\right)\ln T+O_{T}({\sqrt {\ln T}}).} ==== Bounded distributions in [0,1] ==== For D = P ( [ 0 , 1 ] ) {\displaystyle {\mathcal {D}}={\mathcal {P}}([0,1])} (the set of distributions supported on [ 0 , 1 ] {\displaystyle [0,1]} ), and for δ t := ln ⁡ t + ln ⁡ ln ⁡ t {\displaystyle \delta _{t}:=\ln t+\ln \ln t} , we have the following upper bound on the regret of KL-UCB: R T ≤ ( ∑ a : μ a < μ ∗ Δ a K inf ( ν a , μ ∗ , D ) ) ln ⁡ T + O T ( ( ln ⁡ T ) 4 / 5 ln ⁡ ln ⁡ T ) . {\displaystyle R_{T}\leq \left(\sum _{a:\mu _{a}<\mu ^{}}{\frac {\Delta _{a}}{{\mathcal {K}}_{\inf }(\nu _{a},\mu ^{},{\mathcal {D}})}}\right)\ln T+O_{T}{\big (}(\ln T)^{4/5}\ln \ln T{\big )}.} === Runtime === For D = P ( [ 0 , 1 ] ) {\displaystyle {\mathcal {D}}={\mathcal {P}}([0,1])} , the runtime needed per step and for an arm k {\displaystyle k} with n {\displaystyle n} observations is O ( n ( ln ⁡ n ) 2 ) {\displaystyle {\mathcal {O}}{\big (}n(\ln n)^{2}{\big )}} . This is higher than that of other optimal algorithms, such as NPTS with O ( n ) {\displaystyle {\mathcal {O}}(n)} . MED with O ( n ln ⁡ n ) {\displaystyle {\mathcal {O}}(n\ln n)} . and IMED with O ( n ln ⁡ n ) {\displaystyle {\mathcal {O}}(n\ln n)} . The high runtime of KL-UCB is due to a two-level optimisation: for each arm and candidate mean μ {\displaystyle \mu } , the algorithm evaluates K inf ( ν ^ a ( t ) , μ , D ) {\displaystyle {\mathcal {K}}_{\inf }({\hat {\nu }}_{a}(t),\mu ,{\mathcal {D}})} and then maximises μ {\displaystyle \mu } subject to N a ( t ) K inf ( ν ^ a ( t ) , μ , D ) ≤ δ t {\displaystyle N_{a}(t)\,{\mathcal {K}}_{\inf }({\hat {\nu }}_{a}(t),\mu ,{\mathcal {D}})\leq \delta _{t}} . For distributions bounded in [ 0 , 1 ] {\displaystyle [0,1]} the inner problem has no closed form and must be solved numerically, which increases the per-step cost.

    Read more →
  • Elastic cloud storage

    Elastic cloud storage

    An elastic cloud is a cloud computing offering that provides variable service levels based on changing needs. Elasticity is an attribute that can be applied to most cloud services. It states that the capacity and performance of any given cloud service can expand or contract according to a customer's requirements and that this can potentially be changed automatically as a consequence of some software-driven event or, at worst, can be reconfigured quickly by the customer's infrastructure management team. Elasticity has been described as one of the five main principles of cloud computing by Rosenburg and Mateos in The Cloud at Your Service - Manning 2011. == History == Cloud computing was first described by Gillet and Kapor in 1996; however, the first practical implementation was a consequence of a strategy to leverage Amazon's excess data center capacity. Amazon and other pioneers of the commercial use of this technology were primarily interested in providing a “public” cloud service, whereby they could offer customers the benefits of using the cloud, particularly the utility-based pricing model benefit. Other suppliers followed suit with a range of cloud-based models all offering elasticity as a core component, but these suppliers were only offering this service as an element of their public cloud service. Due to perceived weaknesses in security, or at least a lack of proven compliance, many organizations, particularly in the financial and public sectors, have been slow adopters of cloud technologies. These wary organizations can achieve some of the benefits of cloud computing by adopting private cloud technologies. An alternative form of the elastic cloud has been offered by vendors such as EMC and IBM, whereby the service is based around an enterprise's own infrastructure but still retains elements of elasticity and the potential to bill by consumption. == Description == Elasticity in cloud computing is the ability for the organization to adjust its storage requirements in terms of capacity and processing with respect to operational requirements. This has the following benefits: Operational Benefits - Services can be acquired quickly, meaning that the evolving requirements of the business can be addressed almost immediately, giving an organization a potential agility advantage. A properly implemented elastic system will provision/de-provision according to application demands, so if a particular business has activity spikes then the provision can be enabled to match the demand and the capacity can be re-allocated. Research and Development (R&D) Projects - R&D activities are no longer hindered by a requirement to secure a capex budget prior to a project starting. Capability can simply be provisioned from the cloud and released at the end of the exercise. Testing and Deployment - With most large-scale projects a size test needs to be performed prior to final rollout. By taking advantage of the elasticity of the cloud and creating a full-scale avatar of the proposed production system, realistic data and traffic volumes can be provisioned and released as needed. Expensive Resources Allocated - This will normally apply only in the context where a customer is applying at least some of their own servers as part of a cloud infrastructure, specifically where a business (for performance reasons) has decided to invest in solid-state storage as opposed to spinning platters. There are instances when, due to activity spikes, a less critical process may need to be moved from the high-performance resources to more traditional storage. Server Specification - When a customer has elected to own/lease hardware, they can select and specify servers that are specifically tuned to meet the likely needs of their operation (i.e., directly controlling the cost/benefit equation). Utility Based Payments - There is, of course, a key cost driver in this process, and the notion that you should pay for what you consume is acceptable for many organizations. When hardware capacity is sourced internally, organizations need to over-provision. This applies just as much to traditional outsourcing as it does to capex-related expenditure on in-house servers. Cloud Platform – At the heart of any cloud storage system is the ability to manage hyperscale object storage and a Hadoop Distributed Files System (HDFS). Elastic storage capability is particularly well suited to hyperscale and Hadoop environments, where its capability to rapidly respond to changing circumstances and priorities is essential

    Read more →
  • Artificial intelligence in industry

    Artificial intelligence in industry

    Industrial artificial intelligence, or industrial AI, refers to the application of artificial intelligence to industrial business processes. Unlike general artificial intelligence which is a frontier research discipline to build computerized systems that perform tasks requiring human intelligence, industrial AI is more concerned with the application of such technologies to address industrial pain-points for customer value creation, productivity improvement, cost reduction, site optimization, predictive analysis and insight discovery. Artificial intelligence and machine learning have become key enablers to leverage data in production in recent years due to a number of different factors: More affordable sensors and the automated process of data acquisition; More powerful computation capability of computers to perform more complex tasks at a faster speed with lower cost; Faster connectivity infrastructure and more accessible cloud services for data management and computing power outsourcing. == Categories == Possible applications of industrial AI and machine learning in the production domain can be divided into seven application areas: Market and trend analysis Machinery and equipment Intralogistics Production process Supply chain Building Product Each application area can be further divided into specific application scenarios that describe concrete AI/ML scenarios in production. While some application areas have a direct connection to production processes, others cover production adjacent fields like logistics or the factory building. An example from the application scenario Process Design & Innovation are collaborative robots. Collaborative robotic arms are able to learn the motion and path demonstrated by human operators and perform the same task. Predictive and preventive maintenance through data-driven machine learning are application scenarios from the Machinery & Equipment application area. == Challenges == In contrast to entirely virtual systems, in which ML applications are already widespread today, real-world production processes are characterized by the interaction between the virtual and the physical world. Data is recorded using sensors and processed on computational entities and, if desired, actions and decisions are translated back into the physical world via actuators or by human operators. This poses major challenges for the application of ML in production engineering systems. These challenges are attributable to the encounter of process, data and model characteristics: The production domain's high reliability requirements, high risk and loss potential, the multitude of heterogeneous data sources and the non-transparency of ML model functionality impede a faster adoption of ML in real-world production processes. In particular, production data comprises a variety of different modalities, semantics and quality. Furthermore, production systems are dynamic, uncertain and complex, and engineering and manufacturing problems are data-rich but information-sparse. Besides that, due to the variety of use cases and data characteristics, problem-specific data sets are required, which are difficult to acquire, hindering both practitioners and academic researchers in this domain. === Process and industry characteristics === The domain of production engineering can be considered as a rather conservative industry when it comes to the adoption of advanced technology and their integration into existing processes. This is due to high demands on reliability of the production systems resulting from the potentially high economic harm of reduced process effectiveness due to e.g., additional unplanned downtime or insufficient product qualities. In addition, the specifics of machining equipment and products prevent area-wide adoptions across a variety of processes. Besides the technical reasons, the reluctant adoption of ML is fueled by a lack of IT and data science expertise across the domain. === Data characteristics === The data collected in production processes mainly stem from frequently sampling sensors to estimate the state of a product, a process, or the environment in the real world. Sensor readings are susceptible to noise and represent only an estimate of the reality under uncertainty. Production data typically comprises multiple distributed data sources resulting in various data modalities (e.g., images from visual quality control systems, time-series sensor readings, or cross-sectional job and product information). The inconsistencies in data acquisition lead to low signal-to-noise ratios, low data quality and great effort in data integration, cleaning and management. In addition, as a result from mechanical and chemical wear of production equipment, process data is subject to various forms of data drifts. === Machine learning model characteristics === ML models are considered as black-box systems given their complexity and intransparency of input-output relation. This reduces the comprehensibility of the system behavior and thus also the acceptance by plant operators. Due to the lack of transparency and the stochasticity of these models, no deterministic proof of functional correctness can be achieved, complicating the certification of production equipment. Given their inherent unrestricted prediction behavior, ML models are vulnerable against erroneous or manipulated data, further risking the reliability of the production system because of lacking robustness and safety. In addition to high development and deployment costs, the data drifts cause high maintenance costs, which is disadvantageous compared to purely deterministic programs. == Standard processes for data science in production == The development of ML applications – starting with the identification and selection of the use case and ending with the deployment and maintenance of the application – follows dedicated phases that can be organized in standard process models. The process models assist in structuring the development process and defining requirements that must be met in each phase to enter the next phase. The standard processes can be classified into generic and domain-specific ones. Generic standard processes (e.g., CRISP-DM, ASUM-DM, or knowledge discovery in databases (KDD)) describe a generally valid methodology and are thus independent of individual domains. Domain-specific processes on the other hand consider specific peculiarities and challenges of special application areas. The Machine Learning Pipeline in Production is a domain-specific data science methodology that is inspired by the CRISP-DM model and was specifically designed to be applied in fields of engineering and production technology. To address the core challenges of ML in engineering – process, data, and model characteristics – the methodology especially focuses on use-case assessment, achieving a common data and process understanding data integration, data preprocessing of real-world production data and the deployment and certification of real-world ML applications. == Industrial data sources == The foundation of most artificial intelligence and machine learning applications in industrial settings are comprehensive datasets from the respective fields. Those datasets act as the basis for training the employed models. In other domains, like computer vision, speech recognition or language models, extensive reference datasets (e.g. ImageNet, Librispeech, The People's Speech) and data scraped from the open internet are frequently used for this purpose. Such datasets rarely exist in the industrial context because of high confidentiality requirements and high specificity of the data. Industrial applications of artificial intelligence are therefore often faced with the problem of data availability. For these reasons, existing open datasets applicable to industrial applications, often originate from public institutions like governmental agencies or universities and data analysis competitions hosted by companies. In addition to this, data sharing platforms exist. However, most of these platforms have no industrial focus and offer limited filtering abilities regarding industrial data sources.

    Read more →
  • Collective operation

    Collective operation

    Collective operations are building blocks for interaction patterns, that are often used in SPMD algorithms in the parallel programming context. Hence, there is an interest in efficient realizations of these operations. A realization of the collective operations is provided by the Message Passing Interface (MPI). == Definitions == In all asymptotic runtime functions, we denote the latency α {\displaystyle \alpha } (or startup time per message, independent of message size), the communication cost per word β {\displaystyle \beta } , the number of processing units p {\displaystyle p} and the input size per node n {\displaystyle n} . In cases where we have initial messages on more than one node we assume that all local messages are of the same size. To address individual processing units we use p i ∈ { p 0 , p 1 , … , p p − 1 } {\displaystyle p_{i}\in \{p_{0},p_{1},\dots ,p_{p-1}\}} . If we do not have an equal distribution, i.e. node p i {\displaystyle p_{i}} has a message of size n i {\displaystyle n_{i}} , we get an upper bound for the runtime by setting n = max ( n 0 , n 1 , … , n p − 1 ) {\displaystyle n=\max(n_{0},n_{1},\dots ,n_{p-1})} . A distributed memory model is assumed. The concepts are similar for the shared memory model. However, shared memory systems can provide hardware support for some operations like broadcast (§ Broadcast) for example, which allows convenient concurrent read. Thus, new algorithmic possibilities can become available. == Broadcast == The broadcast pattern is used to distribute data from one processing unit to all processing units, which is often needed in SPMD parallel programs to dispense input or global values. Broadcast can be interpreted as an inverse version of the reduce pattern (§ Reduce). Initially only root r {\displaystyle r} with i d {\displaystyle id} 0 {\displaystyle 0} stores message m {\displaystyle m} . During broadcast m {\displaystyle m} is sent to the remaining processing units, so that eventually m {\displaystyle m} is available to all processing units. Since an implementation by means of a sequential for-loop with p − 1 {\displaystyle p-1} iterations becomes a bottleneck, divide-and-conquer approaches are common. One possibility is to utilize a binomial tree structure with the requirement that p {\displaystyle p} has to be a power of two. When a processing unit is responsible for sending m {\displaystyle m} to processing units i . . j {\displaystyle i..j} , it sends m {\displaystyle m} to processing unit ⌈ ( i + j ) / 2 ⌉ {\displaystyle \left\lceil (i+j)/2\right\rceil } and delegates responsibility for the processing units ⌈ ( i + j ) / 2 ⌉ . . j {\displaystyle \left\lceil (i+j)/2\right\rceil ..j} to it, while its own responsibility is cut down to i . . ⌈ ( i + j ) / 2 ⌉ − 1 {\displaystyle i..\left\lceil (i+j)/2\right\rceil -1} . Binomial trees have a problem with long messages m {\displaystyle m} . The receiving unit of m {\displaystyle m} can only propagate the message to other units, after it received the whole message. In the meantime, the communication network is not utilized. Therefore pipelining on binary trees is used, where m {\displaystyle m} is split into an array of k {\displaystyle k} packets of size ⌈ n / k ⌉ {\displaystyle \left\lceil n/k\right\rceil } . The packets are then broadcast one after another, so that data is distributed fast in the communication network. Pipelined broadcast on balanced binary tree is possible in O ( α log ⁡ p + β n ) {\displaystyle {\mathcal {O}}(\alpha \log p+\beta n)} , whereas for the non-pipelined case it takes O ( ( α + β n ) log ⁡ p ) {\displaystyle {\mathcal {O}}((\alpha +\beta n)\log p)} cost. == Reduce == The reduce pattern is used to collect data or partial results from different processing units and to combine them into a global result by a chosen operator. Given p {\displaystyle p} processing units, message m i {\displaystyle m_{i}} is on processing unit p i {\displaystyle p_{i}} initially. All m i {\displaystyle m_{i}} are aggregated by ⊗ {\displaystyle \otimes } and the result is eventually stored on p 0 {\displaystyle p_{0}} . The reduction operator ⊗ {\displaystyle \otimes } must be associative at least. Some algorithms require a commutative operator with a neutral element. Operators like s u m {\displaystyle sum} , m i n {\displaystyle min} , m a x {\displaystyle max} are common. Implementation considerations are similar to broadcast (§ Broadcast). For pipelining on binary trees the message must be representable as a vector of smaller object for component-wise reduction. Pipelined reduce on a balanced binary tree is possible in O ( α log ⁡ p + β n ) {\displaystyle {\mathcal {O}}(\alpha \log p+\beta n)} . == All-Reduce == The all-reduce pattern (also called allreduce) is used if the result of a reduce operation (§ Reduce) must be distributed to all processing units. Given p {\displaystyle p} processing units, message m i {\displaystyle m_{i}} is on processing unit p i {\displaystyle p_{i}} initially. All m i {\displaystyle m_{i}} are aggregated by an operator ⊗ {\displaystyle \otimes } and the result is eventually stored on all p i {\displaystyle p_{i}} . Analog to the reduce operation, the operator ⊗ {\displaystyle \otimes } must be at least associative. All-reduce can be interpreted as a reduce operation with a subsequent broadcast (§ Broadcast). For long messages a corresponding implementation is suitable, whereas for short messages, the latency can be reduced by using a hypercube (Hypercube (communication pattern) § All-Gather/ All-Reduce) topology, if p {\displaystyle p} is a power of two. All-reduce can also be implemented with a butterfly algorithm and achieve optimal latency and bandwidth. All-reduce is possible in O ( α log ⁡ p + β n ) {\displaystyle {\mathcal {O}}(\alpha \log p+\beta n)} , since reduce and broadcast are possible in O ( α log ⁡ p + β n ) {\displaystyle {\mathcal {O}}(\alpha \log p+\beta n)} with pipelining on balanced binary trees. All-reduce implemented with a butterfly algorithm achieves the same asymptotic runtime. == Prefix-Sum/Scan == The prefix-sum or scan operation is used to collect data or partial results from different processing units and to compute intermediate results by an operator, which are stored on those processing units. It can be seen as a generalization of the reduce operation (§ Reduce). Given p {\displaystyle p} processing units, message m i {\displaystyle m_{i}} is on processing unit p i {\displaystyle p_{i}} . The operator ⊗ {\displaystyle \otimes } must be at least associative, whereas some algorithms require also a commutative operator and a neutral element. Common operators are s u m {\displaystyle sum} , m i n {\displaystyle min} and m a x {\displaystyle max} . Eventually processing unit p i {\displaystyle p_{i}} stores the prefix sum ⊗ i ′ <= i {\displaystyle \otimes _{i'<=i}} m i ′ {\displaystyle m_{i'}} . In the case of the so-called exclusive prefix sum, processing unit p i {\displaystyle p_{i}} stores the prefix sum ⊗ i ′ < i {\displaystyle \otimes _{i' Read more →

  • Umbrella review

    Umbrella review

    In medical research, an umbrella review is a review of systematic reviews or meta-analyses. They may also be called overviews of reviews, reviews of reviews, summaries of systematic reviews, or syntheses of reviews. Umbrella reviews are among the highest levels of evidence currently available in medicine. By summarizing information from multiple overview articles, umbrella reviews make it easier to review the evidence and allow for comparison of results between each of the individual reviews. Umbrella reviews may address a broader question than a typical review, such as discussing multiple different treatment comparisons instead of only one. They are especially useful for developing guidelines and clinical practice, and when comparing competing interventions.

    Read more →
  • Machine learning in video games

    Machine learning in video games

    Artificial intelligence and machine learning techniques are used in video games for a wide variety of applications such as non-player character (NPC) control, procedural content generation (PCG) and deep learning-based content generation. Machine learning is a subset of artificial intelligence that uses historical data to build predictive and analytical models. This is in sharp contrast to traditional methods of artificial intelligence such as search trees and expert systems. Information on machine learning techniques in the field of games is mostly known to public through research projects as most gaming companies choose not to publish specific information about their intellectual property. The most publicly known application of machine learning in games is likely the use of deep learning agents that compete with professional human players in complex strategy games. There has been a significant application of machine learning on games such as Atari/ALE, Doom, Minecraft, StarCraft, and car racing. Other games that did not originally exists as video games, such as chess and Go have also been affected by the machine learning. == Overview of relevant machine learning techniques == === Deep learning === Deep learning is a subset of machine learning which focuses heavily on the use of artificial neural networks (ANN) that learn to solve complex tasks. Deep learning uses multiple layers of ANN and other techniques to progressively extract information from an input. Due to this complex layered approach, deep learning models often require powerful machines to train and run on. ==== Convolutional neural networks ==== Convolutional neural networks (CNN) are specialized ANNs that are often used to analyze image data. These types of networks are able to learn translation invariant patterns, which are patterns that are not dependent on location. CNNs are able to learn these patterns in a hierarchy, meaning that earlier convolutional layers will learn smaller local patterns while later layers will learn larger patterns based on the previous patterns. A CNN's ability to learn visual data has made it a commonly used tool for deep learning in games. === Recurrent neural network === Recurrent neural networks are a type of ANN that are designed to process sequences of data in order, one part at a time rather than all at once. An RNN runs over each part of a sequence, using the current part of the sequence along with memory of previous parts of the current sequence to produce an output. These types of ANN are highly effective at tasks such as speech recognition and other problems that depend heavily on temporal order. There are several types of RNNs with different internal configurations; the basic implementation suffers from a lack of long term memory due to the vanishing gradient problem, thus it is rarely used over newer implementations. ==== Long short-term memory ==== A long short-term memory (LSTM) network is a specific implementation of a RNN that is designed to deal with the vanishing gradient problem seen in simple RNNs, which would lead to them gradually "forgetting" about previous parts of an inputted sequence when calculating the output of a current part. LSTMs solve this problem with the addition of an elaborate system that uses an additional input/output to keep track of long term data. LSTMs have achieved very strong results across various fields, and were used by several monumental deep learning agents in games. === Reinforcement learning === Reinforcement learning is the process of training an agent using rewards and/or punishments. The way an agent is rewarded or punished depends heavily on the problem; such as giving an agent a positive reward for winning a game or a negative one for losing. Reinforcement learning is used heavily in the field of machine learning and can be seen in methods such as Q-learning, policy search, Deep Q-networks and others. It has seen strong performance in both the field of games and robotics. === Neuroevolution === Neuroevolution involves the use of both neural networks and evolutionary algorithms. Instead of using gradient descent like most neural networks, neuroevolution models make use of evolutionary algorithms to update neurons in the network. Researchers claim that this process is less likely to get stuck in a local minimum and is potentially faster than state of the art deep learning techniques. == Deep learning agents == Machine learning agents have been used to take the place of a human player rather than function as NPCs, which are deliberately added into video games as part of designed gameplay. Deep learning agents have achieved impressive results when used in competition with both humans and other artificial intelligence agents. === Chess === Chess is a turn-based strategy game that is considered a difficult AI problem due to the computational complexity of its board space. Similar strategy games are often solved with some form of a Minimax Tree Search. These types of AI agents have been known to beat professional human players, such as the historic 1997 Deep Blue versus Garry Kasparov match. Since then, machine learning agents have shown ever greater success than previous AI agents. === Go === Go is another turn-based strategy game which is considered an even more difficult AI problem than chess. The state space of is Go is around 10^170 possible board states compared to the 10^120 board states for Chess. Prior to recent deep learning models, AI Go agents were only able to play at the level of a human amateur. ==== AlphaGo ==== Google's 2015 AlphaGo was the first AI agent to beat a professional Go player. AlphaGo used a deep learning model to train the weights of a Monte Carlo tree search (MCTS). The deep learning model consisted of 2 ANN, a policy network to predict the probabilities of potential moves by opponents, and a value network to predict the win chance of a given state. The deep learning model allows the agent to explore potential game states more efficiently than a vanilla MCTS. The network were initially trained on games of humans players and then were further trained by games against itself. ==== AlphaGo Zero ==== AlphaGo Zero, another implementation of AlphaGo, was able to train entirely by playing against itself. It was able to quickly train up to the capabilities of the previous agent. === StarCraft series === StarCraft and its sequel StarCraft II are real-time strategy (RTS) video games that have become popular environments for AI research. Blizzard and DeepMind have worked together to release a public StarCraft 2 environment for AI research to be done on. Various deep learning methods have been tested on both games, though most agents usually have trouble outperforming the default AI with cheats enabled or skilled players of the game. ==== Alphastar ==== Alphastar was the first AI agent to beat professional StarCraft 2 players without any in-game advantages. The deep learning network of the agent initially received input from a simplified zoomed out version of the gamestate, but was later updated to play using a camera like other human players. The developers have not publicly released the code or architecture of their model, but have listed several state of the art machine learning techniques such as relational deep reinforcement learning, long short-term memory, auto-regressive policy heads, pointer networks, and centralized value baseline. Alphastar was initially trained with supervised learning, it watched replays of many human games in order to learn basic strategies. It then trained against different versions of itself and was improved through reinforcement learning. The final version was hugely successful, but only trained to play on a specific map in a protoss mirror matchup. === Dota 2 === Dota 2 is a multiplayer online battle arena (MOBA) game. Like other complex games, traditional AI agents have not been able to compete on the same level as professional human player. The only widely published information on AI agents attempted on Dota 2 is OpenAI's deep learning Five agent. ==== OpenAI Five ==== OpenAI Five utilized separate long short-term memory networks to learn each hero. It trained using a reinforcement learning technique known as Proximal Policy Learning running on a system containing 256 GPUs and 128,000 CPU cores. Five trained for months, accumulating 180 years of game experience each day, before facing off with professional players. It was eventually able to beat the 2018 Dota 2 esports champion team in a 2019 series of games. === Planetary Annihilation === Planetary Annihilation is a real-time strategy game which focuses on massive scale war. The developers use ANNs in their default AI agent. === Supreme Commander 2 === Supreme Commander 2 is a real-time strategy (RTS) video game. The game uses Multilayer Perceptrons (MLPs) to control a platoon’s reaction to encountered enemy units. Total of four MLPs are used, one for each platoon type: land, naval

    Read more →
  • Enterprise information integration

    Enterprise information integration

    Enterprise information integration (EII) is the ability to support a unified view of data and information for an entire organization. The goal of EII is to get a large set of heterogeneous data sources to appear to a user or system as a single, homogeneous data source. In a data virtualization application of EII, there is a process of information integration, using data abstraction to provide a unified interface (known as uniform data access) for viewing all the data within an organization, and a single set of structures and naming conventions (known as uniform information representation) to represent this data. == Overview == Data within an enterprise can be stored in heterogeneous formats, including relational databases (which themselves come in a large number of varieties), text files, XML files, spreadsheets and a variety of proprietary storage methods, each with their own indexing and data access methods. Standardized data access APIs have emerged that offer a specific set of commands to retrieve and modify data from a generic data source. Many applications exist that implement these APIs' commands across various data sources, most notably relational databases. Such APIs include ODBC, JDBC, XQJ, OLE DB, and more recently ADO.NET. There are also standard formats for representing data within a file that are very important to information integration. The best-known of these is XML, which has emerged as a standard universal representation format. There are also more specific XML "grammars" defined for specific types of data such as Geography Markup Language for expressing geographical features and Directory Service Markup Language for holding directory-style information. In addition, non-XML standard formats exist such as iCalendar for representing calendar information and vCard for business card information. Enterprise Information Integration (EII) applies data integration commercially. Despite the theoretical problems described above, the private sector shows more concern with the problems of data integration as a viable product. EII emphasizes neither correctness nor tractability, but speed and simplicity. === Uniform data access === Uniform data access means connectivity and controllability across numerous target data sources. Necessary to fields such as EII and Electronic Data Interchange (EDI), it is most often used regarding analysis of disparate data types and data sources, which must be rendered into a uniform information representation, and generally must appear homogenous to the analysis tools—when the data being analyzed is typically heterogeneous and widely varying in size, type, and original representation. === Uniform information representation === Uniform information representation allows information from several realms or disciplines to be displayed and worked with as if it came from the same realm or discipline. It takes information from a number of sources, which may have used different methodologies and metrics in their data collection, and builds a single large collection of information, where some records may be more complete than others across all fields of data Uniform information representation is particularly important in EII and Electronic Data Interchange (EDI), where different departments of a large organization may have collected information for different purposes, with different labels and units, until one department realized that data already collected by those other departments could be re-purposed for their own needs—saving the enterprise the effort and cost of re-collecting the same information. === Combining disparate data sets === Each data source is disparate and as such is not designed to support EII. Therefore, data virtualization as well as data federation depends upon accidental data commonality to support combining data and information from disparate data sets. Because of this lack of data value commonality across data sources, the return set may be inaccurate, incomplete, and impossible to validate. One solution is to recast disparate databases to integrate these databases without the need for ETL. The recast databases support commonality constraints where referential integrity may be enforced between databases. The recast databases provide designed data access paths with data value commonality across databases. === Simplicity of deployment === Even if recognized as a solution to a problem, EII as of 2009 currently takes time to apply and offers complexities in deployment. Proposed schema-less solutions include "Lean Middleware". === Handling higher-order information === Analysts experience difficulty—even with a functioning information integration system—in determining whether the sources in the database will satisfy a given application. Answering these kinds of questions about a set of repositories requires semantic information like metadata and/or ontologies. == Applications == EII products enable loose coupling between homogeneous-data consuming client applications and services and heterogeneous-data stores. Such client applications and services include Desktop Productivity Tools (spreadsheets, word processors, presentation software, etc.), development environments and frameworks (Java EE, .NET, Mono, SOAP or RESTful Web services, etc.), business intelligence (BI), business activity monitoring (BAM) software, enterprise resource planning (ERP), Customer relationship management (CRM), business process management (BPM and/or BPEL) Software, and web content management (CMS). == Data access technologies == Service Data Objects (SDO) for Java, C++ and .Net clients and any type of data source XQuery and XQuery API for Java

    Read more →
  • Holographic algorithm

    Holographic algorithm

    In computer science, a holographic algorithm is an algorithm that uses a holographic reduction. A holographic reduction is a constant-time reduction that maps solution fragments many-to-many such that the sum of the solution fragments remains unchanged. These concepts were introduced by Leslie Valiant, who called them holographic because "their effect can be viewed as that of producing interference patterns among the solution fragments". The algorithms are unrelated to laser holography, except metaphorically. Their power comes from the mutual cancellation of many contributions to a sum, analogous to the interference patterns in a hologram. Holographic algorithms have been used to find polynomial-time solutions to problems without such previously known solutions for special cases of satisfiability, vertex cover, and other graph problems. They have received notable coverage due to speculation that they are relevant to the P versus NP problem and their impact on computational complexity theory. Although some of the general problems are #P-hard problems, the special cases solved are not themselves #P-hard, and thus do not prove FP = #P. Holographic algorithms have some similarities with quantum computation, but are completely classical. == Holant problems == Holographic algorithms exist in the context of Holant problems, which generalize counting constraint satisfaction problems (#CSP). A #CSP instance is a hypergraph G=(V,E) called the constraint graph. Each hyperedge represents a variable and each vertex v {\displaystyle v} is assigned a constraint f v . {\displaystyle f_{v}.} A vertex is connected to an hyperedge if the constraint on the vertex involves the variable on the hyperedge. The counting problem is to compute ∑ σ : E → { 0 , 1 } ∏ v ∈ V f v ( σ | E ( v ) ) , ( 1 ) {\displaystyle \sum _{\sigma :E\to \{0,1\}}\prod _{v\in V}f_{v}(\sigma |_{E(v)}),~~~~~~~~~~(1)} which is a sum over all variable assignments, the product of every constraint, where the inputs to the constraint f v {\displaystyle f_{v}} are the variables on the incident hyperedges of v {\displaystyle v} . A Holant problem is like a #CSP except the input must be a graph, not a hypergraph. Restricting the class of input graphs in this way is indeed a generalization. Given a #CSP instance, replace each hyperedge e of size s with a vertex v of degree s with edges incident to the vertices contained in e. The constraint on v is the equality function of arity s. This identifies all of the variables on the edges incident to v, which is the same effect as the single variable on the hyperedge e. In the context of Holant problems, the expression in (1) is called the Holant after a related exponential sum introduced by Valiant. == Holographic reduction == A standard technique in complexity theory is a many-one reduction, where an instance of one problem is reduced to an instance of another (hopefully simpler) problem. However, holographic reductions between two computational problems preserve the sum of solutions without necessarily preserving correspondences between solutions. For instance, the total number of solutions in both sets can be preserved, even though individual problems do not have matching solutions. The sum can also be weighted, rather than simply counting the number of solutions, using linear basis vectors. === General example === It is convenient to consider holographic reductions on bipartite graphs. A general graph can always be transformed it into a bipartite graph while preserving the Holant value. This is done by replacing each edge in the graph by a path of length 2, which is also known as the 2-stretch of the graph. To keep the same Holant value, each new vertex is assigned the binary equality constraint. Consider a bipartite graph G=(U,V,E) where the constraint assigned to every vertex u ∈ U {\displaystyle u\in U} is f u {\displaystyle f_{u}} and the constraint assigned to every vertex v ∈ V {\displaystyle v\in V} is f v {\displaystyle f_{v}} . Denote this counting problem by Holant ( G , f u , f v ) . {\displaystyle {\text{Holant}}(G,f_{u},f_{v}).} If the vertices in U are viewed as one large vertex of degree |E|, then the constraint of this vertex is the tensor product of f u {\displaystyle f_{u}} with itself |U| times, which is denoted by f u ⊗ | U | . {\displaystyle f_{u}^{\otimes |U|}.} Likewise, if the vertices in V are viewed as one large vertex of degree |E|, then the constraint of this vertex is f v ⊗ | V | . {\displaystyle f_{v}^{\otimes |V|}.} Let the constraint f u {\displaystyle f_{u}} be represented by its weighted truth table as a row vector and the constraint f v {\displaystyle f_{v}} be represented by its weighted truth table as a column vector. Then the Holant of this constraint graph is simply f u ⊗ | U | f v ⊗ | V | . {\displaystyle f_{u}^{\otimes |U|}f_{v}^{\otimes |V|}.} Now for any complex 2-by-2 invertible matrix T (the columns of which are the linear basis vectors mentioned above), there is a holographic reduction between Holant ( G , f u , f v ) {\displaystyle {\text{Holant}}(G,f_{u},f_{v})} and Holant ( G , f u T ⊗ ( deg ⁡ u ) , ( T − 1 ) ⊗ ( deg ⁡ v ) f v ) . {\displaystyle {\text{Holant}}(G,f_{u}T^{\otimes (\deg u)},(T^{-1})^{\otimes (\deg v)}f_{v}).} To see this, insert the identity matrix T ⊗ | E | ( T − 1 ) ⊗ | E | {\displaystyle T^{\otimes |E|}(T^{-1})^{\otimes |E|}} in between f u ⊗ | U | f v ⊗ | V | {\displaystyle f_{u}^{\otimes |U|}f_{v}^{\otimes |V|}} to get f u ⊗ | U | f v ⊗ | V | {\displaystyle f_{u}^{\otimes |U|}f_{v}^{\otimes |V|}} = f u ⊗ | U | T ⊗ | E | ( T − 1 ) ⊗ | E | f v ⊗ | V | {\displaystyle =f_{u}^{\otimes |U|}T^{\otimes |E|}(T^{-1})^{\otimes |E|}f_{v}^{\otimes |V|}} = ( f u T ⊗ ( deg ⁡ u ) ) ⊗ | U | ( f v ( T − 1 ) ⊗ ( deg ⁡ v ) ) ⊗ | V | . {\displaystyle =\left(f_{u}T^{\otimes (\deg u)}\right)^{\otimes |U|}\left(f_{v}(T^{-1})^{\otimes (\deg v)}\right)^{\otimes |V|}.} Thus, Holant ( G , f u , f v ) {\displaystyle {\text{Holant}}(G,f_{u},f_{v})} and Holant ( G , f u T ⊗ ( deg ⁡ u ) , ( T − 1 ) ⊗ ( deg ⁡ v ) f v ) {\displaystyle {\text{Holant}}(G,f_{u}T^{\otimes (\deg u)},(T^{-1})^{\otimes (\deg v)}f_{v})} have exactly the same Holant value for every constraint graph. They essentially define the same counting problem. === Specific examples === ==== Vertex covers and independent sets ==== Let G be a graph. There is a 1-to-1 correspondence between the vertex covers of G and the independent sets of G. For any set S of vertices of G, S is a vertex cover in G if and only if the complement of S is an independent set in G. Thus, the number of vertex covers in G is exactly the same as the number of independent sets in G. The equivalence of these two counting problems can also be proved using a holographic reduction. For simplicity, let G be a 3-regular graph. The 2-stretch of G gives a bipartite graph H=(U,V,E), where U corresponds to the edges in G and V corresponds to the vertices in G. The Holant problem that naturally corresponds to counting the number of vertex covers in G is Holant ( H , OR 2 , EQUAL 3 ) . {\displaystyle {\text{Holant}}(H,{\text{OR}}_{2},{\text{EQUAL}}_{3}).} The truth table of OR2 as a row vector is (0,1,1,1). The truth table of EQUAL3 as a column vector is ( 1 , 0 , 0 , 0 , 0 , 0 , 0 , 1 ) T = [ 1 0 ] ⊗ 3 + [ 0 1 ] ⊗ 3 {\displaystyle (1,0,0,0,0,0,0,1)^{T}={\begin{bmatrix}1\\0\end{bmatrix}}^{\otimes 3}+{\begin{bmatrix}0\\1\end{bmatrix}}^{\otimes 3}} . Then under a holographic transformation by [ 0 1 1 0 ] , {\displaystyle {\begin{bmatrix}0&1\\1&0\end{bmatrix}},} OR 2 ⊗ | U | EQUAL 3 ⊗ | V | {\displaystyle {\text{OR}}_{2}^{\otimes |U|}{\text{EQUAL}}_{3}^{\otimes |V|}} = ( 0 , 1 , 1 , 1 ) ⊗ | U | ( [ 1 0 ] ⊗ 3 + [ 0 1 ] ⊗ 3 ) ⊗ | V | {\displaystyle =(0,1,1,1)^{\otimes |U|}\left({\begin{bmatrix}1\\0\end{bmatrix}}^{\otimes 3}+{\begin{bmatrix}0\\1\end{bmatrix}}^{\otimes 3}\right)^{\otimes |V|}} = ( 0 , 1 , 1 , 1 ) ⊗ | U | [ 0 1 1 0 ] ⊗ | E | [ 0 1 1 0 ] ⊗ | E | ( [ 1 0 ] ⊗ 3 + [ 0 1 ] ⊗ 3 ) ⊗ | V | {\displaystyle =(0,1,1,1)^{\otimes |U|}{\begin{bmatrix}0&1\\1&0\end{bmatrix}}^{\otimes |E|}{\begin{bmatrix}0&1\\1&0\end{bmatrix}}^{\otimes |E|}\left({\begin{bmatrix}1\\0\end{bmatrix}}^{\otimes 3}+{\begin{bmatrix}0\\1\end{bmatrix}}^{\otimes 3}\right)^{\otimes |V|}} = ( ( 0 , 1 , 1 , 1 ) [ 0 1 1 0 ] ⊗ 2 ) ⊗ | U | ( ( [ 0 1 1 0 ] [ 1 0 ] ) ⊗ 3 + ( [ 0 1 1 0 ] [ 0 1 ] ) ⊗ 3 ) ⊗ | V | {\displaystyle =\left((0,1,1,1){\begin{bmatrix}0&1\\1&0\end{bmatrix}}^{\otimes 2}\right)^{\otimes |U|}\left(\left({\begin{bmatrix}0&1\\1&0\end{bmatrix}}{\begin{bmatrix}1\\0\end{bmatrix}}\right)^{\otimes 3}+\left({\begin{bmatrix}0&1\\1&0\end{bmatrix}}{\begin{bmatrix}0\\1\end{bmatrix}}\right)^{\otimes 3}\right)^{\otimes |V|}} = ( 1 , 1 , 1 , 0 ) ⊗ | U | ( [ 0 1 ] ⊗ 3 + [ 1 0 ] ⊗ 3 ) ⊗ | V | {\displaystyle =(1,1,1,0)^{\otimes |U|}\left({\begin{bmatrix}0\\1\end{bmatrix}}^{\otimes 3}+{\begin{bmatrix}1\\0\end{bmatrix}}^{\otimes 3}\right)^{\otimes |V|}} = NAND 2 ⊗ | U | EQUAL 3 ⊗ | V | , {\displaystyle ={\text{NAND}}_{2}^{\otim

    Read more →
  • NewSQL

    NewSQL

    NewSQL is a class of relational database management systems that seek to provide the scalability of NoSQL systems for online transaction processing (OLTP) workloads while maintaining the ACID guarantees of a traditional database system. Many enterprise systems that handle high-profile data (e.g., financial and order processing systems) are too large for conventional relational databases, but have transactional and consistency requirements that are not practical for NoSQL systems. The only options previously available for these organizations were to either purchase more powerful computers or to develop custom middleware that distributes requests over conventional DBMS. Both approaches feature high infrastructure costs and/or development costs. NewSQL systems attempt to reconcile the conflicts. == History == The term was first used by 451 Group analyst Matthew Aslett in a 2011 research paper discussing the rise of a new generation of database management systems. One of the first NewSQL systems was the H-Store parallel database system. == Applications == Typical applications are characterized by heavy OLTP transaction volumes. OLTP transactions; are short-lived (i.e., no user stalls) touch small amounts of data per transaction use indexed lookups (no table scans) have a small number of forms (a small number of queries with different arguments). However, some support hybrid transactional/analytical processing (HTAP) applications. Such systems improve performance and scalability by omitting heavyweight recovery or concurrency control. == List of NewSQL-databases == Apache Trafodion Clustrix CockroachDB Couchbase CrateDB Google Spanner MySQL Cluster NuoDB OceanBase Pivotal GemFire XD SequoiaDB SingleStore was formerly known as MemSQL. TIBCO Active Spaces TiDB TokuDB TransLattice Elastic Database VoltDB YDB YugabyteDB == Features == The two common distinguishing features of NewSQL database solutions are that they support online scalability of NoSQL databases and the relational data model (including ACID consistency) using SQL as their primary interface. NewSQL systems can be loosely grouped into three categories: === New architectures === NewSQL systems adopt various internal architectures. Some systems employ a cluster of shared-nothing nodes, in which each node manages a subset of the data. They include components such as distributed concurrency control, flow control, and distributed query processing. === SQL engines === The second category are optimized storage engines for SQL. These systems provide the same programming interface as SQL, but scale better than built-in engines. === Transparent sharding === These systems automatically split databases across multiple nodes using Raft or Paxos consensus algorithm.

    Read more →
  • Inbox by Gmail

    Inbox by Gmail

    Inbox by Gmail was an email service developed by Google. Announced on a limited invitation-only basis on October 22, 2014, it was officially released to the public on May 28, 2015. Inbox was shut down by Google on April 2, 2019. Available on the web, and through mobile apps for Android and iOS, Inbox by Gmail aimed to improve email productivity and organization through several key features. Bundles gathered emails on the same topic together; highlighted surface key details from messages, reminders and assists; and a "snooze" functionality enabled users to control when specific information would appear. Updates to the service enabled an "undo send" feature; a "Smart Reply" feature that automatically generated short reply examples for certain emails; integration with Google Calendar for event organization, previews of newsletters; and a "Save to Inbox" feature that let users save links for later use. Inbox by Gmail received generally positive reviews. At its launch, it was called "minimalist and lovely, full of layers and easy to navigate", with features deemed helpful in finding the right messages—one reviewer noted that the service felt "a lot like the future of email". However, it also received criticism, particularly for a low density of information, algorithms that needed tweaking, and because the service required users to "give up the control" of organizing their own email, meaning that "Anyone who already has a system for organizing their emails will likely find themselves fighting Google's system". Google noted in March 2016 that 10% of all replies on mobile originated from Inbox's Smart Reply feature. Google announced it would discontinue Inbox by Gmail in March 2019, with many of its features integrated into Gmail proper. == Features == Inbox by Gmail scanned the user's incoming Gmail messages for information. It gathered email messages related to the same overall topic into an organized bundle, with a title describing the bundle's content. For example, flight tickets, car rentals, and hotel reservations were grouped under "Travel", giving the user an easier overview of emails. Users could also group emails together manually, to "teach" the Inbox how the user worked. The service highlighted key details and important information in messages, such as flight itineraries, event information, photos and documents. Inbox could retrieve updated information from the Internet, including the real-time status of flights and package deliveries. Users could set reminders to bring up important messages later. When a user needed particular information, Inbox could assist the user by displaying the necessary details. Where Inbox highlights information was not needed immediately, users could "snooze" a message or reminder, with options to make the information reappear at a later time or specific location. In June 2015, Google added an "Undo Send" feature to Inbox, giving the user 10 seconds to undo sending a message. In November 2015, Google added "Smart Reply" functionality to the mobile apps. With Smart Reply, Inbox determined which emails could be answered with a short reply, generating three example responses from which the user could select one with a single tap. Smart Reply (initially available only on the Android and iOS mobile apps) was added to the Inbox website in March 2016, Google announcing that "10% of all your replies on mobile already use Smart Reply". By May 2017, Google said Smart Reply was driving about 12% of replies in inbox on mobile. In April 2016, Google updated Inbox with three new features; Google Calendar event organization, newsletter previews, and a "Save to Inbox" functionality that let the user save links for later use, rather than having to email links to themselves. In December 2017, Google introduced an "Unsubscribe" card that let users easily unsubscribe from mailing lists. The card appeared for email messages (from specific senders) that the user had not opened for a month. A few popular Inbox by Gmail features were subsequently added to Gmail: "Snoozing" of emails Nudges: Gmail could move old messages back to the top of the inbox when it thought a follow up or reply might be required. Hover actions: Placing the mouse cursor over a certain part of the message could quickly effect an action, such as archiving, without its being opened. Smart reply: This feature employed boilerplate text to suggest appropriate replies. Google reportedly wished, at a time then to be decided, to add the "bundles" feature to Gmail, which at the time was available only in Inbox for Gmail. By March 2020, many Inbox features were still missing from Gmail. == Platforms == Inbox by Gmail was announced on a limited invitation-only basis on October 22, 2014, available on the web, and through the Android and iOS mobile operating systems. It was officially released to the public on May 28, 2015. == Reception == David Pierce of The Verge praised the service, writing that it was "minimalist and lovely, full of layers and easy to navigate. It's remarkably fast and smooth on all platforms, and far better on iOS than the Gmail app". However, he criticized the app's low density of information, with only a few emails visible on the screen at a time, making it "a bit of a challenge" for users who need to go through "hundreds of emails" every day. Although positive that "Inbox feels a lot like the future of email", Pierce wrote that there was "plenty of algorithm tweaking and design condensing to do", with particular attention needed on a "compact view" for denser view of information on the screen. Sarah Mitroff of CNET also praised Inbox, writing, "Not only is it visually appealing, it's also full of features that help you find every message you need, when you need it". She added that users must "give up the control" to organize their email, and that it "won't vibe with everyone", but admitted that "if you're willing ... the app will reward you with a smarter and cleaner inbox." Mitroff noted that, initially, users had to coach the app about which bundle was appropriate for certain emails, writing, "It's a tedious process at first, by [sic] in just a few days Inbox starts to get it right." Regarding any downsides of the service, Mitroff wrote that "Inbox has a built-in strategy for managing your emails that works best on its own. Anyone who already has a system for organizing their emails will likely find themselves fighting Google's system". == Discontinuation and legacy == Google ended the service in March 2019. Google called Inbox "a great place to experiment with new ideas" and noted that many of those ideas had been migrated to Gmail. The company wanted, going forward, to focus its resources on a single email system. Several services, like Shortwave, attempted to resurrect some of the features of Inbox by Gmail to attract its old users. Similarly, Inbox Reborn, an actively maintained browser extension developed by a team of volunteer developers from around the world since 2018, aims to recreate the core features and visual style of Inbox by Gmail within the standard Gmail interface. The project continues to focus on preserving functionalities such as email bundling and streamlined workflows to provide users with a familiar productivity experience. Afterwards, most people moved to Spark, Spike, or Newton. According to a product manager at Google, a "more focused approach" regarding email was the companies goal. This is likely the reason they moved away from Inbox.

    Read more →
  • Subject (documents)

    Subject (documents)

    In library and information science documents (such as books, articles and pictures) are classified and searched by subject – as well as by other attributes such as author, genre and document type. This makes "subject" a fundamental term in this field. Library and information specialists assign subject labels to documents to make them findable. There are many ways to do this and in general there is not always consensus about which subject should be assigned to a given document. To optimize subject indexing and searching, we need to have a deeper understanding of what a subject is. The question: "what is to be understood by the statement 'document A belongs to subject category X'?" has been debated in the field for more than 100 years (see below) == Theoretical view == === Charles Ammi Cutter (1837–1903) === For Cutter the stability of subjects depends on a social process in which their meaning is stabilized in a name or a designation. A subject "referred [...] to those intellections [...] that had received a name that itself represented a distinct consensus in usage" (Miksa, 1983a, p. 60) and: the "systematic structure of established subjects" is "resident in the public realm" (Miksa, 1983a, p. 69); "[s]ubjects are by their very nature locations in a classificatory structure of publicly accumulated knowledge (Miksa, 1983a, p. 61). Bernd Frohmann adds: "The stability of the public realm in turn relies upon natural and objective mental structures which, with proper education, govern a natural progression from particular to general concepts. Since for Cutter, mind, society, and SKO [Systems of Knowledge Organization] stand one behind the other, each supporting each, all manifesting the same structure, his discursive construction of subjects invites connections with discourses of mind, education, and society. The Dewey Decimal Classification (DDC), by contrast, severs those connections. Melvil Dewey emphasized more than once that his system maps no structure beyond its own; there is neither a "transcendental deduction" of its categories nor any reference to Cutter's objective structure of social consensus. It is content-free: Dewey disdained any philosophical excogitation of the meaning of his class symbols, leaving the job of finding verbal equivalents to others. His innovation and the essence of the system lay in the notation. The DDC is a poorly semiotic system of expanding nests of ten digits, lacking any referent beyond itself. In it, a subject is wholly constituted in terms of its position in the system. The essential characteristic of a subject is a class symbol which refers only to other symbols. Its verbal equivalent is accidental, a merely pragmatic characteristic... .... The conflict of interpretations over "subjects" became explicit in the battles between "bibliography" (an approach to subjects having much in common with Cutter's) and Dewey's "close classification". William Fletcher spoke for the scholarly bibliographer.... Fletcher's "subjects", like Cutter's, referred to the categories of a fantasized, stable social order, whereas Dewey's subjects were elements of a semiological system of standardized, techno-bureaucratic administrative software for the library in its corporate, rather than high culture, incarnation". (Frohmann, 1994, 112–113). Cutter's early view on what a subject is, is probably wiser than most understandings that dominated the 20th century – and also the understanding reflected in the ISO-standard quoted below. The early statements quoted by Frohmann indicate that subjects are somehow shaped in social processes. When that is said, it should be added that they are not particularly detailed or clear. We only get a vague idea of the social nature of subjects. === S. R. Ranganathan (1892–1972) === A classification system with an explicit theoretical foundation is Ranganathan's Colon Classification. Ranganathan provided an explicit definition of the concept of "subject": Subject – an organized body of ideas, whose extension and intension are likely to fall coherently within the field of interests and comfortably within the intellectual competence and the field of inevitable specialization of a normal person. A related definition is given by one of Ranganathan's students: A subject is an organized and systematized body of ideas. It may consist of one idea or a combination of several... Ranganathan's definition of "subject" is strongly influenced by his Colon Classification system. The colon system is based on the combination of single elements from facets to subject designation. This is the reason why the combined nature of subjects are emphasized so strongly. It leads, however, to absurdities such as the claim that gold cannot be a subject (but is alternatively termed "an isolate"). This aspect of the theory has been criticized by Metcalfe (1973, p. 318). Metcalfe's skepticism regarding Ranganathan's theory is formulated in hard words (op. cit., p. 317): "This pseudo-science imposed itself on British disciples from about 1950 on...". It seems unacceptable that Ranganathan defines the word subject in a way that favors his own system. A scientific concept like "subject" should make it possible to compare different ways of establishing access to information. Whether or not subjects are combined or not should be examined once their definition has been given, it should not determined a priori, in the definition. Besides the emphasis on the combined, organizing and systematizing nature of subjects contains Ranganathan's definition of subject the pragmatic demand, that a subject should be determined in a way that suits a normal person's competency or specialization. Again we see a strange kind of wishful thinking mixing a general understanding of a concept with demands put by his own specific system. One thing is what the word subject means, quite another issue is how to provide subject descriptions that fulfill demands such as the specificity of a given information retrieval language which fulfill demands put on the system, such as precision and recall. If researchers too often define terms in ways that favor specific kinds of systems, that are such definitions not useful to provide more general theories about subjects, subject analysis and IR. Among other things are comparative studies of different kinds of systems made difficult. Based on these arguments, as well as additional arguments which have been used in the literature, we may conclude that Ranganathan's definition of the concept "subject" is not suited for scientific use. Like the definition of "subject" given by the ISO-standard for topic maps, may Ranganathan's definition be useful within his own closed system. The purpose of a scientific and scholarly field is, however, to examine the relative fruitfulness of systems such as topic maps and Colon Classification. For such purpose is another understanding of "subject" necessary. === Patrick Wilson (1927–2003) === In his book Wilson (1968) examined – in particular by thought experiments – the suitability of different methods of examining the subject of a document. The methods were: identifying the author's purpose for writing the document, weighing the relative dominance and subordination of different elements in the picture, which the reading imposes on the reader, grouping or count the document's use of concepts and references, construing a set of rules for selecting elements deemed necessary (as opposed to unnecessary) for the work as a whole. Patrick Wilson shows convincingly that each of these methods are insufficient to determine the subject of a document and is led to conclude ( p. 89): "The notion of the subject of a writing is indeterminate..." or, on p. 92 (about what users may expect to find using a particular position in a library classification system): "For nothing definite can be expected of the things found at any given position". In connection to the last quote has Wilson an interesting footnote in which he writes that authors of documents often use terms in ambiguous ways ("hostility" is used as an example). Even if the librarian could personally develop a very precise understanding of a concept, he would be unable to use it in his classification, because none of the documents use the term in the same precise way. Based on this argumentation is Wilson led to conclude: "If people write on what are for them ill-defined phenomena, a correct description of their subjects must reflect the ill-definedness". Wilson's concept of subject was discussed by Hjørland (1992) who found that it is problematic to give up the precise understanding of such a basic term in LIS. Wilson's arguments led him to an agnostic position which Hjørland found unacceptable and unnecessary. Concerning the authors' use of ambiguous terms, the role of the subject analysis is to determine which documents would be fruitful for users to identify whether or not the documents use one or another term or whether a given term i

    Read more →
  • Magic Quadrant

    Magic Quadrant

    Magic Quadrant (MQ) is a series of market research reports published by research and advisory firm Gartner that rely on proprietary qualitative data analysis methods to demonstrate market trends, such as direction, maturity, and participants. Their analyses are conducted for several specific technology industries and are updated every 1–2 years: once an updated report has been published, its predecessor is "retired". == Rating == Gartner rates vendors upon two criteria: completeness of vision and ability to execute. Completeness of vision – Reflects the vendor's innovation, and whether the vendor drives or follows the market. Ability to execute – Summarizes factors such as the vendor's financial viability, market responsiveness, product development, sales channels and customer base. The two component scores lead to a vendor position in one of four quadrants: === Leaders === Vendors in the "Leaders" quadrant have the highest composite scores for their completeness of vision and ability to execute. A vendor in the Leaders quadrant has the market share, credibility, and marketing & sales capabilities needed to drive the acceptance of new technologies. These vendors demonstrate a clear understanding of market needs, they are innovators and thought leaders, and they have well-articulated plans that customers and prospects can use when designing their infrastructures and strategies. In addition, they have a presence in the five major geographical regions, consistent financial performance, and broad platform support. === Challengers === Vendors in the "Challengers" quadrant have high scores mainly for their ability to execute. They both participate in the market and execute well enough to be a serious threat to vendors in the "Leaders" quadrant. They have strong products, as well as sufficiently credible market position and resources to sustain continued growth. Financial viability is not an issue for vendors in the "Challengers" quadrant, but they lack the size and influence of vendors in the "Leaders" quadrant due to their relative lack of vision. === Visionaries === Vendors in the "Visionaries" quadrant have high scores mainly for their completeness of vision. They deliver innovative products that address operationally or financially important end-user problems at a broad scale, but have not yet demonstrated the ability to capture market share or maintain sustainable levels of profitability. Visionary vendors are frequently privately held companies and acquisition targets for larger, established companies. The likelihood of acquisition often reduces the risks associated with installing their systems. === Niche Players === Vendors in the "Niche Players" quadrant have relatively low scores for both their ability to execute and their completeness of vision. They are often narrowly focused on specific market or vertical segments. This quadrant often also includes vendors that are adapting their existing products to enter the market under consideration, or larger vendors having difficulty developing and executing on their vision. == Gartner Critical Capabilities == Gartner Critical Capabilities complement Magic Quadrant analysis to offer deeper insight into the products and services offered by multiple vendors by a comparative analysis that scores competing products or services against a set of critical differentiators identified by Gartner. Gartner has periodically ended Magic Quadrant listings for IT Service Management, Web Content Management, and other industries as those markets have fully matured or other factors rendered the analytic framework inapplicable. == Criticism == The Magic Quadrant, and analysts in general, skew the market: according to research, by applying their methodologies to describe a market, they change that marketplace to fit their tools. Another criticism is that open source vendors are not considered sufficiently by analysts like Gartner, as has been published in an online discussion between a VP from Talend and a German Research VP from Gartner. On May 29, 2009 (2009-05-29), software vendor ZL Technologies filed a federal lawsuit against Gartner that challenged the "legitimacy" of Gartner's Magic Quadrant rating system. Gartner filed a motion to dismiss by claiming First Amendment protection since it contends that its MQ reports contain "pure opinion", which legally means opinions that are not based on fact. The court threw out the ZL case because it lacked a specific complaint. The decision was upheld on appeal.

    Read more →