AI Grammar Summarizer

AI Grammar Summarizer — independent reviews, comparisons, pricing and step-by-step guides on Aizhi.

• 

  Avizo (software)

  Avizo (pronounce: 'a-VEE-zo') is a general-purpose commercial software application for scientific and industrial data visualization and analysis. Avizo is developed by Thermo Fisher Scientific and was originally designed and developed by the Visualization and Data Analysis Group at Zuse Institute Berlin (ZIB) under the name Amira. Avizo was commercially released in November 2007. For the history of its development, see the Wikipedia article about Amira. == Overview == Avizo is a software application which enables users to perform interactive visualization and computation on 3D data sets. The Avizo interface is modelled on the visual programming. Users manipulate data and module components, organized in an interactive graph representation (called Pool), or in a Tree view. Data and modules can be interactively connected together, and controlled with several parameters, creating a visual processing network whose output is displayed in a 3D viewer. With this interface, complex data can be interactively explored and analyzed by applying a controlled sequence of computation and display processes resulting in a meaningful visual representation and associated derived data. == Application areas == Avizo has been designed to support different types of applications and workflows from 2D and 3D image data processing to simulations. It is a versatile and customizable visualization tool used in many fields: Scientific visualization Materials Research Tomography, Microscopy, etc. Nondestructive testing, Industrial Inspection, and Visual Inspection Computer-aided Engineering and simulation data post-processing Porous medium analysis Civil Engineering Seismic Exploration, Reservoir Engineering, Microseismic Monitoring, Borehole Imaging Geology, Digital Rock Physics (DRP), Earth Sciences Archaeology Food technology and agricultural science Physics, Chemistry Climatology, Oceanography, Environmental Studies Astrophysics == Features == Data import: 2D and 3D image stack and volume data: from microscopes (electron, optical), X-ray tomography (CT, micro-/nano-CT, synchrotron), neutron tomography and other acquisition devices (MRI, radiography, GPR) Geometric models (such as point sets, line sets, surfaces, grids) Numerical simulation data (such as Computational fluid dynamics or Finite element analysis data) Molecular data Time series and animations Seismic data Well logs 4D Multivariate Climate Models 2D/3D data visualization: Volume rendering Digital Volume Correlation Visualization of sections, through various slicing and clipping methods Isosurface rendering Polygonal meshes Scalar fields, Vector fields, Tensor representations, Flow visualization (Illuminated Streamlines, Stream Ribbons) Image processing: 2D/3D Alignment of image slices, Image registration Image filtering Mathematical Morphology (erode, dilate, open, close, tophat) Watershed Transform, Distance Transform Image segmentation 3D models reconstruction: Polygonal surface generation from segmented objects Generation of tetrahedral grids Surface reconstruction from point clouds Skeletonization (reconstruction of dendritic, porous or fracture network) Surface model simplification Quantification and analysis: Measurements and statistics Analysis spreadsheet and charting Material properties computation, based on 3D images: Absolute permeability Thermal conductivity Molecular diffusivity Electrical resistivity/formation factor 3D image-based meshing for CFD and FEA: From 3D imaging modalities (CT, micro-CT, MRI, etc.) Surface and volume meshes generation Export to FEA and CFD solvers for simulation Post-processing for simulation analysis Presentation, automation: MovieMaker, Multiscreen, Video wall, collaboration, and VR support TCL Scripting, C++ extension API Avizo is based on Open Inventor 3D graphics toolkits (FEI Visualization Sciences Group).

  Read more →
• 

  Torus interconnect

  A torus interconnect is a switch-less network topology for connecting processing nodes in a parallel computer system. == Introduction == In geometry, a torus is created by revolving a circle about an axis coplanar to the circle. While this is a general definition in geometry, the topological properties of this type of shape describes the network topology in its essence. === Geometry illustration === In the representations below, the first is a one dimension torus, a simple circle. The second is a two dimension torus, in the shape of a 'doughnut'. The animation illustrates how a two dimension torus is generated from a rectangle by connecting its two pairs of opposite edges. At one dimension, a torus topology is equivalent to a ring interconnect network, in the shape of a circle. At two dimensions, it becomes equivalent to a two dimension mesh, but with extra connection at the edge nodes. === Torus network topology === A torus interconnect is a switch-less topology that can be seen as a mesh interconnect with nodes arranged in a rectilinear array of N = 2, 3, or more dimensions, with processors connected to their nearest neighbors, and corresponding processors on opposite edges of the array connected.[1] In this lattice, each node has 2N connections. This topology is named for the lattice formed in this way, which is topologically homogeneous to an N-dimensional torus. == Visualization == The first 3 dimensions of torus network topology are easier to visualize and are described below: 1D Torus: one dimension, n nodes are connected in closed loop with each node connected to its two nearest neighbors. Communication can take place in two directions, +x and −x. A 1D Torus is the same as ring interconnection. 2D Torus: two dimensions with degree of four, the nodes are imagined laid out in a two-dimensional rectangular lattice of n rows and n columns, with each node connected to its four nearest neighbors, and corresponding nodes on opposite edges connected. Communication can take place in four directions, +x, −x, +y, and −y. The total nodes of a 2D Torus is n2. 3D Torus: three dimensions, the nodes are imagined in a three-dimensional lattice in the shape of a rectangular prism, with each node connected with its six neighbors, with corresponding nodes on opposing faces of the array connected. Each edge consists of n nodes. communication can take place in six directions, +x, −x, +y, −y, +z, −z. Each edge of a 3D Torus consist of n nodes. The total nodes of 3D Torus is n3. ND Torus: N dimensions, each node of an N dimension torus has 2N neighbors, Communication can take place in 2N directions. Each edge consists of n nodes. Total nodes of this torus is nN. The main motivation of having higher dimension of torus is to achieve higher bandwidth, lower latency, and higher scalability. Higher-dimensional arrays are difficult to visualize. The above ruleset shows that each higher dimension adds another pair of nearest neighbor connections to each node. == Performance == A number of supercomputers on the TOP500 list use three-dimensional torus networks, e.g. IBM's Blue Gene/L and Blue Gene/P, and the Cray XT3. IBM's Blue Gene/Q uses a five-dimensional torus network. Fujitsu's K computer and the PRIMEHPC FX10 use a proprietary three-dimensional torus 3D mesh interconnect called Tofu. === 3D Torus performance simulation === Sandeep Palur and Dr. Ioan Raicu from Illinois Institute of Technology conducted experiments to simulate 3D torus performance. Their experiments ran on a computer with 250GB RAM, 48 cores and x86_64 architecture. The simulator they used was ROSS (Rensselaer’s Optimistic Simulation System). They mainly focused on three aspects: Varying network size Varying number of servers Varying message size They concluded that throughput decreases with the increase of servers and network size. Otherwise, throughput increases with the increase of message size. === 6D Torus product performance === Fujitsu Limited developed a 6D torus computer model called "Tofu". In their model, a 6D torus can achieve 100 GB/s off-chip bandwidth, 12 times higher scalability than a 3D torus, and high fault tolerance. The model is used in the K computer and Fugaku. === Cost === While long wrap-around links may be the easiest way to visualize the connection topology, in practice, restrictions on cable lengths often make long wrap-around links impractical. Instead, directly connected nodes—including nodes that the above visualization places on opposite edges of a grid, connected by a long wrap-around link—are physically placed nearly adjacent to each other in a folded torus network. Every link in the folded torus network is very short—almost as short as the nearest-neighbor links in a simple grid interconnect—and therefore low-latency.

  Read more →
• 

  G.hn

  Gigabit Home Networking (G.hn) is a specification for wired home networking that supports speeds up to 2 Gbit/s and operates over four types of legacy wires: telephone wiring, coaxial cables, power lines and plastic optical fiber. Some benefits of a multi-wire standard are lower equipment development costs and lower deployment costs for service providers (by allowing customer self-install). == History == G.hn was developed under the International Telecommunication Union's Telecommunication Standardization sector (the ITU-T) and promoted by the HomeGrid Forum and several other organizations. ITU-T Recommendation (the ITU's term for standard) G.9960, which received approval on October 9, 2009, specified the physical layers and the architecture of G.hn. The Data Link Layer (Recommendation G.9961) was approved on June 11, 2010. Prominent organizations, including CEPca, HomePNA, and UPA, who were creators of some of these interfaces, rallied behind the latest version of the standard, emphasizing its potential and significance in the home networking domain. Moreover, the ITU-T extended the technology with multiple input, multiple output (MIMO) technology to increase data rates and signaling distance. This new feature was approved in March 2012 under G.9963 Recommendation. The development and promotion of G.hn have been significantly supported by the HomeGrid Forum and several other organizations. The technology was not only designed to address home-networking challenges but also found applications beyond this initial scope, showcasing its versatility and potential in the networking domain. == Technical specifications == === Technical overview === G.hn specifies a single physical layer based on fast Fourier transform (FFT) orthogonal frequency-division multiplexing (OFDM) modulation and low-density parity-check code (LDPC) forward error correction (FEC) code. G.hn includes the capability to notch specific frequency bands to avoid interference with amateur radio bands and other licensed radio services. G.hn includes mechanisms to avoid interference with legacy home networking technologies and also with other wireline systems such as VDSL2 or other types of DSL used to access the home. OFDM systems split the transmitted signal into multiple orthogonal sub-carriers. In G.hn each one of the sub-carriers is modulated using QAM. The maximum QAM constellation supported by G.hn is 4096-QAM (12-bit QAM). The G.hn media access control is based on a time division multiple access (TDMA) architecture, in which a "domain master" schedules Transmission Opportunities (TXOPs) that can be used by one or more devices in the "domain". There are two types of TXOPs: Contention-Free Transmission Opportunities (CFTXOP), which have a fixed duration and are allocated to a specific pair of transmitter and receiver. CFTXOP are used for implementing TDMA Channel Access for specific applications that require quality of service (QoS) guarantees. Shared Transmission Opportunities (STXOP), which are shared among multiple devices in the network. STXOP are divided into Time Slots (TS). There are two types of TS: Contention-Free Time Slots (CFTS), which are used for implementing "implicit" token passing Channel Access. In G.hn, a series of consecutive CFTS is allocated to a number of devices. The allocation is performed by the "domain master" and broadcast to all nodes in the network. There are pre-defined rules that specify which device can transmit after another device has finished using the channel. As all devices know "who is next", there is no need to explicitly send a "token" between devices. The process of "passing the token" is implicit and ensures that there are no collisions during Channel access. Contention-Based Time Slots (CBTS), which are used for implementing CSMA/CARP Channel Access. In general, CSMA systems cannot completely avoid collisions, so CBTS are only useful for applications that do not have strict Quality of Service requirements. ==== Optimization for each medium ==== Although most elements of G.hn are common for all three media supported by the standard (power lines, phone lines and coaxial cable), G.hn includes media-specific optimizations for each media. Some of these media-specific parameters include: OFDM Carrier Spacing: 195.31 kHz in coaxial, 48.82 kHz in phone lines, 24.41 kHz in power lines. FEC Rates: G.hn's FEC can operate with code rates 1/2, 2/3, 5/6, 16/18 and 20/21. Although these rates are not media specific, it is expected that the higher code rates will be used in cleaner media (such as coaxial) while the lower code rates will be used in noisy environments such as power lines. Automatic repeat request (ARQ) mechanisms: G.hn supports operation both with and without ARQ (re-transmission). Although this is not media specific, it is expected that ARQ-less operation is sometimes appropriate for cleaner media (such as coaxial) while ARQ operation is appropriate for noisy environments such as power lines. Power levels and frequency bands: G.hn defines different power masks for each medium. MIMO support: Recommendation G.9963 includes provisions for transmitting G.hn signals over multiple AC wires (phase, neutral, ground), if they are physically available. In July 2016, G.9963 was updated to include MIMO support over twisted pairs. ==== Security ==== G.hn uses the Advanced Encryption Standard (AES) encryption algorithm (with a 128-bit key length) using the CCMP protocol to ensure confidentiality and message integrity. Authentication and key exchange is done following ITU-T Recommendation X.1035. G.hn specifies point-to-point security inside a domain, which means that each pair of transmitter and receiver uses a unique encryption key which is not shared by other devices in the same domain. For example, if node Alice sends data to node Bob, node Eve (in the same domain as Alice and Bob) will not be able to easily eavesdrop their communication. G.hn supports the concept of relays, in which one device can receive a message from one node and deliver it to another node farther away in the same domain. Relaying becomes critical for applications with complex network topologies that need to cover large distances, such as those found in industrial or utility applications. While a relay can read the source and target addresses, it cannot read the message's content due to its body being end-to-end-encrypted. ==== Profiles ==== The G.hn architecture includes the concept of profiles. Profiles are intended to address G.hn nodes with significantly different levels of complexity. In G.hn the higher complexity profiles are proper supersets of lower complexity profiles, so that devices based on different profiles can interoperate with each other. Examples of G.hn devices based on high complexity profiles are Residential Gateways or Set-Top Boxes. Examples of G.hn devices based on low complexity profiles are home automation, home security and smart grid devices. ==== Technical parameters ==== The chart depicts a summary of the crucial technical specifications of the G.hn standard. Many of these technical elements are consistent across different physical media, with variations seen in areas such as Tone Spacing and frequency ranges. This uniformity is essential as it allows silicon manufacturers to produce a singular chip capable of implementing all three media types, leading to cost savings. Presently, G.hn chipsets are compatible with all three media types. This compatibility allows system manufacturers to create devices that can adjust to any wiring type simply by modifying a software configuration in the equipment. === Spectrum === The G.hn spectrum depends on the medium as shown in the diagram below: === Protocol stack === G.hn specifies the physical layer and the data link layer, according to the OSI model. The G.hn Data Link Layer (Recommendation G.9961) is divided into three sub-layers: The Application Protocol Convergence (APC) Layer, which accepts frames (usually in Ethernet format) from the upper layer (Application Entity) and encapsulates them into G.hn APC protocol data units (APDUs). The maximum payload of each APDU is 214 bytes. The logical link control (LLC), which is responsible for encryption, aggregation, segmentation and automatic repeat-request. This sub-layer is also responsible for "relaying" of APDUs between nodes that may not be able to communicate through a direct connection. The medium access control (MAC), which schedules channel access. The G.hn physical layer (Recommendation G.9960) is divided into three sub-layers: The Physical Coding Sub-layer (PCS), responsible for generating PHY headers. The Physical Medium Attachment (PMA), responsible for scrambling and forward error correction coding/decoding. The Physical Medium Dependent (PMD), responsible for bit-loading and OFDM modulation. The interface between the Application Entity and the Data Link Layer is called A-interface. The interface between the Data Link Layer and the ph

  Read more →
• 

  Cryptographic bill of materials

  Cryptographic bill of materials (CBOM—also cryptography bill of materials) is a structured inventory of all cryptographic assets present in a software, firmware, device, or system. It enumerates algorithms (and parameters such as key sizes and modes), cryptographic libraries or modules, digital certificates, keys and related material, and protocols in use, and maps their relationships to the components that implement or invoke them. CBOMs are used to improve security analysis, compliance, and cryptographic agility, and are increasingly referenced in guidance for post‑quantum cryptography (PQC) migration. == Definition and scope == A CBOM inventories cryptographic primitives and materials—such as encryption and signature algorithms (with specific variants and modes), key sizes, cryptographic libraries/modules, digital certificates (e.g., X.509), keys and other related cryptographic material, and security protocols (e.g., TLS, IPsec). It also documents dependencies (for example, an application uses an algorithm provided by a library; a protocol uses several algorithms) and can capture certificate lifecycles, cryptographic module certifications (e.g., FIPS 140‑3), and policy conformance metadata. In common practice, a CBOM may be embedded within an SBOM format (such as CycloneDX) or exported as a separate, linked artifact. === Typical CBOM fields === The exact schema varies by implementation, but common fields are summarized below (see CycloneDX CBOM guide and NIST SP 1800‑38B). == Relation to SBOM == A CBOM is complementary to, but distinct from, a software bill of materials (SBOM). Whereas an SBOM lists software components and their versions, a CBOM focuses specifically on the cryptography present and how it is configured and used. For example, an SBOM might enumerate inclusion of a library such as OpenSSL, while the CBOM would identify which algorithms and parameters that library enables (e.g., RSA‑2048, ECDH P‑256, AES‑GCM) and list relevant keys and certificates. The pairing enables both supply‑chain transparency and cryptographic transparency. == History == The term and practice emerged in the early–mid 2020s alongside software‑supply‑chain transparency and PQC planning. The OWASP CycloneDX standard introduced native CBOM support (v1.6 and later), modeling algorithms, keys, certificates, and protocols as first‑class “cryptographic assets” and providing dependency semantics (uses/implements) between software and cryptography. Open tooling from industry and researchers (e.g., IBM's CBOMkit and related generators/viewers) appeared to automate discovery and representation of cryptographic use in the CycloneDX CBOM schema. == Regulatory and policy context == In the United States, policy has emphasized cryptographic inventories as a prerequisite to PQC migration. The White House's National Security Memorandum 10 (2022) directed a government‑wide transition to quantum‑resistant cryptography; the Office of Management and Budget's M‑23‑02 (November 2022) operationalized this by requiring agencies to submit a prioritized inventory of cryptographic systems (with algorithm and key details) by 4 May 2023 and annually thereafter, and tasked CISA/NSA/NIST to develop automated discovery and inventory strategies. A 2024 Office of the National Cyber Director report reiterated that a “comprehensive cryptographic inventory” is the baseline for PQC planning and must be maintained iteratively with both automated and manual discovery. NIST's NCCoE practice guide (SP 1800‑38B, preliminary draft) provides concrete methods for cryptographic discovery and documentation across enterprises, aligning with CBOM‑style representations. CISA later published a strategy to migrate federal agencies to automated cryptography discovery and inventory tools to support continuous reporting. Separately, NSA, CISA, and NIST issued joint guidance encouraging all organisations to prepare cryptographic inventories and roadmaps for PQC, beyond government environments. == Role in quantum readiness and cryptographic agility == Because large‑scale quantum computing threatens widely used public‑key algorithms (e.g., RSA, ECC), organisations are planning multi‑year transitions to post-quantum cryptography. CBOMs enable that planning by identifying where quantum‑vulnerable algorithms appear, prioritising high‑impact systems, and tracking replacements over time. A machine‑readable CBOM also supports cryptographic agility and incident response: if an algorithm, library, or certificate lifecycle becomes non‑compliant or vulnerable, the CBOM indicates which products and systems are affected and where mitigations must be applied first. == Standards and tooling == CycloneDX (OWASP): Native CBOM modelling (v1.6+) for algorithms, certificates, keys/related material, and protocols, with dependency semantics and examples. The project publishes a CBOM guide and use‑case profiles (e.g., certificate and algorithm inventories). NIST NCCoE SP 1800‑38 series: Practice guides for PQC migration include enterprise cryptographic discovery methods that produce CBOM‑like inventories and integrate multiple discovery tools. Government automation initiatives: Following M‑23‑02, CISA issued a strategy to migrate to automated cryptography discovery and inventory tools to support agency reporting and continuous inventory management. Open‑source and vendor tools: IBM's CBOMkit and related components generate, analyse, and visualise CBOMs; the IBM CBOM specification work was upstreamed into CycloneDX 1.6. === Data model and interchange (example) === CycloneDX provides machine‑readable encodings (JSON/XML) for CBOM content. The example below (subset) shows an application depending on a crypto library that provides the AES‑256‑GCM algorithm, and the application also depends on a leaf X.509 certificate. See the CycloneDX CBOM guide, JSON reference, and the “Implementation details” use‑case for the semantics of `dependsOn` and `provides`. == Relationship to cybersecurity supply chain initiatives == CBOMs complement SBOM‑focused supply‑chain transparency introduced by U.S. Executive Order 14028 and NTIA/NIST SBOM work. SBOMs document software components; CBOMs add detail on embedded cryptography to support risk management, policy compliance (e.g., disallowing deprecated algorithms), and PQC transition planning.

  Read more →
• 

  Vision transformer

  A vision transformer (ViT) is a transformer designed for computer vision. A ViT decomposes an input image into a series of patches (rather than text into tokens), serializes each patch into a vector, and maps it to a smaller dimension with a single matrix multiplication. These vector embeddings are then processed by a transformer encoder as if they were token embeddings. ViTs were designed as alternatives to convolutional neural networks (CNNs) in computer vision applications. They have different inductive biases, training stability, and data efficiency. Compared to CNNs, ViTs are less data efficient, but have higher capacity. Some of the largest modern computer vision models are ViTs, such as one with 22B parameters. Subsequent to its publication, many variants were proposed, with hybrid architectures with both features of ViTs and CNNs. ViTs have found application in image recognition, image segmentation, weather prediction, and autonomous driving. == History == Transformers were introduced in Attention Is All You Need (2017), and have found widespread use in natural language processing. A 2019 paper applied ideas from the Transformer to computer vision. Specifically, they started with a ResNet, a standard convolutional neural network used for computer vision, and replaced all convolutional kernels by the self-attention mechanism found in a Transformer. It resulted in superior performance. However, it is not a Vision Transformer. In 2020, an encoder-only Transformer was adapted for computer vision, yielding the ViT, which reached state of the art in image classification, overcoming the previous dominance of CNN. The masked autoencoder (2022) extended ViT to work with unsupervised training. The vision transformer and the masked autoencoder, in turn, stimulated new developments in convolutional neural networks. Subsequently, there was cross-fertilization between the previous CNN approach and the ViT approach. In 2021, some important variants of the Vision Transformers were proposed. These variants are mainly intended to be more efficient, more accurate or better suited to a specific domain. Two studies improved efficiency and robustness of ViT by adding a CNN as a preprocessor. The Swin Transformer achieved state-of-the-art results on some object detection datasets such as COCO, by using convolution-like sliding windows of attention mechanism, and the pyramid process in classical computer vision. == Overview == The basic architecture, used by the original 2020 paper, is as follows. In summary, it is a BERT-like encoder-only Transformer. The input image is of type R H × W × C {\displaystyle \mathbb {R} ^{H\times W\times C}} , where H , W , C {\displaystyle H,W,C} are height, width, channel (RGB). It is then split into square-shaped patches of type R P × P × C {\displaystyle \mathbb {R} ^{P\times P\times C}} . For each patch, the patch is pushed through a linear operator, to obtain a vector ("patch embedding"). The position of the patch is also transformed into a vector by "position encoding" (the paper tried no embedding, 1D embedding, 2D embedding, and relative embedding: 1D was adopted). The two vectors are added, then pushed through several Transformer encoders. The attention mechanism in a ViT repeatedly transforms representation vectors of image patches, incorporating more and more semantic relations between image patches in an image. This is analogous to how in natural language processing, as representation vectors flow through a transformer, they incorporate more and more semantic relations between words, from syntax to semantics. The above architecture turns an image into a sequence of vector representations. To use these for downstream applications, an additional head needs to be trained to interpret them. For example, to use it for classification, one can add a shallow MLP on top of it that outputs a probability distribution over classes. The original paper uses a linear-GeLU-linear-softmax network. == Variants == === Original ViT === The original ViT was an encoder-only Transformer supervise-trained to predict the image label from the patches of the image. As in the case of BERT, it uses a special token in the input side, and the corresponding output vector is used as the only input of the final output MLP head. The special token is an architectural hack to allow the model to compress all information relevant for predicting the image label into one vector. Transformers found their initial applications in natural language processing tasks, as demonstrated by language models such as BERT and GPT-3. By contrast the typical image processing system uses a convolutional neural network (CNN). Well-known projects include Xception, ResNet, EfficientNet, DenseNet, and Inception. Transformers measure the relationships between pairs of input tokens (words in the case of text strings), termed attention. The cost is quadratic in the number of tokens. For images, the basic unit of analysis is the pixel. However, computing relationships for every pixel pair in a typical image is prohibitive in terms of memory and computation. Instead, ViT computes relationships among pixels in various small sections of the image (e.g., 16x16 pixels), at a drastically reduced cost. The sections (with positional embeddings) are placed in a sequence. The embeddings are learnable vectors. Each section is arranged into a linear sequence and multiplied by the embedding matrix. The result, with the position embedding is fed to the transformer. === Architectural improvements === ==== Pooling ==== After the ViT processes an image, it produces some embedding vectors. These must be converted to a single class probability prediction by some kind of network. In the original ViT and Masked Autoencoder, they used a dummy [CLS] token, in emulation of the BERT language model. The output at [CLS] is the classification token, which is then processed by a LayerNorm-feedforward-softmax module into a probability distribution. Global average pooling (GAP) does not use the dummy token, but simply takes the average of all output tokens as the classification token. It was mentioned in the original ViT as being equally good. Multihead attention pooling (MAP) applies a multiheaded attention block to pooling. Specifically, it takes as input a list of vectors x 1 , x 2 , … , x n {\displaystyle x_{1},x_{2},\dots ,x_{n}} , which might be thought of as the output vectors of a layer of a ViT. The output from MAP is M u l t i h e a d e d A t t e n t i o n ( Q , V , V ) {\displaystyle \mathrm {MultiheadedAttention} (Q,V,V)} , where q {\displaystyle q} is a trainable query vector, and V {\displaystyle V} is the matrix with rows being x 1 , x 2 , … , x n {\displaystyle x_{1},x_{2},\dots ,x_{n}} . This was first proposed in the Set Transformer architecture. Later papers demonstrated that GAP and MAP both perform better than BERT-like pooling. A variant of MAP was proposed as class attention, which applies MAP, then feedforward, then MAP again. Re-attention was proposed to allow training deep ViT. It changes the multiheaded attention module. === Masked Autoencoder === The Masked Autoencoder took inspiration from denoising autoencoders and context encoders. It has two ViTs put end-to-end. The first one ("encoder") takes in image patches with positional encoding, and outputs vectors representing each patch. The second one (called "decoder", even though it is still an encoder-only Transformer) takes in vectors with positional encoding and outputs image patches again. ==== Training ==== During training, input images (224px x 224 px in the original implementation) are split along a designated number of lines on each axis, producing image patches. A certain percentage of patches are selected to be masked out by mask tokens, while all others are retained in the image. The network is tasked with reconstructing the image from the remaining unmasked patches. Mask tokens in the original implementation are learnable vector quantities. A linear projection with positional embeddings is then applied to the vector of unmasked patches. Experiments varying mask ratio on networks trained on the ImageNet-1K dataset found 75% mask ratios achieved high performance on both finetuning and linear-probing of the encoder's latent space. The MAE processes only unmasked patches during training, increasing the efficiency of data processing in the encoder and lowering the memory usage of the transformer. A less computationally-intensive ViT is used for the decoder in the original implementation of the MAE. Masked patches are added back to the output of the encoder block as mask tokens and both are fed into the decoder. A reconstruction loss is computed for the masked patches to assess network performance. ==== Prediction ==== In prediction, the decoder architecture is discarded entirely. The input image is split into patches by the same algorithm as in training, but no patches are masked out. A linear projection wi

  Read more →
• 

  Data definition specification

  In computing, a data definition specification (DDS) is a guideline to ensure comprehensive and consistent data definition. It represents the attributes required to quantify data definition. A comprehensive data definition specification encompasses enterprise data, the hierarchy of data management, prescribed guidance enforcement and criteria to determine compliance. == Overview == A data definition specification may be developed for any organization or specialized field, improving the quality of its products through consistency and transparency. It eliminates redundancy (since all contributing areas are referencing the same specification) and provides standardization and degrees of compliance, making it easier and more efficient to create, modify, verify, analyze and share information across the enterprise. To understand how a data definition specification works in an enterprise, we must look at the elements of a DDS. Writing data definitions, defining business terms (or rules) in the context of a particular environment, provides structure for an organization's data architecture. In developing these definitions, the words used must be traceable to clearly defined data. A data definition specification may be used in the following activities: Business intelligence Business process modeling Business rules management Data analysis and modeling Information architecture Metadata modeling Data mastering Report generation == Criteria == A data definition specification requires data definitions to be: Atomic – singular, describing only one concept. Commonly used and ambiguous terms should be defined. While a term refers to one concept, several words may be used in a term: File – A concept identifiable with one word File extension – A concept identifiable with more than one word Traceable – Mapped to a specific data element. In business, a term may be traced to an entity (for example, a customer) or an attribute (such as a customer's name). A term may be a value in a data set (such as gender), or designate the data set itself. Traceability indicates relationships in the data hierarchy. Consistent - Used in a standard syntax; if used in a specific context, the context is noted Accurate - Precise, correct and unambiguous, stating what the term is and is not Clear - Readily understood by the reader Complete - With the term, its description and contextual references Concise - To avoid circular references == Applications == === Enterprise data === A data definition specification was produced by the Open Mobile Alliance to document charging data. The document, the centralized catalog of data elements defined for interfaces, specifies the mapping of these data elements to protocol fields in the interfaces. Created for the exchange of financial data, Market Data Definition Language (MDDL) is an XML specification designed to enable the interchange of information necessary to account, to analyze, and to trade financial instruments of the world's markets. It defines an XML-based interchange format and common data dictionary on the fields needed to describe: (1) financial instruments, (2) corporate events affecting value and tradability, and (3) market-related, economic and industrial indicators. The principal function of MDDL is to allow entities to exchange market data by standardizing formats and definitions. MDDL provides a common format for market data so that it can be efficiently passed from one processing system to another and provides a common understanding of market data content by standardizing terminology and by normalizing the relationships of various data elements to one another ... From the user perspective, the goal of MDDL is to enable users to integrate data from multiple sources by standardizing both the input feeds used for data warehousing (i.e., define what's being provided by vendors) and the output methods by which client applications request the data (i.e., ensure compatibility on how to get data in and out of applications)." === Clinical submissions === The Clinical Data Interchange Standards Consortium, a global, multidisciplinary, non-profit organization, has established standards to support the acquisition, exchange, submission and archiving of clinical research data and metadata. CDISC standards are vendor-neutral, platform-independent and freely available from the CDISC website. The Case Report Tabulation Data Definition Specification (define.xml) draft version 2.0, the oldest data definition specification, is part of the evolution from the 1999 FDA electronic submission (eSub) guidance and electronic Common Technical Document (eCTD) documents specifying that a document describing the content and structure of included data be included in a submission. Define.xml was developed to automate the review process by generating a machine-readable data-definition document. Define.xml has standardized submissions to the Food and Drug Administration, reducing review times from over two years to several months. === Archival data === A data definition specification is the foundation of metadata for scientific data archiving. The Metadata Encoding and Transmission Standard (METS) uses one principle of a DDS: consistent use of key terms to catalog digital objects for global use. The METS schema is a flexible mechanism for encoding descriptive, administrative and structural metadata for a digital library object and expressing complex links between metadata, and can provide a useful standard for the exchange of digital-library objects between repositories. A similar effort is underway to preserve complex data associated with video-game archiving. Preserving Virtual Worlds attempted to address archival-format deficiencies, citing the lack of suitable documentation for interactive fiction and games at the bit level: specifically, the absence of "representation information" needed to map raw bits into higher-level data constructs. Preserving Virtual Worlds 2 is a research project expanding on initial efforts in this field.

  Read more →
• 

  Visual cryptography

  Visual cryptography is a cryptographic technique which allows visual information (pictures, text, etc.) to be encrypted in such a way that the decrypted information appears as a visual image. One of the best-known techniques has been credited to Moni Naor and Adi Shamir, who developed it in 1994. They demonstrated a visual secret sharing scheme, where a binary image was broken up into n shares so that only someone with all n shares could decrypt the image, while any n − 1 shares revealed no information about the original image. Each share was printed on a separate transparency, and decryption was performed by overlaying the shares. When all n shares were overlaid, the original image would appear. There are several generalizations of the basic scheme including k-out-of-n visual cryptography, and using opaque sheets but illuminating them by multiple sets of identical illumination patterns under the recording of only one single-pixel detector, which exposed the image. Using a similar idea, transparencies can be used to implement a one-time pad encryption, where one transparency is a shared random pad, and another transparency acts as the ciphertext. Normally, there is an expansion of space requirement in visual cryptography. But if one of the two shares is structured recursively, the efficiency of visual cryptography can be increased to 100%. Some antecedents of visual cryptography are in patents from the 1960s. Other antecedents are in the work on perception and secure communication. Visual cryptography can be used to protect biometric templates in which decryption does not require any complex computations. == Example == In this example, the binary image has been split into two component images. Each component image has a pair of pixels for every pixel in the original image. These pixel pairs are shaded black or white according to the following rule: if the original image pixel was black, the pixel pairs in the component images must be complementary; randomly shade one ■□, and the other □■. When these complementary pairs are overlapped, they will appear dark gray. On the other hand, if the original image pixel was white, the pixel pairs in the component images must match: both ■□ or both □■. When these matching pairs are overlapped, they will appear light gray. So, when the two component images are superimposed, the original image appears. However, without the other component, a component image reveals no information about the original image; it is indistinguishable from a random pattern of ■□ / □■ pairs. Moreover, if you have one component image, you can use the shading rules above to produce a counterfeit component image that combines with it to produce any image at all. == (2, n) visual cryptography sharing case == Sharing a secret with an arbitrary number of people, n, such that at least 2 of them are required to decode the secret is one form of the visual secret sharing scheme presented by Moni Naor and Adi Shamir in 1994. In this scheme we have a secret image which is encoded into n shares printed on transparencies. The shares appear random and contain no decipherable information about the underlying secret image, however if any 2 of the shares are stacked on top of one another the secret image becomes decipherable by the human eye. Every pixel from the secret image is encoded into multiple subpixels in each share image using a matrix to determine the color of the pixels. In the (2, n) case, a white pixel in the secret image is encoded using a matrix from the following set, where each row gives the subpixel pattern for one of the components: {all permutations of the columns of} : C 0 = [ 1 0 . . . 0 1 0 . . . 0 . . . 1 0 . . . 0 ] . {\displaystyle \mathbf {C_{0}=} {\begin{bmatrix}1&0&...&0\\1&0&...&0\\...\\1&0&...&0\end{bmatrix}}.} While a black pixel in the secret image is encoded using a matrix from the following set: {all permutations of the columns of} : C 1 = [ 1 0 . . . 0 0 1 . . . 0 . . . 0 0 . . . 1 ] . {\displaystyle \mathbf {C_{1}=} {\begin{bmatrix}1&0&...&0\\0&1&...&0\\...\\0&0&...&1\end{bmatrix}}.} For instance in the (2,2) sharing case (the secret is split into 2 shares and both shares are required to decode the secret) we use complementary matrices to share a black pixel and identical matrices to share a white pixel. Stacking the shares we have all the subpixels associated with the black pixel now black while 50% of the subpixels associated with the white pixel remain white. == Cheating the (2, n) visual secret sharing scheme == Horng et al. proposed a method that allows n − 1 colluding parties to cheat an honest party in visual cryptography. They take advantage of knowing the underlying distribution of the pixels in the shares to create new shares that combine with existing shares to form a new secret message of the cheaters choosing. We know that 2 shares are enough to decode the secret image using the human visual system. But examining two shares also gives some information about the 3rd share. For instance, colluding participants may examine their shares to determine when they both have black pixels and use that information to determine that another participant will also have a black pixel in that location. Knowing where black pixels exist in another party's share allows them to create a new share that will combine with the predicted share to form a new secret message. In this way a set of colluding parties that have enough shares to access the secret code can cheat other honest parties. == Visual steganography == 2×2 subpixels can also encode a binary image in each component image. For example, each white pixel of each component image could be represented by two black subpixels, while each black pixel represented by three black subpixels. When overlaid, each white pixel of the secret image is represented by three black subpixels, while each black pixel is represented by all four subpixels black. Each corresponding pixel in the component images is randomly rotated to avoid orientation leaking information about the secret image. == In popular culture == In "Do Not Forsake Me Oh My Darling", a 1967 episode of TV series The Prisoner, the protagonist uses a visual cryptography overlay of multiple transparencies to reveal a secret message – the location of a scientist friend who had gone into hiding.

  Read more →
• 

  Social media mining

  Social media mining is the process of obtaining data from user-generated content on social media in order to extract actionable patterns, form conclusions about users, and act upon the information. Mining supports targeting advertising to users or academic research. The term is an analogy to the process of mining for minerals. Mining companies sift through raw ore to find the valuable minerals; likewise, social media mining sifts through social media data in order to discern patterns and trends about matters such as social media usage, online behaviour, content sharing, connections between individuals, buying behaviour. These patterns and trends are of interest to companies, governments and not-for-profit organizations, as such organizations can use the analyses for tasks such as design strategies, introduce programs, products, processes or services. Social media mining uses concepts from computer science, data mining, machine learning, and statistics. Mining is based on social network analysis, network science, sociology, ethnography, optimization and mathematics. It attempts to formally represent, measure and model patterns from social media data. In the 2010s, major corporations, governments and not-for-profit organizations began mining to learn about customers, clients and others. Platforms such as Google, Facebook (partnered with Datalogix and BlueKai) conduct mining to target users with advertising. Scientists and machine learning researchers extract insights and design product features. Users may not understand how platforms use their data. Users tend to click through Terms of Use agreements without reading them, leading to ethical questions about whether platforms adequately protect users' privacy. During the 2016 United States presidential election, Facebook allowed Cambridge Analytica, a political consulting firm linked to the Trump campaign, to analyze the data of an estimated 87 million Facebook users to profile voters, creating controversy when this was revealed. == Background == As defined by Kaplan and Haenlein, social media is the "group of internet-based applications that build on the ideological and technological foundations of Web 2.0, and that allow the creation and exchange of user-generated content." There are many categories of social media including, but not limited to, social networking (Facebook or LinkedIn), microblogging (Twitter), photo sharing (Flickr, Instagram, Photobucket, or Picasa), news aggregation (Google Reader, StumbleUpon, or Feedburner), video sharing (YouTube, MetaCafe), livecasting (Ustream or Twitch), virtual worlds (Kaneva), social gaming (World of Warcraft), social search (Google, Bing, or Ask.com), and instant messaging (Google Talk, Skype, or Yahoo! messenger). The first social media website was introduced by GeoCities in 1994. It enabled users to create their own homepages without having a sophisticated knowledge of HTML coding. The first social networking site, SixDegrees.com, was introduced in 1997. Since then, many other social media sites have been introduced, each providing service to millions of people. These individuals form a virtual world in which individuals (social atoms), entities (content, sites, etc.) and interactions (between individuals, between entities, between individuals and entities) coexist. Social norms and human behavior govern this virtual world. By understanding these social norms and models of human behavior and combining them with the observations and measurements of this virtual world, one can systematically analyze and mine social media. Social media mining is the process of representing, analyzing, and extracting meaningful patterns from data in social media, resulting from social interactions. It is an interdisciplinary field encompassing techniques from computer science, data mining, machine learning, social network analysis, network science, sociology, ethnography, statistics, optimization, and mathematics. Social media mining faces grand challenges such as the big data paradox, obtaining sufficient samples, the noise removal fallacy, and evaluation dilemma. Social media mining represents the virtual world of social media in a computable way, measures it, and designs models that can help us understand its interactions. In addition, social media mining provides necessary tools to mine this world for interesting patterns, analyze information diffusion, study influence and homophily, provide effective recommendations, and analyze novel social behavior in social media. == Uses == Social media mining is used across several industries including business development, social science research, health services, and educational purposes. Once the data received goes through social media analytics, it can then be applied to these various fields. Often, companies use the patterns of connectivity that pervade social networks, such as assortativity—the social similarity between users that are induced by influence, homophily, and reciprocity and transitivity. These forces are then measured via statistical analysis of the nodes and connections between these nodes. Social analytics also uses sentiment analysis, because social media users often relay positive or negative sentiment in their posts. This provides important social information about users' emotions on specific topics. These three patterns have several uses beyond pure analysis. For example, influence can be used to determine the most influential user in a particular network. Companies would be interested in this information in order to decide who they may hire for influencer marketing. These influencers are determined by recognition, activity generation, and novelty—three requirements that can be measured through the data mined from these sites. Analysts also value measures of homophily: the tendency of two similar individuals to become friends. Users have begun to rely on information of other users' opinions in order to understand diverse subject matter. These analyses can also help create recommendations for individuals in a tailored capacity. By measuring influence and homophily, online and offline companies are able to suggest specific products for individuals consumers, and groups of consumers. Social media networks can use this information themselves to suggest to their users possible friends to add, pages to follow, and accounts to interact with. == Perception == Modern social media mining is a controversial practice that has led to exponential gains in user growth for tech giants such as Facebook, Inc., Twitter, and Google. Companies such as these, considered "Big Tech" are companies that build algorithms that take advantage of user input to understand their preferences, and keep them on the platform as much as possible. These inputs, that can be as simple as time spent on a given screen, provide the data being mined, and lead to companies profiting heavily from using that data to capitalize on extremely accurate predictions about user behavior. The growth of platforms accelerated rapidly once these strategies were put in place; Most of the largest platforms now average over 1 billion active users per month as of 2021. It has been claimed by a multitude of anti-algorithm personalities, like Tristan Harris or Chamath Palihapitiya, that certain companies (specifically Facebook) valued growth above all else, and ignored potential negative impacts from these growth engineering tactics. At the same time, users have now created their own data arbitrages with the help of their own data, through content monetization and becoming influencers. Users typically have access to a varied set of analytics specific to people that interact with them on social media, and can use these as building blocks for their own targeting and growth strategies through ads and posts that cater to their audiences. Influencers also commonly promote products and services for established brands, creating one of the largest digital industries: Influencer marketing. Instagram, Facebook, Twitter, YouTube, Google, and others have long given access to platform analytics, and allowed third parties to access that information as well, at times unbeknownst to even the user whose data is being viewed/bought. == Research == === Research areas === Social media event detection – Social networks enable users to freely communicate with each other and share their recent news, ongoing activities or views about different topics. As a result, they can be seen as a potentially viable source of information to understand the current emerging topics/events. Public health monitoring and surveillance - Using large-scale analysis of social media to study large cohorts of patients and the general public, e.g. to obtain early warning signals of drug-drug interactions and adverse drug reactions, or understand human reproduction and sexual interest. Community structure (Community Detection/Evolution/Evaluation) – Identifying communities on social networks, how t

  Read more →
• 

  User-defined function

  A user-defined function (UDF) is a function provided by the user of a program or environment, in a context where the usual assumption is that functions are built into the program or environment. UDFs are usually written for the requirement of its creator. == BASIC language == In some old implementations of the BASIC programming language, user-defined functions are defined using the "DEF FN" syntax. More modern dialects of BASIC are influenced by the structured programming paradigm, where most or all of the code is written as user-defined functions or procedures, and the concept becomes practically redundant. == COBOL language == In the COBOL programming language, a user-defined function is an entity that is defined by the user by specifying a FUNCTION-ID paragraph. A user-defined function must return a value by specifying the RETURNING phrase of the procedure division header and they are invoked using the function-identifier syntax. See the ISO/IEC 1989:2014 Programming Language COBOL standard for details. As of May 2022, the IBM Enterprise COBOL for z/OS 6.4 (IBM COBOL) compiler contains support for user-defined functions. == Databases == In relational database management systems, a user-defined function provides a mechanism for extending the functionality of the database server by adding a function, that can be evaluated in standard query language (usually SQL) statements. The SQL standard distinguishes between scalar and table functions. A scalar function returns only a single value (or NULL), whereas a table function returns a (relational) table comprising zero or more rows, each row with one or more columns. User-defined functions in SQL are declared using the CREATE FUNCTION statement. For example, a user-defined function that converts Celsius to Fahrenheit (a temperature scale used in USA) might be declared like this: Once created, a user-defined function may be used in expressions in SQL statements. For example, it can be invoked where most other intrinsic functions are allowed. This also includes SELECT statements, where the function can be used against data stored in tables in the database. Conceptually, the function is evaluated once per row in such usage. For example, assume a table named Elements, with a row for each known chemical element. The table has a column named BoilingPoint for the boiling point of that element, in Celsius. The query would retrieve the name and the boiling point from each row. It invokes the CtoF user-defined function as declared above in order to convert the value in the column to a value in Fahrenheit. Each user-defined function carries certain properties or characteristics. The SQL standard defines the following properties: Language - defines the programming language in which the user-defined function is implemented; examples include SQL, C, C# and Java. Parameter style - defines the conventions that are used to pass the function parameters and results between the implementation of the function and the database system (only applicable if language is not SQL). Specific name - a name for the function that is unique within the database. Note that the function name does not have to be unique, considering overloaded functions. Some SQL implementations require that function names are unique within a database, and overloaded functions are not allowed. Determinism - specifies whether the function is deterministic or not. The determinism characteristic has an influence on the query optimizer when compiling a SQL statement. SQL-data access - tells the database management system whether the function contains no SQL statements (NO SQL), contains SQL statements but does not access any tables or views (CONTAINS SQL), reads data from tables or views (READS SQL DATA), or actually modifies data in the database (MODIFIES SQL DATA). User-defined functions should not be confused with stored procedures. Stored procedures allow the user to group a set of SQL commands. A procedure can accept parameters and execute its SQL statements depending on those parameters. A procedure is not an expression and, thus, cannot be used like user-defined functions. Some database management systems allow the creation of user defined functions in languages other than SQL. Microsoft SQL Server, for example, allows the user to use .NET languages including C# for this purpose. DB2 and Oracle support user-defined functions written in C or Java programming languages. === SQL Server 2000 === There are three types of UDF in Microsoft SQL Server 2000: scalar functions, inline table-valued functions, and multistatement table-valued functions. Scalar functions return a single data value (not a table) with RETURNS clause. Scalar functions can use all scalar data types, with exception of timestamp and user-defined data types. Inline table-valued functions return the result set of a single SELECT statement. Multistatement table-valued functions return a table, which was built with many TRANSACT-SQL statements. User-defined functions can be invoked from a query like built‑in functions such as OBJECT_ID, LEN, DATEDIFF, or can be executed through an EXECUTE statement like stored procedures. Performance Notes: User-defined functions are subroutines made of one or more Transact-SQL statements that can be used to encapsulate code for reuse. It takes zero or more arguments and evaluates a return value. Has both control-flow and DML statements in its body similar to stored procedures. Does not allow changes to any Global Session State, like modifications to database or external resource, such as a file or network. Does not support output parameter. DEFAULT keyword must be specified to pass the default value of parameter. Errors in UDF cause UDF to abort which, in turn, aborts the statement that invoked the UDF. === Apache Hive === Apache Hive defines, in addition to the regular user-defined functions (UDF), also user-defined aggregate functions (UDAF) and table-generating functions (UDTF). Hive enables developers to create their own custom functions with Java. === Apache Doris === Apache Doris, an open-source real-time analytical database, allows external users to contribute their own UDFs written in C++ to it.

  Read more →
• 

  Sentiment analysis

  Sentiment analysis (also known as opinion mining) is the use of natural language processing, text analysis, computational linguistics, and biometrics to systematically identify, extract, quantify, and study affective states and subjective information. Sentiment analysis is widely applied to voice of the customer materials such as reviews and survey responses, online and social media, and healthcare materials for applications that range from marketing to customer service to clinical medicine. With the rise of deep language models, such as RoBERTa, more difficult data domains can be analyzed, e.g., news texts where authors typically express their opinion/sentiment less explicitly. == Types == A basic task in sentiment analysis is classifying the polarity of a given text at the document, sentence, or feature/aspect level—whether the expressed opinion in a document, a sentence or an entity feature/aspect is positive, negative, or neutral. Advanced, "beyond polarity" sentiment classification looks, for instance, at emotional states such as enjoyment, anger, disgust, sadness, fear, and surprise. Precursors to sentimental analysis include the General Inquirer, which provided hints toward quantifying patterns in text and, separately, psychological research that examined a person's psychological state based on analysis of their verbal behavior. Subsequently, the method described in a patent by Volcani and Fogel, looked specifically at sentiment and identified individual words and phrases in text with respect to different emotional scales. A current system based on their work, called EffectCheck, presents synonyms that can be used to increase or decrease the level of evoked emotion in each scale. Many other subsequent efforts were less sophisticated, using a mere polar view of sentiment, from positive to negative, such as work by Turney, and Pang who applied different methods for detecting the polarity of product reviews and movie reviews respectively. This work is at the document level. One can also classify a document's polarity on a multi-way scale, which was attempted by Pang and Snyder among others: Pang and Lee expanded the basic task of classifying a movie review as either positive or negative to predict star ratings on either a 3- or a 4-star scale, while Snyder performed an in-depth analysis of restaurant reviews, predicting ratings for various aspects of the given restaurant, such as the food and atmosphere (on a five-star scale). First steps to bringing together various approaches—learning, lexical, knowledge-based, etc.—were taken in the 2004 AAAI Spring Symposium where linguists, computer scientists, and other interested researchers first aligned interests and proposed shared tasks and benchmark data sets for the systematic computational research on affect, appeal, subjectivity, and sentiment in text. Even though in most statistical classification methods, the neutral class is ignored under the assumption that neutral texts lie near the boundary of the binary classifier, several researchers suggest that, as in every polarity problem, three categories must be identified. Moreover, it can be proven that specific classifiers such as the Max Entropy and SVMs can benefit from the introduction of a neutral class and improve the overall accuracy of the classification. There are in principle two ways for operating with a neutral class. Either, the algorithm proceeds by first identifying the neutral language, filtering it out and then assessing the rest in terms of positive and negative sentiments, or it builds a three-way classification in one step. This second approach often involves estimating a probability distribution over all categories (e.g. naive Bayes classifiers as implemented by the NLTK). Whether and how to use a neutral class depends on the nature of the data: if the data is clearly clustered into neutral, negative and positive language, it makes sense to filter the neutral language out and focus on the polarity between positive and negative sentiments. If, in contrast, the data are mostly neutral with small deviations towards positive and negative affect, this strategy would make it harder to clearly distinguish between the two poles. A different method for determining sentiment is the use of a scaling system whereby words commonly associated with having a negative, neutral, or positive sentiment are given an associated number on a −10 to +10 scale (most negative up to most positive) or simply from 0 to a positive upper limit such as +4. This makes it possible to adjust the sentiment of a given term relative to its environment (usually on the level of the sentence). When a piece of unstructured text is analyzed using natural language processing, each concept in the specified environment is given a score based on the way sentiment words relate to the concept and its associated score. This allows movement to a more sophisticated understanding of sentiment, because it is now possible to adjust the sentiment value of a concept relative to modifications that may surround it. Words, for example, that intensify, relax or negate the sentiment expressed by the concept can affect its score. Alternatively, texts can be given a positive and negative sentiment strength score if the goal is to determine the sentiment in a text rather than the overall polarity and strength of the text. There are various other types of sentiment analysis, such as aspect-based sentiment analysis, grading sentiment analysis (positive, negative, neutral), multilingual sentiment analysis and detection of emotions. === Subjectivity/objectivity identification === This task is commonly defined as classifying a given text (usually a sentence) into one of two classes: objective or subjective. This problem can sometimes be more difficult than polarity classification. The subjectivity of words and phrases may depend on their context and an objective document may contain subjective sentences (e.g., a news article quoting people's opinions). Moreover, as mentioned by Su, results are largely dependent on the definition of subjectivity used when annotating texts. However, Pang showed that removing objective sentences from a document before classifying its polarity helped improve performance. Subjective and objective identification, emerging subtasks of sentiment analysis to use syntactic, semantic features, and machine learning knowledge to identify if a sentence or document contains facts or opinions. Awareness of recognizing factual and opinions is not recent, having possibly first presented by Carbonell at Yale University in 1979. The term objective refers to the incident carrying factual information. Example of an objective sentence: 'To be elected president of the United States, a candidate must be at least thirty-five years of age.' The term subjective describes the incident contains non-factual information in various forms, such as personal opinions, judgment, and predictions, also known as 'private states'. In the example down below, it reflects a private states 'We Americans'. Moreover, the target entity commented by the opinions can take several forms from tangible product to intangible topic matters stated in Liu (2010). Furthermore, three types of attitudes were observed by Liu (2010), 1) positive opinions, 2) neutral opinions, and 3) negative opinions. Example of a subjective sentence: 'We Americans need to elect a president who is mature and who is able to make wise decisions.' This analysis is a classification problem. Each class's collections of words or phrase indicators are defined for to locate desirable patterns on unannotated text. For subjective expression, a different word list has been created. Lists of subjective indicators in words or phrases have been developed by multiple researchers in the linguist and natural language processing field states in Riloff et al. (2003). A dictionary of extraction rules has to be created for measuring given expressions. Over the years, in subjective detection, the features extraction progression from curating features by hand to automated features learning. At the moment, automated learning methods can further separate into supervised and unsupervised machine learning. Patterns extraction with machine learning process annotated and unannotated text have been explored extensively by academic researchers. However, researchers recognized several challenges in developing fixed sets of rules for expressions respectably. Much of the challenges in rule development stems from the nature of textual information. Six challenges have been recognized by several researchers: 1) metaphorical expressions, 2) discrepancies in writings, 3) context-sensitive, 4) represented words with fewer usages, 5) time-sensitive, and 6) ever-growing volume. Metaphorical expressions. The text contains metaphoric expression may impact on the performance on the extraction. Besides, metaphors take in different forms, which may have been contribu

  Read more →
• 

  Social trading

  Social trading is a form of investing that allows investors to observe the trading behavior of their peers and expert traders. The primary objective is to follow their investment strategies using copy trading or mirror trading. Social trading requires little or no knowledge about financial markets. == History == One of the first social trading platforms was Collective2] which began offering a social trading functionality to retail traders as early as 2003 (preceding ZuluTrade by four years). In 2010, social trading started to achieve a greater degree of mainstream appeal with eToro, followed by Wikifolio in 2012. Europe-based NAGA, listed on Frankfurt Stock Exchange since 2017, claims more than EUR 27 billion was traded on its platform in the second half of 2019. Some of the other contemporary social trading platforms and tech providers are Trading Motion, Brokeree Solutions, iSystems, and FX Junction, among others. === Research === MIT Computer Scientist and researcher Yaniv Altshuler described social trading networks as complex adaptive systems, and in his 2014 research on eToro's OpenBook, wrote that "Having the inherent ability to share ideas and information between each others, OpenBook's users are given a new source of information they can use in order to enhance their trading performance. As the users are not playing against each other but rather – against the market, this situation becomes a non zero-sum game, hence incentivizing the users to share as much information as possible." His paper concludes that "social trading provides much better opportunities for profiting compared with individual trading," but that users make "excellent but sometimes not optimal decisions in selecting experts when they can see others' choices." A 2015 World Economic Forum report described social trading networks as disruptors, which "have emerged to provide low-cost, sophisticated alternatives to traditional wealth managers. These solutions cater to a broader customer base and empower customers to have more control of their wealth management," and "pose a tangible threat to the traditional practices of the wealth management industry". Economist Nouriel Roubini's thinktank predicted in 2016 that "newer forms of investment, such as socially responsible investments and social trading will bring some of the largest industry growth in the coming years." A 2017 St. John's University study found that 'leader' traders, or those with followers, are more susceptible to the disposition effect than investors that are not being followed by any other traders, with the authors suggesting the observation may be explained by "leaders feeling responsible towards their followers and an urge to not let them down, by fear of losing followers when admitting a bad investment decision and signaling confidence in their initial investment choice, or by an attempt of newly appointed leaders to manage their self-image." Social trading may potentially also change how much risk investors take. A recent experimental study argues that merely providing information on the success of others may lead to a significant increase in risk taking. This increase in risk taking may even be larger when subjects are provided with the option to directly copy others. == Characteristics == Social trading is an alternative way of analyzing financial data by looking at what other traders are doing and comparing and copying their techniques and strategies. Prior to the advent of social trading, investors and traders were relying on fundamental or technical analysis to form their investment decisions. Using social trading investors and traders could integrate into their investment decision-process social indicators from trading data-feeds of other traders. Social trading platforms or networks can be considered a subcategory of social networking services. Social trading allows traders to trade online with the help of others and some have claimed shortens the learning curve from novice to experienced trader. Traders can interact with others, watch others take trades, then duplicate their trades and learn what prompted the top performer to take a trade in the first place. By copying trades, traders can learn which strategies work and which do not work. Social trading is used to do speculation; in the moral context speculative practices are considered negatively and to be avoided by each individual. who conversely should maintain a long-term horizon avoiding any types of short term speculation. Social Media has permeated the trading world such that two main types of trading has evolved: Traditional Trades Single (or non-social) trade: Trader A places a normal trade by himself or herself; This can by manual or automated Social Trading There are two main types of social trading: Copy trade: Trader A places exactly the same trade as trader B's one single trade; (iii) Mirror trade: Trader A automatically executes trader B's every single trade, i.e., trader A follows exactly trader B's trading activities. Other variations offered on some platforms allow users to copy another trader's portfolio (copy portfolio), and follow a trader's dividends (copy dividends), where whenever a followed trader withdraws money from his or her account, a proportional amount of money will be withdrawn from the balance of their follower, in real time. === Key features === Information flow: Unencumbered access to information is important in financial markets and that makes the free exchange of information of interest to small scale as well as individual investors. Cooperative trading: Social trading offers traders the opportunity to work together in trading teams which can trade the markets collaboratively, whether by pooling funds, dividing research or through sharing information. Monetization: As with social networks in the broader sense, monetization strategies are not always clear. As with social networks in general, it is possible, however, that the long-term worth of such websites may come from the variety and depth of data about their users which their active communities are likely to generate. Transparency: Social trading platforms reveal traders' performance stats, open and past positions, and market sentiment, giving members complete information to assess the credibility of the contributors they follow on the platform.

  Read more →
• 

  Web presence

  A web presence is a location on the World Wide Web where a person, business, or some other entity is represented (see also web property and point of presence). Examples of a web presence for a person could be a personal website, a blog, a profile page, a wiki page, or a social media point of presence (e.g. a LinkedIn profile, a Facebook account, or a Twitter account). Examples of a web presence for a business or some other entity could be a corporate website, a microsite, a page on a review site, a wiki page, or a social media point of presence (e.g., a LinkedIn company page and/or group, a Facebook business/brand/product page, or a Twitter account). Every web presence is associated with a unique web address to distinguish one point of presence from another. == Owned vs. unowned == Web presence can either be owned or unowned. Owned media exists when a single person or group can control the content that is published on its web presence (e.g. a corporate website or a personal Twitter account). However, when a single person or group cannot solely control the content, the creator is different from the owner. This is considered unowned media (see earned media). A Wikipedia page or a Yelp page about a person, company, or product would be an example of a known (or "earned") web presence. Occasionally, a first form of media known as "paid media" is often included in the discussion of media types: "earned vs. owned vs. paid". Paid media is commonly found in the form of advertisements, but it is not considered a form of web presence. == Management == Web presence management is the process of establishing and maintaining a digital footprint on the web. The three factors that are considered include the following: where a person or business has web presence; how each web presence represents its enterprise; and what is published at a point of presence. Web presence management is the discipline of determining and governing: the distribution of policy documents which platforms are most appropriate (e.g. internal vs. external blog, YouTube vs. Vimeo) the single inventory of personal or corporate web presence (e.g. partners or advocates) where on the web a business and any relatable assets are represented where on the web a business and any relatable assets are impersonated or pirated web properties with the particular entities they represent who has control over which web properties new web properties which are not in the personal or corporate inventory (e.g. someone creates a new presence) authorized and unauthorized changes to the creation (e.g. branding) of a web presence a workflow for creating a web property that follows its corporate standards === Management system === The purpose of a web presence management system is to manage the web presence of a person or business. This includes the collection of domain names, websites, social media, and other web pages where he, she, or it is being represented. The tool generally offers the following key functions: new presence discovery, inventory management, change detection, access control, stakeholder coordination, and compliance workflow. A web presence management system is meant to have a broader reach so that it emphasizes where a presence has been established, will be established, must be maintained, or must be remediated. An example of a web presence management system is the Brandle Presence Manager. In order to publish content to the various points of web presence, multiple content management systems and sometimes even social media management systems are often used. The primary focus of most content and social media management systems is limited to their specific web platforms. === Domain names === Another aspect of web presence management is managing the collection of domain names registered to the person or business. Any entity may register multiple domain names for the same property. As a result, they can link alternative spellings, different top-level domains, aliases, brands, or products to the same website. Similarly, negative or derogatory domain names may also be registered. This is done to prevent certain domain names from being used against the person or business. It is common for a larger business to have domain names registered by multiple employees at multiple domain name registrars, possibly a result of organizational or geographical requirements. Consequently, a web presence management system can be used to monitor all domain names registered by the business, regardless of the registrars used. == Discovery == Web presence discovery is the process of monitoring the web for a new point of presence about a person or business. Web presence discovery is often included in a web presence management system. Whether a new domain is registered, a new website is published, or a new social media account is established, it occurs outside of the person's or business’ control. As a result, its purpose is to assess a new point of presence and appropriately handle any violations. Web presence discovery differs from content listening. The former involves looking for new properties on the web, whereas the latter refers to analyzing content that already exists to hear how a person or business is seen often in near real time. Examples of content listening systems include Sysomos and Radian6, which is now a subsidiary of Salesforce.com. === Brand protection === A person or business may choose to watch for a new web presence that might appear to misrepresent or mislead an audience, such as counterfeiters, spoofers, or malicious hackers. One of the early software in the online brand protection marketplace was MarkMonitor, now part of Thomson Reuters. This software helped detect rogue domain names and websites. However, the modern day growth of social media has seen a rise in the number of fraudulent brand impersonations. It has become much easier for a new web presence to be created on those platforms, which results in a greater frequency of them today. As a preventive measure, online brand protection providers are now adding social media to their domain and website discovery options. === Security === The widespread growth of social media has also made it easier for unauthorized individuals to impersonate an employee. Consequently, social media has now become a recognized threat vector in that it can be used to socially engineer an attack on a business. To counter this, companies are able to use web presence monitoring tools to detect new points of presence on the web and thereby defend against socially engineered attacks. === Distributed inventory management === A web presence monitoring system can be used by a business to associate a new web property with its corporate inventory. It is designed to address autonomous, distributed behaviors. This usually applies to larger businesses whose geographically diverse employees are more prone to creating new points of presence on the web. For example, a retail chain may allow each local store to create and manage their web presence to market to and communicate with their local customer base. Similarly, a global business may have teams in each country or region who create and manage a web presence to adapt to local languages or cultures. == Monitoring == Web presence monitoring is the process of monitoring a known inventory of web presence to detect any changes that are made. Web presence monitoring is often included in a web presence management system and can serve multiple purposes for both larger corporations and certain individuals, such as celebrities. It is important to note that presence monitoring differs from content listening. The former involves monitoring the properties (e.g. branding) of a web property in an established inventory, whereas the latter refers to analyzing content that already exists to hear how a person or business is seen often in near real time. Additionally, presence monitoring focuses on owned media and content listening on earned media. === Corporate, brand, and regulatory compliance === Many companies ensure that certain standards are met for a property on the web that represents their business. For companies in regulated industries, such as finance and healthcare, the company may be required by law to ensure that all publicized content, regardless of platform or technology, follow specific requirements. The widespread growth of social media has seen a rise in the number of fraudulent corporate impersonations. It has become much easier for a new web presence to be created on these platforms, and so these are much more prevalent than they used to be. As a preventive measure, a web presence monitoring system alerts the company when a known property is changed, allowing for the property to be reviewed and amended so that it follows the proper standards. . A web presence monitoring system helps alert the company when a known property is changed, so it can be reviewed and brought back, if necessary, into compliance with the appro

  Read more →
• 

  Toy problem

  In scientific disciplines, a toy problem or a puzzlelike problem is a problem that is not of immediate scientific interest, yet is used as an expository device to illustrate a trait that may be shared by other, more complicated, instances of the problem, or as a way to explain a particular, more general, problem solving technique. A toy problem is useful to test and demonstrate methodologies. Researchers can use toy problems to compare the performance of different algorithms. They are also good for game designing. For instance, while engineering a large system, the large problem is often broken down into many smaller toy problems which have been well understood in detail. Often these problems distill a few important aspects of complicated problems so that they can be studied in isolation. Toy problems are thus often very useful in providing intuition about specific phenomena in more complicated problems. As an example, in the field of artificial intelligence, classical puzzles, games and problems are often used as toy problems. These include sliding-block puzzles, N-Queens problem, missionaries and cannibals problem, tic-tac-toe, chess, Tower of Hanoi and others.

  Read more →
• 

  Key Transparency

  Key Transparency allows communicating parties to verify public keys used in end-to-end encryption. In many end-to-end encryption services, to initiate communication a user will reach out to a central server and request the public keys of the user with which they wish to communicate. If the central server is malicious or becomes compromised, a man-in-the-middle attack can be launched through the issuance of incorrect public keys. The communications can then be intercepted and manipulated. Additionally, legal pressure could be applied by surveillance agencies to manipulate public keys and read messages. With Key Transparency, public keys are posted to a public log that can be universally audited. Communicating parties can verify public keys used are accurate.

  Read more →
• 

  Data profiling

  Data profiling is the process of examining the data available from an existing information source (e.g. a database or a file) and collecting statistics or informative summaries about that data. The purpose of these statistics may be to: Find out whether existing data can be easily used for other purposes Improve the ability to search data by tagging it with keywords, descriptions, or assigning it to a category Assess data quality, including whether the data conforms to particular standards or patterns Assess the risk involved in integrating data in new applications, including the challenges of joins Discover metadata of the source database, including value patterns and distributions, key candidates, foreign-key candidates, and functional dependencies Assess whether known metadata accurately describes the actual values in the source database Understanding data challenges early in any data intensive project, so that late project surprises are avoided. Finding data problems late in the project can lead to delays and cost overruns. Have an enterprise view of all data, for uses such as master data management, where key data is needed, or data governance for improving data quality. == Introduction == Data profiling refers to the analysis of information for use in a data warehouse in order to clarify the structure, content, relationships, and derivation rules of the data. Profiling helps to not only understand anomalies and assess data quality, but also to discover, register, and assess enterprise metadata. The result of the analysis is used to determine the suitability of the candidate source systems, usually giving the basis for an early go/no-go decision, and also to identify problems for later solution design. == How data profiling is conducted == Data profiling utilizes methods of descriptive statistics such as minimum, maximum, mean, mode, percentile, standard deviation, frequency, variation, aggregates such as count and sum, and additional metadata information obtained during data profiling such as data type, length, discrete values, uniqueness, occurrence of null values, typical string patterns, and abstract type recognition. The metadata can then be used to discover problems such as illegal values, misspellings, missing values, varying value representation, and duplicates. Different analyses are performed for different structural levels. E.g. single columns could be profiled individually to get an understanding of frequency distribution of different values, type, and use of each column. Embedded value dependencies can be exposed in a cross-columns analysis. Finally, overlapping value sets possibly representing foreign key relationships between entities can be explored in an inter-table analysis. Normally, purpose-built tools are used for data profiling to ease the process. The computational complexity increases when going from single column, to single table, to cross-table structural profiling. Therefore, performance is an evaluation criterion for profiling tools. == When is data profiling conducted? == According to Kimball, data profiling is performed several times and with varying intensity throughout the data warehouse developing process. A light profiling assessment should be undertaken immediately after candidate source systems have been identified and DW/BI business requirements have been satisfied. The purpose of this initial analysis is to clarify at an early stage if the correct data is available at the appropriate detail level and that anomalies can be handled subsequently. If this is not the case the project may be terminated. Additionally, more in-depth profiling is done prior to the dimensional modeling process in order assess what is required to convert data into a dimensional model. Detailed profiling extends into the ETL system design process in order to determine the appropriate data to extract and which filters to apply to the data set. Additionally, data profiling may be conducted in the data warehouse development process after data has been loaded into staging, the data marts, etc. Conducting data at these stages helps ensure that data cleaning and transformations have been done correctly and in compliance of requirements. == Benefits and examples == Data profiling can improve data quality, shorten the implementation cycle of major projects, and improve users' understanding of data. Discovering business knowledge embedded in data itself is one of the significant benefits derived from data profiling. It can improve data accuracy in corporate databases.

  Read more →