AI Analytics Hub

AI Analytics Hub — independent reviews, comparisons, pricing and step-by-step guides on Aizhi.

Patch management

Patch management (or patch management policy or patch policy or patch management process) is concerned with the identification, acquisition, distribution, testing and installation of patches to systems. Proper patch management can be a net productivity boost for an organization. Patches can be used to defend against and eliminate potential vulnerabilities of a system, so that no threats may exploit them. Problems can arise during patch management, including buggy patches that either fail to fix their problem or introduce new issues. Patch management tools help orchestrate all of the procedures involved in patch management. == Description == Patch management is defined as a sub-practice of various disciplines including vulnerability management (part of security management), lifecycle management (with further possible sub-classification into application lifecycle management and release management), change management, and systems management. The practice is broadly concerned with the identification, acquisition, distribution, and installation of patches to systems. Some definitions of patch management are as a software-level practice, while others are as a systems-level process: software, drivers, and firmware. == Cost–benefit analysis == While reserving time for patching takes up enterprise resources, there are balancing factors which can make proper patch management into a net productivity boost for an organization. Up-to-date systems often perform more efficiently, less costly, with less errors, less security risks, and better user workflow. Additionally, compliance with changing local and federal regulations are more likely to be satisfied. Patching security vulnerabilities has been one among many competing priorities for organizations, leading to longer periods before patching for some organizations. Equifax was too slow to implement its 2015 patch management plan to be able to mitigate or prevent the 2017 Equifax data breach, leading to scrutiny from regulators. == Relation to security management == Patches can be used to defend against and eliminate potential vulnerabilities of a system, so that no threats may exploit them; therefore, patch management can be considered a sub-discipline of vulnerability management. Every patchable device in a system presents an attack surface that must be secured. === Time plan === Automatic updates are where the patch is applied automatically with little to know actions or planning required. This approach is recommended for many individuals and organizations. Some organizations also have to prioritize which patches to prioritize given limited resources. Patch Tuesday is the most common process when major companies like Microsoft and Adobe release patches on a known date so that companies can plan resources around implementing the patches more quickly. Linux is open-sourced and patches can be released at any time, leading some to rely on mailing lists or other ways to be alerted to updates. === Inventory === Taking an inventory of software and hardware, including versions can make it easier to correlate with bugs or patches as they become known. Taking stock of how much education and support others in an organization need to install their patches can also help for planning how to implement the patch or design systems to begin with. Streamlining the process by using tools that can communicate with each other can also help to reduce the time of exposure to known vulnerabilities. == Challenges == There are a multitude of problems that can arise during patch management. A common issue is buggy patches, which either fail to fix their problem or introduce new issues. Another issue is deployment synchronization, since various subsystems may receive instructions to update at different times. Similarly, the difficulty of patch management across many devices may grow at an uncontrollable rate depending on organizational size. One prominent demonstration of the challenges facing proper patch management was the buggy Falcon Sensor patch by CrowdStrike which caused one of the worst IT outages of all time. == Implementations == A patch management tool (alternatively patch manager, patch management system, patch management software, or centralized patch management) help orchestrate all of the procedures involved in patch management. Tools can be in-house (applied locally by local administrators), or external, as with managed service providers (applied externally by a provider). === Patch management software === Windows Update for Business, System Center Configuration Manager, and Windows Server Update Services offer control over patch deployment, with features enabling testing, scheduling updates, and setting custom configurations on Windows platforms. === Managed service providers === == Regulatory requirements (United States) == Timely patching of software vulnerabilities is a requirement under multiple regulatory frameworks in the United States. The Health Insurance Portability and Accountability Act (HIPAA) Security Rule requires covered entities to protect electronic protected health information by implementing security measures sufficient to reduce risks to a reasonable and appropriate level, which industry guidance has long interpreted to include timely patch management. A proposed new HIPAA Security Rule would make patch management requirements explicit, mandating that covered entities and business associates deploy security patches and updates within a defined risk-based timeline and maintain written procedures for prioritizing, testing, and applying patches to systems that store, process, or transmit ePHI. The 2025 proposal continues to receive industry pushback as of December 2025. HIPAA was last updated in 2013. The Payment Card Industry Data Security Standard (PCI DSS) requires organizations to protect system components from known vulnerabilities by installing applicable security patches within one month of release for critical patches. The Cybersecurity and Infrastructure Security Agency (CISA) maintains a Known Exploited Vulnerabilities (KEV) catalog that compels U.S. federal agencies to remediate listed vulnerabilities within specified timelines. Agencies are typically required to patch within 3 weeks, though some vulnerabilities must be fixed within 24 hours.
Read more →
Social media reach

Social media reach is a media analytics metric that refers to the number of users who have come across a particular content on a particular social media platform. Social media platforms have their own individual ways of tracking, analyzing and reporting the traffic on each of the individual platforms. As these platforms are a main source of communication between companies and their target audiences, by conducting research, companies are able to utilize analytical information, such as the reach of their posts, to better understand the interactions between the users and their content. There are multiple underlying factors that will determine what shows up on a newsfeed or timeline. Algorithms, for example, are a type of factor that can alter the reach of a post due to the way the algorithm is coded, which can affect who sees a post and when. Other examples of factors that can impede the reach can include the time at which posts are made, as well as how frequent the posts are between one another. In comparison, an impression is the total number of circumstances where content has been shown on a social timeline, meanwhile, engagement looks at how people interact with the content that they see on a social platform such as like, share or retweet. == Reach on Facebook == Facebook has their own analytic platform which allows the user to see how other users are interacting with their posts, with the use of multiple metrics. This is not something the average user uses, but rather a tool that is used by pages or public figures. For example, Facebook pages that represent a business often look at the activity their posts have generated. There are three types of reach that can be looked at on the Facebook analytic platform. === Types of reach === ==== Organic Reach ==== This type of reach regards the number of distinct users that have seen a specific post on their feed. Organic reach, in other words is the number of people who have seen the post being analyzed on their Facebook newsfeed. Data gathered from this type of reach can give intel to those doing the analysis, such as the demographics of those who have seen the post. ==== Paid Reach ==== This type of reach regards the number of times that distinct users have come across sponsored posts, ads or content. In other words, paid reach is the number of times Facebook users have seen a post that has been paid for by a company. Data collected can give insight, to advertisers or marketers for example, on the activity based around the reach of their post. ==== Viral Reach ==== This type of reach regards the number of views by distinct users on posts that have been commented on or shared by their friends on Facebook. In other words, viral reach looks at the number of people who have seen a post after a friend of theirs commented or shared the original post, therefore it showed on their timeline. Viral reach can be looked at in terms of a collective number of times that the post has been on individual user's timelines. Data collected from viral reach can be used in multiple ways, for example, it can be used to analyze the type of content that gets shared or commented on and can be further used to compare to other posts. === Engaged users === This refers to the number of individual users who have clicked and interacted with a post on Facebook. == Reach on Twitter == Twitter gives access to any of their users to analytics of their tweets as well as their followers. Their dashboard is user friendly, which allows anyone to take a look at the analytics behind their Twitter account. This open access is useful for both the average user and companies as it can provide a quick glance or general outlook of who has seen their tweets. The way that Twitter works is slightly different than the way of Facebook in terms of the reach. On Twitter, especially for users with a higher profile, they are not only engaging with the people who follow them, but also with the followers of their own followers. The reach metric on Twitter looks at the quantity of Twitter users who have been engaged, but also the number of users that follow them as well. This metric is useful to see the if the tweets/content being shared on Twitter are contributing to the growth of audience on this platform. == Reach on Instagram == Instagram gives their users access to their reach, in the Instagram Insights section. Instagram insights can be used to learn more about an account's followers and performance. Reach indicates the total number of unique Instagram accounts that have seen your Instagram post or story. You can find this data by looking at each individual post insights. == Uses of reach == The reach can be a useful metric to analyze for marketers and advertisers. Social media is a platform that is used by marketers to directly target their intended audience with ease. These platforms not only allow marketers to get a better understanding of their audience, but also allow advertisers to insert their ads onto the timelines of specific users to later be able to conduct research to see the reach of their posts/content. The basic goal of marketers is to increase their reach as much as possible to impact bigger audiences of their dream customers and, in the end, make more sales. When doing organic social media marketing, using paid methods like ads or doing influencer marketing whether it is paid or free, it allows marketers to track the performance of their strategy and tweak it based on what works and what does not. == Analytics and reach == Social analytics looks at the data collected based on the interactions of users on social media platforms. A lot of information can be gathered which can provide intel based on user activities on social media. When looking into analytics in regard to social media, each company or group has a different goal in mind to engage their audience. At a glance, the three might seem as if they are very similar, however the differences between them are significant. There are many aspects that can be analyzed from the data gathered from social media platforms, depending on what is being observed, the correct metric would then be selected to further analyze. One example of the many metrics that can be used through social analytics is the reach. == Reach formula == To calculate social media reach one can use the following formula: R = I f ¯ {\displaystyle R={\frac {I}{\bar {f}}}} where R {\displaystyle R} — is social media reach, I {\displaystyle I} stands for the number of impressions, f ¯ {\displaystyle {\bar {f}}} is the average frequency of impressions per user. f ¯ {\displaystyle {\bar {f}}} represents the number of events when the ad is shown to a particular user. The average value should be calculated over the time period with stable settings of advertisement campaign. == Commenting For Better Reach == Commenting For Better Reach also known as "CFBR" is a widely used strategy for organically boosting post reach on social media platforms. Algorithms tend to favor posts with substantial likes and comments, granting them broader exposure compared to less engaging content. Primarily seen on LinkedIn, a platform geared toward professional networking and business connections, the use of CFBR signals active engagement aimed at enhancing post visibility. It is important to note that genuine and meaningful comments are key to effective engagement. Spammy or irrelevant comments not only detract from the conversation but may also limit a post's potential reach and impact.
Read more →
Control break

In computer programming, a control break is a change in the value of one of the keys on which a file is sorted, which requires some extra processing. For example, with an input file sorted by post code, the number of items found in each postal district might need to be printed on a report, and a heading shown for the next district. Quite often there is a hierarchy of nested control breaks in a program, such as streets within districts within areas, with the need for a grand total at the end. Structured programming techniques have been developed to ensure correct processing of control breaks in languages such as COBOL and to ensure that conditions such as empty input files and sequence errors are handled properly. With fourth-generation languages such as SQL, the programming language should handle most of the details of control breaks automatically.
Read more →
PGP word list

The PGP Word List ("Pretty Good Privacy word list", also called a biometric word list for reasons explained below) is a list of words for conveying data bytes in a clear unambiguous way via a voice channel. They are analogous in purpose to the NATO phonetic alphabet, except that a longer list of words is used, each word corresponding to one of the 256 distinct numeric byte values. == History and structure == The PGP Word List was designed in 1995 by Patrick Juola, a computational linguist, and Philip Zimmermann, creator of PGP. The words were carefully chosen for their phonetic distinctiveness, using genetic algorithms to select lists of words that had optimum separations in phoneme space. The candidate word lists were randomly drawn from Grady Ward's Moby Pronunciator list as raw material for the search, successively refined by the genetic algorithms. The automated search converged to an optimized solution in about 40 hours on a DEC Alpha, a particularly fast machine in that era. The Zimmermann–Juola list was originally designed to be used in PGPfone, a secure VoIP application, to allow the two parties to verbally compare a short authentication string to detect a man-in-the-middle attack (MiTM). It was called a biometric word list because the authentication depended on the two human users recognizing each other's distinct voices as they read and compared the words over the voice channel, binding the identity of the speaker with the words, which helped protect against the MiTM attack. The list can be used in many other situations where a biometric binding of identity is not needed, so calling it a biometric word list may be imprecise. Later, it was used in PGP to compare and verify PGP public key fingerprints over a voice channel. This is known in PGP applications as the "biometric" representation. When it was applied to PGP, the list of words was further refined, with contributions by Jon Callas. More recently, it has been used in Zfone and the ZRTP protocol, the successor to PGPfone. The list is actually composed of two lists, each containing 256 phonetically distinct words, in which each word represents a different byte value between 0 and 255. Two lists are used because reading aloud long random sequences of human words usually risks three kinds of errors: 1) transposition of two consecutive words, 2) duplicate words, or 3) omitted words. To detect all three kinds of errors, the two lists are used alternately for the even-offset bytes and the odd-offset bytes in the byte sequence. Each byte value is actually represented by two different words, depending on whether that byte appears at an odd or an even offset from the beginning of the byte sequence. The two lists are readily distinguished by the number of syllables; the odd list has words of three syllables, the even list has two. The two lists have a maximum word length of 11 and 9 letters, respectively. Using a two-list scheme was suggested by Zhahai Stewart. == Examples == Each byte in a bytestring is encoded as a single word. A sequence of bytes is rendered in network byte order, from left to right. For example, the leftmost (i.e. byte 0) is considered "even" and is encoded using the PGP Even Word table. The next byte to the right (i.e. byte 1) is considered "odd" and is encoded using the PGP Odd Word table. This process repeats until all bytes are encoded. Thus, "E582" produces "topmost Istanbul", whereas "82E5" produces "miser travesty". A PGP public key fingerprint that displayed in hexadecimal as E582 94F2 E9A2 2748 6E8B 061B 31CC 528F D7FA 3F19 would display in PGP Words (the "biometric" fingerprint) as topmost Istanbul Pluto vagabond treadmill Pacific brackish dictator goldfish Medusa afflict bravado chatter revolver Dupont midsummer stopwatch whimsical cowbell bottomless The order of bytes in a bytestring depends on endianness. == Other word lists for data == There are several other word lists for conveying data in a clear unambiguous way via a voice channel: the NATO phonetic alphabet maps individual letters and digits to individual words the S/KEY system maps 64 bit numbers to 6 short words of 1 to 4 characters each from a publicly accessible 2048-word dictionary. The same dictionary is used in RFC 1760 and RFC 2289. the Diceware system maps five base-6 random digits (almost 13 bits of entropy) to a word from a dictionary of 7,776 distinct words. the Electronic Frontier Foundation has published a set of improved word lists based on the same concept FIPS 181: Automated Password Generator converts random numbers into somewhat pronounceable "words". mnemonic encoding converts 32 bits of data into 3 words from a vocabulary of 1626 words. what3words encodes geographic coordinates in 3 dictionary words. the BIP39 standard permits encoding a cryptographic key of fixed size (128 or 256 bits, usually the unencrypted master key of a Cryptocurrency wallet) into a short sequence of readable words known as the seed phrase, for the purpose of storing the key offline. This is used in cryptocurrencies such as Bitcoin or Monero. Like the PGP word list, the Bytewords standard maps each possible byte to a word. There is only one list, rather than two. The words are uniformly four letters long and can be uniquely identified by their first and last letters
Read more →
Pandas (software)

Pandas (styled as pandas) is a software library written for the Python programming language for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series. It is free software released under the three-clause BSD license. The name is derived from the term "panel data", an econometrics term for data sets that include observations over multiple time periods for the same individuals, as well as a play on the phrase "Python data analysis". Wes McKinney started building what would become Pandas at AQR Capital while he was a researcher there from 2007 to 2010. The development of Pandas introduced into Python many comparable features of working with DataFrames that were established in the R programming language. The library is built upon another library, NumPy. == History == Developer Wes McKinney started working on Pandas in 2008 while at AQR Capital Management out of the need for a high performance, flexible tool to perform quantitative analysis on financial data. Before leaving AQR, he was able to convince management to allow him to open source the library in 2009. Another AQR employee, Chang She, joined the effort in 2012 as the second major contributor to the library. In 2015, Pandas signed on as a fiscally sponsored project of NumFOCUS, a 501(c)(3) nonprofit charity in the United States. == Data model == Pandas is built around data structures called Series and DataFrames. Data for these collections can be imported from various file formats such as comma-separated values, JSON, Parquet, SQL database tables or queries, and Microsoft Excel. === Series === A Series is a one-dimensional array-like object that stores a sequence of values together with an associated set of labels, called an index. It is built on top of NumPy's array and affords many similar functionalities, but instead of using implicit integer positions, a Series allows explicit index labels of many data types. A Series can be created from Python lists, dictionaries, or NumPy arrays. If no index is provided, pandas automatically assigns a default integer index ranging from 0 to n-1, where n is the number of items in the Series. A simple example with customized labels is: To access a value or list of values from a Series, use its index or list of indices: Series can be used arithmetically, as in the statement series_3 = series_1 + series_2. This will align data points with corresponding index values in series_1 and series_2 (similar to a join in relational algebra), then add them together to produce new values in series_3. A Series has various attributes, such as name (Series name), dtype (data type of values), shape (number of rows), values, and index. They can be used in many of the same operations as NumPy arrays, with additional methods for reindexing, label-based selection, and handling missing data. === DataFrame === A DataFrame is a two-dimensional, tabular data structure with labeled rows and columns. Each column is stored internally as a Series and may hold a different data type (numeric, string, boolean, etc.). DataFrames can be created by a variety of means, including dictionaries of lists, NumPy arrays, and external files such as CSV or Excel spreadsheets: To retrieve a DataFrame column as a Series, use either 1) the index (dict-like notation) or 2) the name of column if the name is a valid Python identifier (attribute-like access). DataFrames support operations such as column assignment, row and column deletion, label-based indexing with loc, position-based indexing with iloc, reshaping, grouping, and joining. Merge operations implement a subset of relational algebra and allow one-to-one, many-to-one, and many-to-many joins. Some common attributes of a DataFrame include dtypes (data type of each column), shape (dimensions of the DataFrame returned as a tuple with form (number of rows, number of columns)), index/columns (labels of the DataFrame's rows/columns, respectively, returned as an Index object), values (data in the DataFrame returned as a 2D array), and empty (returns True if the DataFrame is empty). === Index === Index objects hold metadata for Series and Dataframe objects, such as axis labels and names, and are automatically created from input data. By default, a pandas index is a series of integers ascending from 0, similar to the indices of Python arrays. However, indices can also use any NumPy data type, including floating point, timestamps, or strings. Indices are also immutable, which allows them to be safely shared across multiple objects. pandas' syntax for mapping index values to relevant data is the same syntax Python uses to map dictionary keys to values. For example, if s is a Series, s['a'] will return the data point at index a. Unlike dictionary keys, index values are not guaranteed to be unique. If a Series uses the index value a for multiple data points, then s['a'] will instead return a new Series containing all matching values. A DataFrame's column names are stored and implemented identically to an index. As such, a DataFrame can be thought of as having two indices: one column-based and one row-based. Because column names are stored as an index, these are not required to be unique. If data is a Series, then data['a'] returns all values with the index value of a. However, if data is a DataFrame, then data['a'] returns all values in the column(s) named a. To avoid this ambiguity, Pandas supports the syntax data.loc['a'] as an alternative way to filter using the index. Pandas also supports the syntax data.iloc[n], which always takes an integer n and returns the nth value, counting from 0. This allows a user to act as though the index is an array-like sequence of integers, regardless of how it is actually defined. pandas also supports hierarchical indices with multiple values per data point through the "MultiIndex" class. MultiIndex objects allow a single DataFrame to represent multiple dimensions, similar to a pivot table in Microsoft Excel, where each level can optionally carry its own unique name. In practice, data with more than 2 dimensions is often represented using DataFrames with hierarchical indices, instead of the higher-dimension Panel and Panel4D data structures. == Functionality == pandas supports a variety of indexing and subsetting techniques, allowing data to be selected by label, index, or Boolean conditions. For example, df[df['col1'] > 5] will return all rows in the DataFrame df for which the value of the column col1 exceeds 5. The library also implements grouping operations based on the split-apply-combine approach, enabling users to aggregate, transform, or restructure data according to column values or functions applied to index labels. For example, df['col1'].groupby(df['col2']) groups the data in 'col1' by their values in 'col2', while df.groupby(lambda i: i % 2) groups all data in the whole DataFrame by whether their index is even. The library also provides extensive tools for transforming, filtering and summarizing data. Users may apply arbitrary functions to Series and DataFrames, and because the library is built on top of Numpy, most NumPy functions can be applied directly to pandas objects as well. The library also includes built-in operations for arithmetic operations, string processing, and descriptive statistics such as mean, median, and standard deviation. These built-in functions are designed to handle missing data, usually represented by the floating-point value NaN. In addition, pandas includes tools for reorganizing data into different structural formats, with methods that can reshape tabular data between "wide" and "long" formats and pivot values based on column labels. pandas also implements a flexible set of relational operations for combining datasets. For instance, merge() links row in DataFrames based on one or more shared keys or indices, supporting one-to-one, one-to-many, and many-to-many relationships in a manner analogous to join operations in relational databases like SQL. DataFrames can also be concatenated or stacked together along an axis through the concat() method, and overlapping data can be further spliced together using combine_first() to fill in missing values. Furthermore, the library includes specialized support for working with time-series data. Features include the ability to interpolate values and filter using a range of timestamps, such as data['1/1/2023':'2/2/2023'] , which will return all dates between January 1 and February 2. Missing values in time-series data are represented by a dedicated NaT (Not a Timestamp) object, instead of the NaN value it uses elsewhere. == Criticisms == Pandas has been criticized for its inefficiency. The entire dataset must be loaded in RAM, and the library does not optimize query plans or support parallel computing across multiple cores. Wes McKinney, the creator of Pandas, has recommended Apache Arrow as an alternative to address these performance concerns and ot
Read more →
Information leakage

Information leakage happens whenever a system that is designed to be closed to an eavesdropper reveals some information to unauthorized parties nonetheless. In other words: Information leakage occurs when secret information correlates with, or can be correlated with, observable information. For example, when designing an encrypted instant messaging network, a network engineer without the capacity to crack encryption codes could see when messages are transmitted, even if he could not read them. == Risk vectors == A modern example of information leakage is the leakage of secret information via data compression, by using variations in data compression ratio to reveal correlations between known (or deliberately injected) plaintext and secret data combined in a single compressed stream. Another example is the key leakage that can occur when using some public-key systems when cryptographic nonce values used in signing operations are insufficiently random. Bad randomness cannot protect proper functioning of a cryptographic system, even in a benign circumstance, it can easily produce crackable keys that cause key leakage. Information leakage can sometimes be deliberate: for example, an algorithmic converter may be shipped that intentionally leaks small amounts of information, in order to provide its creator with the ability to intercept the users' messages, while still allowing the user to maintain an illusion that the system is secure. This sort of deliberate leakage is sometimes known as a subliminal channel. Generally, only very advanced systems employ defenses against information leakage. Following are the commonly implemented countermeasures : Use steganography to hide the fact that a message is transmitted at all. Use chaffing to make it unclear to whom messages are transmitted (but this does not hide from others the fact that messages are transmitted). For busy re-transmitting proxies, such as a Mixmaster node: randomly delay and shuffle the order of outbound packets - this will assist in disguising a given message's path, especially if there are multiple, popular forwarding nodes, such as are employed with Mixmaster mail forwarding. When a data value is no longer going to be used, erase it from the memory.
Read more →
Intranet

An intranet is a computer network for sharing information, easier communication, collaboration tools, operational systems, and other computing services within an organization, usually to the exclusion of access by outsiders. The term is used in contrast to public networks, such as the Internet, but uses the same technology based on the Internet protocol suite. An organization-wide intranet can constitute a focal point of internal communication and collaboration, and provide a single starting point to access internal and external resources. In its simplest form, an intranet is established with the technologies for local area networks (LANs) and wide area networks (WANs). Many modern intranets have search engines, user profiles, blogs, mobile apps with notifications, and events planning within their infrastructure. An intranet is sometimes contrasted to an extranet. While an intranet is generally restricted to employees of the organization, extranets may also be accessed by customers, suppliers, or other approved parties. Extranets extend a private network onto the Internet with special provisions for authentication, authorization and accounting (AAA protocol). == Uses == Intranets are increasingly being used to deliver tools, such as for collaboration (to facilitate working in groups and teleconferencing) or corporate directories, sales and customer relationship management, or project management. Intranets are also used as corporate culture-change platforms. For example, a large number of employees using an intranet forum application to host a discussion about key issues could come up with new ideas related to management, productivity, quality, and other corporate issues. In large intranets, website traffic is often similar to public website traffic and can be better understood by using web metrics software to track overall activity. User surveys also improve intranet website effectiveness. Larger businesses allow users within their intranet to access public internet through firewall servers. They have the ability to screen incoming and outgoing messages, keeping security intact. When part of an intranet is made accessible to customers and others outside the business, it becomes part of an extranet. Businesses can send private messages through the public network using special encryption/decryption and other security safeguards to connect one part of their intranet to another. Intranet user-experience, editorial, and technology teams work together to produce in-house sites. Most commonly, intranets are managed by the communications, HR or CIO departments of large organizations, or some combination of these. Because of the scope and variety of content and the number of system interfaces, the intranets of many organizations are much more complex than their respective public websites. Intranets and the use of intranets are growing rapidly. According to the Intranet Design Annual 2007 from Nielsen Norman Group, the number of pages on participants' intranets averaged 200,000 over the years 2001 to 2003 and has grown to an average of 6 million pages over 2005–2007. == Benefits == Intranets can help users locate and view information faster and use applications relevant to their roles and responsibilities. With a web browser interface, users can access data held in any database the organization wants to make available at any time and — subject to security provisions — from anywhere within company workstations, increasing employees' ability to perform their jobs faster, more accurately, and with confidence that they have the right information. It also helps improve services provided to users. Using hypermedia and Web technology, Web publishing allows for the maintenance of and easy access to cumbersome corporate knowledge, such as employee manuals, benefits documents, company policies, business standards, news feeds, and even training, all of which can be accessed throughout a company using common Internet standards (Acrobat files, Flash files, CGI applications). Because each business unit can update the online copy of a document, the most recent version is usually available to employees using the intranet. Intranets are also used as a platform for developing and deploying applications to support business operations and decisions across the internetworked enterprise. Information is easily accessible to all authorised users, enabling collaboration. Being able to communicate in real-time through integrated third-party tools, such as an instant messenger, promotes the sharing of ideas and removes blockages to communication to help boost a business's productivity. Intranets can serve as powerful tools for communicating (such as through chat, email and/or blogs) within a given organization about vertically strategic initiatives that have a global reach throughout said organization. The type of information that can easily be conveyed is the purpose of the initiative and what it is aiming to achieve, who is driving it, results achieved to date, and whom to speak to for more information. By providing this information on the intranet, staff can keep up-to-date with the strategic focus of their organization. For example, when Nestlé had a number of food processing plants in Scandinavia, their central support system had to deal with a number of queries every day. When Nestlé decided to invest in an intranet, they quickly realized the savings. Gerry McGovern says that the savings from the reduction in query calls was substantially greater than the investment in the intranet. Users can view information and data via a web browser rather than maintaining physical documents such as procedure manuals, internal phone list and requisition forms. This can potentially save the business money on printing, duplicating documents, and the environment, as well as document maintenance overhead. For example, the HRM company PeopleSoft "derived significant cost savings by shifting HR processes to the intranet". McGovern goes on to say the manual cost of enrolling in benefits was found to be US$109.48 per enrollment. "Shifting this process to the intranet reduced the cost per enrollment to $21.79; a saving of 80 percent". Another company that saved money on expense reports was Cisco. "In 1996, Cisco processed 54,000 reports and the amount of dollars processed was USD19 million". Many companies dictate computer specifications which, in turn, may allow Intranet developers to write applications that only have to work on one browser such that there are no cross-browser compatibility issues. Being able to specifically address one's "viewer" is a great advantage. Since intranets are user-specific (requiring database/network authentication prior to access), users know exactly who they are interfacing with and can personalize their intranet based on role (job title, department) or individual ("Congratulations Jane, on your 3rd year with our company!"). Since "involvement in decision making" is one of the main drivers of employee engagement, offering tools (like forums or surveys) that foster peer-to-peer collaboration and employee participation can make employees feel more valued and involved. == Planning and creation == Most organizations devote considerable resources into the planning and implementation of their intranet as it is of strategic importance to the organization's success. Some of the planning would include topics such as determining the purpose and goals of the intranet, identifying persons or departments responsible for implementation and management and devising functional plans, page layouts and designs. The appropriate staff would also ensure that implementation schedules and phase-out of existing systems were organized, while defining and implementing security of the intranet and ensuring it lies within legal boundaries and other constraints. In order to produce a high-value end product, systems planners should determine the level of interactivity (e.g. wikis, on-line forms) desired. Planners may also consider whether the input of new data and updating of existing data is to be centrally controlled or devolve. These decisions sit alongside to the hardware and software considerations (like content management systems), participation issues (like good taste, harassment, confidentiality), and features to be supported. Intranets are often static sites; they are a shared drive, serving up centrally stored documents alongside internal articles or communications (often one-way communication). By leveraging firms which specialise in 'social' intranets, organisations are beginning to think of how their intranets can become a 'communication hub' for their entire team. The actual implementation would include steps such as securing senior management support and funding, conducting a business requirement analysis and identifying users' information needs. From the technical perspective, there would need to be a coordinated installation of the web server and user access netw
Read more →
Batch cryptography

Batch cryptography is a field of cryptology focused on the design of cryptographic protocols that perform operations—such as encryption, decryption, key exchange, and authentication—on multiple inputs simultaneously, rather than processing each input individually. Batching cryptographic operations can significantly reduce the marginal cost of handling individual inputs—a principle that was first introduced by Amos Fiat in 1989.
Read more →
Teaspiller

Teaspiller was a US-based web application for customers to find accountants and hire them to do their taxes and accounting online. In 2013 the company was acquired by Intuit, Inc and added to its TurboTax product line. The Teaspiller employees and code were all acquired and the product was renamed as "TurboTax CPA select". It enabled accountants to work remotely with clients (share files, send secure messages, schedule appointments), as well as find new clients looking for their specific skills through a complex search algorithm. This was done through extended profiles containing licensing information, professional histories, user ratings, peer endorsements, association memberships, and practice areas. The service had been called an H&R Block killer by Business Insider as it helped customers find accountants to prepare tax returns online. As of 2011 it had 20,000 US accountants listed on the site. The application was built using the Django framework. == History == Teaspiller was built by Vemdara, LLC, a web company based in New York and founded in 2009 by Amit Vemuri (a former VP at Travelocity). The web application was launched in 2010. In 2013 the company was acquired by Intuit as part of their TurboTax product line and renamed as "TurboTax CPA select".
Read more →
Weird SoundCloud

Weird SoundCloud, or SoundClown, is a mashup parody music scene taking place on the online distribution platform SoundCloud. The scene has been described by its producers and music journalists to be a satirical take on electronic dance music, and useless, throwaway internet content. One critic, Audra Schroeder, categorized it as an in-joke that is "deconstructing and reshaping memes and popular music, recontextualizing the sacred texts of millennial chat rooms." == Origins == In a January 2014 interview, DJ Kevin Wang suggested that the Weird SoundCloud has "been around in the last one to two years", but started to gain much more popularity the previous year through electronic dance music internet blogs. Weird SoundCloud producer Ideaot suggested that some in the phenomenon came from the YouTube poop scene. Another producer in the community, DJ @@ (AT-AT), reasoned that producers joining the scene "want to express their musicality, see it as a more mature form of YouTube Poop," or are "just looking for recognition on social media sites." AT-AT said that it was "a fun thing to do, and after I stopped making proper music I felt I needed a bit of an outlet for my creativity. The fact that people enjoyed it and/or treated it as a travesty (Direct quote from one of my tracks) spurs me on." == Characteristics == Weird SoundCloud is a mash-up and parody music genre labeled by journalist Audra Schroeder as an in-joke that is "deconstructing and reshaping memes and popular music, recontextualizing the sacred texts of millennial chat rooms." Most tracks range from around 30 seconds to one minute in length. The people who make weird SoundCloud are known as SoundClowns, a term coined by producer Dicksoak. Ideaot described the weird SoundCloud community as "largely just people who are friends with each other." Noisey critic Ryan Bassil spotlight the variety of music coming out of the weird SoundCloud landscape: "One minute you could be listening to the Seinfeld theme reimagined as an aneurysm inducing dubstep corker, the next, you're recovering from hearing a version of Tenacious D's "Tribute" that's akin to having a stroke." Bassil analyzes that the tracks "often take the past and repurpose it into something that, although not altogether useful, sounds fresh and reflective of the abstract, confusing panoramic that encapsulates the modern internet." Bassil compared the lexicon of SoundClown's track titles to that of Reddit and Twitter users. According to Dicksoak, most works of the style are critiques of EDM or "are just uploaded because they sound funny." However, Bassil disagreed, writing that there are also many tracks that keep repurposing a certain meme, such as "mom's spaghetti" or the re-use of vocals from recordings by hip hop group Death Grips. He describe the scene's re-use of memes as a satirical take on pointless online content that is only on the internet to "do nothing other than fill the void": They're changing the format of the original work's intended message or audience - a technique often employed by top-tier digital media companies - and in doing so they're sarcastically, ironically, taking the piss out of what Web 2.0's turned into - an open arena where the most ridiculous, unashamed, often pointless piggy-back content can rack up thousands and thousands of clicks. == Notable examples == There are mash-ups that "disrupt the flow of popular music", in the words of writer Schroeder, such as a "flutedrop" remix of the Miley Cyrus song "Wrecking Ball" and Shaliek's mashup of music by Bruno Mars and Korn. In November 2013, Wang released a set of mp3 files on SoundCloud named Best Drops Ever, which included tracks like "A Drop So Epic a Bunch of NYU Bros Already Bought a 3-Day Weekend Pass for It" and "A Drop So Crazy You'll Kill Your Family". All of the tracks start as normal electronic dance music build-ups, before they drop into a "bait and switch" audio or film clip such as Filet-O-Fish commercials, the Whitney Houston song "I Will Always Love You" and the film Bambi (1942) that ruins the anticipation. The collection is a parody of the over-importance and over-focus of the drop and lack of care of the overall quality of a song common in the modern electronic dance music scene. Wang has released more than 45 tracks in the weird SoundCloud, some of them receiving around a million plays. Subgenres of Weird SoundCloud include Macklecore, mash-ups and remixes that include the works of American hip-hop recording artist Macklemore, and Biggiewave, which include samples of songs from the album Ready to Die (1994) by The Notorious B.I.G. Common audio and meme sources used include Skrillex, the Martin Garrix track "Animals", Thomas the Tank Engine, Shrek, Macklemore, "Gangnam Style", the Bruno Mars track "Uptown Funk", the Disturbed track "Down with the Sickness", Space Jam, the Childish Gambino track "Bonfire", the Death Grips track "Takyon" and air horn sound effects. == Reception == Bassil praised the SoundClown scene as "loveable and strangely honest", reasoning that it "just reminds me that we're all humans on the internet, all searching for #content that means something, something to connect with, but usually only dredging up bastardised versions of things we've already read, seen, or watched before." Bassil also described the weird SoundCloud as a more successful version of a similar scene known as weird YouTube; the reason for the success of SoundClowns is due to SoundCloud's discovery algorithm: "Small collectives and trends are able to form, and there's an abundance of tracks from artists who are almost forging careers out of it, as opposed to uploading one viral hit." Publications have made lists of weird SoundCloud works, such as BuzzFeed's "23 Of The Weirdest Songs On Soundcloud", Obsev's "Weird SoundCloud Mashups That Must've Been Made While Drunk", and Thump's "9 of the Best and Most Upsetting Soundclowns we Could Find", where writer Isabelle Hellyer called it the "most influential genre of music in human history." A Your EDM writer called it "oddly addicting."
Read more →
Knapsack cryptosystems

Knapsack cryptosystems are cryptosystems whose security is based on the hardness of solving the knapsack problem. They remain quite unpopular because simple versions of these algorithms have been broken for several decades. However, that type of cryptosystem is a good candidate for post-quantum cryptography. The most famous knapsack cryptosystem is the Merkle-Hellman Public Key Cryptosystem, one of the first public key cryptosystems, published the same year as the RSA cryptosystem. However, this system has been broken by several attacks: one from Shamir, one by Adleman, and the low density attack. However, there exist modern knapsack cryptosystems that are considered secure so far: among them is Nasako-Murakami 2006. Knapsack cryptosystems, when not subject to classical cryptoanalysis, are believed to be difficult even for quantum computers. That is not the case for systems that rely on factoring large integers, like RSA, or computing discrete logarithms, like ECDSA, problems solved in polynomial time with Shor's algorithm.
Read more →
ESign (India)

Aadhaar eSign is an online electronic signature service in India to facilitate an Aadhaar holder to digitally sign a document. The signature service is facilitated by authenticating the Aadhaar holder via the Aadhaar-based e-KYC (electronic Know Your Customer) service. To eSign a document, one has to have an Aadhaar card and a mobile number registered with Aadhaar. With these two things, an Indian citizen can sign a document remotely without being physically present. == Procedure == The notification issued by Government of India in this regard stipulates the following procedure for the e-authentication using Aadhaar e-KYC services. Authentication of an electronic record by e-authentication technique, which shall be done by the applicable use of e-authentication, hash function, and asymmetric cryptosystem techniques, leading to issuance of digital signature certificate by Certifying Authority, a trusted third party service by subscriber's key pair generation, storing of the key pairs on hardware security module and creation of digital signature provided that the trusted third party shall be offered by the certifying authority (the trusted third party shall send application form and certificate signing request to the Certifying Authority for issuing a digital signature certificate to the subscriber), issuance of digital signature certificate by Certifying Authority shall be based on e-authentication, particulars given in the prescribed format, digitally signed verified information from Aadhaar e-KYC services and electronic consent of digital signature certificate applicant, the manner and requirements for e-authentication shall be as issued by the Controller from time to time, the security procedure for creating the subscriber's key pair shall be in accordance with the e-authentication guidelines issued by the Controller, the standards referred to in rule 6 of the Information Technology (Certifying Authorities) Rules, 2000 shall be complied with, in so far as they relate to the certification function of public key of Digital Signature Certificate applicant, and the manner in which information is authenticated by means of digital signature shall comply with the standards specified in rule 6 of the Information Technology (Certifying Authorities) Rules, 2000 in so far as they relate to the creation, storage and transmission of Digital Signature. == eSign Service Providers == Organisations and individuals seeking to obtain the eSigning Service can utilize the services of various service providers. There are empanelled service providers with whom organisations can register as an Application Service Prover after submitting the requisite documents, getting UAT access, building the application around the service and going through an IT Audit by an CERT-IN empanelled auditor. However, the process of registering as an Application Service Provider is cumbersome, and requires huge investments of time, money and resources in complying with the regulations and building a suitable application. Most organisations prefer using services of plug-n-play gateway providers who take the responsibility of complying with the regulations, hence simplifying the process for the market.
Read more →
Automation

Automation describes a wide range of technologies that reduce human intervention in processes, mainly by predetermining decision criteria, subprocess relationships, and related actions, as well as embodying those predeterminations in machines. Automation has been achieved by various means including mechanical, hydraulic, pneumatic, electrical, electronic devices, and computers, usually in combination. Complicated systems, such as modern factories, airplanes, and ships typically use combinations of all of these techniques. The benefits of automation includes labor savings, reducing waste, savings in electricity costs, savings in material costs, and improvements to quality, accuracy, and precision. Automation includes the use of various equipment and control systems such as machinery, processes in factories, boilers, and heat-treating ovens, switching on telephone networks, steering, stabilization of ships, aircraft and other applications and vehicles with reduced human intervention. Examples range from a household thermostat controlling a boiler to a large industrial control system with tens of thousands of input measurements and output control signals. In the simplest type of an automatic control loop, a controller compares a measured value of a process with a desired set value and processes the resulting error signal to change some input to the process, in such a way that the process stays at its set point despite disturbances. This closed-loop control is an application of negative feedback to a system. The mathematical basis of control theory began in the 18th century and advanced rapidly in the 20th. The term automation, inspired by the earlier word automatic (coming from automaton), was not widely used before 1947, when Ford established an automation department. It was during this time that the industry was rapidly adopting feedback controllers, Technological advancements introduced in the 1930s revolutionized various industries significantly. The World Bank's World Development Report of 2019 shows evidence that the new industries and jobs in the technology sector outweigh the economic effects of workers being displaced by automation. Job losses and downward mobility blamed on automation have been cited as one of many factors in the resurgence of nationalist, protectionist and populist politics in the US, UK and France, among other countries since the 2010s. == History == === Early history === It was a preoccupation of the Greeks and Arabs (in the period between about 300 BC and about 1200 AD) to keep an accurate track of time. In Ptolemaic Egypt, about 270 BC, Ctesibius described a float regulator for a water clock, a device not unlike the ball and cock in a modern flush toilet. This was the earliest feedback-controlled mechanism. The appearance of the mechanical clock in the 14th century made the water clock and its feedback control system obsolete. The Persian Banū Mūsā brothers, in their Book of Ingenious Devices (850 AD), described a number of automatic controls. Two-step level controls for fluids, a form of discontinuous variable structure controls, were developed by the Banu Musa brothers. They also described a feedback controller. The design of feedback control systems up through the Industrial Revolution was by trial-and-error, together with a great deal of engineering intuition. It was not until the mid-19th century that the stability of feedback control systems was analyzed using mathematics, the formal language of automatic control theory. The centrifugal governor was invented by Christiaan Huygens in the seventeenth century, and used to adjust the gap between millstones. === Industrial Revolution in Western Europe === The introduction of prime movers, or self-driven machines advanced grain mills, furnaces, boilers, and the steam engine created a new requirement for automatic control systems including temperature regulators (invented in 1624; see Cornelius Drebbel), pressure regulators (1681), float regulators (1700) and speed control devices. Another control mechanism was used to tent the sails of windmills. It was patented by Edmund Lee in 1745. Also in 1745, Jacques de Vaucanson invented the first automated loom. Around 1800, Joseph Marie Jacquard created a punch-card system to program looms. In 1771 Richard Arkwright invented the first fully automated spinning mill driven by water power, known at the time as the water frame. An automatic flour mill was developed by Oliver Evans in 1785, making it the first completely automated industrial process. A centrifugal governor was used by Mr. Bunce of England in 1784 as part of a model steam crane. The centrifugal governor was adopted by James Watt for use on a steam engine in 1788 after Watt's partner Boulton saw one at a flour mill Boulton & Watt were building. The governor could not actually hold a set speed; the engine would assume a new constant speed in response to load changes. The governor was able to handle smaller variations such as those caused by fluctuating heat load to the boiler. Also, there was a tendency for oscillation whenever there was a speed change. As a consequence, engines equipped with this governor were not suitable for operations requiring constant speed, such as cotton spinning. Several improvements to the governor, plus improvements to valve cut-off timing on the steam engine, made the engine suitable for most industrial uses before the end of the 19th century. Advances in the steam engine stayed well ahead of science, both thermodynamics and control theory. The governor received relatively little scientific attention until James Clerk Maxwell published a paper that established the beginning of a theoretical basis for understanding control theory. === 20th century === Relay logic was introduced with factory electrification, which underwent rapid adaptation from 1900 through the 1920s. Central electric power stations were also undergoing rapid growth and the operation of new high-pressure boilers, steam turbines and electrical substations created a great demand for instruments and controls. Central control rooms became common in the 1920s, but as late as the early 1930s, most process controls were on-off. Operators typically monitored charts drawn by recorders that plotted data from instruments. To make corrections, operators manually opened or closed valves or turned switches on or off. Control rooms also used color-coded lights to send signals to workers in the plant to manually make certain changes. The development of the electronic amplifier during the 1920s, which was important for long-distance telephony, required a higher signal-to-noise ratio, which was solved by negative feedback noise cancellation. This and other telephony applications contributed to the control theory. In the 1940s and 1950s, German mathematician Irmgard Flügge-Lotz developed the theory of discontinuous automatic controls, which found military applications during the Second World War to fire control systems and aircraft navigation systems. Controllers, which were able to make calculated changes in response to deviations from a set point rather than on-off control, began being introduced in the 1930s. Controllers allowed manufacturing to continue showing productivity gains to offset the declining influence of factory electrification. Factory productivity was greatly increased by electrification in the 1920s. U.S. manufacturing productivity growth fell from 5.2%/yr 1919–29 to 2.76%/yr 1929–41. Alexander Field notes that spending on non-medical instruments increased significantly from 1929 to 1933 and remained strong thereafter. The First and Second World Wars saw major advancements in the field of mass communication and signal processing. Other key advances in automatic controls include differential equations, stability theory and system theory (1938), frequency domain analysis (1940), ship control (1950), and stochastic analysis (1941). Starting in 1958, various systems based on solid-state digital logic modules for hard-wired programmed logic controllers (the predecessors of programmable logic controllers [PLC]) emerged to replace electro-mechanical relay logic in industrial control systems for process control and automation, including early Telefunken/AEG Logistat, Siemens Simatic, Philips/Mullard/Valvo Norbit, BBC Sigmatronic, ACEC Logacec, Akkord Estacord, Krone Mibakron, Bistat, Datapac, Norlog, SSR, or Procontic systems. In 1959 Texaco's Port Arthur Refinery became the first chemical plant to use digital control. Conversion of factories to digital control began to spread rapidly in the 1970s as the price of computer hardware fell. === Significant applications === The automatic telephone switchboard was introduced in 1892 along with dial telephones. By 1929, 31.9% of the Bell system was automatic. Automatic telephone switching originally used vacuum tube amplifiers and electro-mechanical switches, which consumed a large amount of electricity. Call volume eve
Read more →
Data

Data ( DAY-tə, US also DAT-ə, India: DEE-tə) is a collection of discrete or continuous values that conveys information, describing the quantity, quality, fact, statistics, other basic units of meaning, or simply sequences of symbols that may be further interpreted formally. A data point or datum is an individual value in a collection of data. Data is usually organized into structures such as tables that provide additional context and meaning, and may itself be used as data in larger structures. Data may be used as variables in a computational process. Data may represent abstract ideas or concrete measurements. Data is commonly used in scientific research, economics, and virtually every other form of human organizational activity. Examples of data sets include price indices (such as the consumer price index), unemployment rates, literacy rates, and census data. In this context, data represents the raw facts and figures from which useful information can be extracted. Data is collected using techniques such as measurement, observation, query, or analysis, and is typically represented as numbers or characters that may be further processed. Field data is data that is collected in an uncontrolled, in-situ environment. Experimental data is data that is generated in the course of a controlled scientific experiment. Data is analyzed using techniques such as calculation, reasoning, discussion, presentation, visualization, or other forms of post-analysis. Prior to analysis, raw data (or unprocessed data) is typically cleaned: Outliers are removed, and obvious instrument or data entry errors are corrected. Data can be seen as the smallest unit of factual information that can be used as a basis for calculation, reasoning, or discussion. Data can range from abstract ideas to concrete measurements, including, but not limited to, statistics. Thematically connected data presented in some relevant context can be viewed as information. Contextually connected pieces of information can then be described as data insights or intelligence. The stock of insights and intelligence that accumulate over time, resulting from the synthesis of data into information, can then be described as knowledge. Data has been described as "the new oil of the digital economy". Data, as a general concept, refers to the fact that some existing information or knowledge is represented or coded in some form suitable for better usage or processing. Advances in computing technologies have led to the advent of big data, which generally refers to very large quantities of data, typically at the petabyte scale. If restricted to traditional data analysis methods and computing, working with such large (and growing) datasets is difficult, even impossible. In response, the relatively new field of data science uses machine learning (and other artificial intelligence) methods that allow for efficient applications of analytic methods to big data. == Etymology and terminology == The Latin word data is the plural of datum, "(thing) given," and the neuter past participle of dare, "to give". The first English use of the word "data" is from the 1640s. The word "data" was first used to mean "transmissible and storable computer information" in 1946. The expression "data processing" was first used in 1954. When "data" is used more generally as a synonym for "information", it is treated as a mass noun in singular form. This usage is common in everyday language and in technical and scientific fields such as software development and computer science. One example of this usage is the term "big data". When used more specifically to refer to the processing and analysis of sets of data, the term retains its plural form. This usage is common in the natural sciences, life sciences, social sciences, software development and computer science, and grew in popularity in the 20th and 21st centuries. Some style guides do not recognize the different meanings of the term and simply recommend the form that best suits the target audience of the guide. For example, APA style as of the 7th edition requires "data" to be treated as a plural form. == Meaning == Data, information, knowledge, and wisdom are closely related concepts, but each has its role concerning the other, and each term has its meaning. According to a common view, data is collected and analyzed; data only becomes information suitable for making decisions once it has been analyzed in some fashion. One can say that the extent to which a set of data is informative to someone depends on the extent to which it is unexpected by that person. The amount of information contained in a data stream may be characterized by its Shannon entropy. Knowledge is the awareness of its environment that some entity possesses, whereas data merely communicates that knowledge. For example, the entry in a database specifying the height of Mount Everest is a datum that communicates a precisely measured value. This measurement may be included in a book along with other data on Mount Everest to describe the mountain in a manner useful for those who wish to decide on the best method to climb it. Awareness of the characteristics represented by this data is knowledge. Data are often assumed to be the least abstract concept, information the next least, and knowledge the most abstract. In this view, data becomes information by interpretation; e.g., the height of Mount Everest is generally considered "data", a book on Mount Everest geological characteristics may be considered "information", and a climber's guidebook containing practical information on the best way to reach Mount Everest's peak may be considered "knowledge". "Information" bears a diversity of meanings that range from everyday usage to technical use. This view, however, has also been argued to reverse how data emerges from information, and information from knowledge. Generally speaking, the concept of information is closely related to notions of constraint, communication, control, data, form, instruction, knowledge, meaning, mental stimulus, pattern, perception, and representation. Beynon-Davies uses the concept of a sign to differentiate between data and information; data is a series of symbols, while information occurs when the symbols are used to refer to something. Before the development of computing devices and machines, people had to manually collect data and impose patterns on it. With the development of computing devices and machines, these devices can also collect data. In the 2010s, computers were widely used in many fields to collect data and sort or process it, in disciplines ranging from marketing, analysis of social service usage by citizens to scientific research. These patterns in the data are seen as information that can be used to enhance knowledge. These patterns may be interpreted as "truth" (though "truth" can be a subjective concept) and may be authorized as aesthetic and ethical criteria in some disciplines or cultures. Events that leave behind perceivable physical or virtual remains can be traced back through data. Marks are no longer considered data once the link between the mark and observation is broken. Mechanical computing devices are classified according to how they represent data. An analog computer represents a datum as a voltage, distance, position, or other physical quantity. A digital computer represents a piece of data as a sequence of symbols drawn from a fixed alphabet. The most common digital computers use a binary alphabet, that is, an alphabet of two characters typically denoted "0" and "1". More familiar representations, such as numbers or letters, are then constructed from the binary alphabet. Some special forms of data are distinguished. A computer program is a collection of data, that can be interpreted as instructions. Most computer languages make a distinction between programs and the other data on which programs operate, but in some languages, notably Lisp and similar languages, programs are essentially indistinguishable from other data. It is also useful to distinguish metadata, that is, a description of other data. A similar yet earlier term for metadata is "ancillary data." The prototypical example of metadata is the library catalog, which is a description of the contents of books. == Data sources == With respect to ownership of data collected in the course of marketing or other corporate collection, data has been characterized according to party depending on how close the data is to the source or if it has been generated through additional processing. "Zero-party data" refers to data that customers "intentionally and proactively shares". This kind of data can come from a variety of sources, including: subscriptions, preference centers, quizzes, surveys, pop-up forms, and interactive digital experiences. "First-party data" may be collected by a company directly from its customers. The secure exchange of first-party data among companies can be done using data clean rooms. "S
Read more →
Data transformation (computing)

In computing, data transformation is the process of converting data from one format or structure into another format or structure. It is a fundamental aspect of most data integration and data management tasks such as data wrangling, data warehousing, data integration and application integration. Data transformation can be simple or complex based on the required changes to the data between the source (initial) data and the target (final) data. Data transformation is typically performed via a mixture of manual and automated steps. Tools and technologies used for data transformation can vary widely based on the format, structure, complexity, and volume of the data being transformed. A master data recast is another form of data transformation where the entire database of data values is transformed or recast without extracting the data from the database. All data in a well-designed database is directly or indirectly related to a limited set of master database tables by a network of foreign key constraints. Each foreign key constraint is dependent upon a unique database index from the parent database table. Therefore, when the proper master database table is recast with a different unique index, the directly and indirectly related data are also recast or restated. The directly and indirectly related data may also still be viewed in the original form since the original unique index still exists with the master data. Also, the database recast must be done in such a way as to not impact the applications architecture software. When the data mapping is indirect via a mediating data model, the process is also called data mediation. == Data transformation process == Data transformation can be divided into the following steps, each applicable as needed based on the complexity of the transformation required. Data discovery Data mapping Code generation Code execution Data review These steps are often the focus of developers or technical data analysts who may use multiple specialized tools to perform their tasks. The steps can be described as follows: Data discovery is the first step in the data transformation process. Typically the data is profiled using profiling tools or sometimes using manually written profiling scripts to better understand the structure and characteristics of the data and decide how it needs to be transformed. Data mapping is the process of defining how individual fields are mapped, modified, joined, filtered, aggregated etc. to produce the final desired output. Developers or technical data analysts traditionally perform data mapping since they work in the specific technologies to define the transformation rules (e.g. visual ETL tools, transformation languages). Code generation is the process of generating executable code (e.g. SQL, Python, R, or other executable instructions) that will transform the data based on the desired and defined data mapping rules. Typically, the data transformation technologies generate this code based on the definitions or metadata defined by the developers. Code execution is the step whereby the generated code is executed against the data to create the desired output. The executed code may be tightly integrated into the transformation tool, or it may require separate steps by the developer to manually execute the generated code. Data review is the final step in the process, which focuses on ensuring the output data meets the transformation requirements. It is typically the business user or final end-user of the data that performs this step. Any anomalies or errors in the data that are found and communicated back to the developer or data analyst as new requirements to be implemented in the transformation process. == Types of data transformation == === Batch data transformation === Traditionally, data transformation has been a bulk or batch process, whereby developers write code or implement transformation rules in a data integration tool, and then execute that code or those rules on large volumes of data. This process can follow the linear set of steps as described in the data transformation process above. Batch data transformation is the cornerstone of virtually all data integration technologies such as data warehousing, data migration and application integration. When data must be transformed and delivered with low latency, the term "microbatch" is often used. This refers to small batches of data (e.g. a small number of rows or a small set of data objects) that can be processed very quickly and delivered to the target system when needed. === Benefits of batch data transformation === Traditional data transformation processes have served companies well for decades. The various tools and technologies (data profiling, data visualization, data cleansing, data integration etc.) have matured and most (if not all) enterprises transform enormous volumes of data that feed internal and external applications, data warehouses and other data stores. === Limitations of traditional data transformation === This traditional process also has limitations that hamper its overall efficiency and effectiveness. The people who need to use the data (e.g. business users) do not play a direct role in the data transformation process. Typically, users hand over the data transformation task to developers who have the necessary coding or technical skills to define the transformations and execute them on the data. This process leaves the bulk of the work of defining the required transformations to the developer, which often in turn do not have the same domain knowledge as the business user. The developer interprets the business user requirements and implements the related code/logic. This has the potential of introducing errors into the process (through misinterpreted requirements), and also increases the time to arrive at a solution. This problem has given rise to the need for agility and self-service in data integration (i.e. empowering the user of the data and enabling them to transform the data themselves interactively). There are companies that provide self-service data transformation tools. They are aiming to efficiently analyze, map and transform large volumes of data without the technical knowledge and process complexity that currently exists. While these companies use traditional batch transformation, their tools enable more interactivity for users through visual platforms and easily repeated scripts. Still, there might be some compatibility issues (e.g. new data sources like IoT may not work correctly with older tools) and compliance limitations due to the difference in data governance, preparation and audit practices. === Interactive data transformation === Interactive data transformation (IDT) is an emerging capability that allows business analysts and business users the ability to directly interact with large datasets through a visual interface, understand the characteristics of the data (via automated data profiling or visualization), and change or correct the data through simple interactions such as clicking or selecting certain elements of the data. Although interactive data transformation follows the same data integration process steps as batch data integration, the key difference is that the steps are not necessarily followed in a linear fashion and typically don't require significant technical skills for completion. There are a number of companies that provide interactive data transformation tools, including Trifacta, Alteryx and Paxata. They are aiming to efficiently analyze, map and transform large volumes of data while at the same time abstracting away some of the technical complexity and processes which take place under the hood. Interactive data transformation solutions provide an integrated visual interface that combines the previously disparate steps of data analysis, data mapping and code generation/execution and data inspection. That is, if changes are made at one step (like for example renaming), the software automatically updates the preceding or following steps accordingly. Interfaces for interactive data transformation incorporate visualizations to show the user patterns and anomalies in the data so they can identify erroneous or outlying values. Once they've finished transforming the data, the system can generate executable code/logic, which can be executed or applied to subsequent similar data sets. By removing the developer from the process, interactive data transformation systems shorten the time needed to prepare and transform the data, eliminate costly errors in the interpretation of user requirements and empower business users and analysts to control their data and interact with it as needed. == Transformational languages == There are numerous languages available for performing data transformation. Many transformation languages require a grammar to be provided. In many cases, the grammar is structured using something closely resembling Backus–Naur form (BNF). There are numerous languages
Read more →