AI Data Analyst Zalando

AI Data Analyst Zalando — independent reviews, comparisons, pricing and step-by-step guides on Aizhi.

• 

  Round-trip engineering

  Round-trip engineering (RTE) in the context of model-driven architecture is a functionality of software development tools that synchronizes two or more related software artifacts, such as, source code, models, configuration files, documentation, etc. between each other. The need for round-trip engineering arises when the same information is present in multiple artifacts and when an inconsistency may arise in case some artifacts are updated. For example, some piece of information was added to/changed in only one artifact (source code) and, as a result, it became missing in/inconsistent with the other artifacts (in models). == Overview == Round-trip engineering is closely related to traditional software engineering disciplines: forward engineering (creating software from specifications), reverse engineering (creating specifications from existing software), and reengineering (understanding existing software and modifying it). Round-trip engineering is often wrongly defined as simply supporting both forward and reverse engineering. In fact, the key characteristic of round-trip engineering that distinguishes it from forward and reverse engineering is the ability to synchronize existing artifacts that evolved concurrently by incrementally updating each artifact to reflect changes made to the other artifacts. Furthermore, forward engineering can be seen as a special instance of RTE in which only the specification is present and reverse engineering can be seen as a special instance of RTE in which only the software is present. Many reengineering activities can also be understood as RTE when the software is updated to reflect changes made to the previously reverse engineered specification. === Types === Various books describe two types of RTE: partial or uni-directional RTE: changes made to a higher level representation of a code and model are reflected in lower level, but not otherwise; the latter might be allowed, but with limitations that may not affect higher-level abstractions full or bi-directional RTE: regardless of changes, both higher and lower-level code and model representations are synchronized if any of them altered === Auto synchronization === Another characteristic of round-trip engineering is automatic update of the artifacts in response to automatically detected inconsistencies. In that sense, it is different from forward- and reverse engineering which can be both manual (traditionally) and automatic (via automatic generation or analysis of the artifacts). The automatic update can be either instantaneous or on-demand. In instantaneous RTE, all related artifacts are immediately updated after each change made to one of them. In on-demand RTE, authors of the artifacts may concurrently update the artifacts (even in a distributed setting) and at some point choose to execute matching to identify inconsistencies and choose to propagate some of them and reconcile potential conflicts. === Iterative approach === Round trip engineering may involve an iterative development process. After you have synchronized your model with revised code, you are still free to choose the best way to work – make further modifications to the code or make changes to your model. You can synchronize in either direction at any time and you can repeat the cycle as many times as necessary. == Software == Many commercial tools and research prototypes support this form of RTE; a 2007 book lists Rational Rose, Together, ESS-Model, BlueJ, and Fujaba among those capable, with Fujaba said to be capable to also identify design patterns. == Limitations == A 2005 book on Visual Studio notes for instance that a common problem in RTE tools is that the model reversed is not the same as the original one, unless the tools are aided by leaving laborious annotations in the source code. The behavioral parts of UML impose even more challenges for RTE. Usually, UML class diagrams are supported to some degree; however, certain UML concepts, such as associations and containment do not have straightforward representations in many programming languages which limits the usability of the created code and accuracy of code analysis/reverse engineering (e.g., containment is hard to recognize in the code). A more tractable form of round-trip engineering is implemented in the context of framework application programming interfaces (APIs), whereby a model describing the usage of a framework API by an application is synchronized with that application's code. In this setting, the API prescribes all correct ways the framework can be used in applications, which allows precise and complete detection of API usages in the code as well as creation of useful code implementing correct API usages. Two prominent RTE implementations in this category are framework-specific modeling languages and Spring Roo (Java). Round-trip engineering is critical for maintaining consistency among multiple models and between the models and the code in Object Management Group's (OMG) Model-driven architecture. OMG proposed the QVT (query/view/transformation) standard to handle model transformations required for MDA. To date, a few implementations of the standard have been created. (Need to present practical experiences with MDA in relation to RTE). == Controversies == === Code generation controversy === Code generation (forward-engineering) from models means that the user abstractly models solutions, which are connoted by some model data, and then an automated tool derives from the models parts or all of the source code for the software system. In some tools, the user can provide a skeleton of the program source code, in the form of a source code template where predefined tokens are then replaced with program source code parts during the code generation process. UML (if used for MDA) diagrams specification was criticized for lack the detail which is needed to contain the same information as is covered with the program source. Some developers even claim that "the Code is the design". == Disadvantages == There is a serious risk that the generated code will rapidly differ from the model or that the reverse-engineered model will lose its reflection on the code or a mix of these two problems as result of cycled reengineering efforts. Regarding behavioral/dynamic part of UML for features like statechart diagram there is no equivalents in programming languages. Their translation during code-generation will result in common programming statement (.e.g if,switch,enum) being either missing or misinterpreted. If edited and imported back may result in different or incomplete model. The same goes for code snippets used for code generation stage for the pattern-implementation and user-specific logic: intermixed they may not be easily reverse-engineered back. There is also general lack of advanced tooling for modelling that are comparable to that of modern IDEs (for testing, debugging, navigation, etc.) for general-purpose programming languages and domain-specific languages. == Examples in software engineering == Perhaps the most common form of round-trip engineering is synchronization between UML (Unified Modeling Language) models and the corresponding source code and entity–relationship diagrams in data modelling and database modelling. Round-trip engineering based on Unified Modeling Language (UML) needs three basic tools for software development: Source Code Editor; UML Editor for the Attributes and Methods; Visualisation of UML structure

  Read more →
• 

  Contrast set learning

  Contrast set learning is a form of association rule learning that seeks to identify meaningful differences between separate groups by reverse-engineering the key predictors that identify for each particular group. For example, given a set of attributes for a pool of students (labeled by degree type), a contrast set learner would identify the contrasting features between students seeking bachelor's degrees and those working toward PhD degrees. == Overview == A common practice in data mining is to classify, to look at the attributes of an object or situation and make a guess at what category the observed item belongs to. As new evidence is examined (typically by feeding a training set to a learning algorithm), these guesses are refined and improved. Contrast set learning works in the opposite direction. While classifiers read a collection of data and collect information that is used to place new data into a series of discrete categories, contrast set learning takes the category that an item belongs to and attempts to reverse engineer the statistical evidence that identifies an item as a member of a class. That is, contrast set learners seek rules associating attribute values with changes to the class distribution. They seek to identify the key predictors that contrast one classification from another. For example, an aerospace engineer might record data on test launches of a new rocket. Measurements would be taken at regular intervals throughout the launch, noting factors such as the trajectory of the rocket, operating temperatures, external pressures, and so on. If the rocket launch fails after a number of successful tests, the engineer could use contrast set learning to distinguish between the successful and failed tests. A contrast set learner will produce a set of association rules that, when applied, will indicate the key predictors of each failed tests versus the successful ones (the temperature was too high, the wind pressure was too high, etc.). Contrast set learning is a form of association rule learning. Association rule learners typically offer rules linking attributes commonly occurring together in a training set (for instance, people who are enrolled in four-year programs and take a full course load tend to also live near campus). Instead of finding rules that describe the current situation, contrast set learners seek rules that differ meaningfully in their distribution across groups (and thus, can be used as predictors for those groups). For example, a contrast set learner could ask, “What are the key identifiers of a person with a bachelor's degree or a person with a PhD, and how do people with PhD's and bachelor’s degrees differ?” Standard classifier algorithms, such as C4.5, have no concept of class importance (that is, they do not know if a class is "good" or "bad"). Such learners cannot bias or filter their predictions towards certain desired classes. As the goal of contrast set learning is to discover meaningful differences between groups, it is useful to be able to target the learned rules towards certain classifications. Several contrast set learners, such as MINWAL or the family of TAR algorithms, assign weights to each class in order to focus the learned theories toward outcomes that are of interest to a particular audience. Thus, contrast set learning can be thought of as a form of weighted class learning. === Example: Supermarket Purchases === The differences between standard classification, association rule learning, and contrast set learning can be illustrated with a simple supermarket metaphor. In the following small dataset, each row is a supermarket transaction and each "1" indicates that the item was purchased (a "0" indicates that the item was not purchased): Given this data, Association rule learning may discover that customers that buy onions and potatoes together are likely to also purchase hamburger meat. Classification may discover that customers that bought onions, potatoes, and hamburger meats were purchasing items for a cookout. Contrast set learning may discover that the major difference between customers shopping for a cookout and those shopping for an anniversary dinner are that customers acquiring items for a cookout purchase onions, potatoes, and hamburger meat (and do not purchase foie gras or champagne). == Treatment learning == Treatment learning is a form of weighted contrast-set learning that takes a single desirable group and contrasts it against the remaining undesirable groups (the level of desirability is represented by weighted classes). The resulting "treatment" suggests a set of rules that, when applied, will lead to the desired outcome. Treatment learning differs from standard contrast set learning through the following constraints: Rather than seeking the differences between all groups, treatment learning specifies a particular group to focus on, applies a weight to this desired grouping, and lumps the remaining groups into one "undesired" category. Treatment learning has a stated focus on minimal theories. In practice, treatment are limited to a maximum of four constraints (i.e., rather than stating all of the reasons that a rocket differs from a skateboard, a treatment learner will state one to four major differences that predict for rockets at a high level of statistical significance). This focus on simplicity is an important goal for treatment learners. Treatment learning seeks the smallest change that has the greatest impact on the class distribution. Conceptually, treatment learners explore all possible subsets of the range of values for all attributes. Such a search is often infeasible in practice, so treatment learning often focuses instead on quickly pruning and ignoring attribute ranges that, when applied, lead to a class distribution where the desired class is in the minority. === Example: Boston housing data === The following example demonstrates the output of the treatment learner TAR3 on a dataset of housing data from the city of Boston (a nontrivial public dataset with over 500 examples). In this dataset, a number of factors are collected for each house, and each house is classified according to its quality (low, medium-low, medium-high, and high). The desired class is set to "high", and all other classes are lumped together as undesirable. The output of the treatment learner is as follows: Baseline class distribution: low: 29% medlow: 29% medhigh: 21% high: 21% Suggested Treatment: [PTRATIO=[12.6..16), RM=[6.7..9.78)] New class distribution: low: 0% medlow: 0% medhigh: 3% high: 97% With no applied treatments (rules), the desired class represents only 21% of the class distribution. However, if one filters the data set for houses with 6.7 to 9.78 rooms and a neighborhood parent-teacher ratio of 12.6 to 16, then 97% of the remaining examples fall into the desired class (high-quality houses). == Algorithms == There are a number of algorithms that perform contrast set learning. The following subsections describe two examples. === STUCCO === The STUCCO contrast set learner treats the task of learning from contrast sets as a tree search problem where the root node of the tree is an empty contrast set. Children are added by specializing the set with additional items picked through a canonical ordering of attributes (to avoid visiting the same nodes twice). Children are formed by appending terms that follow all existing terms in a given ordering. The formed tree is searched in a breadth-first manner. Given the nodes at each level, the dataset is scanned and the support is counted for each group. Each node is then examined to determine if it is significant and large, if it should be pruned, and if new children should be generated. After all significant contrast sets are located, a post-processor selects a subset to show to the user - the low order, simpler results are shown first, followed by the higher order results which are "surprising and significantly different." The support calculation comes from testing a null hypothesis that the contrast set support is equal across all groups (i.e., that contrast set support is independent of group membership). The support count for each group is a frequency value that can be analyzed in a contingency table where each row represents the truth value of the contrast set and each column variable indicates the group membership frequency. If there is a difference in proportions between the contrast set frequencies and those of the null hypothesis, the algorithm must then determine if the differences in proportions represent a relation between variables or if it can be attributed to random causes. This can be determined through a chi-square test comparing the observed frequency count to the expected count. Nodes are pruned from the tree when all specializations of the node can never lead to a significant and large contrast set. The decision to prune is based on: The minimum deviation size: The maximum difference between the support

  Read more →
• 

  Squeaky Dolphin

  Squeaky Dolphin is a program developed by the Government Communications Headquarters (GCHQ), a British intelligence and security organization, to collect and analyze data from social media networks. The program was first revealed to the general public on NBC on 27 January 2014 based on documents previously leaked by Edward Snowden. == Scope of surveillance == According to a document of the GCHQ dated August 2012, the program enables broad, real-time surveillance of the following items: YouTube video views The Like button on Facebook. Facebook has since then encrypted the data. Blogspot/Blogger visits Twitter, which has however encrypted its communications since this presentation was made The program can be supplemented with commercially available analytic software to determine which videos are popular among residents of specific cities. The dashboard software chosen was made by Splunk. The presentation, which was originally shown to an NSA audience and was made public by the NBC, contains a note saying the program was "Not interested in individuals just broad trends!". However, "according to other Snowden documents" obtained by NBC, in 2010, "GCHQ exploited unencrypted data from Twitter to identify specific users around the world and target them with propaganda."

  Read more →
• 

  Social media use in African politics

  Since the Egyptian Revolution in 2011 and the Tunisian Revolution, social media, especially Facebook, Twitter, and YouTube, began to gain traction as a political tool in Africa. Various political actors have used social media to pursue a wide range of political objectives. State actors can use social media to encourage political discourse, campaign, or implement censorship and surveillance. Non-state actors, such as civil society organizations and opposition movements, can use social media to address political concerns and to organize widespread uprisings, such as the 2014 Burkinabé uprising. Meanwhile, extremist organizations can use social media to further their propaganda and recruitment. However, social media has been criticized for its limited accessibility and for facilitating the spread of misinformation, causing some skepticism about its effectiveness. Due to low entry barriers and user-generated content, social media provides a platform where people from different social classes can engage and interact with one another. Under traditional media, the public had limited opportunities to voice their political opinions. Social media enables people to both create and consume content. The public has become increasingly comfortable and confident in expressing political opinions online, often away from government scrutiny. Scholars argue that social media use has democratizing effects in African countries. == State actors == === Promoting political discourse === Through social media, the government and its citizens can discuss policy ideas, policy implementation, and political actions. Regardless of geographical location and distance, people are able to voice their opinions to the government. Social media includes citizens who were previously not able to express their discontent or share their ideas to the government. As state actors keep the public informed, social media can increase civic engagement. With more civic engagement, policies can be discussed without politicization. Before the commonplace use of social media, African countries faced weak feedback mechanisms that effectively excluded the average African citizen from policy discourse. In South Africa, the government uses social media to connect with constituencies. The South African president runs an official Twitter, Facebook, YouTube, and Flickr accounts to engage with the public. === Campaigning === Political parties also use social media for political campaigns during election periods. In South Africa, the ANC (African National Congress) and DA (Democratic Alliance) use social media for political purposes. These parties specifically use Facebook as a tool for campaigning and engaging with the public to improve their relationship with citizens. Nigerian President Goodluck Jonathan employed social media to campaign for the presidential election in 2011, which he won. When President Goodluck Jonathan announced his bid for the presidency on social media in 2010, it reached about 217,000 people. As his campaign progressed, President Goodluck Jonathan was able to increase his followers to half a million by early 2011. === Censorship & Surveillance === While state actors can use social media to encourage their party or discourse, social media can be used to censor and surveil citizens. For example, the ANC and DA use Facebook to monitor South Africans. The government is able to track down people who have spoken against the government and translate this information into physical action to stop any possibility of a revolution. Social media platforms can be shut down to manipulate the flow of information. In Chad, citizens cannot access information through online platforms. This censorship blocked "Facebook, Twitter, WhatsApp and Viber". In the Democratic Republic of Congo, the government shut down the internet before contested elections. In Zimbabwe, the government shut down the internet to hide civilian protests against fuel price increases. == Non-state actors == === Civil society organizations (CSOs) === Civil society organizations have also used social media networks in an effort to recruit supporters and communicate with the public. CSOs can use social media to mobilize people to support their cause, such as the Ghanaian Committee for Joint Action (CJA). In 2005 and 2006, the CJA gathered support to protest against the 50% fuel price increase. CSOs can play the role of a counterforce against state actors and state propaganda during times of crises, such as protests and military clashes. In some cases, CSOs release their own videos and photos on social media which challenges traditional forms of media. CSOs have also served to monitor elections to reduce corruption and violence during election day. For instance, the Zambian Bantu Watch started the #bantuwatch social media campaign to monitor the 2011 presidential election. Zambians used Facebook and Twitter to report polling station results to mitigate election fraud and election violence. In South Africa, CSOs created 'amandla.mobi' to campaign for public policies by creating petitions. Through 'amandla.mobi', CSOs are able to circulate petitions on social media to collect signatures. South African CSOs reported how social media helped their organizations to gain support and share ideas. However, CSOs struggle to attract media attention and often have to pay for media coverage. === Opposition forces against the government === Social media is also used by the public or opposition forces against the government. Through horizontal social media, organizing can lead to street protests and revolutions, some of which are successful. For instance, during the Egyptian revolution of 2011, "The Day of the Revolution Against Torture, Poverty, Corruption, and Unemployment" and "We Are All Khaled Said" gathered support against President Hosni Mubarak. In particular, "We Are All Khaled Said" had Egyptian citizens gather around the death of Khaled Said who was brutally tortured and killed by the Egyptian government because Said wanted to uncover government corruption. As unrest erupted into public demonstrations, President Hosni Mubarak was forced to resign. Witnessing the success of social media during the Egyptian revolution, the Tunisian Revolution, or the Jasmine Revolution, mobilized through Facebook and Twitter. Likewise, in South Africa, Malawi, and Mozambique, these countries have used social media as "new protest drums." Due to social media's low entry barrier, opposition forces against the government can facilitate political discourse that can lead to accountability. Whistleblowers and opposition forces are able to expose corruption through social media, where they face less repression while reaching a larger audience. For example, the youth of Zimbabwe and South Africa use Facebook to discuss politics without judgment. Specifically, in Zimbabwe, political youth used Facebook to avoid state surveillance. Social media is used as a supplemental tool for activism. In 2015, South African student activists started the hashtag #RhodesMustFall to push the issue of colonialism and racism at the forefront of the public. === Extremist organizations === Social media is easily accessible and created by user-based content. Therefore, marginalized groups are able to use social media to spread extremist ideas. For instance, Boko Haram created the Media Office of West Africa Province and perpetuated propaganda through Twitter and YouTube. Boko Haram's online propaganda campaign targets and persuades young dissuaded Nigerians to join their cause. It is important to note that social media has also been used against Boko Haram. In April 2014, Boko Haram kidnapped 276 schoolgirls and an international campaign fought for their return through #BringBackOurGirls. Another extremist group, Al-Shabaab, has created an online presence through Twitter and YouTube. Through these social media networks, Al-Shabaab recruits new members to their extremist group through their propaganda which emphasizes the group's successes. Albeit their efforts, Al-Shabaab has not been very successful in coordinating their members but they are successful in financing their group. Furthermore, the Islamic State of Iraq and the Levant (ISIL) use social media to target and recruit individuals to their cause. ISIL's social media usage is more diverse compared to Boko Haram and Al-Shabaab; ISIL uses "Facebook, Twitter, YouTube, WhatsApp, Telegram, JustPaste.it, Kik and Ask.fm." Since ISIL's Twitter accounts kept getting shut down, ISIL uses Telegram and WhatsApp chat rooms to privately conduct meetings. Due to the spread of extremist ideology, Zhuravskaya et al. acknowledge social media's potential to be misused. == Challenges == Although social media can be used as a political tool, it faces challenges in Africa. Due to low literacy rates in Africa, social media networks exclude many of the population members. In addition, lack of access to electricity and the internet can fur

  Read more →
• 

  U-Net

  U-Net is a convolutional neural network that was developed for image segmentation. The network is based on a fully convolutional neural network whose architecture was modified and extended to work with fewer training images and to yield more precise segmentation. Segmentation of a 512 × 512 image takes less than a second on a modern (2015) GPU using the U-Net architecture. The U-Net architecture has also been employed in diffusion models for iterative image denoising. This technology underlies many modern image generation models, such as DALL-E, Midjourney, and Stable Diffusion. U-Net is also being explored for language models. Tokenization is not a separate step, allowing the model to more easily understand spelling and concurrently vectorizing / tokenizing higher level concepts. == Description == The U-Net architecture stems from the so-called "fully convolutional network". The main idea is to supplement a usual contracting network by successive layers, where pooling operations are replaced by upsampling operators. Hence these layers increase the resolution of the output. A successive convolutional layer can then learn to assemble a precise output based on this information. One important modification in U-Net is that there are a large number of feature channels in the upsampling part, which allow the network to propagate context information to higher resolution layers. As a consequence, the expansive path is more or less symmetric to the contracting part, and yields a u-shaped architecture. The network only uses the valid part of each convolution without any fully connected layers. To predict the pixels in the border region of the image, the missing context is extrapolated by mirroring the input image. This tiling strategy is important to apply the network to large images, since otherwise the resolution would be limited by the GPU memory. Recently, there had also been an interest in receptive field based U-Net models for medical image segmentation. == Network architecture == The network consists of a contracting path and an expansive path, which gives it the u-shaped architecture. The contracting path is a typical convolutional network that consists of repeated application of convolutions, each followed by a rectified linear unit (ReLU) and a max pooling operation. During the contraction, the spatial information is reduced while feature information is increased. The expansive pathway combines the feature and spatial information through a sequence of up-convolutions and concatenations with high-resolution features from the contracting path. == Applications == There are many applications of U-Net in biomedical image segmentation, such as brain image segmentation (''BRATS'') and liver image segmentation ("siliver07") as well as protein binding site prediction. U-Net implementations have also found use in the physical sciences, for example in the analysis of micrographs of materials. Variations of the U-Net have also been applied for medical image reconstruction. Here are some variants and applications of U-Net as follows: Pixel-wise regression using U-Net and its application on pansharpening; 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation; TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation. Image-to-image translation to estimate fluorescent stains In binding site prediction of protein structure. == History == U-Net was created by Olaf Ronneberger, Philipp Fischer, Thomas Brox in 2015 and reported in the paper "U-Net: Convolutional Networks for Biomedical Image Segmentation". It is an improvement and development of FCN: Evan Shelhamer, Jonathan Long, Trevor Darrell (2014). "Fully convolutional networks for semantic segmentation".

  Read more →
• 

  Cognos ReportNet

  Cognos ReportNet (CRN) was a web-based software product for creating and managing ad hoc and custom-made reports. ReportNet was developed by the Ottawa-based company Cognos (formerly Cognos Incorporated), an IBM company. The web-based reporting tool was launched in September 2003. Since IBM's acquisition of Cognos, ReportNet has been renamed IBM Cognos ReportNet like all other Cognos products. ReportNet uses web services standards such as XML and Simple Object Access Protocol and also supports dynamic HTML and Java. ReportNet is compatible with multiple databases including Oracle, SAP, Teradata, Microsoft SQL server, DB2 and Sybase. The product provides interface in over 10 languages, has Web Services architecture to meet the needs of multi-national, diversified enterprises and helps reduce total cost of ownership. Multiple versions of Cognos ReportNet have since been released by the company. Cognos ReportNet was awarded the Software and Information Industry Association (SIIA) 2005 Codie awards for the "Best Business Intelligence or Knowledge Management Solution" category. CRN's capabilities have been further used in IBM Cognos 8 BI (2005), the latest reporting tool. CRN comes with its own software development kit (SDK). == Launch == Early adopters of Cognos ReportNet for their corporate reporting needs included Bear Stearns, BMW and Alfred Publishing. Around this same time of launch, Cognos competitor Business Objects released version 6.1 of its enterprise reporting tool. Cognos ReportNet has been successful since its launch, raising revenues in 2004 from licensing fees. == Controversy == Cognos rival Business Objects announced in 2005 that BusinessObjects XI significantly outperformed Cognos ReportNet in benchmark tests conducted by VeriTest, an independent software testing firm. The tests performed showed Cognos ReportNet performed poorly when processing styled reports, complex business reports and combination of both. The tests reported a massive 21 times higher report throughput for BusinessObjects XI than Cognos ReportNet at capacity loads. Cognos soon dismissed the claims by stating Business Objects dictated the environment and testing criteria and Cognos did not provide the software to participate in benchmark test. Cognos later performed their own test to demonstrate Cognos ReportNet capabilities. == Components == Cognos Report Studio – A Web-based product for creating complex professional looking reports. Cognos Query Studio - A Web-based product for creating ad-hoc reports. Cognos Framework Manager – A metadata modeling tool to create BI metadata for reporting and dashboard applications. Cognos Connection – Main portal used to access reports, schedule reports and perform administrator activities. == Versions == Cognos ReportNet 1.1 – Java EE-style professional web-based authoring tool. (base version) Cognos ReportNet IBM Special Edition – comes with an embedded version of IBM WebSphere as its application server and IBM DB2 as its data store. Cognos Linux – for Intel-based Linux platforms.

  Read more →
• 

  Data preservation

  Data preservation is the act of conserving and maintaining both the safety and integrity of data. Preservation is done through formal activities that are governed by policies, regulations and strategies directed towards protecting and prolonging the existence and authenticity of data and its metadata. Data can be described as the elements or units in which knowledge and information is created, and metadata are the summarizing subsets of the elements of data; or the data about the data. The main goal of data preservation is to protect data from being lost or destroyed and to contribute to the reuse and progression of the data. == History == Most historical data collected over time has been lost or destroyed. War and natural disasters combined with the lack of materials and necessary practices to preserve and protect data has caused this. Usually, only the most important data sets were saved, such as government records and statistics, legal contracts and economic transactions. Scientific research and doctoral theses data have mostly been destroyed from improper storage and lack of data preservation awareness and execution. Over time, data preservation has evolved and has generated importance and awareness. We now have many different ways to preserve data and many different important organizations involved in doing so. The first digital data preservation storage solutions appeared in the 1950s, which were usually flat or hierarchically structured. While there were still issues with these solutions, it made storing data much cheaper, and more easily accessible. In the 1970s relational databases as well as spreadsheets appeared. Relational data bases structure data into tables using structured query languages which made them more efficient than the preceding storage solutions, and spreadsheets hold high volumes of numeric data which can be applied to these relational databases to produce derivative data. More recently, non-relational (non-structured query language) databases have appeared as complements to relational databases which hold high volumes of unstructured or semi-structured data. == Importance == The scope of data preservation is vast. Everything from governmental to business records to art essentially can be represented as data, and is amenable to be lost. This then leads to loss of human history, for perpetuity. Data can be lost on a small or independent scale whether it's personal data loss, or data loss within businesses and organizations, as well as on a larger or national or global scale which can negatively and potentially permanently affect things such as environmental protection, medical research, homeland security, public health and safety, economic development and culture. The mechanisms of data loss are also as many as they are varied, spanning from disaster, wars, data breaches, negligence, all the way through simple forgetting to natural decay. Ways in which data collections can be used when preserved and stored properly can be seen through the U.S. Geological Survey, which stores data collections on natural hazards, natural resources, and landscapes. The data collected by the Survey is used by federal and state land management agencies towards land use planning and management, and continually needs access to historical reference data. == Related Concepts == In contrast, data holdings are collections of gathered data that are informally kept, and not necessarily prepared for long-term preservation. For example, a collection or back-up of personal files. Data holdings are generally the storage methods used in the past when data has been lost due to environmental and other historical disasters. Furthermore, data retention differs from data preservation in the sense that by definition, to retain an object (data) is to hold or keep possession or use of the object. To preserve an object is to protect, maintain and keep up for future use. Retention policies often circle around when data should be deleted on purpose as well, and held from public access, while preservation prioritizes permanence and more widely shared access. Thus, data preservation exceeds the concept of having or possessing data or back up copies of data. Data preservation ensures reliable access to data by including back-up and recovery mechanisms that precede the event of a disaster or technological change. == Methods == === Digital === Digital preservation, is similar to data preservation, but is mainly concerned with technological threats, and solely digital data. Essentially digital data is a set of formal activities to enable ongoing or persistent use and access of digital data exceeding the occurrence of technological malfunction or change. Digital preservation is aware of the inevitable change in technology and protocols, and prepares for data that will need to be accessible across new types of technologies and platforms while the integrity of the data and metadata are being conserved. Technology, while providing great process in conserving data that may not have been possible in the past, is also changing at such a quick rate that digital data may not be accessible anymore due to the format being incompatible with new software. Without the use of data preservation much of our existing digital data is at risk. The majority of methods used towards data preservation today are digital methods, which are so far the most effective methods that exist. === Archives === Archives are a collection of historical documents and records. Archives contribute and work towards the preservation of data by collecting data that is well organized, while providing the appropriate metadata to confirm it. An example of an important data archive is The LONI Image Data Archive, which is an archive that collects data regarding clinical trials and clinical research studies. === Catalogues, directories and portals === Catalogues, directories and portals are consolidated resources which are kept by individual institutions, and are associated with data archives and holdings. In other words, the data is not presented on the site, but instead might act as metadata and aggregators, and may administer thorough inventories. === Repositories === Repositories are places where data archives and holdings can be accessed and stored. The goal of repositories is to make sure that all requirements and protocols of archives and holdings are being met, and data is being certified to ensure data integrity and user trust. Single-site Repositories A repository that holds all data sets on a single site. An example of a major single-site repository the Data Archiving and Networking Services which is a repository which provides ongoing access to digital research resources for the Netherlands. Multi-Site Repositories A repository that hosts data set on multiple institutional sites. An example of a well known multi-site repository is OpenAIRE which is a repository that hosts research data and publications collaborating all of the EU countries and more. OpenAIRE promotes open scholarship and seeks to improves discover-ability and re-usability of data. Trusted Digital Repository A repository that seeks to provide reliable, trusted access over a long period of time. The repository can be single or multi-sited but must cooperate with the Reference Model for an Open Archival Information System, as well as adhere to a set of rules or attributes that contribute to its trust such as having persistent financial responsibility, organizational buoyancy, administrative responsibility security and safety. An example of a trusted digital repository is The Digital Repository of Ireland (DRI) which is a multi-site repository that hosts Ireland's humanity and social science data sets. === Cyber Infrastructures === Cyber infrastructures which consists of archive collections which are made available through the system of hardware, technologies, software, policies, services and tools. Cyber infrastructures are geared towards the sharing of data supporting peer-to-peer collaborations and a cultural community. An example of a major cyber-infrastructure is The Canadian Geo-spatial Data Infrastructure which provides access to spatial data in Canada.

  Read more →
• 

  Philco computers

  Philco was one of the pioneers of transistorized computers, also known as second-generation computers. After the company developed the surface-barrier transistor, which was much faster than previous point-contact types, it was awarded contracts for military and government computers. Commercialized derivatives of some of these designs became successful business and scientific computers. The TRANSAC (Transistor Automatic Computer) Model S-1000 was released as a scientific computer. The TRANSAC S-2000 mainframe computer system was first produced in 1958, and a family of compatible machines, with increasing performance, was released over the next several years. However, the mainframe computer market was dominated by IBM. Other companies could not deploy resources for development, customer support and marketing on the scale that IBM could afford, making competition in this segment difficult after the introduction of the IBM 360 family. Philco went bankrupt and was purchased in 1961 by Ford Motor Company, but the computer division carried on until the Philco division of Ford exited the computer business in 1963. The Ford company maintained one Philco mainframe in use until 1981. == The surface-barrier transistor == The surface-barrier transistor developed by Philco in 1953 had a much higher frequency response than the original point-contact transistors. The transistor was made of a thin crystal of germanium, which was electrolytically etched with pits on either side forming a very thin base region, on the order of 5 micrometers. Philco's process for etching was United States patent number 2,885,571. Philco surface-barrier transistors were used in TX-0, and in early models of what would become the DEC PDP product line. Although relatively fast, the small size of the devices limited their power to circuits operating at a few tens of milliwatts. == Military and government == Between 1955 and 1957, Philco built transistor computers for use in aircraft, models C-1000, C-1100, and C-1102, intended for airborne real-time applications. By 1957, the C-1102 had been used by a civilian sector customer. The BASICPAC AN/TYK 6V (first delivery in 1961), COMPAC AN/TYK 4V (not completed), and LOGICPAC systems were built for the US Army as transportable computer systems for use with their Fieldata concept of integrated information management. BASICPAC was a transistorized computer with up to 28,672 words of 38-bit core memory (including sign and parity), available in several configurations from a minimum system, to a truck-borne mobile version, to a fully expanded system. Basic clock periods was 1 microsecond (which gives a clock rate of 1 MHz), with 12 microsecond memory access and a fixed-point multiplication taking 242 microseconds. Input/output was by paper tape reader and punch, or through a teletypewriter. With additional hardware, magnetic tape storage was also available, with up to seven I/O devices. The instruction set had 31 basic operation codes and nine opcodes for I/O === CXPQ === Philco was contracted by the US Navy to build the CXPQ computer. One model was completed and installed at the David Taylor Model Basin. This design was later adapted to become the commercial TRANSAC S-2000. Only one CXPQ was built. The CXPQ is a 48-bit transistorized computer. === SOLO === In 1955, the National Security Agency through the US Navy contracted with Philco to produce a computer suitable for use as a workstation, with an architecture based on the vacuum-tube computer system called Atlas II already in use at the NSA, and similar to the commercial UNIVAC 1103. At the time, Philco was the largest producer of surface barrier transistors, which were the only type available with the speed and quantities required for a computer. The SOLO prototype was delivered in 1958, but required extensive debugging at NSA. Difficulties were encountered with core memory and power supplies. SOLO used paper tape and teleprinter machines for input and output. SOLO cost about $1 million US, and contained 8,000 transistors. While the system was extensively used for training, testing, research and development, no additional units were ordered. SOLO was removed from active service in 1963. The design of the SOLO became commercialized as Philco's TRANSAC Model S-1000. == Commercial == === S-1000 === The TRANSAC S-1000 was a scientific computer with a 36-bit word length and 4096 words of core memory. It was packaged in a container about the size of a large office desk, and used only 1.2 kilowatts, much less than vacuum-tube-based computers of similar capacity. In a 1961 survey, about 15 S-1000 computer installations had been identified. It weighed about 1,650 pounds (750 kg). === S-2000 === The TRANSAC S-2000 was a large mainframe system intended for both business and scientific work. It had a 48-bit word length and supported calculations in fixed point, floating point and binary-coded decimal formats. The original S-2000 "TRANSAC" (Transistor Automatic Computer) released in 1958 was later designated Model 210; it was used internally at Philco. Similar to the Control Data Corporation Model 1604, it was a 48-bit fully transistorized computer. Three succeeding models were released in the series, all compatible with the software of the original model. The Model 211 was introduced in 1960, using micro-alloy diffused field-effect transistors, requiring significant redesign of circuits compared to the original. The TRANSAC S-2000/Philco 210/211 weighed about 2,000 pounds (910 kg). By 1964, eighteen Model 210, eighteen Model 211 and seven Model 212 systems had been sold. After Philco was purchased by Ford Motor Company, the Model 212 was introduced in 1962 and released in 1963. It had 65,535 words of 48-bit memory. Initially made with 6-microsecond core memory, it had better performance than the IBM 7094 transistor computer. It was later upgraded in 1964 to 2-microsecond core memory, which gave the machine floating-point performance greater than the IBM 7030 Stretch computer. A Model 213 was announced in 1964 but never built. By that time competition from IBM had made the Philco computer operations no longer profitable for Ford, and the division was closed down. The Model 212 could carry out a floating-point multiplication in 22 microseconds. Each word contained two 24-bit instructions with 16 bits of address information and eight bits for the opcode. There were 225 different valid opcodes in the Model 212; invalid opcodes were detected and halted the machine. The CPU had an accumulator register of 48 bits, three general-purpose registers of 24 bits, and 32 index registers of 15 bits. Main memory size ranged from 4K words to 64K words. Only the first model had a magnetic drum memory; later editions used tape drives. The Model 212 weighed about 6,500 pounds (3.3 short tons; 2.9 t). Software for the S-2000 initially consisted of TAC (Translator-Assembler-Compiler), and ALTAC, a FORTRAN II-like language with some differences from the IBM 704 FORTRAN implementation. A COBOL compiler was also available, targeted at business applications. The Philco 2400 was the input/output system for the S-2000. Operations such as reading cards or printing were carried out through magnetic tapes, thereby offloading the S-2000 from relatively slow input/output processing. The 2400 had a 24-bit word length and could be supplied with 4K to 32K characters (1K to 8K words) of core memory, rated at 3-microsecond cycle time. The instruction set was aimed at character I/O use. The idea of base registers, implemented in Philco computers, influenced the design of IBM/360. The last Philco TRANSAC S-2000 Model 212 was taken out of service in December 1981, after 19 years of service at Ford.

  Read more →
• 

  CatDV

  CatDV is a media asset manager program for handling multimedia production workflows developed by Square Box Systems. Quantum Corporation acquired Square Box Systems in 2020. == Versions == The full family of CatDV Products is as follows: CatDV Standalone Products CatDV Professional Edition CatDV Pegasus CatDV Networked Products CatDV Essential - entry level server product CatDV Enterprise Server - for MySQL databases and most common server platforms including Linux, Windows and Mac OS X CatDV Pegasus Server - adds features such as high performance full-text indexing, access control lists, and more CatDV Worker Node - automated workflow and transcoding engine CatDV Web Client - provides access to the CatDV database via a web browser. There is no need to install special software on the desktop, making it easy to deploy to a large number of users. CatDV Professional Edition & Pegasus Clients - designed to support the multi-user capabilities of the CatDV Enterprise and Workgroup Servers from the desktop Using plugins and scripting, which often require additional professional services support to set up, complex integrations with a wide variety of third party systems (including archive, cloud storage, and artificial intelligence) are possible. == Awards == CatDV won two awards in 2010, a blue ribbon from Creative COW Magazine and a "Best of Show Vidy Award" from Videography. In April 2012 Square Box won a Queen's Award for Enterprise for CatDV.

  Read more →
• 

  Pivot to video

  "Pivot to video" is a phrase referring to the trend, starting in 2015, of media publishing companies cutting staff resources for written content (generally published on their own web sites) in favor of short-form video content (often published on third-party platforms such as Facebook, Instagram, Twitter, YouTube, Snapchat, and TikTok). These moves were generally presented by publishers as a response to changes in social media traffic or to changes in the media consumption habits of younger audiences. However, many media commentators have argued that this shift was primarily motivated by advertising revenue, and that only advertisers, not consumers, prefer video over text. The pivot's contribution to job loss in the media industry has given the phrase "pivot to video" an association with decline, especially in a business context. Commentators have also noted a lack of transparency and accuracy in the viewership metrics reported by platforms such as Facebook, pointing out that abrupt shifts in platforms' proprietary algorithms can have devastating effects on publishers' viewership, traffic, and revenue. Following a scandal in which Facebook revealed it had artificially inflated numbers to its advertisers about how long viewers watched ads, many journalists and industry analysts concluded that the shift to video was based on such misleading or inaccurate metrics, which created a false impression that there was customer demand for additional video content. == History == Streaming media technology has been available since the early 1990s, though it was relatively low-fidelity and not widely available until the mid-2000s. In 2007, traditional media publishers including the New York Times, Washington Post and Time Inc. created new divisions to develop web videos, and Facebook launched its video platform. Twitter purchased micro-video service Vine in October 2012, began adding native video streaming in late 2014, and acquired video-streaming service Periscope in January 2015. An August 2014 profile on BuzzFeed noted the publisher's large investment into video production, and observed that "the future of BuzzFeed may not even be on BuzzFeed.com. One of the company’s nascent ideas, BuzzFeed Distributed, will be a team of 20 people producing content that lives entirely on other popular platforms, like Tumblr, Instagram or Snapchat." On 7 January 2015, Facebook issued a statement about "the shift to video," reporting that "since June 2014, Facebook has averaged more than 1 billion video views every day." Media critic John Herrman argued that "What the shift to Facebook video means is that Facebook is more interested in hosting the things media companies make than just spreading them, that it views links to outside pages as a problem to be solved, and that it sees Facebook-hosted video as an example of the solution." In February 2015, the digital video-journalism publisher NowThis announced that it would operate without a home page, producing content to be published directly on social media platforms. In April 2016, Mashable fired much of its editorial staff, attempting to pivot away from hard news coverage while "growing Mashable across every platform" and doubling down on branded content and video. By December 2017, following a sale to Ziff Davis, Mashable retreated from this focus on video; Bernard Gershon, president of GershonMedia, said that the announcement of many such "pivots" were actually aimed primarily at investors. By 2017, "advertiser interest in video [was] insatiable... Any CFO is going to say 'How can we get more video?'" according to an executive of the publishers' trade association Digital Content Next. Publishers such as Vanity Fair, the Washington Post, and Sports Illustrated began adapting their own articles into cheap video content, either dictated by a newsreader or animated as a slideshow with captions, which could be shared on social platforms or even played alongside the articles themselves. June 2017 saw numerous high-profile pivots to video. Vocativ laid off at least 20 staff, including its entire newsroom, explaining that "as the industry evolves, we are undertaking a strategic shift to focus exclusively on video content that will be distributed via social media and other platforms." Fox Sports eliminated its entire writing staff to focus on creating "premium video across all platforms." And MTV News announced a restructuring that would cut its writing team. Less than two years earlier, MTV News had hired Grantland co-founder Dan Fierman to lead a significant investment in "longform" political and cultural reporting, but Fierman left in April 2017, and in June MTV announced it was "shifting resources into short-form video content more in line with young people's media consumption habits." In July, Vice Media laid off at least 60 employees, including the editor-in-chief of Vice Sports, while expanding video production. August 2017 saw Mic cut ten writers and directed the remainder of the newsroom to generate videos for social platforms. CEO Chris Altchek said "When you think about how many hours people spend watching video versus reading, the audience has already spoken." The move was ultimately unsuccessful, and Mic laid off the majority of its staff a year later before being sold to Bustle Media Group for a fraction of its former value. In September 2017, the for-profit wiki-hosting company Fandom began adding commercially produced videos to its otherwise user-generated wiki subdomains, explicitly citing the need to "keep up with user and advertiser expectations" by "diversifying our content," claiming without substantiation that "consumer patterns are changing," necessitating the addition of "complementary video" to accommodate that supposed need. Objection to the content in these videos and its sharp contrast against the content of the wiki sites to which they were applied led to vocal user backlash, leading Fandom CCO Dorth Raphaely to offer the following non-committal response: "I agree that with these videos in particular we did not deliver the right type of content experience." Movie Pilot CEO Tobi Bauckhage explained his company's fall 2017 layoffs as part of moving "from a text-based publishing model to video... a reaction to the fact that Facebook has changed their algorithms in favor of video instead of referral traffic over the last 12 months and we were losing money in the publishing bit of our business." As part of the company's change in direction, the majority of its staff was laid off and its parent company was sold to Webedia. In November 2017, magazine publisher Condé Nast cut jobs, reduced the frequency of several magazines, and shut down the print edition of Teen Vogue, then invested significant new resources in video production, with a senior executive saying "In the next 24 months, I hope that video is half our business... It’s critical. It’s the macro trend of content consumption." In February 2018, Vox Media cut approximately 50 employees, primarily those assigned to "social video," as Vox CEO Jim Bankoff admitted that those efforts were not "viable audience or revenue growth drivers." In August 2020, Facebook Inc. (now Meta Platforms) pivoted Instagram to video in an effort to replicate the success of TikTok and appeal to a younger audience, introducing "reels" as a form of video and promoting them aggressively. Reels accounted more than half the 20 most-viewed posts on Facebook; however, most of these reels were anonymous aggregations of content from TikTok. Elon Musk declared in early 2024 that X (formerly Twitter) was now a "video-first platform", which has been described by critics as a "pivot to video". == As euphemism == In 2017, Journalist Brian Feldman said that "'Pivoting to video' has become a business strategy for digital publishers common enough in recent months to be a kind of cliché — a slick way to describe something else: layoffs." In response, writers use the phrase as gallows humor shorthand for death or cancellation, as in "how do i tell my bf i want our relationship to pivot to video" (SkyNews' Mollie Goodfellow) or "Horse broke its leg, so we had to take it out back and help it 'pivot to video'" (blogger Anil Dash). == Facebook metrics controversy == In September 2016, Facebook admitted that it had reported artificially inflated numbers to its advertisers about how long viewers watched ads leading to an overestimation of 60-80%. Plaintiffs in a later court case allege the discrepancy was as high as 150-900%. Facebook apologized in an official statement and in multiple staff appearances at New York Advertising Week. Two months later, Facebook disclosed additional discrepancies in audience metrics. In October 2018, a California federal court unsealed the text of a class action lawsuit filed by advertisers against Facebook, alleging that Facebook had known since 2015 that its viewership numbers were highly inflated, that internal records showed it "was far from an hon

  Read more →
• 

  NATGRID

  The National Intelligence Grid or NATGRID is an integrated intelligence master database structure for counter-terrorism purposes which connects databases of various core security agencies under the Government of India. It collects and analyses comprehensive patterns procured from 21 different organizations that can be readily accessed by security agencies round the clock. As of September 2025 its CEO is Hirdesh Kumar. NATGRID came into existence after the 2008 Mumbai attacks. The Government of India in July 2016 appointed Ashok Patnaik as the Chief Executive Officer (CEO) of NATGRID. The appointment is being seen as the government's effort to revive the project. Patnaik's appointment was valid till 31 December 2018. As of 2019, NATGRID is headed by an Indian Police Service (IPS) officer Ashish Gupta. The Ministry of Home Affairs on 5 February 2020 announced in Parliament that Project NATGRID with all its required physical infrastructures been completed as of 31 March 2020 and the NATGRID solution went live as of 31 December 2020. == Reason for establishment == The landscape of Terrorism in India and the subsequent response by Law enforcement in India have necessitated a sophisticated data-integration framework, positioning NATGRID as a vital tool for national security agencies. This shift towards Mass surveillance in India is rooted in a broader policy evolution of state monitoring, which is technologically enabled by the India Stack—the foundational digital infrastructure providing the API-based backbone for government service delivery and identity verification. This ecosystem is further bolstered by advanced Signal intelligence capabilities and the implementation of SIM binding, a security protocol that anchors a user’s digital identity to a specific mobile device and verified SIM card to prevent identity fraud and unauthorized access. Collectively, these elements form a 360-degree surveillance and authentication grid designed to preemptively identify threats by synthesizing historical, financial, and real-time communication data across disparate platforms. === Terror attacks in India === The 2008 Mumbai attacks led to the exposure of several weaknesses in India's intelligence gathering and action networks. NATGRID is part of the radical overhaul of the security and intelligence apparatuses of India that was mooted by the then Home Minister P. Chidambaram in 2009. The National Investigation Agency (NIA) and the National Counter Terrorism Centre (NCTC) are two organisations established in the aftermath of the Mumbai attacks of 2008. Before the Mumbai attacks, a Pakistani origin American Lashkar-e-Taiba (LeT) operative David Coleman Headley had visited India several times and done a recce of the places that came under attack on 26/11. Despite having travelled to India several times and having returned to the US through Pakistan or West Asia, his trips failed to raise the suspicion of Indian agencies as they lacked a system that could reveal a pattern in his unusual travel itineraries and trips to the country. It was argued that if they had a system like the NATGRID in place, Headley would have been apprehended well before the attacks. === Need for the integrated intelligence system === During the inauguration of NATGRID campus in Bengaluru, the Minister of Home Affairs, Amit Shah stated that a new national database is in the process of being made which will bring a change in the current ways of functioning of agencies once it's ready also adding that the government has entrusted the task of developing and operating a state-of-the-art and innovative technology system. It is accessible to 11 central agencies in the first phase and in later phases will be made accessible to police of all States and Union Territories and only authorized personnel are allowed access to the platform on a case-to-case basis for investigations into suspected cases of terrorism. NATGRID has a total fund allocation of ₹3,400 crore (US$355 million). d == Legal framework == Relevant legal framework: Digital Personal Data Protection Act, 2023 – The legislative framework governing how digital data is handled. Information Technology Act - Interception Rules, 2002 – The specific regulations under the Information Technology Act that govern these agencies. National Security Act of 1980, evidence-based preventative detention of suspects Right to Information Act, 2005, for obtaining information from the government and used by activists and whistleblowers == Structure and functions == === Multi-agency integrated intelligence database === NATGRID is an intelligence sharing network that collates data from the standalone databases of the various agencies and ministries of the Indian government. It is a counter terrorism measure that collects and collates a host of information from government databases including tax and bank account details, credit/debit card transactions, visa and immigration records and itineraries of rail and air travel. It also has access to the Crime and Criminal Tracking Network and Systems, a database that links crime information, including First Information Reports, across 14,000 police stations in India. This combined data will be made available to 11 central agencies, which are: the Research and Analysis Wing (R&AW), Intelligence Bureau (IB), National Investigation Agency (NIA), Central Bureau of Investigation (CBI), Narcotics Control Bureau (NCB), Financial Intelligence Unit (India) (FIU), Enforcement Directorate (ED), Central Board of Direct Taxes (CBDT), Central Board of Indirect Taxes and Customs (CBIC), Directorate of Revenue Intelligence (DRI) and Directorate General of GST Intelligence. Also as stated by the MHA, NATGRID will have an in-built mechanism for continuous upgradation. In the later phases of NATGRID integration, the central government further plans to integrate 950 additional organizations into it. === Key components and users === ==== Some important backend data feeds to the NATGRID (middleware) ==== National Crime Records Bureau's Crime and Criminal Tracking Network and Systems (CCTNS) national-integrated law-and-order database for the state-level police forces: CCTNS is a mission-mode project under the National e-Governance Plan that interconnects over 15,000 police stations across India. It serves as the primary source for NATGRID to access digitized FIR (First Information Report) data and criminal history records from state-level law enforcement. NSA's National Technical Research Organisation (NTRO) national security-based database feed to NATGRID: NTRO serves as a primary technical data provider to NATGRID, offering specialized intercepts and satellite imagery. While NATGRID functions as a centralized data-integration middleware under the Ministry of Home Affairs, NTRO reports to the National Security Advisor within the Prime Minister's Office. DRDO's NETRA (Network Traffic Analysis) ELINT-based mass surveillance system for monitor internal internet traffic for keywords related to terrorism and criminal activity within Indian borders: Developed by the Centre for Artificial Intelligence and Robotics (CAIR), NETRA is an internet monitoring system capable of scanning traffic for specific trigger words. It provides digital behavioral triggers that NATGRID can cross-reference against structural data like financial or travel records. NETRA is a massive software network used to intercept and analyze internet traffic (emails, social media, blogs) for keywords like "bomb," "attack," or "kill." The intelligence gathered by NETRA regarding suspicious digital patterns or "keyword hits" can be fed into NATGRID. This allows an investigator to see if a person flagged by NETRA also has suspicious travel (from airline databases) or financial records (from bank databases) linked within NATGRID. Department of Telecommunications (DoT's Central Monitoring System (CMS) for lawfully intercepting national and international telecomm data: CMS is the centralized system for lawful interception of all telecommunications (phone calls, SMS, and data) in India, managed by the Department of Telecommunications (DoT). While CMS focuses on the content and metadata of real-time communication, NATGRID focuses on historical/structural data (tax, travel, identity). They represent two halves of a 360-degree surveillance profile: CMS listens to what a suspect says, while NATGRID tracks where they go and what they own. The CMS allows for the lawful interception of telecommunications metadata and content in real-time. In the broader surveillance architecture, CMS provides the "active" communication profile while NATGRID provides the "static" historical profile. Telecom Enforcement Resource and Monitoring (TERM) - Telecomm Regulatory & Verification Node for telecomm KYC: TERM cells verify subscriber identity (KYC) and maintain the integrity of telecom databases. NATGRID relies on these audited records to ensure the accuracy of telephone-to-identity mapping. TERM

  Read more →
• 

  Big memory

  Big-memory computers are machines with a large amount of random-access memory (RAM). The computers are required for databases, graph analytics, or more generally, high-performance computing, data science, and big data. Some database systems called in-memory databases are designed to run mostly in memory, rarely if ever retrieving data from disk or flash memory. See list of in-memory databases. == Details == The performance of big-memory systems depends on how the central processing units (CPUs) access the memory, via a conventional memory controller or via non-uniform memory access (NUMA). Performance also depends on the size and design of the CPU cache. Performance also depends on operating system (OS) design. The huge pages feature in Linux and other OSes can improve the efficiency of virtual memory. The transparent huge pages feature in Linux can offer better performance for some big-memory workloads. The "Large-Page Support" in Microsoft Windows enables server applications to establish large-page memory regions which are typically three orders of magnitude larger than the native page size.

  Read more →
• 

  Language engineering

  Language engineering involves the creation of natural language processing systems, whose cost and outputs are measurable and predictable. It is a distinct field contrasted to natural language processing and computational linguistics. A recent trend of language engineering is the use of Semantic Web technologies for the creation, archiving, processing, and retrieval of machine processable language data. Meta-Language Engineering is a proposed extension of Language Engineering first recorded in 2025, associated with the work of Delyone de Paula Canedo Filho. The term is used to designate an approach that, in addition to natural language processing, encompasses the symbolic, cognitive, and epistemological structuring of language systems.

  Read more →
• 

  Content management

  Content management (CM) are a set of processes and technologies that support the collection, managing, and publishing of information in any form or medium. When stored and accessed via computers, this information may be more specifically referred to as digital content, or simply as content. Digital content may take the form of text (such as electronic documents), images, multimedia files (such as audio or video files), or any other file type that follows a content lifecycle requiring management. The process of content development and management is complex enough that various commercial software vendors (large and small), such as Interwoven and Microsoft, offer content management software to control and automate significant aspects of the content lifecycle. == Process == Content management practices and goals vary by mission and by organizational governance structure. News organizations, e-commerce websites, and educational institutions all use content management, but in different ways. This leads to differences in terminology and in the names and number of steps in the process. For example, some digital content is created by one or more authors. Over time that content may be edited. One or more individuals may provide some editorial oversight, approving the content for publication. Publishing may take many forms: it may be the act of "pushing" content out to others, or simply granting digital access rights to certain content to one or more individuals. Later that content may be superseded by another version of the content and thus retired or removed from use (as when this wiki page is modified). Content management is an inherently collaborative process. It often consists of the following basic roles and responsibilities: Creator – responsible for creating and editing content. Editor – responsible for tuning the content message and the style of delivery, including translation and localization. Publisher – responsible for releasing the content for use. Administrator – responsible for managing access permissions to folders, collections and files, usually accomplished by assigning access rights to user groups or roles. Admins may also assist and support users in various ways. Consumer, viewer or guest – the person who reads or otherwise consumes the content after it is published or shared. A critical aspect of content management is the ability to manage versions of content as it evolves (see also version control). Authors and editors often need to restore older versions of edited products due to a process failure or an undesirable series of edits. Time-sensitive content may also require updates as the subject matter evolves over time. Another equally important aspect of content management involves the creation, maintenance, and application of review standards. Each member of the content creation and review process has a unique role and set of responsibilities in the development or publication of the content. Each review team member requires clear and concise review standards. These must be maintained on an ongoing basis to ensure the long-term consistency and health of the knowledge base. A content management system is a set of automated processes that may support the following features: Import and creation of documents and multimedia material Identification of all key users and their roles The ability to assign roles and responsibilities to different instances of content categories or types Definition of workflow tasks often coupled with messaging so that content managers are alerted to changes in content The ability to track and manage multiple versions of a single instance of content The ability to publish the content to a repository to support access The ability to personalize content based on a set of rules Increasingly, the repository is an inherent part of the system, and incorporates enterprise search and retrieval. Content management systems take the following forms: Web content management system—software for web site management (often what content management implicitly means) Output of a newspaper editorial staff organization Workflow for article publication Document management systems Knowledge management software Single source content management system—content stored in chunks within a relational database Variant management system—where personnel tag source content (usually text and graphics) to represent variants stored as single source "master" content modules, resolved to the desired variant at publication (for example: automobile owners manual content for 12 model years stored as single master content files and "called" by model year as needed)—often used in concert with database chunk storage (see above) for large content objects == Governance structures == Content management expert Marc Feldman defines three primary content management governance structures: localized, centralized, and federated—each having its unique strengths and weaknesses. === Localized governance === By putting control in the hands of those closest to the content, the context experts, localized governance models empower and unleash creativity. These benefits come, however, at the cost of a partial-to-total loss of managerial control and oversight. === Centralized governance === When the levers of control are strongly centralized, content management systems are capable of delivering an exceptionally clear and unified brand message. Moreover, centralized content management governance structures allow for a large number of cost-savings opportunities in large enterprises, realized, for example, through (1) the avoidance of duplicated efforts in creating, editing, formatting, repurposing and archiving content; (2) process management and the streamlining of all content related labor; and/or (3) an orderly deployment or updating of the content management system. === Federated governance === Federated governance models potentially realize the benefits of both localized and centralized control while avoiding the weaknesses of both. While content management software systems are inherently structured to enable federated governance models, realizing these benefits can be difficult because it requires, for example, negotiating the boundaries of control with local managers and content creators. In the case of larger enterprises, in particular, the failure to fully implement or realize a federated governance structure equates to a failure to realize the full return on investment and cost savings that content management systems enable. == Implementation == Content management implementations must be able to manage content distributions and digital rights in content life cycle. Content management systems are usually involved with digital rights management in order to control user access and digital rights. In this step, the read-only structures of digital rights management systems force some limitations on content management, as they do not allow authors to change protected content in their life cycle. Creating new content using managed (protected) content is also an issue that gets protected contents out of management controlling systems. A few content management implementations cover all these issues.

  Read more →
• 

  Information leakage

  Information leakage happens whenever a system that is designed to be closed to an eavesdropper reveals some information to unauthorized parties nonetheless. In other words: Information leakage occurs when secret information correlates with, or can be correlated with, observable information. For example, when designing an encrypted instant messaging network, a network engineer without the capacity to crack encryption codes could see when messages are transmitted, even if he could not read them. == Risk vectors == A modern example of information leakage is the leakage of secret information via data compression, by using variations in data compression ratio to reveal correlations between known (or deliberately injected) plaintext and secret data combined in a single compressed stream. Another example is the key leakage that can occur when using some public-key systems when cryptographic nonce values used in signing operations are insufficiently random. Bad randomness cannot protect proper functioning of a cryptographic system, even in a benign circumstance, it can easily produce crackable keys that cause key leakage. Information leakage can sometimes be deliberate: for example, an algorithmic converter may be shipped that intentionally leaks small amounts of information, in order to provide its creator with the ability to intercept the users' messages, while still allowing the user to maintain an illusion that the system is secure. This sort of deliberate leakage is sometimes known as a subliminal channel. Generally, only very advanced systems employ defenses against information leakage. Following are the commonly implemented countermeasures : Use steganography to hide the fact that a message is transmitted at all. Use chaffing to make it unclear to whom messages are transmitted (but this does not hide from others the fact that messages are transmitted). For busy re-transmitting proxies, such as a Mixmaster node: randomly delay and shuffle the order of outbound packets - this will assist in disguising a given message's path, especially if there are multiple, popular forwarding nodes, such as are employed with Mixmaster mail forwarding. When a data value is no longer going to be used, erase it from the memory.

  Read more →