This is a list of data science software and platforms used in data science, which includes programming languages, programming environments, machine learning frameworks, data engineering tools, statistical software, data analysis, plotting, MLOps systems, and more. == Programming languages == == Development environments == These interactive notebooks, IDEs, and platforms provide specialised development environments. Apache Zeppelin Architect — Eclipse (software) CoCalc Dataiku Data Science Studio FreeMat GNU Octave Google Colab DataSpell Jupyter Notebook / JupyterLab Kaggle Notebooks MATLAB O-Matrix PyCharm RStudio SAS (software) and SAS Studio Spyder Visual Studio Code == Machine and deep learning software == The Machine learning / deep learning tools support development in those fields. == Data engineering == Examples of Data engineering tools. Apache Airflow Apache Flink Apache Hadoop Apache Kafka Apache NiFi Apache Spark Dask Data build tool (dbt) == Data mining == Examples of Data mining tools. === Free and open-source === === Proprietary === == Database management == === List of RDBMS === ==== Proprietary ==== == Data warehouses == Data warehouse environments include: Amazon Redshift Snowflake Google BigQuery Microsoft Azure Synapse Teradata Vertica == Data lakes == Data lake environments include: Apache Hadoop Cloudera Databricks Delta Lake Amazon S3 Google Cloud Storage Azure Data Lake == Algorithms == Apriori algorithm – frequent itemset mining and association rule learning in market basket analysis Backpropagation – algorithm for training artificial neural networks using gradient descent Decision Trees – tree-based algorithm for classification and regression Expectation–maximization algorithm – iterative procedure for maximum likelihood estimation with latent variables Gradient descent – iterative optimization algorithm for minimizing a loss function ID3 algorithm – used to generate a decision tree from a dataset K-Means – clustering algorithm based on minimizing within-cluster distances K-Nearest Neighbors (KNN) – instance-based learning and classification method Linear regression – estimation method for predicting a dependent variable based on independent variables Logistic regression – classification algorithm for predicting a binary outcome Naive Bayes – probabilistic classifier based on Bayes' theorem Ordinary least squares – estimation method for parameters in linear regression PageRank – graph-based algorithm for link analysis and search ranking Principal component analysis – technique to reduce high-dimensional data while preserving variance Q-learning – reinforcement learning algorithm for learning optimal actions Random forest – ensemble of decision trees for improved classification or regression Sequential minimal optimization – solver for training support vector machines Stochastic gradient descent – randomized variant of gradient descent for large-scale machine learning Support Vector Machines (SVM) – algorithm for finding a hyperplane to separate classes == Statistical software == === Open-source === === Public domain === CSPro Dataplot Epi Map X-13ARIMA-SEATS === Freeware === BV4.1 MINUIT WinBUGS Winpepi === Proprietary === == Data processing == Tools for Data processing and analysis: == Data and information visualization == Software for Data visualization: == Plotting software == Software for plotting data to support processing and visualise results. == Maps and geospatial visualization == ArcGIS Carto Epi Map GeoDA Google Earth Engine Leaflet Mapbox MountainsMap QGIS == Machine learning == MLOps and model deployment: BentoML Data Version Control (DVC) Kubeflow MLflow Seldon Core Streamlit TensorFlow Serving Weights & Biases == Data repositories == Kaggle – platform for data science competitions, datasets, and notebooks. OpenML – collaborative platform for sharing datasets, algorithms, and experiments. University of California, Irvine Machine Learning Repository Zenodo – open-access repository supported by CERN and the EU. == Educational data science software == Kaggle – online platform for data science education, competitions, datasets, and collaborative learning. KNIME – open-source data analytics platform used for teaching data science, machine learning, and workflow-based analysis. RapidMiner – used in academic research and education for data mining and machine learning. Statistics Online Computational Resource (SOCR) – online tools and instructional resources for statistics education. Tanagra (machine learning) – data mining software developed for research and teaching purposes. TinkerPlots – explore and analyze data through visual modeling.
Quack.com
Quack.com was an early voice portal company. The domain name later was used for Quack, an iPad search application from AOL. == History == It was founded in 1998 by Steven Woods, Jeromy Carriere and Alex Quilici as a Pittsburgh, Pennsylvania, USA, based voice portal infrastructure company named Quackware. Quack was the first company to try to create a voice portal: a consumer-based destination "site" in which consumers could not only access information by voice alone, but also complete transactions. Quackware launched a beta phone service in 1999 that allowed consumers to purchase books from sites such as Amazon and CDs from sites such as CDNow by answering a short set of questions. Quack followed with a set of information services from movie listings (inspired by, but expanding upon, Moviefone) to news, weather and stock quotes. This concept introduced a series of lookalike startups including Tellme Networks which raised more money than any Internet startup in history on a similar concept. Quack received its first venture funding from HDL Capital in 1999 and moved operations to Mountain View in Silicon Valley, California in 1999. A deal with Lycos was announced in May 2000. In September 2000 Quack was acquired for $200 million by America Online (AOL) and moved onto the Netscape campus with what was left of the Netscape team. Quack was attacked in the Canadian press for being representative of the Canadian "brain drain" to the US during the Internet bubble, focusing its recruiting efforts on the University of Waterloo, hiring more than 50 engineers from Waterloo in less than 10 months. Quack competitor Tellme Networks raised enormous funds in what became a highly competitive market in 2000, with the emergence of more than a dozen additional competitors in a 12-month period. Following its acquisition by America Online in an effort led by Ted Leonsis to bring Quack into AOL Interactive, the Quack voice service became AOLbyPhone as one of AOL's "web properties" along with MapQuest, Moviefone and others. Quack secured several patents that underlie the technical challenges of delivering interactive voice services. Constructing a voice portal required integrations and innovations not only in speech recognition and speech generation, but also in databases, application specification, constraint-based reasoning and artificial intelligence and computational linguistics. "Quack"'s name derived from the company goal of providing not only voice-based services, but more broadly "Quick Ubiquitous Access to Consumer Knowledge". The patents assigned to Quack.com include: System and method for voice access to Internet-based information, System and method for advertising with an Internet Voice Portal and recognizing the axiom that in interactive voice systems one must "know the set of possible answers to a question before asking it". System and method for determining if one web site has the same information as another web site. Quack.com was spoofed in The Simpsons in March 2002 in the episode "Blame It on Lisa" in which a "ComQuaak" sign is replaced by another equally crazy telecom company name. == 2010 onwards == In July 2010, quack.com became the focus of a new AOL iPad application, that was a web search experience. The product delivers web results and blends in picture, video and Twitter results. It enables you to preview the web results before you go to the site, search within each result, and flip through the results pages, making full use of the iPad's touch screen features. The iPad app was free via iTunes, but support discontinued in 2012.
Harvest now, decrypt later
Harvest now, decrypt later (HNDL) is a surveillance strategy that relies on the acquisition and long-term storage of currently unreadable encrypted data awaiting possible breakthroughs in decryption technology that would render it readable in the future—a hypothetical date referred to as Y2Q (a reference to Y2K), or Q-Day. The most common concern is the prospect of developments in quantum computing which would allow current strong encryption algorithms to be broken at some time in the future, making it possible to decrypt any stored material that had been encrypted using those algorithms. However, the improvement in decryption technology need not be due to a quantum-cryptographic advance; any other form of attack capable of enabling decryption would be sufficient. The existence of this strategy has led to concerns about the need to urgently deploy post-quantum cryptography; even though no practical quantum attacks yet exist, some data stored now may still remain sensitive even decades into the future. As of 2022, the U.S. federal government has proposed a roadmap for organizations to start migrating toward quantum-cryptography-resistant algorithms to mitigate these threats. This new version of Commercial National Security Algorithm Suite uses publicly-available algorithms and is allowed for government use up to the TOP SECRET level. == Terminology and scope == The term “harvest now, decrypt later” encompasses various surveillance or espionage operations in which ciphertext or encrypted communications are collected today with the view that they may one day be decrypted, given sufficient advances in computing power or cryptanalysis. The abbreviation HNDL is sometimes used in technical and policy documents. The “Y2Q” (or “Q-Day”) label draws an analogy to the Y2K date-change issue, emphasising a potential future point at which current cryptography may collapse. The strategy is particularly relevant for data with long confidentiality lifetimes, such as diplomatic communications, personal health records, critical infrastructure logs, or intellectual property. == Mitigation strategies == The primary defense against HNDL attacks is the transition to post-quantum cryptography (PQC), which utilizes algorithms believed to be secure against quantum computer attacks. However, because PQC protects the data payload digitally, rather than the transmission itself, the encrypted data can still be harvested and stored. A complementary approach involves physical layer security (also known as optical layer encryption or photonic shielding). Unlike algorithmic encryption, this method modifies the optical waveform itself—often by burying the signal within optical noise or using spectral phase encoding—to render the transmission unrecordable by standard receivers. By preventing the attacker from capturing a valid signal in the first place, this approach aims to eliminate the "harvest" phase of the threat. Commercial implementations of harvest-proof optical encryption have been developed by firms such as CyberRidge to secure long-haul fiber networks. Field trials have demonstrated 100 Gbps throughput over legacy DWDM networks using this method.
White-box cryptography
In cryptography, the white-box model refers to an extreme attack scenario, in which an adversary has full unrestricted access to a cryptographic implementation, most commonly of a block cipher such as the Advanced Encryption Standard (AES). A variety of security goals may be posed (see the section below), the most fundamental being "unbreakability", requiring that any (bounded) attacker should not be able to extract the secret key hardcoded in the implementation, while at the same time the implementation must be fully functional. In contrast, the black-box model only provides an oracle access to the analyzed cryptographic primitive (in the form of encryption and/or decryption queries). There is also a model in-between, the so-called gray-box model, which corresponds to additional information leakage from the implementation, more commonly referred to as side-channel leakage. White-box cryptography is a practice and study of techniques for designing and attacking white-box implementations. It has many applications, including digital rights management (DRM), pay television, protection of cryptographic keys in the presence of malware, mobile payments and cryptocurrency wallets. Examples of DRM systems employing white-box implementations include CSS and Widevine. White-box cryptography is closely related to the more general notions of obfuscation, in particular, to Black-box obfuscation, proven to be impossible, and to Indistinguishability obfuscation, constructed recently under well-founded assumptions but so far being infeasible to implement in practice. As of January 2023, there are no publicly known unbroken white-box designs of standard symmetric encryption schemes. On the other hand, there exist many unbroken white-box implementations of dedicated block ciphers designed specifically to achieve incompressibility (see § Security goals). == Security goals == Depending on the application, different security goals may be required from a white-box implementation. Specifically, for symmetric-key algorithms the following are distinguished: Unbreakability is the most fundamental goal requiring that a bounded attacker should not be able to recover the secret key embedded in the white-box implementation. Without this requirement, all other security goals are unreachable since a successful attacker can simply use a reference implementation of the encryption scheme together with the extracted key. One-wayness requires that a white-box implementation of an encryption scheme can not be used by a bounded attacker to decrypt ciphertexts. This requirement essentially turns a symmetric encryption scheme into a public-key encryption scheme, where the white-box implementation plays the role of the public key associated to the embedded secret key. This idea was proposed already in the famous work of Diffie and Hellman in 1976 as a potential public-key encryption candidate. Code lifting security is an informal requirement on the context, in which the white-box program is being executed. It demands that an attacker can not extract a functional copy of the program. This goal is particularly relevant in the DRM setting. Code obfuscation techniques are often used to achieve this goal. A commonly used technique is to compose the white-box implementation with so-called external encodings. These are lightweight secret encodings that modify the function computed by the white-box part of an application. It is required that their effect is canceled in other parts of the application in an obscure way, using code obfuscation techniques. Alternatively, the canceling counterparts can be applied on a remote server. Incompressibility requires that an attacker can not significantly compress a given white-box implementation. This can be seen as a way to achieve code lifting security (see above), since exfiltrating a large program from a constrained device (for example, an embedded or a mobile device) can be time-consuming and may be easy to detect by a firewall. Examples of incompressible designs include SPACE cipher, SPNbox, WhiteKey and WhiteBlock. These ciphers use large lookup tables that can be pseudorandomly generated from a secret master key. Although this makes the recovery of the master key hard, the lookup tables themselves play the role of an equivalent secret key. Thus, unbreakability is achieved only partially. Traceability (Traitor tracing) requires that each distributed white-box implementation contains a digital watermark allowing identification of the guilty user in case the white-box program is being leaked and distributed publicly. == History == The white-box model with initial attempts of white-box DES and AES implementations were first proposed by Chow, Eisen, Johnson and van Oorshot in 2003. The designs were based on representing the cipher as a network of lookup tables and obfuscating the tables by composing them with small (4- or 8-bit) random encodings. Such protection satisfied a property that each single obfuscated table individually does not contain any information about the secret key. Therefore, a potential attacker has to combine several tables in their analysis. The first two schemes were broken in 2004 by Billet, Gilbert, and Ech-Chatbi using structural cryptanalysis. The attack was subsequently called "the BGE attack". The numerous consequent design attempts (2005-2022) were quickly broken by practical dedicated attacks. In 2016, Bos, Hubain, Michiels and Teuwen showed that an adaptation of standard side-channel power analysis attacks can be used to efficiently and fully automatically break most existing white-box designs. This result created a new research direction about generic attacks (correlation-based, algebraic, fault injection) and protections against them. == Competitions == Four editions of the WhibOx contest were held in 2017, 2019, 2021 and 2024 respectively. These competitions invited white-box designers both from academia and industry to submit their implementation in the form of (possibly obfuscated) C code. At the same time, everyone could attempt to attack these programs and recover the embedded secret key. Each of these competitions lasted for about 4-5 months. WhibOx 2017 / CHES 2017 Capture the Flag Challenge targeted the standard AES block cipher. Among 94 submitted implementations, all were broken during the competition, with the strongest one staying unbroken for 28 days. WhibOx 2019 / CHES 2019 Capture the Flag Challenge again targeted the AES block cipher. Among 27 submitted implementations, 3 programs stayed unbroken throughout the competition, but were broken after 51 days since the publication. WhibOx 2021 / CHES 2021 Capture the Flag Challenge changed the target to ECDSA, a digital signature scheme based on elliptic curves. Among 97 submitted implementations, all were broken within at most 2 days. WhibOx 2024 / CHES 2024 Capture the Flag Challenge again targeted ECDSA. Among 47 submitted implementations, all were broken during the competition, with the strongest one staying unbroken for almost 5 days.
Social recruiting
Social recruiting (social hiring or social media recruitment) is recruiting candidates by using social platforms as talent databases or for advertising. Social recruiting uses social media profiles, blogs, and other Internet sites to find information on candidates. It also uses social media to advertise jobs either through HR vendors or through crowdsourcing where job seekers and others share job openings within their online social networks. Social recruiting's effectiveness and return on investment have been difficult to determine, since applicants do not usually apply through the social channels which first attracted them. In May 2013, Maximum Employment Marketing Group released the Social Recruitment Monitor, which ranks the reach, engagement, and interactivity of employers' social recruiting efforts around the world. == Social recruitment software == The social recruitment software market (a form of e-recruitment) is often included in the wider talent management software sector. Bersin & Associates valued the wider talent management market at over $2bn in 2007. Social recruitment increasingly sits at an intersection of a number of fast-moving areas including social networking, recruitment and now cloud computing. Additionally, mobile recruiting has become another hot topic, especially with the rise in tablet and smartphone usage. In 2012, there was a rise of tech companies using social recruiting applications to find and screen applicants. As more companies saw value in filling jobs by putting them on the social platforms where millions of people spend at least 37 minutes daily, there developed a much larger focus on social recruiting among the talent acquisition community. By mid-2013, many major enterprise companies such as Pepsi, Gap, AIG, and Oracle had begun effectively utilizing social recruiting software, making it clear that large corporations were open to automating or streamlining (and ultimately investing in) their social recruiting processes.
DBGallery
DBGallery, short for Database Gallery, is a cloud-based Software as a Service (SaaS) and on-prem webserver for teams of various sizes. DBGallery enables users to centrally store, manage, catalog, archive, and securely share image, video, and document files. It facilitates version control, detects duplicates, and offers an intuitive and advanced search functionality, making assets easily accessible to all users. It takes advantage of current AI technologies to automatically add significant metadata to images, facilitates custom-trained AI models, and offers bespoke AI features. Additionally, DBGallery provides team management tools, workflow management, an activity audit trail, and other collaborative features that foster a productive environment for both internal and external stakeholders. == History == DBGallery's first public release was December 2007. Since then each year has seen continuous enhancements. 2013 added support for additional non-English languages in its meta-data. 2014 added support for creating custom data fields for tagging and search. In 2015 included the ability to auto-tag images using Reverse Geocoding. 2018 added artificial intelligence (AI) image recognition as a further addition to auto-tagging. March 2020 added complete image collection management via the web (e.g. file and folder drag and drop), a new collection dashboard, custom data layouts, and an improved audit trail. 2021 saw user experience improvements provided by improved styling and performance enhancements. Version 12 was released in October 2021. It added the ability to upload unlimited file sizes and made significant performance improvements for very large collections. June 2022 saw the release of a global duplicate images search. In late 2022, DBGallery began offering significantly reduced cloud storage cost, at a third of its previous prices, which played into its recent high-volume/high-capacity capabilities and its clients' subsequent demand for additional storage. 2023 saw improvements in user and role management, introduced it's mobile app (PWA), and improved custom-trained object detection. Release 14.0 in the spring of 2024 had large sharing improvements and a new find related images feature. Winter 2025's v15 release introduced AI-generated image descriptions, image-to-text, and facial recognition.
Data lake
A data lake is a system or repository of data stored in its natural/raw format, usually object blobs or files. A data lake is usually a single store of data including raw copies of source system data, sensor data, social data etc., and transformed data used for tasks such as reporting, visualization, advanced analytics, and machine learning. A data lake can include structured data from relational databases (rows and columns), semi-structured data (CSV, logs, XML, JSON), unstructured data (emails, documents, PDFs), and binary data (images, audio, video). A data lake can be established on premises (within an organization's data centers) or in the cloud (using cloud services). == Background == James Dixon, then chief technology officer at Pentaho, coined the term by 2011 to contrast it with data mart, which is a smaller repository of interesting attributes derived from raw data. In promoting data lakes, he argued that data marts have several inherent problems, such as information siloing. PricewaterhouseCoopers (PwC) said that data lakes could "put an end to data silos". In their study on data lakes, they noted that enterprises were "starting to extract and place data for analytics into a single, Hadoop-based repository." == Examples == Many companies use cloud storage services such as Google Cloud Storage and Amazon S3 or a distributed file system such as Apache Hadoop distributed file system (HDFS). There is a gradual academic interest in the concept of data lakes. For example, Personal DataLake at Cardiff University is a new type of data lake which aims at managing big data of individual users by providing a single point of collecting, organizing, and sharing personal data. Early data lakes, such as Hadoop 1.0, had limited capabilities because it only supported batch-oriented processing (Map Reduce). Interacting with it required expertise in Java, map reduce and higher-level tools like Apache Pig, Apache Spark and Apache Hive (which were also originally batch-oriented). == Criticism == Poorly managed data lakes have been facetiously called data swamps. In June 2015, David Needle characterized "so-called data lakes" as "one of the more controversial ways to manage big data". PwC was also careful to note in their research that not all data lake initiatives are successful. They quote Sean Martin, CTO of Cambridge Semantics: We see customers creating big data graveyards, dumping everything into Hadoop distributed file system (HDFS) and hoping to do something with it down the road. But then they just lose track of what’s there. The main challenge is not creating a data lake, but taking advantage of the opportunities it presents. They describe companies that build successful data lakes as gradually maturing their lake as they figure out which data and metadata are important to the organization. Another criticism is that the term data lake is used with many different meanings. It may be used to refer to, for example: any tools or data management practices that are not data warehouses; a particular technology for implementation; a raw data reservoir; a hub for ETL offload; or a central hub for self-service analytics. While critiques of data lakes are warranted, in many cases they apply to other data projects as well. For example, the definition of data warehouse is also changeable, and not all data warehouse efforts have been successful. In response to various critiques, McKinsey noted that the data lake should be viewed as a service model for delivering business value within the enterprise, not a technology outcome. == Data lakehouses == Data lakehouses are a hybrid approach that can ingest a variety of raw data formats like a data lake, while also providing ACID transactions and enforced data quality like a data warehouse.