Apache Hama is a distributed computing framework based on bulk synchronous parallel computing techniques for massive scientific computations e.g., matrix, graph and network algorithms. Originally a sub-project of Hadoop, it became an Apache Software Foundation top level project in 2012. It was created by Edward J. Yoon, who named it (short for "Hadoop Matrix Algebra"), and Hama also means hippopotamus in Yoon's native Korean language (하마), following the trend of naming Apache projects after animals and zoology (such as Apache Pig). Hama was inspired by Google's Pregel large-scale graph computing framework described in 2010. When executing graph algorithms, Hama showed a fifty-fold performance increase relative to Hadoop. Retired in April 2020, project resources are made available as part of the Apache Attic. Yoon cited issues of installation, scalability, and a difficult programming model for its lack of adoption. == Architecture == Hama consists of three major components: BSPMaster, GroomServers and Zookeeper. === BSPMaster === BSPMaster is responsible for: Maintaining groom server status Controlling super steps in a cluster Maintaining job progress information Scheduling jobs and assigning tasks to groom servers Disseminating execution class across groom servers Controlling fault Providing users with the cluster control interface. A BSP Master and multiple grooms are started by the script. Then, the bsp master starts up with a RPC server for groom servers. Groom servers starts up with a BSPPeer instance and a RPC proxy to contact the bsp master. After started, each groom periodically sends a heartbeat message that encloses its groom server status, including maximum task capacity, unused memory, and so on. Each time the BSP master receives a heartbeat message, it brings the groom server status up-to-date. The bsp master makes use of groom servers' status in order to assign tasks to idle groom servers - and returns a heartbeat response containing assigned tasks and others actions for a groom server to do. Currently BSP master has a FIFO job scheduler and simple task assignment algorithms. === GroomServer === A groom server (shortly referred to as groom) is a process that performs BSP tasks assigned by BSPMaster. Each groom contacts the BSPMaster, and it takes assigned tasks and reports its status by means of periodical piggybacks with BSPMaster. Each groom is designed to run with HDFS or other distributed storages. Basically, a groom server and a data node should be run on one physical node. === Zookeeper === A Zookeeper is used to manage the efficient barrier synchronisation of the BSPPeers.
Curse of dimensionality
The curse of dimensionality refers to various phenomena that arise when analyzing and organizing data in high-dimensional spaces that do not occur in low-dimensional settings such as the three-dimensional physical space of everyday experience. The expression was coined by Richard E. Bellman when considering problems in dynamic programming. The curse generally refers to issues that arise when the number of datapoints is small (in a suitably defined sense) relative to the intrinsic dimension of the data. Dimensionally cursed phenomena occur in domains such as numerical analysis, sampling, combinatorics, machine learning, data mining and databases. The common theme of these problems is that when the dimensionality increases, the volume of the space increases so fast that the available data becomes sparse. In order to obtain a reliable result, the amount of data needed often grows exponentially with the dimensionality. Also, organizing and searching data often relies on detecting areas where objects form groups with similar properties; in high dimensional data, however, all objects appear to be sparse and dissimilar in many ways, which prevents common data organization strategies from being efficient. == Domains == === Combinatorics === In some problems, each variable can take one of several discrete values, or the range of possible values is divided to give a finite number of possibilities. Taking the variables together, a huge number of combinations of values must be considered. This effect is also known as the combinatorial explosion. Even in the simplest case of d {\displaystyle d} binary variables, the number of possible combinations already is 2 d {\displaystyle 2^{d}} , exponential in the dimensionality. Naively, each additional dimension doubles the effort needed to try all combinations. === Sampling === There is an exponential increase in volume associated with adding extra dimensions to a mathematical space. For example, 102 = 100 evenly spaced sample points suffice to sample a unit interval (try to visualize a "1-dimensional" cube, i.e. a line) with no more than 10−2 = 0.01 distance between points; an equivalent sampling of a 10-dimensional unit hypercube with a lattice that has a spacing of 10−2 = 0.01 between adjacent points would require 1020 = [(102)10] sample points. In general, with a spacing distance of 10−n the 10-dimensional hypercube appears to be a factor of 10n(10−1) = [(10n)10/(10n)] "larger" than the 1-dimensional hypercube, which is the unit interval. In the above example n = 2: when using a sampling distance of 0.01 the 10-dimensional hypercube appears to be 1018 "larger" than the unit interval. This effect is a combination of the combinatorics problems above and the distance function problems explained below. === Optimization === When solving dynamic optimization problems by numerical backward induction, the objective function must be computed for each combination of values. This is a significant obstacle when the dimension of the "state variable" is large. === Machine learning === In machine learning problems that involve learning a "state-of-nature" from a finite number of data samples in a high-dimensional feature space with each feature having a range of possible values, typically an enormous amount of training data is required to ensure that there are several samples with each combination of values. In an abstract sense, as the number of features or dimensions grows, the amount of data we need to generalize accurately grows exponentially. A typical rule of thumb is that there should be at least 5 training examples for each dimension in the representation. In machine learning and insofar as predictive performance is concerned, the curse of dimensionality is used interchangeably with the peaking phenomenon, which is also known as Hughes phenomenon. This phenomenon states that with a fixed number of training samples, the average (expected) predictive power of a classifier or regressor first increases as the number of dimensions or features used is increased but beyond a certain dimensionality it starts deteriorating instead of improving steadily. Nevertheless, in the context of a simple classifier (e.g., linear discriminant analysis in the multivariate Gaussian model under the assumption of a common known covariance matrix), Zollanvari et al. showed both analytically and empirically that as long as the relative cumulative efficacy of an additional feature set (with respect to features that are already part of the classifier) is greater (or less) than the size of this additional feature set, the expected error of the classifier constructed using these additional features will be less (or greater) than the expected error of the classifier constructed without them. In other words, both the size of additional features and their (relative) cumulative discriminatory effect are important in observing a decrease or increase in the average predictive power. In metric learning, higher dimensions can sometimes allow a model to achieve better performance. After normalizing embeddings to the surface of a hypersphere, FaceNet achieves the best performance using 128 dimensions as opposed to 64, 256, or 512 dimensions in one ablation study. A loss function for unitary-invariant dissimilarity between word embeddings was found to be minimized in high dimensions. === Data mining === In data mining, the curse of dimensionality refers to a data set with too many features. Consider the first table, which depicts 200 individuals and 2000 genes (features) with a 1 or 0 denoting whether or not they have a genetic mutation in that gene. A data mining application to this data set may be finding the correlation between specific genetic mutations and creating a classification algorithm such as a decision tree to determine whether an individual has cancer or not. A common practice of data mining in this domain would be to create association rules between genetic mutations that lead to the development of cancers. To do this, one would have to loop through each genetic mutation of each individual and find other genetic mutations that occur over a desired threshold and create pairs. They would start with pairs of two, then three, then four until they result in an empty set of pairs. The complexity of this algorithm can lead to calculating all permutations of gene pairs for each individual or row. Given the formula for calculating the permutations of n items with a group size of r is: n ! ( n − r ) ! {\displaystyle {\frac {n!}{(n-r)!}}} , calculating the number of three pair permutations of any given individual would be 7988004000 different pairs of genes to evaluate for each individual. The number of pairs created will grow by an order of factorial as the size of the pairs increase. The growth is depicted in the permutation table (see right). As we can see from the permutation table above, one of the major problems data miners face regarding the curse of dimensionality is that the space of possible parameter values grows exponentially or factorially as the number of features in the data set grows. This problem critically affects both computational time and space when searching for associations or optimal features to consider. Another problem data miners may face when dealing with too many features is that the number of false predictions or classifications tends to increase as the number of features grows in the data set. In terms of the classification problem discussed above, keeping every data point could lead to a higher number of false positives and false negatives in the model. This may seem counterintuitive, but consider the genetic mutation table from above, depicting all genetic mutations for each individual. Each genetic mutation, whether they correlate with cancer or not, will have some input or weight in the model that guides the decision-making process of the algorithm. There may be mutations that are outliers or ones that dominate the overall distribution of genetic mutations when in fact they do not correlate with cancer. These features may be working against one's model, making it more difficult to obtain optimal results. This problem is up to the data miner to solve, and there is no universal solution. The first step any data miner should take is to explore the data, in an attempt to gain an understanding of how it can be used to solve the problem. One must first understand what the data means, and what they are trying to discover before they can decide if anything must be removed from the data set. Then they can create or use a feature selection or dimensionality reduction algorithm to remove samples or features from the data set if they deem it necessary. One example of such methods is the interquartile range method, used to remove outliers in a data set by calculating the standard deviation of a feature or occurrence. === Distance function === When a measure such as a Euclidean distance is defined using many coordinat
PNGOUT
PNGOUT is a freeware command line optimizer for PNG images written by Ken Silverman. The transformation is lossless, meaning that the resulting image is visually identical to the source image. According to its author, this program can often get higher compression than other optimizers by 5–10%. It is possible to compress some inflated PNGs to a size below 1% of the original file. PNGOUT was also available as a plug-in for the freeware image viewer IrfanView and can be enabled as an option when saving files. It allows editing of various PNGOUT settings via a dialog box. PNGOUT integration was removed in IrfanView version 4.58 in favour of OptiPNG. In 2006, a commercial version of PNGOUT with a graphical user interface, known as PNGOUTWin, was released by Ardfry Imaging, a small company Silverman co-founded in 2005. There is also a freeware GUI frontend to PNGOUT available, known as PNGGauntlet. == Main operation == The main function of PNGOUT is to reduce the size of image data contained in the IDAT chunk. This chunk is compressed using the deflate algorithm. Deflate algorithms can vary in speed and compression ratio, with higher compression ratios generally implying lower speed. Ken Silverman wrote a deflate compressor for PNGOUT that is slower than the ones used in most graphics software, but produces smaller files. PNGOUT also performs automatic bit depth, color, and palette reduction where appropriate.
Drops (app)
Drops is a language learning app that was created in Estonia by Daniel Farkas and Mark Szulyovszky in 2015. It is the second product from the company, after their first app, LearnInvisible, had issues in retaining a user's engagement over the required time period. The languages available include Native Hawaiian and Māori, and was classified as one of the fifty "Most Innovative Companies" for 2019 by Fast Company. The company partnered with Global Eagle Entertainment to include Travel Talk, a feature intended to focus on words and phrases frequently used by travelers. At the beginning of the COVID-19 pandemic in March 2020, the number of users increased by 55 percent in the United States and 92 percent in the United Kingdom. Droplets, a language app for children, includes profiles for multiple teachers working with remote students. The company also produces an app called Scripts, intended to help users learn to write alphabets. The app was purchased by the Norwegian company Kahoot! on 24 November 2020.
Conference app
A conference app, also known as an event app or meeting app, is a mobile app developed to help attendees and meeting planners manage their conference experience. It typically includes conference proceedings and venue information, allowing users to create personalized schedules and engage with other users. A conference app can be a native app or web-based. In recent years, conference apps have gained in popularity as a sustainable solution for event management by reducing paper produced by printed materials. Advanced features often include real-time notifications for updates or changes, integration with virtual meeting platforms for hybrid or fully online events, and analytics tools for organizers to measure attendance and engagement. Additionally, some apps support sponsorship and exhibitor features, enabling businesses to showcase their products or services directly within the app.
Educational robotics
Educational robotics teaches the design, analysis, application and operation of robots. Robots include articulated robots, mobile robots or autonomous vehicles. Educational robotics can be taught from elementary school to graduate programs. Robotics may also be used to motivate and facilitate the instruction other, often foundational, topics such as computer programming, artificial intelligence or engineering design. == Education and training == Robotics engineers design robots, maintain them, develop new applications for them, and conduct research to expand the potential of robotics. Robots have become a popular educational tool in some middle and high schools, as well as in numerous youth summer camps, raising interest in programming, artificial intelligence and robotics among students. First-year computer science courses at several universities now include programming of a robot in addition to traditional software engineering-based coursework. == Category of Educational robotics == The categories of educational robots seen as having more than one category. It can be alienated into different categories based on their physical design and coding method. Generally they are categorised as arm robots, wheeled mobile robots and humanoid robots. Tangibly, coded robots uses a physical means of coding instead of the screens coding. === Initiatives in schools === Leachim, was a robot teacher programmed with the class curricular, as well as certain biographical information on the 40 students whom it was programmed to teach. Leachim could synthesize human speech using Diphone synthesis. It was invented by Michael J. Freeman in 1974 and was tested in a fourth grade classroom in the Bronx, New York. === Post-secondary degree programs === From approximately 1960 through 2005, robotics education at post-secondary institutions took place through elective courses, thesis experiences and design projects offered as part of degree programs in traditional academic disciplines, such as mechanical engineering, electrical engineering, industrial engineering or computer science. Since 2005, more universities have begun granting degrees in robotics as a discipline in its own right, often under the name "Robotic Engineering". Based on a 2015 web-based survey of robotics educators, the degree programs and their estimates annual graduates are listed alphabetically below. Note that only official degree programs where the word "robotics" appears on the transcript or diploma are listed here; whereas degree programs in traditional disciplines with course concentrations or thesis topics related to robotics are deliberately omitted. === Certification === The Robotics Certification Standards Alliance (RCSA) is an international robotics certification authority that confers various industry- and educational-related robotics certifications. === Summer robotics camp === Several summer camp programs include robotics as part of their core curriculum. In addition, youth summer robotics programs are frequently offered by celebrated museums such as the American Museum of Natural History and The Tech Museum of Innovation in Silicon Valley, CA, just to name a few. There are of benefits that come from attending robotics camps. It teaches students how to use teamwork, resilience and motivation, and decision-making. Students learn teamwork because most camps involve exciting activities requiring teamwork. Resilience and motivation is expected because by completing the challenging programs, students feel talented and accomplished after they complete the program. Also students are given unique situations making them make decisions to further their situation. === Educational robotics in special education === Educational robotics can be a useful tool in early and special education. According to a journal on new perspectives in science education, educational robotics can help to develop abilities that promote autonomy and assist their integration into society. Social and personal skills can also be developed through educational robotics. Using Lego Mindstorms NXT, schoolteachers were able to work with middle school aged children in order to develop programs and improve the children's social and personal skills. Additionally, problem solving skills and creativity were utilized through the creation of artwork and scenery to house the robots. Other studies show the benefits of educational robotics in special education as promoting superior cognitive functions, including executive functions. This can lead to an increased ability in "problem solving, reasoning and planning in typically developing preschool children." Through eight weeks of weekly forty-five-minute group sessions using the Bee-Bot, an increase in interest, attention, and interaction between both peers and adults was found in the school and preschool-aged children with Down Syndrome. This study suggests that educational robotics in the classroom can also lead to an improvement in visuo-spatial memory and mental planning. Furthermore, executive functions seemed to be possible in one child during this study.
Plotly
Plotly is a technical computing company headquartered in Montreal, Quebec, that develops online data analytics and visualization tools. Plotly provides online graphing, analytics, and statistics tools for individuals and collaboration, as well as scientific graphing libraries for Python, R, MATLAB, Perl, Julia, Arduino, JavaScript and REST. == History == Plotly was founded by Alex Johnson, Jack Parmer, Chris Parmer, and Matthew Sundquist. The founders' backgrounds are in science, energy, and data analysis and visualization. Early employees include Christophe Viau, a Canadian software engineer and Ben Postlethwaite, a Canadian geophysicist. Plotly was named one of the Top 20 Hottest Innovative Companies in Canada by the Canadian Innovation Exchange. Plotly was featured in "startup row" at PyCon 2013, and sponsored the SciPy 2018 conference. Plotly raised $5.5 million during its Series A funding, led by MHS Capital, Siemens Venture Capital, Rho Ventures, Real Ventures, and Silicon Valley Bank. The Boston Globe and Washington Post newsrooms have produced data journalism using Plotly. In 2020, Plotly was named a Best Place to Work by the Canadian SME National Business Awards, and nominated as Business of the Year. == Products == Plotly offers open-source and enterprise products. Dash is an open-source Python, R, and Julia framework for building web-based analytic applications. Many specialized open-source Dash libraries exist that are tailored for building domain-specific Dash components and applications. Some examples are Dash DAQ, for building data acquisition GUIs to use with scientific instruments, and Dash Bio, which enables users to build custom chart types, sequence analysis tools, and 3D rendering tools for bioinformatics applications. Dash Enterprise is Plotly's paid product for building, testing, deploying, managing and scaling Dash applications organization-wide. Chart Studio Cloud is a free, online tool for creating interactive graphs. It has a point-and-click graphical user interface for importing and analyzing data into a grid and using stats tools. Graphs can be embedded or downloaded. Chart Studio Enterprise is a paid product that allows teams to create, style, and share interactive graphs on a single platform. It offers expanded authentication and file export options, and does not limit sharing and viewing. Data visualization libraries Plotly.js is an open-source JavaScript library for creating graphs and powers Plotly.py for Python, as well as Plotly.R for R, MATLAB, Node.js, Julia, and Arduino and a REST API. Plotly can also be used to style interactive graphs with Jupyter notebook. Figure converters which convert matplotlib, ggplot2, and IGOR Pro graphs into interactive, online graphs. == Data visualization libraries == Plotly provides a collection of supported chart types across several programming languages: == Dash == Dash is a Python framework built on top of React, a JavaScript library. Dash also works for R, and most recently supports Julia. While still described as a Python framework, Python isn't used for the other languages: "... describing Dash as a Python framework misses a key feature of its design: the Python side (the back end/server) of Dash was built to be lightweight and stateless [allowing] multiple back-end languages to coexist on an equal footing". It is possible to integrate D3.js charts as Dash components. Dash provides the default CSS (plus HTML and JavaScript), but for custom styling Dash applications, CSS can be added, or Dash Enterprise used. === Dash Enterprise === Dash Enterprise is Plotly's paid product for building, testing, deploying, managing and scaling Dash applications organization-wide. The product integrates with enterprise IT systems to enable organizations to build, deploy and scale low-code Dash applications. With open-source Dash, analytic applications can be run from a local machine, but cannot be easily accessed by others in the organization. ==== Enterprise IT integration ==== Dash Enterprise installs on cloud environments and on-premises. Amazon Web Services, Google Cloud Platform, and Microsoft Azure are supported, as are multiple Linux on-premises servers. Authentication integrations include LDAP, AD, PKI, Okta, SAML, OAuth2, SSO, and email authentication, and Dash application access is managed through a GUI rather than code. Dash Enterprise connects to major big data backends, including Salesforce, PostgreSQL, Databricks via PySpark, Snowflake, Dask, Datashader, and Vaex. In 2020, Plotly partnered with NVIDIA to integrate Dash with RAPIDS, and NVIDIA participated in Plotly's Series C funding round. ==== Low-code capabilities ==== Dash Enterprise enables low-code development of Dash applications, which is not possible with open-source Dash. Enterprise users can write applications in multiple development environments, including Jupyter Notebook. Dash Enterprise ships with several “development engines” for drag-and-drop application editing, application design, and automated reporting, as well as dozens of artificial intelligence and machine learning application templates. ==== Deployment and scaling ==== Dash application code is deployed to Dash Enterprise using the git-push command. Dash application deployments are containerized to avoid dependency conflicts, and can be embedded in existing web platforms without iframes. Deployed applications can be managed and accessed in a single portal called App Manager, where administrators can control user authentication and view usage analytics. Dash Enterprise scales horizontally with Kubernetes. Jobs queuing, GPU acceleration, and CPU parallelization support high performance computing requirements. Plotly also offers professional services for application development and workshop training.