AI Code Update

AI Code Update — independent reviews, comparisons, pricing and step-by-step guides on Aizhi.

  • Language technology

    Language technology

    Language technology, often called human language technology (HLT), studies methods of how computer programs or electronic devices can analyze, produce, modify or respond to human texts and speech. Working with language technology often requires broad knowledge not only about linguistics but also about computer science. It consists of natural language processing (NLP) and computational linguistics (CL) on the one hand, many application oriented aspects of these, and more low-level aspects such as encoding and speech technology on the other hand. Note that these elementary aspects are normally not considered to be within the scope of related terms such as natural language processing and (applied) computational linguistics, which are otherwise near-synonyms. As an example, for many of the world's lesser known languages, the foundation of language technology is providing communities with fonts and keyboard setups so their languages can be written on computers or mobile devices. Other tools also are part of modern language technology and include machine translation, speech recognition, text processing and natural language processing. Large scale AI models have recently advanced the field and enhanced the ability of machines to interpret complex human context.

    Read more →
  • Graphics

    Graphics

    Graphics (from Ancient Greek γραφικός (graphikós) 'pertaining to drawing, painting, writing, etc.') are visual images or designs on some surface, such as a wall, canvas, screen, paper, or stone, to inform, illustrate, or entertain. In contemporary usage, it includes a pictorial representation of data, as in design and manufacture, in typesetting and the graphic arts, and in educational and recreational software. Images that are generated by a computer are called computer graphics. Examples are photographs, drawings, line art, mathematical graphs, line graphs, charts, diagrams, typography, numbers, symbols, geometric designs, maps, engineering drawings, or other images. Graphics often combine text, illustration, and color. Graphic design may consist of the deliberate selection, creation, or arrangement of typography alone, as in a brochure, flyer, poster, web site, or book without any other element. The objective can be clarity or effective communication, association with other cultural elements, or merely the creation of a distinctive style. Graphics can be functional or artistic. The latter can be a recorded version, such as a photograph, or an interpretation by a scientist to highlight essential features, or an artist, in which case the distinction with imaginary graphics may become blurred. It can also be used for architecture. == History == The earliest graphics known to anthropologists studying prehistoric periods are cave paintings and markings on boulders, bone, ivory, and antlers, which were created during the Upper Palaeolithic period from 40,000 to 10,000 B.C. or earlier. Many of these were found to record astronomical, seasonal, and chronological details. Some of the earliest graphics and drawings are known to the modern world, from almost 6,000 years ago, are that of engraved stone tablets and ceramic cylinder seals, marking the beginning of the historical periods and the keeping of records for accounting and inventory purposes. Records from Egypt predate these and papyrus was used by the Egyptians as a material on which to plan the building of pyramids; they also used slabs of limestone and wood. From 600 to 250 BC, the Greeks played a major role in geometry. They used graphics to represent their mathematical theories such as the Circle Theorem and the Pythagorean theorem. In art, "graphics" is often used to distinguish work in a monotone and made up of lines, as opposed to painting. === Drawing === Drawing generally involves making marks on a surface by applying pressure from a tool or moving a tool across a surface. In which a tool is always used as if there were no tools it would be art. Graphical drawing is an instrumental guided drawing. === Printmaking === Woodblock printing, including images is first seen in China after paper was invented (about A.D. 105). In the West, the main techniques have been woodcut, engraving and etching, but there are many others. ==== Etching ==== Etching is an intaglio method of printmaking in which the image is incised into the surface of a metal plate using an acid. The acid eats the metal, leaving behind roughened areas, or, if the surface exposed to the acid is very thin, burning a line into the plate. The use of the process in printmaking is believed to have been invented by Daniel Hopfer (c. 1470–1536) of Augsburg, Germany, who decorated armour in this way. Etching is also used in the manufacturing of printed circuit boards and semiconductor devices. === Line art === Line art is a rather non-specific term sometimes used for any image that consists of distinct straight and curved lines placed against a (usually plain) background, without gradations in shade (darkness) or hue (color) to represent two-dimensional or three-dimensional objects. Line art is usually monochromatic, although lines may be of different colors. === Illustration === An illustration is a visual representation such as a drawing, painting, photograph or other work of art that stresses the subject more than form. The aim of an illustration is to elucidate or decorate a story, poem or piece of textual information (such as a newspaper article), traditionally by providing a visual representation of something described in the text. The editorial cartoon, also known as a political cartoon, is an illustration containing a political or social message. Illustrations can be used to display a wide range of subject matter and serve a variety of functions, such as: giving faces to characters in a story displaying a number of examples of an item described in an academic textbook (e.g. A Typology) visualizing step-wise sets of instructions in a technical manual communicating subtle thematic tone in a narrative linking brands to the ideas of human expression, individuality, and creativity making a reader laugh or smile for fun (to make laugh) funny === Graphs === A graph or chart is a graphic that represents tabular or numeric data. Charts are often used to make it easier to understand large quantities of data and the relationships between different parts of the data. === Diagrams === A diagram is a simplified and structured visual representation of concepts, ideas, constructions, relations, statistical data, etc., used to visualize and clarify the topic. === Symbols === A symbol, in its basic sense, is a representation of a concept or quantity; i.e., an idea, object, concept, quality, etc. In more psychological and philosophical terms, all concepts are symbolic in nature, and representations for these concepts are simply token artifacts that are allegorical to (but do not directly codify) a symbolic meaning, or symbolism. === Maps === A map is a simplified depiction of a space, a navigational aid which highlights relations between objects within that space. Usually, a map is a two-dimensional, geometrically accurate representation of a three-dimensional space. One of the first 'modern' maps was made by Waldseemüller. === Photography === One difference between photography and other forms of graphics is that a photographer, in principle, just records a single moment in reality, with seemingly no interpretation. However, a photographer can choose the field of view and angle, and may also use other techniques, such as various lenses to choose the view or filters to change the colors. In recent times, digital photography has opened the way to an infinite number of fast, but strong, manipulations. Even in the early days of photography, there was controversy over photographs of enacted scenes that were presented as 'real life' (especially in war photography, where it can be very difficult to record the original events). Shifting the viewer's eyes ever so slightly with simple pinpricks in the negative could have a dramatic effect. The choice of the field of view can have a strong effect, effectively 'censoring out' other parts of the scene, accomplished by cropping them out or simply not including them in the photograph. This even touches on the philosophical question of what reality is. The human brain processes information based on previous experience, making us see what we want to see or what we were taught to see. Photography does the same, although the photographer interprets the scene for their viewer. === Engineering drawings === An engineering drawing is a type of drawing and is technical in nature, used to fully and clearly define requirements for engineered items. It is usually created in accordance with standardized conventions for layout, nomenclature, interpretation, appearance (such as typefaces and line styles), size, etc. === Computer graphics === There are two types of computer graphics: raster graphics, where each pixel is separately defined (as in a digital photograph), and vector graphics, where mathematical formulas are used to draw lines and shapes, which are then interpreted at the viewer's end to produce the graphic. Using vectors results in infinitely sharp graphics and often smaller files, but, when complex, like vectors take time to render and may have larger file sizes than a raster equivalent. In 1950, the first computer-driven display was attached to MIT's Whirlwind I computer to generate simple pictures. This was followed by MIT's TX-0 and TX-2, interactive computing which increased interest in computer graphics during the late 1950s. In 1962, Ivan Sutherland invented Sketchpad, an innovative program that influenced alternative forms of interaction with computers. In the mid-1960s, large computer graphics research projects were begun at MIT, General Motors, Bell Labs, and Lockheed Corporation. Douglas T. Ross of MIT developed an advanced compiler language for graphics programming. S.A.Coons, also at MIT, and J. C. Ferguson at Boeing, began work in sculptured surfaces. GM developed their DAC-1 system, and other companies, such as Douglas, Lockheed, and McDonnell, also made significant developments. In 1968, ray tracing was first described by Arthur Appel of the IBM Research Center, Yorktown Heights, N

    Read more →
  • Seccomp

    Seccomp

    seccomp (short for secure computing) is a computer security facility in the Linux kernel. seccomp allows a process to make a one-way transition into a "secure" state where it cannot make any system calls except exit(), sigreturn(), read() and write() to already-open file descriptors. Should it attempt any other system calls, the kernel will either just log the event or terminate the process with SIGKILL or SIGSYS. In this sense, it does not virtualize the system's resources but isolates the process from them entirely. seccomp mode is enabled via the prctl(2) system call using the PR_SET_SECCOMP argument, or (since Linux kernel 3.17) via the seccomp(2) system call. seccomp mode used to be enabled by writing to a file, /proc/self/seccomp, but this method was removed in favor of prctl(). In some kernel versions, seccomp disables the RDTSC x86 instruction, which returns the number of elapsed processor cycles since power-on, used for high-precision timing. seccomp-bpf is an extension to seccomp that allows filtering of system calls using a configurable policy implemented using Berkeley Packet Filter rules. It is used by OpenSSH and vsftpd as well as the Google Chrome/Chromium web browsers on ChromeOS and Linux. (In this regard seccomp-bpf achieves similar functionality, but with more flexibility and higher performance, to the older systrace—which seems to be no longer supported for Linux.) Some consider seccomp comparable to OpenBSD pledge(2) and FreeBSD capsicum(4). == History == seccomp was first devised by Andrea Arcangeli in January 2005 for use in public grid computing and was originally intended as a means of safely running untrusted compute-bound programs. It was merged into the Linux kernel mainline in kernel version 2.6.12, which was released on March 8, 2005. == Software using seccomp or seccomp-bpf == Android uses a seccomp-bpf filter in the zygote since Android 8.0 Oreo. systemd's sandboxing options are based on seccomp. QEMU, the Quick Emulator, the core component to the modern virtualization together with KVM uses seccomp on the parameter --sandbox Docker – software that allows applications to run inside of isolated containers. Docker can associate a seccomp profile with the container using the --security-opt parameter. Arcangeli's CPUShare was the only known user of seccomp for a while. Writing in February 2009, Linus Torvalds expresses doubt whether seccomp is actually used by anyone. However, a Google engineer replied that Google is exploring using seccomp for sandboxing its Chrome web browser. Firejail is an open source Linux sandbox program that utilizes Linux namespaces, Seccomp, and other kernel-level security features to sandbox Linux and Wine applications. As of Chrome version 20, seccomp-bpf is used to sandbox Adobe Flash Player. As of Chrome version 23, seccomp-bpf is used to sandbox the renderers. Snap specify the shape of their application sandbox using "interfaces" which snapd translates to seccomp, AppArmor and other security constructs vsftpd uses seccomp-bpf sandboxing as of version 3.0.0. OpenSSH has supported seccomp-bpf since version 6.0. Mbox uses ptrace along with seccomp-bpf to create a secure sandbox with less overhead than ptrace alone. LXD, a Ubuntu "hypervisor" for containers Firefox and Firefox OS, which use seccomp-bpf Tor supports seccomp since 0.2.5.1-alpha Lepton, a JPEG compression tool developed by Dropbox uses seccomp Kafel is a configuration language, which converts readable policies into seccompb-bpf bytecode Subgraph OS uses seccomp-bpf Flatpak uses seccomp for process isolation Bubblewrap is a lightweight sandbox application developed from Flatpak minijail uses seccomp for process isolation SydBox uses seccomp-bpf to improve the runtime and security of the ptrace sandboxing used to sandbox package builds on Exherbo Linux distribution. File, a Unix program to determine filetypes, uses seccomp to restrict its runtime environment Zathura, a minimalistic document viewer, uses seccomp filter to implement different sandbox modes Tracker, a indexing and preview application for the GNOME desktop environment, uses seccomp to prevent automatic exploitation of parsing vulnerabilities in media files

    Read more →
  • Film-out

    Film-out

    Film-out is the process in the computer graphics, video production and filmmaking disciplines of transferring images or animation from videotape or digital files to a traditional film print. Film-out is a broad term that encompasses the conversion of frame rates, color correction, as well as the actual printing, also called scannior recording. The film-out process is different depending on the regional standard of the master videotape in question – NTSC, PAL, or SECAM – or likewise on the several emerging region-independent formats of high definition video (HD video); thus each type is covered separately, taking into account regional film-out industries, methods and technical considerations. == Live action video == Many modern documentaries and low-budget films are shot on videotape or other digital video media, instead of film stock, and completed as digital video. Video production means substantially lower costs than 16 mm or 35 mm film production on all levels. Until recently, the relatively low cost of video ended when the issue of a theatrical presentation was raised, which required a print for film projection. With the growing presence of digital projection, this is becoming less of a factor. === Standard definition (SD) video === Film-out of standard-definition video – or any source that has an incompatible frame rate – is the up-conversion of video media to film for theatrical viewing. The video-to-film conversion process consists of two major steps: first, the conversion of video into digital film frames which are then stored on a computer or on HD videotape; and secondly, the printing of these digital film frames onto actual film. To understand these two steps, it is important to understand how video and film differ. Film (sound film, at least) has remained unchanged for almost a century and creates the illusion of moving images through the rapid projection of still images, frames, upon a screen, typically 24 per second. Traditional interlaced SD video has no real frame rate, (though the term frame is applied to video, it has a different meaning). Instead, video consists of a very fast succession of horizontal lines that continually cascade down the television screen – streaming top to bottom, before jumping back to the top and then streaming down to the bottom again, repeatedly, almost 60 alternating screen-fulls every second for NTSC, or exactly 50 such screen-fulls per second for PAL and SECAM. Since visual movement in video is infused in this continuous cascade of scan lines, there is no discrete image or real frame that can be identified at any one time. Therefore, when transferring video to film, it is necessary to invent individual film frames, 24 for every second of elapsed time. The bulk of the work done by a film-out company is this first step, creating film frames out of the stream of interlaced video. Each company employs its own (often proprietary) technology for turning interlaced video into high-resolution digital video files of 24 discrete images every second, called 24 progressive video or 24p. The technology must filter out all the visually unappealing artifacting that results from the inherent mismatch between video and film movement. Moreover, the conversion process usually requires human intervention at every edit point of a video program, so that each type of scene can be calibrated for maximum visual quality. The use of archival footage in video especially calls for extra attention. Step two, the scanning to film, is the rote part of the process. This is the mechanical step where lasers print each of the newly created frames of the 24p video, stored on computer files or HD videotape, onto rolls of film. Most companies that do film-out, do all the stages of the process themselves for a lump sum. The job includes converting interlaced video into 24p and often a color correction session – (calibrating the image for theatrical projection), before scanning to physical film, (possibly followed by color correction of the film print made from the digital intermediary) – is offered. At the very least, film-out can be understood as the process of converting interlaced video to 24p and then scanning it to film. ==== NTSC video ==== NTSC is the most challenging of the formats when it comes to standards conversion and, specifically, converting to film prints. NTSC runs at the approximate rate of 29.97 video frames (consisting of two interlaced screen-fulls of scan lines, called fields, per frame) per second. In this way, NTSC resolves actual live action movement at almost – but not quite – 60 alternating half-resolution images every second. Because of this 29.97 rate, no direct correlation to film frames at 24 frames per second can be achieved. NTSC is hardest to reconcile with film, thus motivating its own unique processes. ==== PAL and SECAM video ==== PAL and SECAM run at 25 interlaced video frames per second, which can be slowed down or frame-dropped, then deinterlaced, to correlate frame for frame with film running at 24 actual frames per second. PAL and SECAM are less complex and demanding than NTSC for film-out. PAL and SECAM conversions do agitate, though, with the unpleasant choice between slowing down video (and audio pitch, noticeably) by four percent, from 25 to 24 frames per second, in order to maintain a 1:1 frame match, slightly changing the rhythm and feel of the program; or maintaining original speed by periodically dropping frames, thereby creating jerkiness and possible loss of vital detail in fast-moving action or precise edits. === High definition (HD) digital video === High definition digital video can be shot at a variety of frame rates, including 29.97 interlaced (like NTSC) or progressive; or 25 interlaced (like PAL) or progressive; or even 24-progressive (just like film). HD, if shot in 24-progressive, scans nearly perfectly to film without the need for a frame or field conversion process. Other issues remain though, based on the different resolutions, color spaces, and compression schemes that exist in the high-definition video world. == Computer graphics and animation == Artists working with CGI-Computer-generated imagery animation computers create pictures frame by frame. Once the finished product is done, the frames are outputted, normally in a DPX file. These picture data files can then be put on to film using a film recorder for film out. SGI computers started the high-end CGI-Computer-generated imagery animation systems, but with faster computers and the growth of Linux-based systems, many others are on the market now. Movies fully rendered and animated in CGI such as Toy Story, and Antz utilize the film-out method to produce 35mm copies for archival and release prints. Most CGI work is done in 2K Display resolution files (about the size of QXGA) and then output to the Film-out device for creation of 35 mm elements. With 4K Display resolution digital intermediates on the rise, newer types of film-out recorders are being developed to accept 4k resolution files. A 2K movie requires a Storage Area Network storage several terabytes in size to be properly stored and played out. Computer graphics files are handled the same way but in single frames and may use DPX, TIFF or other file formats. == Digital intermediates == Film-out-recording is the last step of digital intermediate workflow. DPX files that were scanned on a motion picture film scanner are stored on a storage area network (often abbreviated as SAN). The scanned DPX footage is edited and composited-FX on workstations, then mastered back on film. Film restoration is also done this way. A "film intermediate" is an analog variation of a digital intermediate, where a project shot on digital video is printed onto film stock and transferred back to digital video to emulate film. The term was coined after it was used on the Oscar-winning 2012 short film "Curfew". The process was also used on the films Dune (2021) and The Batman (2022). == Images for graphic design and print industries == The days of newspapers and magazines shooting 35mm film are almost gone. Digital cameras can now shoot all the images needed, storing them as files (e.g. JPEG, DPX or another format) that are readily edited prior to use. Once the final copy is approved, it can be filmed out for publishing. Digital stills are not the only way to get pictures used in the graphic design and print industries. Film scanners and computer graphics programs are also common sources for graphic design and print industries. == Types of devices == The following devices are used in film-out processes: CRT recorder. Camera and a special TV display Kinescope – early type Electronic Video Recording or EVR – early type EBR Electron Beam Film Recorder 16 mm by 3M Laser film recorder, like Kodak's high-end Lightning II recorder and Arri's Arrilaser. DLP Film recorder, like Cinevation's real-time Cinevator. == History == Lately it has become possible to transfer video images, inclu

    Read more →
  • Test data management

    Test data management

    Test data management (TDM) is a process in software testing concerned with the creation, preparation, and control of data used for testing software systems. It involves supplying datasets required to execute test cases and verifying system behaviour under defined conditions. Test data management is an integral part of the software development lifecycle (SDLC) and is utilized in both manual and automated testing processes. It is applied in environments that use continuous integration and DevOps practices, where test execution requires consistent and repeatable data conditions. == Overview == Test data management includes the generation, selection, and preparation of data for testing purposes, as well as its distribution across test environments. It also involves controlling data versions and ensuring that datasets correspond to specific test scenarios. In many cases, production data is adapted for testing through techniques such as masking or subsetting to reduce size and remove sensitive content. Test data management ensures that test cases are executed with relevant, consistent, and readily available data. This reduces variability in test results and supports reproducibility across test cycles. == Importance == The role of test data management has expanded with the growth of complex, data-driven systems and regulatory requirements governing data usage. Testing often depends on data that reflects real-world conditions, but direct use of production data may introduce security and privacy risks. As a result, organizations apply methods such as data masking and anonymization to meet compliance requirements, including those set by the California Privacy Rights Act (CPRA) and Europe’s General Data Protection Regulation (GDPR). Inadequate control of test data can lead to incomplete test coverage, unreliable test results, or delays in testing processes due to unavailable or inconsistent datasets. == Techniques and tools == Test data management leverages various techniques for preparing and controlling data used in testing. These include the generation of synthetic data, the extraction of subsets from production datasets, and the modification of data to remove or obscure sensitive information. A key technical requirement in these processes is maintaining referential integrity, or ensuring that relationships between data entities remain consistent across different tables and systems after masking or subsetting. Data virtualization is also used to provide access to datasets without full replication. These methods may be implemented using software tools that automate data preparation, masking, and distribution.

    Read more →
  • Clipmap

    Clipmap

    In computer graphics, clipmapping is a method of clipping a mipmap to a subset of data pertinent to the geometry being displayed. This is useful for loading as little data as possible when memory is limited, such as on a graphics processing unit. The technique is used for LODing in NVIDIA’s implementation of voxel cone tracing. The high-resolution levels of the mipmapped scene representation are clipped to a region near the camera, while lower resolution levels are clipped further away. == MegaTexture == MegaTexture is a clipmap implementation developed by id Software. It was introduced in their id Tech 4 engine and also appeared in id Tech 5 and id Tech 6 before being removed in id Tech 7. MegaTexture is a texture allocation technique that uses a single, extremely large texture rather than repeating multiple smaller textures. It is also featured in Splash Damage's game Enemy Territory: Quake Wars, and was developed by id Software former technical director John Carmack. MegaTexture employs a single large texture space for static terrain. The texture is stored on removable media or a computer's hard drive and streamed as needed, allowing large amounts of detail and variation over a large area with comparatively little RAM usage. Depending on the pixel resolution per square meter, covering a large area could require several gigabytes of memory. However, RAM is also filled by the rest of the game and the underlying operating system, limiting the amount available for texturing. As the player moves around the game, different sections of the MegaTexture are loaded into memory. They are then scaled to the correct size and applied to the 3D models of the terrain. Id has presented a more advanced technique that builds upon the MegaTexture idea and virtualizes both the geometry and the textures to obtain unique geometry down to the equivalent of the texel: the sparse voxel octree (SVO). It works by raycasting the geometry represented by voxels (instead of triangles) stored in an octree. The goal is to stream parts of the octree into video memory, going further down along the tree for nearby objects to give them more details, and to use higher level, larger voxels for farther objects, which give an automatic level of detail (LOD) system for both geometry and textures at the same time. The geometric detail that can be obtained using this method is nearly infinite, which removes the need for faking 3-dimensional details with techniques such as normal mapping. Despite that most voxel rendering tests use very large amounts of memory (up to several GB), Jon Olick of id Software claimed the technology is able to compress such SVO to 1.15 bits per voxel of position data. == Virtual texturing == Unlike clipmaps, which clip each mip level around a viewpoint-dependent clipcenter and therefore work best for terrain, virtual texturing preprocesses texture data into equally sized tiles that can be streamed for arbitrary textured geometry. Rage, powered by the id Tech 5 engine, uses a more advanced technique called virtual texturing. Textures can measure up to 128000×128000 pixels and are also used for in-game models and sprites, etc. and not just the terrain. Wolfenstein: The New Order and the 2016 version of Doom also use these. Carmageddon: Reincarnation also uses virtual texturing, though unlike id's virtual texturing system, which is designed for unique texture-mapping everywhere, their system is designed to use storage space sparingly while still offering good blend of texture variation and resolution.

    Read more →
  • Secure environment

    Secure environment

    In computing, a secure environment is any system which implements the controlled storage and use of information. In the event of computing data loss, a secure environment is used to protect personal or confidential data. It may also be known as a trusted execution environment (TEE). Often, secure environments employ cryptography as a means to protect information. This is typically used for processing confidential or restricted information. Some secure environments employ cryptographic hashing, simply to verify that the information has not been altered since it was last modified.

    Read more →
  • Visual analytics

    Visual analytics

    Visual analytics is a multidisciplinary science and technology field that emerged from information visualization and scientific visualization. It focuses on how analytical reasoning can be facilitated by interactive visual interfaces. == Overview == Visual analytics is "the science of analytical reasoning facilitated by interactive visual interfaces." It can address problems whose size, complexity, and need for closely coupled human and machine analysis may make them otherwise intractable. Visual analytics advances scientific and technological development across multiple domains, including analytical reasoning, human–computer interaction, data transformations, visual representation for computation and analysis, analytic reporting, and the transition of new technologies into practice. As a research agenda, visual analytics brings together several scientific and technical communities from computer science, information visualization, cognitive and perceptual sciences, interactive design, graphic design, and social sciences. Visual analytics integrates new computational and theory-based tools with innovative interactive techniques and visual representations to enable human-information discourse. The design of the tools and techniques is based on cognitive, design, and perceptual principles. This science of analytical reasoning provides the reasoning framework upon which one can build both strategic and tactical visual analytics technologies for threat analysis, prevention, and response. Analytical reasoning is central to the analyst's task of applying human judgments to reach conclusions from a combination of evidence and assumptions. Visual analytics has some overlapping goals and techniques with information visualization and scientific visualization. There is currently no clear consensus on the boundaries between these fields, but broadly speaking the three areas can be distinguished as follows: Scientific visualization deals with data that has a natural geometric structure (e.g., MRI data, wind flows). Information visualization handles abstract data structures such as trees or graphs. Visual analytics is especially concerned with coupling interactive visual representations with underlying analytical processes (e.g., statistical procedures, data mining techniques) such that high-level, complex activities can be effectively performed (e.g., sense making, reasoning, decision making). Visual analytics seeks to marry techniques from information visualization with techniques from computational transformation and analysis of data. Information visualization forms part of the direct interface between user and machine, amplifying human cognitive capabilities in six basic ways: by increasing cognitive resources, such as by using a visual resource to expand human working memory, by reducing search, such as by representing a large amount of data in a small space, by enhancing the recognition of patterns, such as when information is organized in space by its time relationships, by supporting the easy perceptual inference of relationships that are otherwise more difficult to induce, by perceptual monitoring of a large number of potential events, and by providing a manipulable medium that, unlike static diagrams, enables the exploration of a space of parameter values These capabilities of information visualization, combined with computational data analysis, can be applied to analytic reasoning to support the sense-making process. == History == As an interdisciplinary approach, visual analytics has its roots in information visualization, cognitive sciences, and computer science. The term and scope of the field was defined in the early 2000s through researchers such as Jim Thomas, Kristin A. Cook, John Stasko, Pak Chung Wong, Daniel A. Keim and David S. Ebert. As a reaction to the September 11, 2001 attacks the United States Department of Homeland Security was established in late 2002, combining dozens of previously separated government agencies. Building upon earlier work on visual data mining by Daniel A. Keim starting in the late 1990s, this simultaneously lead to the development of a research agenda for visual analytics. As part of these efforts the National Visualization and Analytics Center (NVAC) at Pacific Northwest National Laboratory was established in 2004, whose charter was to develop system to mitigate information overload after the September 11, 2001 attacks in the intelligence community. Their research work determined core challenges, posed open research questions, and positioned visual analytics as a new research domain, in particular through the 2005 research agenda Illuminating the Path. In 2006, the IEEE VIS community led by Pak Chung Wong and Daniel A. Keim launched the annual IEEE Conference on Visual Analytics Science and Technology (VAST), providing a dedicated venue for research into visual analytics, which in 2020 merged to form the IEEE Visualization conference. In 2008, scope and challenges of visual analytics were conceptually defined by Daniel A. Keim and Jim Thomas in their influential book about visual data mining. The domain was further refined as part of the European Commissions FP7 VisMaster program in the late 2000s. == Topics == === Scope === Visual analytics is a multidisciplinary field that includes the following focus areas: Analytical reasoning techniques that enable users to obtain deep insights that directly support assessment, planning, and decision making Data representations and transformations that convert all types of conflicting and dynamic data in ways that support visualization and analysis Techniques to support production, presentation, and dissemination of the results of an analysis to communicate information in the appropriate context to a variety of audiences. Visual representations and interaction techniques that take advantage of the human eye's broad bandwidth pathway into the mind to allow users to see, explore, and understand large amounts of information at once. === Analytical reasoning techniques === Analytical reasoning techniques are the method by which users obtain deep insights that directly support situation assessment, planning, and decision making. Visual analytics must facilitate high-quality human judgment with a limited investment of the analysts’ time. Visual analytics tools must enable diverse analytical tasks such as: Understanding past and present situations quickly, as well as the trends and events that have produced current conditions Identifying possible alternative futures and their warning signs Monitoring current events for emergence of warning signs as well as unexpected events Determining indicators of the intent of an action or an individual Supporting the decision maker in times of crisis. These tasks will be conducted through a combination of individual and collaborative analysis, often under extreme time pressure. Visual analytics must enable hypothesis-based and scenario-based analytical techniques, providing support for the analyst to reason based on the available evidence. === Data representations === Data representations are structured forms suitable for computer-based transformations. These structures must exist in the original data or be derivable from the data themselves. They must retain the information and knowledge content and the related context within the original data to the greatest degree possible. The structures of underlying data representations are generally neither accessible nor intuitive to the user of the visual analytics tool. They are frequently more complex in nature than the original data and are not necessarily smaller in size than the original data. The structures of the data representations may contain hundreds or thousands of dimensions and be unintelligible to a person, but they must be transformable into lower-dimensional representations for visualization and analysis. === Theories of visualization === Theories of visualization include: Jacques Bertin's Semiology of Graphics (1967) Nelson Goodman's Languages of Art (1977) Jock D. Mackinlay's Automated design of optimal visualization (APT) (1986) Leland Wilkinson's Grammar of Graphics (1998) Hadley Wickham's Layered Grammar of Graphics (2010) === Visual representations === Visual representations translate data into a visible form that highlights important features, including commonalities and anomalies. These visual representations make it easy for users to perceive salient aspects of their data quickly. Augmenting the cognitive reasoning process with perceptual reasoning through visual representations permits the analytical reasoning process to become faster and more focused. == Process == The input for the data sets used in the visual analytics process are heterogeneous data sources (i.e., the internet, newspapers, books, scientific experiments, expert systems). From these rich sources, the data sets S = S1, ..., Sm are chosen, whereas each Si , i ∈ (1, ..., m) consists of attrib

    Read more →
  • LRE Map

    LRE Map

    The LRE Map (Language Resources and Evaluation) is a freely accessible large database on resources dedicated to Natural language processing. The original feature of LRE Map is that the records are collected during the submission of different major Natural language processing conferences. The records are then cleaned and gathered into a global database called "LRE Map". The LRE Map is intended to be an instrument for collecting information about language resources and to become, at the same time, a community for users, a place to share and discover resources, discuss opinions, provide feedback, discover new trends, etc. It is an instrument for discovering, searching and documenting language resources, here intended in a broad sense, as both data and tools. The large amount of information contained in the Map can be analyzed in many different ways. For instance, the LRE Map can provide information about the most frequent type of resource, the most represented language, the applications for which resources are used or are being developed, the proportion of new resources vs. already existing ones, or the way in which resources are distributed to the community. == Context == Several institutions worldwide maintain catalogues of language resources (ELRA, LDC, NICT Universal Catalogue, ACL Data and Code Repository, OLAC, LT World, etc.) However, it has been estimated that only 10% of existing resources are known, either through distribution catalogues or via direct publicity by providers (web sites and the like). The rest remains hidden, the only occasions where it briefly emerges being when a resource is presented in the context of a research paper or report at some conference. Even in this case, nevertheless, it might be that a resource remains in the background simply because the focus of the research is not on the resource per se. == History == The LRE Map originated under the name "LREC Map" during the preparation of LREC 2010 conference. More specifically, the idea was discussed within the FlaReNet project, and in collaboration with ELRA and the Institute of Computational Linguistics of CNR in Pisa, the Map was put in place at LREC 2010. The LREC organizers asked the authors to provide some basic information about all the resources (in a broad sense, i.e. including tools, standards and evaluation packages), either used or created, described in their papers. All these descriptors were then gathered in a global matrix called the LREC Map. The same methodology and requirements from the authors has been then applied and extended to other conferences, namely COLING-2010, EMNLP-2010, RANLP-2011, LREC 2012, LREC 2014 and LREC 2016. After this generalization to other conferences, the LREC Map has been renamed as the LRE Map. == Size and content == The size of the database increases over time. The data collected amount to 4776 entries. Each resource is described according to the following attributes: Resource type, e.g. lexicon, annotation tool, tagger/parser. Resource production status, e.g. newly created finished, existing-updated. Resource availability, e.g. freely available, from data center. Resource modality, e.g. speech, written, sign language. Resource use, e.g. named entity recognition, language identification, machine translation. Resource language, e.g. English, 23 European Union languages, official languages of India. == Uses == The LRE map is a very important tool to chart the NLP field. Compared to other studied based on subjective scorings, the LRE map is made of real facts. The map has a great potential for many uses, in addition to being an information gathering tool: It is a great instrument for monitoring the evolution of the field (useful for funders), if applied in different contexts and times. It can be seen as a huge joint effort, the beginning of an even larger cooperative action not just among few leaders but among all the researchers. It is also an "educational" means towards the broad acknowledgment of the need of meta-research activities with the active involvement of many. It is also instrumental in introducing the new notion of "citation of resources" that could provide an award and a means of scholarly recognition for researchers engaged in resource creation. It is used to help the organization of the conferences of the field like LREC. == Derived matrices == The data were then cleaned and sorted by Joseph Mariani (CNRS-LIMSI IMMI) and Gil Francopoulo (CNRS-LIMSI IMMI + Tagmatica) in order to compute the various matrices of the final FLaReNet reports. One of them, the matrix for written data at LREC 2010 is as follows: English is the most studied language. Secondly, come French and German languages and then Italian and Spanish. == Future == The LRE Map has been extended to Language Resources and Evaluation Journal and other conferences.

    Read more →
  • Pocketbook (application)

    Pocketbook (application)

    Pocketbook was a Sydney-based free budget planner and personal finance app launched in 2012. The app helped users setup and manage budgets, track spending and manage bills. As of 2016 Pocketbook claimed to support over 250,000 Australians, in January 2018 that number was 435,000. After being acquired by Zip Co Ltd in 2016, it was announced in 2022 that the app was to be shut down and all user accounts deleted. == History == Pocketbook was founded by Alvin Singh and Bosco Tan in 2012. It was conceived in 2011 in a Wolli Creek apartment as a tool for Alvin and Bosco to take control of their money. In 2013, Pocketbook raised $500,000 from technology fund Tank Stream Ventures, and a group of investors including TV personality David Koch, Geoff Levy, David Shein and Peter Cooper. In September 2016 Digital retail finance and payment industry player zipMoney (now trading as Zip Co Limited) acquired Pocketbook in a $7.5m deal == Features == The app synced with the bank account of users and would organize spending into different categories. Users could also be reminded of bill payments, analyse spending and set spending limits. They can also be alerted of fraudulent transactions and deductions. The app employs security measures like end to end encryption, CloudFlare protection, fraud detection, identity protection etc. Pocketbook was available via web and mobile version. == Awards == Personal Finance Innovator of the Year by Fintech Business Awards 2017 Innovator of the Year by OPTUS MyBusiness Awards 2017 Best Finance App of 2016 by Australian Fintech Best Personal Finance App: Pocketbook won the 2016 Finder Innovation Awards, presented at a gala dinner hosted by media personality and The New Inventors presenter James O'Loghlin. Best Mobile App of the Year Winner: StartCon hosted the first annual Australasian Startup Awards. Over 200 nominations in 14 categories and an overall winner were reviewed, and winners were determined by public voting, with over 63,000 votes in total. Best New Startup 2014 by StartupSmart. Finalist in the SWIFT Innotribe startup competition in Dubai in 2013.

    Read more →
  • Security awareness

    Security awareness

    Security awareness is the knowledge and attitude members of an organization possess regarding the protection of the physical, and especially informational, assets of that organization. However, it is very tricky to implement because organizations are not able to impose such awareness directly on employees as there are no ways to explicitly monitor people's behavior. That being said, the literature does suggest several ways that such security awareness could be improved. Many organizations require formal security awareness training for all workers when they join the organization and periodically thereafter, usually annually. Another main force that is found to have a strong correlation with employees' security awareness is managerial security participation. It also bridges security awareness with other organizational aspects. == Relationship between Security Awareness and Human Factors == Employees' behavior, cognitive biases, and decision-making processes influence the effectiveness of security measures. Research indicates that psychological factors, such as optimism bias, overconfidence, and habitual behaviors, can undermine security awareness initiatives. To address these challenges, organizations are increasingly using behavioral analytics and security nudges—subtle prompts like password reminders and phishing warnings—to encourage secure behavior. Human error remains the leading cause of cybersecurity incidents. A 2023 IBM Security report found that 95% of breaches are due to human mistakes, including falling for phishing emails, using weak passwords, and mishandling sensitive data. Organizations emphasize security awareness training as a key strategy to mitigate this risk. It is particularly important for leadership to foster a culture of cybersecurity and to provide targeted training to increase security awareness among all employees across the organization. == Coverage == Topics covered in security awareness training include: The nature of sensitive material and physical assets they may come in contact with, such as trade secrets, privacy concerns and government classified information Employee and contractor responsibilities in handling sensitive information, including review of employee nondisclosure agreements Requirements for proper handling of sensitive material in physical form, including marking, transmission, storage and destruction Proper methods for protecting sensitive information on computer systems, including password policy and use of two-factor authentication Other computer security concerns, including malware, phishing, social engineering, etc. Workplace security, including building access, wearing of security badges, reporting of Incidents, forbidden articles, etc. Consequences of failure to properly protect information, including potential loss of employment, economic consequences to the firm, damage to individuals whose private records are divulged, and possible civil and criminal penalties Security awareness means understanding that there is the potential for some people to deliberately or accidentally steal, damage, or misuse the data that is stored within a company's computer systems and throughout its organization. Therefore, it would be prudent to support the assets of the institution (information, physical, and personal) by trying to stop that from happening. According to the European Network and Information Security Agency, "Awareness of the risks and available safeguards is the first line of defence for the security of information systems and networks." "The focus of Security Awareness consultancy should be to achieve a long term shift in the attitude of employees towards security, whilst promoting a cultural and behavioural change within an organisation. Security policies should be viewed as key enablers for the organisation, not as a series of rules restricting the efficient working of your business." == Role of Gamification and Interactive Training == Modern security awareness programs increasingly utilize gamification, phishing simulations, and interactive learning modules. Studies have shown that engaging employees through serious games, reward systems, and real-world attack simulations improves retention and application of security practices. One example is phishing simulation training, where employees receive simulated phishing emails to test their ability to recognize threats. Research indicates that repeated exposure to such exercises leads to long-term improvements in security awareness. == Legislation and Compliance Requirements == Many industries mandate security awareness training to comply with regulations such as: General Data Protection Regulation (GDPR) – requires organizations to ensure data protection awareness among employees. Health Insurance Portability and Accountability Act (HIPAA) – mandates security awareness programs for healthcare providers. Payment Card Industry Data Security Standard (PCI-DSS) – enforces security training for businesses handling payment card information. == Measuring security awareness == In a 2016 study, researchers developed a method of measuring security awareness. Specifically they measured "understanding about circumventing security protocols, disrupting the intended functions of systems or collecting valuable information, and not getting caught" (p. 38). The researchers created a method that could distinguish between experts and novices by having people organize different security scenarios into groups. Experts will organize these scenarios based on centralized security themes where novices will organize the scenarios based on superficial themes. Security awareness is also assessed through real-time security metrics, such as tracking phishing click rates, password reuse tendencies, and policy adherence rates. Organizations are adopting continuous monitoring strategies to provide immediate feedback to employees about risky behavior and suggest corrective actions. == Evolving cyber threats and security awareness strategies == As cyber threats continue to evolve, security awareness programs must adapt to new attack vectors, such as AI-driven cyberattacks, deepfakes, and insider threats. ENISA's Threat Landscape report highlights the increasing prominence of these emerging threats, stressing the need for security measures that address both traditional attacks like ransomware and malware, as well as more sophisticated techniques such as Living Off Trusted Sites (LOTS) and advanced evasion methods used by cybercriminals.

    Read more →
  • Digital Michelangelo Project

    Digital Michelangelo Project

    The Digital Michelangelo Project was a pioneering initiative undertaken during the 1998–1999 academic year to digitize the sculptures and architecture of Michelangelo using advanced laser scanning technology. The project was led by a team of 30 faculty, staff, and students from Stanford University and the University of Washington, with the aim of creating high-resolution 3D models of Michelangelo's works for scholarly, educational, and preservation purposes. == Objectives == The primary goals of the Digital Michelangelo Project were: To apply recent advancements in laser rangefinder technology for digitizing large cultural artifacts. To create detailed digital archives of Michelangelo's sculptures and architectural spaces for future study and analysis. To explore potential educational and curatorial applications for 3D scanned data. === Artworks digitized === The project involved scanning several iconic works by Michelangelo, including: David The Unfinished Slaves (Atlas, Awakening, Bearded, and Youthful) St. Matthew The allegorical statues from the Medici tombs (Night, Day, Dawn, and Dusk) The architectural interiors of the Tribuna del David at the Galleria dell'Accademia and the New Sacristy in the Medici Chapels. == Technology and methodology == === 3D scanning === The project's primary scanner was a laser triangulation rangefinder mounted on a motorized gantry, custom-built by Cyberware Inc. The scanner used a laser sheet to project onto an object, capturing its shape through triangulation. Multiple scans were taken from various angles and combined into a single, detailed 3D mesh. The resolution achieved was fine enough to capture even Michelangelo's chisel marks, with triangles approximately 0.25 mm on each side. In addition to shape data, color data was captured using a spotlight and a secondary camera, enabling the creation of textured 3D models. === Data processing === The project developed a software suite for processing the scanned data. This included: Aligning and merging multiple scans into a seamless 3D model. Filling holes in the geometry caused by inaccessible areas. Correcting color data for lighting inconsistencies and shadowing. Non-photorealistic rendering techniques were also applied, highlighting surface features such as Michelangelo’s chisel marks for enhanced visualization. == Logistical challenges == The scale and complexity of the project presented several challenges: Data size: The dataset for David alone comprised 2 billion polygons and 7,000 color images, occupying 60 GB of storage. Artifact safety: Ensuring the safety of the statues during scanning required extensive crew training, foam-encased equipment, and collision-prevention mechanisms. == Applications and impact == The digitized models have numerous potential applications: Art history: Allowing precise measurements and geometric analysis, such as determining chisel types or evaluating structural balance. Education: Providing new ways to study art, including interactive viewing from unconventional angles and with custom lighting. Museum curation: Enhancing visitor experiences through interactive kiosks and virtual models. The project demonstrated the potential for 3D technology to preserve and disseminate cultural heritage. == Data distribution == The project's models are available through Stanford University for scholarly purposes, under strict licensing due to Italian intellectual property laws. === ScanView === To provide public access to the 3D models while respecting usage restrictions, the project developed ScanView, a client/server rendering system. ScanView allows users to view and interact with high-resolution 3D models without downloading the data. The client component consists of a freely available viewer program and simplified 3D models. Users can navigate these models locally, adjusting position, orientation, lighting, and surface appearance. When a user finalizes a view, the client queries a remote server for a high-resolution rendering of the model, which is sent back to overwrite the simplified version on the user’s screen. A typical query-response cycle takes 1–2 seconds, depending on network conditions. To protect the models from unauthorized reconstruction, the system employs several security measures, including: Encrypting queries Perturbing viewpoint and lighting parameters Adding noise and warping rendered images Compressing images before transmission ScanView operates on Windows-based PCs and provides access to selected models, including David and St. Matthew, as well as other artifacts such as fragments of the Forma Urbis Romae and items from the Stanford 3D Scanning Repository. == Sponsors == The Digital Michelangelo Project was supported by Stanford University, Interval Research Corporation, and the Paul G. Allen Foundation for the Arts.

    Read more →
  • Distributed manufacturing

    Distributed manufacturing

    Distributed manufacturing, also known as distributed production, cloud producing, distributed digital manufacturing, and local manufacturing, is a form of decentralized manufacturing practiced by enterprises using a network of geographically dispersed manufacturing facilities that are coordinated using information technology. It can also refer to local manufacture via the historic cottage industry model, or manufacturing that takes place in the homes of consumers. == Enterprise == In enterprise environments, the primary attribute of distributed manufacturing is the ability to create value at geographically dispersed locations. For example, shipping costs could be minimized when products are built geographically close to their intended markets. Also, products manufactured in a number of small facilities distributed over a wide area can be customized with details adapted to individual or regional tastes. Manufacturing components in different physical locations and then managing the supply chain to bring them together for final assembly of a product is also considered a form of distributed manufacturing. Digital networks combined with additive manufacturing allow companies a decentralized and geographically independent distributed production (cloud manufacturing). == Consumer == Within the maker movement and DIY culture, small scale production by consumers often using peer-to-peer resources is being referred to as distributed manufacturing. Consumers download digital designs from an open design repository website like Youmagine or Thingiverse and produce a product for low costs through a distributed network of 3D printing services such as 3D Hubs, Geomiq. In the most distributed form of distributed manufacturing the consumer becomes a prosumer and manufacturers products at home with an open-source 3-D printer such as the RepRap. In 2013 a desktop 3-D printer could be economically justified as a personal product fabricator and the number of free and open hardware designs were growing exponentially. Today there are millions of open hardware product designs at hundreds of repositories and there is some evidence consumers are 3-D printing to save money. For example, 2017 case studies probed the quality of: (1) six common complex toys; (2) Lego blocks; and (3) the customizability of open source board games and found that all filaments analyzed saved the prosumer over 75% of the cost of commercially available true alternative toys and over 90% for recyclebot filament. Overall, these results indicate a single 3D printing repository, MyMiniFactory, is saving consumers well over $60 million/year in offset purchases of only toys. These 3-D printers can now be used to make sophisticated high-value products like scientific instruments. Similarly, a study in 2022 found that 81% of open source designs provided economic savings and the total savings for the 3D printing community is more than $35 million from downloading only the top 100 products at YouMagine. In general, the savings are largest when compared to conventional products when prosumers use recycled materials in 'distributed recycling and additive manufacturing' (DRAM). == Emergency Distributed Manufacturing During COVID-19 Pandemic == Distributed manufacturing became far more visible during the COVID-19 pandemic because it offered a practical response to the breakdown of centralized global supply chains. As lock downs, border restrictions, and factory shutdowns disrupted conventional production, decentralized networks using local facilities such as Open Source Medical Supplies stepped in and manufactured over 48 million products. Additive manufacturing /3D printing were used to produce urgently needed items such as face shields, ventilators and their components, nasopharyngeal swabs, and other personal protective equipment. This demonstrated that distributed manufacturing could reduce lead times, improve responsiveness, and lessen dependence on distant suppliers during crisis conditions for a wide range of products. Peer-reviewed studies on pandemic-era manufacturing note that additive manufacturing was especially valuable because digital design files could be shared rapidly and produced close to the point of need, enabling hospitals, universities, small firms, and maker communities to supplement strained medical supply chains. The pandemic also helped shift distributed manufacturing from being seen as a niche or experimental model to a credible strategy for resilience, flexibility, and emergency response. At the same time, scholars caution that its wider adoption depends on solving issues related to quality assurance, regulation, material consistency, and coordination across distributed production sites. Overall, COVID-19 popularized distributed manufacturing by showing that localized, digitally enabled production could complement traditional manufacturing systems when speed, adaptability, and supply-chain resilience were critical. == Social change == Some call attention to the conjunction of commons-based peer production with distributed manufacturing techniques. The self-reinforced fantasy of a system of eternal growth can be overcome with the development of economies of scope, and here, the civil society can play an important role contributing to the raising of the whole productive structure to a higher plateau of more sustainable and customised productivity. Further, it is true that many issues, problems and threats rise due to the large democratization of the means of production, and especially regarding the physical ones. For instance, the recyclability of advanced nanomaterials is still questioned; weapons manufacturing could become easier; not to mention the implications on counterfeiting and on "intellectual property". It might be maintained that in contrast to the industrial paradigm whose competitive dynamics were about economies of scale, commons-based peer production and distributed manufacturing could develop economies of scope. While the advantages of scale rest on cheap global transportation, the economies of scope share infrastructure costs (intangible and tangible productive resources), taking advantage of the capabilities of the fabrication tools. And following Neil Gershenfeld in that "some of the least developed parts of the world need some of the most advanced technologies", commons-based peer production and distributed manufacturing may offer the necessary tools for thinking globally but act locally in response to certain problems and needs. As well as supporting individual personal manufacturing social and economic benefits are expected to result from the development of local production economies. In particular, the humanitarian and development sector are becoming increasingly interested in how distributed manufacturing can overcome the supply chain challenges of last mile distribution. Further, distributed manufacturing has been proposed as a key element in the Cosmopolitan localism or cosmolocalism framework to reconfigure production by prioritizing socio-ecological well-being over corporate profits, over-production and excess consumption. == Technology == By localizing manufacturing, distributed manufacturing may enable a balance between two opposite extreme qualities in technology development: Low technology and High tech. This balance is understood as an inclusive middle, a "mid-tech", that may go beyond the two polarities, incorporating them into a higher synthesis. Thus, in such an approach, low-tech and high-tech stop being mutually exclusive. They instead become a dialectic totality. Mid-tech may be abbreviated to "both…and…" instead of "neither…nor…". Mid-tech combines the efficiency and versatility of digital/automated technology with low-tech's potential for autonomy and resilience. == Contracting in Distributed Manufacturing == Research into contracting and order processing models tailored for distributed manufacturing has highlighted the need for flexible, role-based frameworks and advanced digital tools. These tools and frameworks are essential for addressing issues related to quality assurance, payment structures, legal compliance, and coordination among multiple actors. By addressing these challenges, contracting models for distributed manufacturing can unlock its potential for more localized, efficient, and sustainable production systems. A system prototype has been developed to simplify contracting for distributed manufacturing. This tool allows buyers to manage orders across multiple manufacturers using a single interface, automating workflows to ensure clarity and accountability for everyone involved. This research was led by the Internet of Production, as part of the mAkE project (African European Maker Innovation Ecosystem), funded by the European Horizon 2020 research and innovation programme.

    Read more →
  • Two-phase locking

    Two-phase locking

    In databases and transaction processing, two-phase locking (2PL) is a pessimistic concurrency control method that guarantees conflict-serializability. It is also the name of the resulting set of database transaction schedules (histories). The protocol uses locks, applied by a transaction to data, which may block (interpreted as signals to stop) other transactions from accessing the same data during the transaction's life. By the 2PL protocol, locks are applied and removed in two phases: Expanding phase: locks are acquired and no locks are released. Shrinking phase: locks are released and no locks are acquired. Two types of locks are used by the basic protocol: Shared and Exclusive locks. Refinements of the basic protocol may use more lock types. Using locks that block processes, 2PL, S2PL, and SS2PL may be subject to deadlocks that result from the mutual blocking of two or more transactions. == Read and write locks == Locks are used to guarantee serializability. A transaction is holding a lock on an object if that transaction has acquired a lock on that object which has not yet been released. For 2PL, the only used data-access locks are read-locks (shared locks) and write-locks (exclusive locks). Below are the rules for read-locks and write-locks: A transaction is allowed to read an object if and only if it is holding a read-lock or write-lock on that object. A transaction is allowed to write an object if and only if it is holding a write-lock on that object. A schedule (i.e., a set of transactions) is allowed to hold multiple locks on the same object simultaneously if and only if none of those locks are write-locks. If a disallowed lock attempts on being held simultaneously, it will be blocked. == Variants == Note that all conflict serializable schedules are also view serializable (but not vice-versa). === Two-phase locking === According to the two-phase locking protocol, each transaction handles its locks in two distinct, consecutive phases during the transaction's execution: Expanding phase (aka Growing phase): locks are acquired and no locks are released (the number of locks can only increase). Shrinking phase (aka Contracting phase): locks are released and no locks are acquired. The two phase locking rules can be summarized as: each transaction must never acquire a lock after it has released a lock. The serializability property is guaranteed for a schedule with transactions that obey this rule. Typically, without explicit knowledge in a transaction on end of phase 1, the rule is safely determined only when a transaction has completed processing and requested commit. In this case, all the locks can be released at once (phase 2). === Conservative two-phase locking === Conservative two-phase locking (C2PL) differs from 2PL in that transactions obtain all the locks they need before the actual execution begins. This is to ensure that a transaction that already holds some locks will not block waiting for other locks. C2PL prevents deadlocks. In cases of heavy lock contention, C2PL reduces the time locks are held on average, relative to 2PL and Strict 2PL, because transactions that hold locks are never blocked. In light lock contention, C2PL holds more locks than is necessary, because it is difficult to predict which locks will be needed in the future, thus leading to higher overhead. A C2PL transaction will not obtain any locks if it cannot obtain all the locks it needs in its initial request. Furthermore, each transaction needs to declare its read and write set (the data items that will be read/written), which is not always possible. Because of these limitations, C2PL is not used very frequently. === Strict two-phase locking === To comply with the strict two-phase locking (S2PL) protocol, a transaction needs to comply with 2PL, and release its write (exclusive) locks only after the transaction has ended (i.e., either committed or aborted). On the other hand, read (shared) locks are released regularly during the shrinking phase. Unlike 2PL, S2PL provides strictness (a special case of cascade-less recoverability). This protocol is not appropriate in B-trees because it causes Bottleneck (while B-trees always starts searching from the parent root). === Strong strict two-phase locking === or Rigorousness, or Rigorous scheduling, or Rigorous two-phase locking To comply with strong strict two-phase locking (SS2PL), a transaction's read and write locks are released only after that transaction has ended (i.e., either committed or aborted). A transaction obeying SS2PL has only a phase 1 and lacks a phase 2 until the transaction has completed. Every SS2PL schedule is also an S2PL schedule, but not vice versa.

    Read more →
  • Autocommit

    Autocommit

    In the context of data management, autocommit is a mode of operation of a database connection. Each individual database interaction (i.e., each SQL statement) submitted through the database connection in autocommit mode will be executed in its own transaction that is implicitly committed. A SQL statement executed in autocommit mode cannot be rolled back. Autocommit mode incurs per-statement transaction overhead and can often lead to undesirable performance or resource utilization impact on the database. Nonetheless, in systems such as Microsoft SQL Server, as well as connection technologies such as ODBC and Microsoft OLE DB, autocommit mode is the default for all statements that change data, in order to ensure that individual statements will conform to the ACID (atomicity-consistency-isolation-durability) properties of transactions. The alternative to autocommit mode (non-autocommit) means that the SQL client application itself is responsible for ending transactions explicitly via the commit or rollback SQL commands. Non-autocommit mode enables grouping of multiple data manipulation SQL commands into a single atomic transaction. Some DBMS (e.g. MariaDB) force autocommit for every DDL statement, even in non-autocommit mode. In this case, before each DDL statement, previous DML statements in transaction are autocommitted. Each DDL statement is executed in its own new autocommit transaction.

    Read more →