AI App Video Generator

AI App Video Generator — independent reviews, comparisons, pricing and step-by-step guides on Aizhi.

  • Level set (data structures)

    Level set (data structures)

    In computer science, a level set is a data structure designed to represent discretely sampled dynamic level sets of functions. A common use of this form of data structure is in efficient image rendering. The underlying method constructs a signed distance field that extends from the boundary, and can be used to solve the motion of the boundary in this field. == Chronological developments == The powerful level-set method is due to Osher and Sethian 1988. However, the straightforward implementation via a dense d-dimensional array of values, results in both time and storage complexity of O ( n d ) {\displaystyle O(n^{d})} , where n {\displaystyle n} is the cross sectional resolution of the spatial extents of the domain and d {\displaystyle d} is the number of spatial dimensions of the domain. === Narrow band === The narrow band level set method, introduced in 1995 by Adalsteinsson and Sethian, restricted most computations to a thin band of active voxels immediately surrounding the interface, thus reducing the time complexity in three dimensions to O ( n 2 ) {\displaystyle O(n^{2})} for most operations. Periodic updates of the narrowband structure, to rebuild the list of active voxels, were required which entailed an O ( n 3 ) {\displaystyle O(n^{3})} operation in which voxels over the entire volume were accessed. The storage complexity for this narrowband scheme was still O ( n 3 ) . {\displaystyle O(n^{3}).} Differential constructions over the narrow band domain edge require careful interpolation and domain alteration schemes to stabilise the solution. === Sparse field === This O ( n 3 ) {\displaystyle O(n^{3})} time complexity was eliminated in the approximate "sparse field" level set method introduced by Whitaker in 1998. The sparse field level set method employs a set of linked lists to track the active voxels around the interface. This allows incremental extension of the active region as needed without incurring any significant overhead. While consistently O ( n 2 ) {\displaystyle O(n^{2})} efficient in time, O ( n 3 ) {\displaystyle O(n^{3})} storage space is still required by the sparse field level set method. See for implementation details. === Sparse block grid === The sparse block grid method, introduced by Bridson in 2003, divides the entire bounding volume of size n 3 {\displaystyle n^{3}} into small cubic blocks of m 3 {\displaystyle m^{3}} voxels each. A coarse grid of size ( n / m ) 3 {\displaystyle (n/m)^{3}} then stores pointers only to those blocks that intersect the narrow band of the level set. Block allocation and deallocation occur as the surface propagates to accommodate to the deformations. This method has a suboptimal storage complexity of O ( ( n m ) 3 + m 3 n 2 ) {\displaystyle O\left((nm)3+m^{3}n^{2}\right)} , but retains the constant time access inherent to dense grids. === Octree === The octree level set method, introduced by Strain in 1999 and refined by Losasso, Gibou and Fedkiw, and more recently by Min and Gibou uses a tree of nested cubes of which the leaf nodes contain signed distance values. Octree level sets currently require uniform refinement along the interface (i.e. the narrow band) in order to obtain sufficient precision. This representation is efficient in terms of storage, O ( n 2 ) , {\displaystyle O(n^{2}),} and relatively efficient in terms of access queries, O ( log n ) . {\displaystyle O(\log \,n).} An advantage of the level method on octree data structures is that one can solve the partial differential equations associated with typical free boundary problems that use the level set method. The CASL research group has developed this line of work in computational materials, computational fluid dynamics, electrokinetics, image-guided surgery and controls. === Run-length encoded === The run-length encoding (RLE) level set method, introduced in 2004, applies the RLE scheme to compress regions away from the narrow band to just their sign representation while storing with full precision the narrow band. The sequential traversal of the narrow band is optimal and storage efficiency is further improved over the octree level set. The addition of an acceleration lookup table allows for fast O ( log ⁡ r ) {\displaystyle O(\log r)} random access, where r is the number of runs per cross section. Additional efficiency is gained by applying the RLE scheme in a dimensional recursive fashion, a technique introduced by Nielsen & Museth's similar DT-Grid. === Hash Table Local Level Set === The Hash Table Local Level Set method was introduced in 2011 by Eyiyurekli and Breen and extended in 2012 by Brun, Guittet, and Gibou, only computes the level set data in a band around the interface, as in the Narrow Band Level-Set Method, but also only stores the data in that same band. A hash table data structure is used, which provides an O ( 1 ) {\displaystyle O(1)} access to the data. However, Brun et al. conclude that their method, while being easier to implement, performs worse than a quadtree implementation. They find that as it is, [...] a quadtree data structure seems more adapted than the hash table data structure for level-set algorithms. Three main reasons for worse efficiency are listed: to obtain accurate results, a rather large band is required close to the interface, which counterbalances the absence of grid nodes far from the interface; the performances are deteriorated by extrapolation procedures on the outer edges of the local grid and the width of the band restricts the time step and slows down the method. === Point-based === Corbett in 2005 introduced the point-based level set method. Instead of using a uniform sampling of the level set, the continuous level set function is reconstructed from a set of unorganized point samples via moving least squares.

    Read more →
  • Private message

    Private message

    In computer networking, a private message (PM), or direct message (DM), refers to a private communication, often text-based, sent or received by a user of a private communication channel on any given platform. Unlike public posts, PMs are only viewable by the participants. Long a function present on IRCs and Internet forums, private channels for PMs have also been prevalent features on instant messaging (IM) and on social media networks. It may be either synchronous (e.g. on an IM) or asynchronous (e.g. on an Internet forum). The term private message (PM) originated as a feature on internet forums, while the term direct message (DM) originated as a feature on Twitter. Due to the popularity of the latter service, DM has since been appropriated by other platforms, such as Instagram, and is often genericized in popular usage. == Overview == There are two main types of private messages, and one obscure type: One type includes those found on IRCs and Internet forums, as well as on social media services like Twitter, Facebook, and Instagram, where the focus is public posting, PMs allow users to communicate privately without leaving the platform. The second type are those relayed through instant messaging platforms such as WhatsApp and Snapchat, where users join the networks primarily to exchange PMs. A third type, peer-to-peer messaging, occurs when users create and own the infrastructure used to transmit and store the messages; while features vary depending on application, they give the user full control over the data they transmit. An example of software that enables this kind of messaging is Classified-ads. Besides serving as a tool to connect privately with friends and family, PMs have gained momentum in the workplace. Working professionals use PMs to reach coworkers in other spaces and increase efficiency during meetings. Although useful, using PMs in the workplace may blur the boundary between work and private lives. Some common forms of private messaging today include Facebook messaging (sometimes referred to as "inboxing"), Twitter direct messaging, and Instagram direct messaging. These forms of private messaging provide a private space on a usually public site. For instance, most activity on Twitter is public, but Twitter DMs provide a private space for communication between two users. This differs from mediums like email, texting, and Snapchat, where most or all activity is always private. Modern forms of private messaging may include multimedia messages, such as pictures or videos. == History == Email was first developed to send messages between different computers on ARPANET in 1971. Access to ARPANET was primarily limited to universities and other research institutions. Starting in 1983 or 1984, FidoNet allowed home computer users to send and receive email via bulletin board systems. Information services such as CompuServe, America Online, and Prodigy also helped to popularizes online messaging. The advent of the public World Wide Web in 1993 increased access to email via internet service providers, and later via webmail. Instant messaging systems became popular in the mid 1990s, as Internet access improved and personal computers became more common. The introduction of Skype in 2003 popularized Internet-based voice and video messaging. Direct messaging is now a feature of all major social networking services. == Privacy concerns == In January 2014, Matthew Campbell and Michael Hurley filed a class-action lawsuit against Facebook for breaching the Electronic Communications Privacy Act. They alleged that private messages which contained URLs were being read and used to generate profit, through data mining and user profiling, and that it was misleading for Facebook to refer to the functionality as "private" with the implication that the communication was "free from surveillance". In 2012, some Facebook users misinterpreted a redesign of the Facebook wall as publicly sharing private messages from 2008–2009. These were found to be public wall posts from those years, made at a time when it was not possible to like or comment on a wall post, making the notes look like private messages.

    Read more →
  • Data preservation

    Data preservation

    Data preservation is the act of conserving and maintaining both the safety and integrity of data. Preservation is done through formal activities that are governed by policies, regulations and strategies directed towards protecting and prolonging the existence and authenticity of data and its metadata. Data can be described as the elements or units in which knowledge and information is created, and metadata are the summarizing subsets of the elements of data; or the data about the data. The main goal of data preservation is to protect data from being lost or destroyed and to contribute to the reuse and progression of the data. == History == Most historical data collected over time has been lost or destroyed. War and natural disasters combined with the lack of materials and necessary practices to preserve and protect data has caused this. Usually, only the most important data sets were saved, such as government records and statistics, legal contracts and economic transactions. Scientific research and doctoral theses data have mostly been destroyed from improper storage and lack of data preservation awareness and execution. Over time, data preservation has evolved and has generated importance and awareness. We now have many different ways to preserve data and many different important organizations involved in doing so. The first digital data preservation storage solutions appeared in the 1950s, which were usually flat or hierarchically structured. While there were still issues with these solutions, it made storing data much cheaper, and more easily accessible. In the 1970s relational databases as well as spreadsheets appeared. Relational data bases structure data into tables using structured query languages which made them more efficient than the preceding storage solutions, and spreadsheets hold high volumes of numeric data which can be applied to these relational databases to produce derivative data. More recently, non-relational (non-structured query language) databases have appeared as complements to relational databases which hold high volumes of unstructured or semi-structured data. == Importance == The scope of data preservation is vast. Everything from governmental to business records to art essentially can be represented as data, and is amenable to be lost. This then leads to loss of human history, for perpetuity. Data can be lost on a small or independent scale whether it's personal data loss, or data loss within businesses and organizations, as well as on a larger or national or global scale which can negatively and potentially permanently affect things such as environmental protection, medical research, homeland security, public health and safety, economic development and culture. The mechanisms of data loss are also as many as they are varied, spanning from disaster, wars, data breaches, negligence, all the way through simple forgetting to natural decay. Ways in which data collections can be used when preserved and stored properly can be seen through the U.S. Geological Survey, which stores data collections on natural hazards, natural resources, and landscapes. The data collected by the Survey is used by federal and state land management agencies towards land use planning and management, and continually needs access to historical reference data. == Related Concepts == In contrast, data holdings are collections of gathered data that are informally kept, and not necessarily prepared for long-term preservation. For example, a collection or back-up of personal files. Data holdings are generally the storage methods used in the past when data has been lost due to environmental and other historical disasters. Furthermore, data retention differs from data preservation in the sense that by definition, to retain an object (data) is to hold or keep possession or use of the object. To preserve an object is to protect, maintain and keep up for future use. Retention policies often circle around when data should be deleted on purpose as well, and held from public access, while preservation prioritizes permanence and more widely shared access. Thus, data preservation exceeds the concept of having or possessing data or back up copies of data. Data preservation ensures reliable access to data by including back-up and recovery mechanisms that precede the event of a disaster or technological change. == Methods == === Digital === Digital preservation, is similar to data preservation, but is mainly concerned with technological threats, and solely digital data. Essentially digital data is a set of formal activities to enable ongoing or persistent use and access of digital data exceeding the occurrence of technological malfunction or change. Digital preservation is aware of the inevitable change in technology and protocols, and prepares for data that will need to be accessible across new types of technologies and platforms while the integrity of the data and metadata are being conserved. Technology, while providing great process in conserving data that may not have been possible in the past, is also changing at such a quick rate that digital data may not be accessible anymore due to the format being incompatible with new software. Without the use of data preservation much of our existing digital data is at risk. The majority of methods used towards data preservation today are digital methods, which are so far the most effective methods that exist. === Archives === Archives are a collection of historical documents and records. Archives contribute and work towards the preservation of data by collecting data that is well organized, while providing the appropriate metadata to confirm it. An example of an important data archive is The LONI Image Data Archive, which is an archive that collects data regarding clinical trials and clinical research studies. === Catalogues, directories and portals === Catalogues, directories and portals are consolidated resources which are kept by individual institutions, and are associated with data archives and holdings. In other words, the data is not presented on the site, but instead might act as metadata and aggregators, and may administer thorough inventories. === Repositories === Repositories are places where data archives and holdings can be accessed and stored. The goal of repositories is to make sure that all requirements and protocols of archives and holdings are being met, and data is being certified to ensure data integrity and user trust. Single-site Repositories A repository that holds all data sets on a single site. An example of a major single-site repository the Data Archiving and Networking Services which is a repository which provides ongoing access to digital research resources for the Netherlands. Multi-Site Repositories A repository that hosts data set on multiple institutional sites. An example of a well known multi-site repository is OpenAIRE which is a repository that hosts research data and publications collaborating all of the EU countries and more. OpenAIRE promotes open scholarship and seeks to improves discover-ability and re-usability of data. Trusted Digital Repository A repository that seeks to provide reliable, trusted access over a long period of time. The repository can be single or multi-sited but must cooperate with the Reference Model for an Open Archival Information System, as well as adhere to a set of rules or attributes that contribute to its trust such as having persistent financial responsibility, organizational buoyancy, administrative responsibility security and safety. An example of a trusted digital repository is The Digital Repository of Ireland (DRI) which is a multi-site repository that hosts Ireland's humanity and social science data sets. === Cyber Infrastructures === Cyber infrastructures which consists of archive collections which are made available through the system of hardware, technologies, software, policies, services and tools. Cyber infrastructures are geared towards the sharing of data supporting peer-to-peer collaborations and a cultural community. An example of a major cyber-infrastructure is The Canadian Geo-spatial Data Infrastructure which provides access to spatial data in Canada.

    Read more →
  • Media intelligence

    Media intelligence

    Media intelligence uses data mining and data science to analyze public, social and editorial media content. It refers to marketing systems that synthesize billions of online conversations into relevant information. This allow organizations to measure and manage content performance, understand trends, and drive communications and business strategy. Media intelligence can include software as a service using big data terminology. This includes questions about messaging efficiency, share of voice, audience geographical distribution, message amplification, influencer strategy, journalist outreach, creative resonance, and competitor performance in all these areas. Media intelligence differs from business intelligence in that it uses and analyzes data outside company firewalls. Examples of that data are user-generated content on social media sites, blogs, comment fields, and wikis etc. It may also include other public data sources like press releases, news, blogs, legal filings, reviews and job postings. Media intelligence may also include competitive intelligence, wherein information that is gathered from publicly available sources such as social media, press releases, and news announcements are used to better understand the strategies and tactics being deployed by competing businesses. Media intelligence is enhanced by means of emerging technologies like ambient intelligence, machine learning, semantic tagging, natural language processing, sentiment analysis and machine translation. == Technologies used == Different media intelligence platforms use different technologies for monitoring, curating content, engaging with content, data analysis and measurement of communications and marketing campaign success. These technology providers may obtain content by scraping content directly from websites or by connecting to the API provided by social media, or other content platforms that are created for 3rd party developers to develop their own applications and services that access data. Technology companies may also get data from a data reseller. Some social media monitoring and analytics companies use calls to data providers each time an end-user develops a query. Others archive and index social media posts to provide end users with on-demand access to historical data and enable methodologies and technologies leveraging network and relational data. Additional monitoring companies use crawlers and spidering technology to find keyword references, known as semantic analysis or natural language processing. Basic implementation involves curating data from social media on a large scale and analyzing the results to make sense out of it.

    Read more →
  • Vatican News App

    Vatican News App

    The Vatican News App is an official mobile application software issued by the Vatican's Dicastery for Communication. Formerly titled The Pope App, the app was launched on January 23, 2013, under the auspices of the Pontifical Council for Social Communications, a now-defunct dicastery that was merged into the Secretariat (now Dicastery) for Communication in March 2016. Initially, The Pope App was available only on iOS devices, but became available for Android phones at the end of February 2013. The app is available for download on iOS and Android in five languages: English, French, Italian, Portuguese and Spanish. It was originally promoted as an application with focus on the figure of the Pope which made it possible to follow the Pope's events while they are taking place. Alerts notified the followers by informing and offering access to "official papal-related content in a variety of formats". The app also enabled its users to see areas of the Vatican through webcams allocated throughout St. Peter's Square in Rome that broadcast images. In early 2018, The Pope App was relaunched as the Vatican News App, accompanied by a redesign that eliminated many of the previous version's features, reducing the app to a more conventional news service, with increased emphasis on news from the Vatican and the worldwide Catholic Church and less focus on the day-to-day activities of the Pope.

    Read more →
  • Strategic Air Command Digital Information Network

    Strategic Air Command Digital Information Network

    The Strategic Air Command DIgital Network (SACDIN) was a United States military computer network that provided computerized record communications, replacing the Data Transmission Subsystem and part of the Data Display Subsystem of the SAC Automated Command and Control System. SACDIN enabled a rapid flow of communications from headquarters SAC to its fielded forces, such as B-52 bases and ICBM Launch Control Centers. == Logistics == Major portions of SACDIN were developed, engineered and installed by the International Telephone and Telegraph (ITT) company, under contract to the Electronic Systems Center. == Chronology == 1969 - Headquarters SAC submits a request to the Joint Chiefs of Staff to study an expanded communications system, known as the SAC Total Information Network (SATIN). It would interconnect Air Force Satellite Communications (AFSATCOM), Advanced Airborne Command Post (AABNCP), Airborne Command Post (ABNCP), high frequency/single sideband radio HF/SSB radio, SAC Automated Command and Control System (SACCS), Automatic Digital Information Network (AUTODIN), Survivable Low Frequency Communications System (SLFCS) and Command Data Buffer (CDB) 1977 1 November - SATIN IV was effectively terminated by Congress. The restructured program was renamed SAC Digital Network (SACDIN), and was formulated to meet SAC's minimum essential data communications requirements, but also had the capability to grow in a modular fashion. 1986 ?? ??? - SACDIN replaces much of the SAC Automated Command and Control System (SACCS) and the SAC Automated Total Information Network (SATIN)

    Read more →
  • Business intelligence

    Business intelligence

    Business intelligence (BI) consists of strategies, methodologies, and technologies used by enterprises for data analysis and management of business information to inform business strategies and business operations. Common functions of BI technologies include reporting, online analytical processing, analytics, dashboard development, data mining, process mining, complex event processing, business performance management, benchmarking, text mining, predictive analytics, and prescriptive analytics. BI tools can handle large amounts of structured and sometimes unstructured data to help organizations identify, develop, and otherwise create new strategic business opportunities. They aim to allow for the easy interpretation of these big data. Identifying new opportunities and implementing an effective strategy based on insights is assumed to potentially provide businesses with a competitive market advantage and long-term stability, and help them take strategic decisions. Business intelligence can be used by enterprises to support a wide range of business decisions ranging from operational to strategic. Basic operating decisions include product positioning or pricing. Strategic business decisions involve priorities, goals, and directions at the broadest level. In all cases, business intelligence is considered most effective when it combines data from the market in which a company operates (external data) with data from internal company sources, such as financial and operational information. When integrated, external and internal data provide a comprehensive view that creates ‘intelligence’ not possible from any single data source alone. Among their many uses, business intelligence tools empower organizations to gain insight into new markets, to assess demand and suitability of products and services for different market segments, and to gauge the impact of marketing efforts. BI applications use data gathered from a data warehouse (DW) or from a data mart, and the concepts of BI and DW combine as "BI/DW" or as "BIDW". A data warehouse contains a copy of analytical data that facilitates decision support. == History == The earliest known use of the term business intelligence is in Richard Millar Devens' Cyclopædia of Commercial and Business Anecdotes (1865). Devens used the term to describe how the banker Sir Henry Furnese gained profit by receiving and acting upon information about his environment, prior to his competitors: Throughout Holland, Flanders, France, and Germany, he maintained a complete and perfect train of business intelligence. The news of the many battles fought was thus received first by him, and the fall of Namur added to his profits, owing to his early receipt of the news. The ability to collect and react accordingly based on the information retrieved, Devens says, is central to business intelligence. When Hans Peter Luhn, a researcher at IBM, used the term business intelligence in an article published in 1958, he employed the Webster's Dictionary definition of intelligence: "the ability to apprehend the interrelationships of presented facts in such a way as to guide action towards a desired goal." In 1989, Howard Dresner (later a Gartner analyst) proposed business intelligence as an umbrella term to describe "concepts and methods to improve business decision making by using fact-based support systems." It was not until the late 1990s that this usage was widespread. == Definition == According to Solomon Negash and Paul Gray, business intelligence (BI) can be defined as systems that combine: Data gathering Data storage Knowledge management with analysis to evaluate complex corporate and competitive information for presentation to planners and decision makers, with the objective of improving the timeliness and the quality of the input to the decision process." According to Forrester Research, business intelligence is "a set of methodologies, processes, architectures, and technologies that transform raw data into meaningful and useful information used to enable more effective strategic, tactical, and operational insights and decision-making." Under this definition, business intelligence encompasses information management (data integration, data quality, data warehousing, master-data management, text- and content-analytics, et al.). Therefore, Forrester refers to data preparation and data usage as two separate but closely linked segments of the business-intelligence architectural stack. Some elements of business intelligence are: Multidimensional aggregation and allocation Denormalization, tagging, and standardization Realtime reporting with analytical alert A method of interfacing with unstructured data sources Group consolidation, budgeting, and rolling forecasts Statistical inference and probabilistic simulation Key performance indicators optimization Version control and process management Open item management Forrester distinguishes this from the business-intelligence market, which is "just the top layers of the BI architectural stack, such as reporting, analytics, and dashboards." === Compared with competitive intelligence === Though the term business intelligence is sometimes a synonym for competitive intelligence (because they both support decision making), BI uses technologies, processes, and applications to analyze mostly internal, structured data and business processes while competitive intelligence gathers, analyzes, and disseminates information with a topical focus on company competitors. If understood broadly, competitive intelligence can be considered as a subset of business intelligence. === Compared with business analytics === Business intelligence and business analytics are sometimes used interchangeably, but there are alternate definitions. Thomas Davenport, professor of information technology and management at Babson College argues that business intelligence should be divided into querying, reporting, Online analytical processing (OLAP), an "alerts" tool, and business analytics. In this definition, business analytics is the subset of BI focusing on statistics, prediction, and optimization, rather than the reporting functionality. == Unstructured data == Business operations can generate a very large amount of data in the form of emails, memos, notes from call centers, news, user groups, chats, reports, web pages, presentations, image files, video files, and marketing material. According to Merrill Lynch, more than 85% of all business information exists in these forms; a company might only use such a document a single time. Because of the way it is produced and stored, this information is either unstructured or semi-structured. The management of semi-structured data is an unsolved problem in the information technology industry. According to projections from Gartner (2003), white-collar workers spend 30–40% of their time searching, finding, and assessing unstructured data. BI uses both structured and unstructured data. The former is easy to search, and the latter contains a large quantity of the information needed for analysis and decision-making. Because of the difficulty of properly searching, finding, and assessing unstructured or semi-structured data, organizations may not draw upon these vast reservoirs of information, which could influence a particular decision, task, or project. This can ultimately lead to poorly informed decision-making. Therefore, when designing a business intelligence/DW solution, the specific problems associated with semi-structured and unstructured data must be accommodated, as well as those associated with structured data. === Limitations of semi-structured and unstructured data === There are several challenges to developing BI with semi-structured data. According to Inmon & Nesavich, some of those are: Physically accessing unstructured textual data – unstructured data is stored in a huge variety of formats. Terminology – Among researchers and analysts, there is a need to develop standardized terminology. Volume of data – As stated earlier, up to 85% of all data exists as semi-structured data. Couple that with the need for word-to-word and semantic analysis. Searchability of unstructured textual data – A simple search on some data, e.g. apple, results in links where there is a reference to that precise search term. (Inmon & Nesavich, 2008) gives an example: "a search is made on the term felony. In a simple search, the term felony is used, and everywhere there is a reference to felony, a hit to an unstructured document is made. But a simple search is crude. It does not find references to crime, arson, murder, embezzlement, vehicular homicide, and such, even though these crimes are types of felonies". === Metadata === To solve problems with searchability and assessment of data, it is necessary to know something about the content. This can be done by adding context through the use of metadata. Many systems already capture some metadata (e.g. filename, author, size, etc.), but more usef

    Read more →
  • Chaos Communication Congress

    Chaos Communication Congress

    The Chaos Communication Congress is an annual hacker conference organized by the Chaos Computer Club. The congress features a variety of lectures and workshops on technical and political issues related to security, cryptography, privacy and online freedom of speech. It has taken place regularly at the end of the year since 1984, with the current date and duration (27–30 December) established in 2005. It is considered one of the largest events of its kind, alongside DEF CON in Las Vegas. == History == The congress is held in Germany. It started in 1984 in Hamburg, moved to Berlin in 1998, and back to Hamburg in 2012, having exceeded the capacity of the Berlin venue with more than 4500 attendees. Since then, it attracts an increasing number of people: around 6600 attendees in 2012, over 13000 in 2015, and more than 15000 in 2017. From 2017 to 2019, it took place at the Trade Fair Grounds in Leipzig, since the Hamburg venue (CCH) was closed for renovation in 2017 and the existing space was not enough for the growing congress. The congress moved back to Hamburg in 2023, after the renovation of CCH was finished. A large range of speakers are featured. The event is organized by volunteers called Chaos Angels. The non-members entry fee for four days was €100 in 2016, and was raised to €120 in 2018 to include a public transport ticket for the Leipzig area. An important part of the congress are the assemblies, semi-open spaces with clusters of tables and internet connections for groups and individuals to collaborate and socialize in projects, workshops and hands-on talks. These assembly spaces, introduced at the 2012 meeting, combine the hack center project space and distributed group spaces of former years. From 1997 to 2004 the congress also hosted the annual German Lockpicking Championships. 2005 was the first year the Congress lasted four days instead of three and lacked the German Lockpicking Championships. 2020 was the first year where the Congress did not take place at a physical location due to the COVID-19 pandemic, giving way to the first Remote Chaos Experience (rC3). The Chaos Computer Club announced to return to the now newly renovated Congress Center Hamburg for the 37th edition of the Chaos Communication Congress. The announcement confirms the usual date of 27-30 December, notably omitting the year it will be held. On 18 October 2022, they confirmed that the congress will indeed not be held in 2022. On 6 October 2023, the CCC announced that 37C3 will take place again on the usual dates in 2023. === Timeline ===

    Read more →
  • Creately

    Creately

    Creately is a SaaS visual collaboration tool with diagramming and design capabilities designed by Cinergix. The application is mostly known for creating flowcharts, organization charts, project charts, UML diagrams, mind maps, and other business visuals. == History == The initial beta version of Creately was released by Chandika Jayasundara. Hiraash Thawfeek, Nick Foster and Charanjit Singh joined the project in the same year. Chandika Jayasundara is CEO of Cinergix. The headquarters of the company is located at Mentone, Victoria, Australia. == Features and reception == Creately provides predefined templates and diagram elements for incorporating in the projects. It provides drag and drop feature with which both predefined and custom made shapes can be included to build the desired diagram while the same workspace can be shared with multiple persons for collaboration. Some experts have reviewed the application by commenting on its lacking in accessible integration options as its downside. The company claims Creately to have integration feature with Slack, Confluence while not having the integration with Zapier and OneDrive yet. It is compatible with Google Drive and Dropbox. The software is available as both freemium and paid option.

    Read more →
  • Social media optimization

    Social media optimization

    Social media optimization (SMO) is the use of online platforms to generate income or publicity to increase the awareness of a brand, event, product or service. Types of social media involved include RSS feeds, blogging sites, social bookmarking sites, social news websites, video sharing websites such as YouTube and social networking sites such as Facebook, Instagram, TikTok and X (Twitter). SMO is similar to search engine optimization (SEO) in that the goal is to drive web traffic, and draw attention to a company or creator. SMO's focal point is on gaining organic links to social media content. In contrast, SEO's core is about reaching the top of the search engine hierarchy. In general, social media optimization refers to optimizing a website and its content to encourage more users to use and share links to the website across social media and networking sites. SMO is used to strategically create online content ranging from well-written text to eye-catching digital photos or video clips that encourages and entices people to engage with a website. Users share this content, via its weblink, with social media contacts and friends. Common examples of social media engagement are "liking and commenting on posts, retweeting, embedding, sharing, and promoting content". Social media optimization is also an effective way of implementing online reputation management (ORM), meaning that if someone posts bad reviews of a business, an SMO strategy can ensure that the negative feedback is not the first link to come up in a list of search engine results. In the 2010s, with social media sites overtaking TV as a source for news for young people, news organizations have become increasingly reliant on social media platforms for generating web traffic. Publishers such as The Economist employ large social media teams to optimize their online posts and maximize traffic, while other major publishers now use advanced artificial intelligence (AI) technology to generate higher volumes of web traffic. == Relationship with search engine optimization == Social media optimization is an increasingly important factor in search engine optimization, which is the process of designing a website in a way so that it has as high a ranking as possible on search engines. Search engines are increasingly utilizing the recommendations of users of social networks such as Reddit, Facebook, Tumblr, Twitter, YouTube, LinkedIn, Pinterest and Instagram to rank pages in the search engine result pages. The implication is that when a webpage is shared or "liked" by a user on a social network, it counts as a "vote" for that webpage's quality. Thus, search engines can use such votes accordingly to properly ranked websites in search engine results pages. Furthermore, since it is more difficult to tip the scales or influence the search engines in this way, search engines are putting more stock into social search. This, coupled with increasingly personalized search based on interests and location, has significantly increased the importance of a social media presence in search engine optimization. Due to personalized search results, location-based social media presences on websites such as Yelp, Google Places, Foursquare, and Yahoo! Local have become increasingly important. While social media optimization is related to search engine marketing, it differs in several ways. Primarily, SMO focuses on driving web traffic from sources other than search engines, though improved search engine ranking is also a benefit of successful social media optimization. Further, SMO is helpful to target particular geographic regions in order to target and reach potential customers. This helps in lead generation (finding new customers) and contributes to high conversion rates (i.e., converting previously uninterested individuals into people who are interested in a brand or organization). == Relationship with viral marketing == Social media optimization is in many ways connected to the technique of viral marketing or "viral seeding" where word of mouth is created through the use of networking in social bookmarking, video and photo sharing websites. An effective SMO campaign can harness the power of viral marketing; for example, 80% of activity on Pinterest is generated through "repinning." Furthermore, by following social trends and utilizing alternative social networks, websites can retain existing followers while also attracting new ones. This allows businesses to build an online following and presence, all linking back to the company's website for increased traffic. For example, with an effective social bookmarking campaign, not only can website traffic be increased, but a site's rankings can also be increased. In a similar way, the engagement with blogs creates a similar result by sharing content through the use of RSS in the blogosphere. Social media optimization is considered an integral part of an online reputation management (ORM) or search engine reputation management (SERM) strategy for organizations or individuals who care about their online presence. SMO is one of six key influencers that affect Social Commerce Construct (SCC). Online activities such as consumers' evaluations and advices on products and services constitute part of what creates a Social Commerce Construct (SCC). Social media optimization is not limited to marketing and brand building. Increasingly, smart businesses are integrating social media participation as part of their knowledge management strategy (i.e., product/service development, recruiting, employee engagement and turnover, brand building, customer satisfaction and relations, business development and more). Additionally, social media optimization can be implemented to foster a community of the associated site, allowing for a healthy business-to-consumer (B2C) relationship. == Origins and implementation == According to technologist Danny Sullivan, the term "social media optimization" was first used and described by marketer Rohit Bhargava on his marketing blog in August 2006. In the same post, Bhargava established the five important rules of social media optimization. Bhargava believed that by following his rules, anyone could influence the levels of traffic and engagement on their site, increase popularity, and ensure that it ranks highly in search engine results. An additional 11 SMO rules have since been added to the list by other marketing contributors. The 16 rules of SMO, according to one source, are as follows: Increase your linkability Make tagging and bookmarking easy Reward inbound links Help your content to "travel" via sharing Encourage the mashup, where users are allowed to remix content Be a user resource, even if it doesn't help you (e.g., provide resources and information for users) Reward helpful and valuable users Participate (join the online conversation) Know how to target your audience Create new, quality content ("web scraping" of existing online content is ignored by good search engines) Be "real" in the tone and style of the posts Don't forget your roots; be humble Don't be afraid to experiment, innovate, try new things and "stay fresh" Develop an SMO strategy Choose your SMO tactics wisely Make SMO a key part of your marketing process and develop company best practices Bhargava's initial five rules were more specifically designed to SMO, while the list is now much broader and addresses everything that can be done across different social media platforms. According to author and CEO of TopRank Online Marketing, Lee Odden, a Social Media Strategy is also necessary to ensure optimization. This is a similar concept to Bhargava's list of rules for SMO. The Social Media Strategy may consider: Objectives e.g. creating brand awareness and using social media for external communications. Listening e.g. monitoring conversations relating to customers and business objectives. Audience e.g. finding out who the customers are, what they do, who they are influenced by, and what they frequently talk about. It is important to work out what customers want in exchange for their online engagement and attention. Participation and content e.g. establishing a presence and community online and engaging with users by sharing useful and interesting information. Measurement e.g. keeping a record of likes and comments on posts, and the number of sales to monitor growth and determine which tactics are most useful in optimizing social media. According to Lon Safko and David K. Brake in The Social Media Bible, it is also important to act like a publisher by maintaining an effective organizational strategy, to have an original concept and unique "edge" that differentiates one's approach from competitors, and to experiment with new ideas if things do not work the first time. If a business is blog-based, an effective method of SMO is using widgets that allow users to share content to their personal social media platforms. This will ultimately reach a wider target audience and drive mor

    Read more →
  • Social recruiting

    Social recruiting

    Social recruiting (social hiring or social media recruitment) is recruiting candidates by using social platforms as talent databases or for advertising. Social recruiting uses social media profiles, blogs, and other Internet sites to find information on candidates. It also uses social media to advertise jobs either through HR vendors or through crowdsourcing where job seekers and others share job openings within their online social networks. Social recruiting's effectiveness and return on investment have been difficult to determine, since applicants do not usually apply through the social channels which first attracted them. In May 2013, Maximum Employment Marketing Group released the Social Recruitment Monitor, which ranks the reach, engagement, and interactivity of employers' social recruiting efforts around the world. == Social recruitment software == The social recruitment software market (a form of e-recruitment) is often included in the wider talent management software sector. Bersin & Associates valued the wider talent management market at over $2bn in 2007. Social recruitment increasingly sits at an intersection of a number of fast-moving areas including social networking, recruitment and now cloud computing. Additionally, mobile recruiting has become another hot topic, especially with the rise in tablet and smartphone usage. In 2012, there was a rise of tech companies using social recruiting applications to find and screen applicants. As more companies saw value in filling jobs by putting them on the social platforms where millions of people spend at least 37 minutes daily, there developed a much larger focus on social recruiting among the talent acquisition community. By mid-2013, many major enterprise companies such as Pepsi, Gap, AIG, and Oracle had begun effectively utilizing social recruiting software, making it clear that large corporations were open to automating or streamlining (and ultimately investing in) their social recruiting processes.

    Read more →
  • Cryptographic Service Provider

    Cryptographic Service Provider

    A cryptographic service provider (CSP) is a package that "provides a concrete implementation of certain cryptographic services." A CSP offers operations and protocols to support a variety of use cases. The cryptographic application programming interface (API) provided by the CSP provides common solutions for different platforms, for example hardware and cloud services. == Microsoft Windows == In Microsoft Windows, a Cryptographic Service Provider is a software library that implements the Microsoft CryptoAPI (CAPI). CSPs implement encoding and decoding functions, which computer application programs may use, for example, to implement strong user authentication or for secure email. CSPs are independent modules that can be used by different applications. A user program calls CryptoAPI functions and these are redirected to CSPs functions. Since CSPs are responsible for implementing cryptographic algorithms and standards, applications do not need to be concerned about security details. Furthermore, each application can define which CSP it is going to use on its calls to CryptoAPI. In fact, all cryptographic activity is implemented in CSPs. CryptoAPI only works as a bridge between the application and the CSP. CSPs are implemented basically as a special type of DLL with special restrictions on loading and use. Every CSP must be digitally signed by Microsoft and the signature is verified when Windows loads the CSP. In addition, after being loaded, Windows periodically re-scans the CSP to detect tampering, either by malicious software such as computer viruses or by the user him/herself trying to circumvent restrictions (for example on cryptographic key length) that might be built into the CSP's code. To obtain a signature, non-Microsoft CSP developers must supply paperwork to Microsoft promising to obey various legal restrictions and giving valid contact information. As of circa 2000, Microsoft did not charge any fees to supply these signatures. For development and testing purposes, a CSP developer can configure Windows to recognize the developer's own signatures instead of Microsoft's, but this is a somewhat complex and obscure operation unsuitable for nontechnical end users. The CAPI/CSP architecture had its origins in the era of restrictive US government controls on the export of cryptography. Microsoft's default or "base" CSP then included with Windows was limited to 512-bit RSA public-key cryptography and 40-bit symmetric cryptography, the maximum key lengths permitted in exportable mass market software at the time. CSPs implementing stronger cryptography were available only to U.S. residents, unless the CSPs themselves had received U.S. government export approval. The system of requiring CSPs to be signed only on presentation of completed paperwork was intended to prevent the easy spread of unauthorized CSPs implemented by anonymous or foreign developers. As such, it was presented as a concession made by Microsoft to the government, in order to get export approval for the CAPI itself. After the Bernstein v. United States court decision establishing computer source code as protected free speech and the transfer of cryptographic regulatory authority from the U.S. State Department to the more pro-export Commerce Department, the restrictions on key lengths were dropped, and the CSPs shipped with Windows now include full-strength cryptography. The main use of third-party CSPs is to interface with external cryptography hardware such as hardware security modules (HSM) or smart cards. === Smart Card CSP === These cryptographic functions can be realized by a smart card, thus the Smart Card CSP is the Microsoft way of a PKCS#11. Microsoft Windows is identifying the correct Smart Card CSP, which have to be used, analyzing the answer to reset (ATR) of the smart card, which is registered in the Windows Registry. Installing a new CSP, all ATRs of the supported smart cards are enlisted in the registry. === Use of CSP in MS Office password protection === Cryptographic service providers can be used for encryption of Word, Excel, and PowerPoint documents starting from Microsoft Office XP. A standard encryption algorithm with a 40-bit key is used by default, but enabling a CSP enhances key length and thus makes decryption process more continuous. This only applies to passwords that are required to open document because this password type is the only one that encrypts a password-protected document.

    Read more →
  • C-RAN

    C-RAN

    C-RAN (Cloud-RAN), also referred to as Centralized-RAN, is an architecture for cellular networks. C-RAN is a centralized, cloud computing-based architecture for radio access networks that supports 2G, 3G, 4G, 5G and future wireless communication standards. Its name comes from the four 'C's in the main characteristics of C-RAN system, "Clean, Centralized processing, Collaborative radio, and a real-time Cloud Radio Access Network". == Background == Traditional cellular, or Radio Access Networks (RAN), consist of many stand-alone base stations (BTS). Each BTS covers a small area, whereas a group BTS provides coverage over a continuous area. Each BTS processes and transmits its own signal to and from the mobile terminal, and forwards the data payload to and from the mobile terminal and out to the core network via the backhaul. Each BTS has its own cooling, back haul transportation, backup battery, monitoring system, and so on. Because of limited spectral resources, network operators 'reuse' the frequency among different base stations, which can cause interference between neighboring cells. There are several limitations in the traditional cellular architecture. First, each BTS is costly to build and operate. Moore's law helps reduce the size and power of an electrical system, but the supporting facilities of the BTS are not improved quite as well. Second, when more BTS are added to a system to improve its capacity, interference among BTS is more severe as BTS are closer to each other and more of them are using the same frequency. Third, because users are mobile, the traffic of each BTS fluctuates (called 'tide effect'), and as a result, the average utilization rate of individual BTS is pretty low. However, these processing resources cannot be shared with other BTS. Therefore, all BTS are designed to handle the maximum traffic, not average traffic, resulting in a waste of processing resources and power at idle times. == Evolution of base station architecture == === All-in-one macro base station === In the 1G and 2G cellular networks, base stations had an all-in-one architecture. Analog, digital, and power functions were housed in a single cabinet as large as a refrigerator. Usually the base station cabinet was placed in a dedicated room along with all necessary supporting facilitates such as power, backup battery, air conditioning, environment surveillance, and backhaul transmission equipment. The RF signal is generated by the base station RF unit and propagates through pairs of RF cables up to the antennas on the top of a base station tower or other mounting points. This all-in-one architecture was mostly found in macro cell deployments. === Distributed base station === For 3G, a distributed base station architecture was introduced by Ericsson, Nokia, Huawei, and other leading telecom equipment vendors. In this architecture the radio function unit, also known as the remote radio head (RRH), is separated from the digital function unit, or baseband unit (BBU) by fiber. Digital baseband signals are carried over fiber, using the Open Base Station Architecture Initiative (OBSAI) or Common Public Radio Interface (CPRI) standard. The RRH can be installed on the top of tower close to the antenna, reducing the loss compared to the traditional base station where the RF signal has to travel through a long cable from the base station cabinet to the antenna at the top of the tower. The fiber link between RRH and BBU also allows more flexibility in network planning and deployment as they can be placed a few hundred meters or a few kilometers away. Most modern base stations now use this decoupled architecture. === C-RAN/Cloud-RAN === C-RAN may be viewed as an architectural evolution of the above distributed base station system. It takes advantage of many technological advances in wireless, optical and IT communications systems. For example, it uses the latest CPRI standard, low cost Coarse or Dense Wavelength Division Multiplexing (CWDM/ DWDM) technology, and mmWave to allow transmission of baseband signal over long distance thus achieving large scale centralised base station deployment. It applies recent Data Centre Network technology to allow a low cost, high reliability, low latency and high bandwidth interconnect network in the BBU pool. It utilizes open platforms and real-time virtualization technology rooted in cloud computing to achieve dynamic shared resource allocation and support multi-vendor, multi-technology environments. == Architecture overview == C-RAN architecture has the following characteristics that are distinct from other cellular architectures: Large scale centralized deployment: Allows many RRHs to connect to a centralized BBU pool. The maximum distance can be 20km in fiber link for 4G (LTE/LTE-A) systems, and even longer distances (40~80km) for 3G (WCDMA/TD-SCDMA) and 2G (GSM/CDMA) systems. Native support to Collaborative Radio technologies: Any BBU can talk with any other BBU within the BBU pool with very high bandwidth (10 Gbit/s and above) and low latency (10 μs level). This is enabled by the interconnection of BBUs in the pool. This is one major difference from BBU Hotelling, or base station Hotelling; in the latter case, the BBUs of different base stations are simply stacked together and have no direct link between them to allow physical layer co-ordination. Real-time virtualization capability based on open platform: This is different from traditional base stations built on proprietary hardware, where the software and hardware are close-sourced and provided by single vendors. In contrast, a C-RAN BBU pool is built on open hardware, like x86/ARM CPU based servers, and interface cards that handle fiber links to RRHs and inter-connections in the pool. Real-time virtualization ensures that resources in the pool can be allocated dynamically to base station software stacks, say 4G/3G/2G function modules from different vendors, according to network load. However, to satisfy the strict timing requirements of wireless communication systems, the real-time performance for C-RAN is at the level of tens of microseconds, which is two orders of magnitude better than the millisecond level 'real-time' performance usually seen in Cloud Computing environments. == Similar architecture and systems == KT, a telecom operator in the Republic of Korea, introduced a Cloud Computing Center (CCC) system in their 3G (WCDMA/HSPA) and 4G (LTE/LTE-A) network in 2011 and 2012. The concept of CCC is basically the same as C-RAN. SK Telecom has also deployed Smart Cloud Access Network (SCAN) and Advanced-SCAN in their 4G (LTE/LTE-A) network in Korea no later than 2012. In 2014, Airvana (now CommScope) introduced OneCell, a C-RAN-based small cell system designed for enterprises and public spaces. == Competing architectures in cellular network evolution == === All-in-one BTS === One major alternative solution that is addressing similar challenges of RAN, is the small size, all-in-one outdoor BTS. Thanks to the achievements in the semiconductor industry, all the functionality of a BTS, including RF, baseband processing, MAC processing and package level processing, can now be implemented in a volume of <50 liters. This makes the system small and weatherproof, reduces the difficulty of BTS site choice and construction, eliminates the air conditioning requirement, and thus reduces operational costs. However, because each BTS is still working on its own, it cannot readily make use of the collaboration algorithms to reduce the interference between neighboring BTSs. It is also relatively hard to upgrade or repair because the all-in-one BTS units are usually mounted near the antenna. More processing units in less-protected environments also implies a higher failure rate compared to C-RAN, which only has the RRU deployed outdoors. The advantage of Cloud RAN lies in its ability to implement LTE-Advanced features such as Coordinated MultiPoint (CoMP) with very low latency between multiple radio heads. However, the economic benefit of improvements such as CoMP can be negated by the higher backhaul costs for some operators. === Small cell === The main competition between small cell and C-RAN occurs in two deployment scenarios: outdoor hotspot coverage and indoor coverage. == Academic research and publications == As one of the promising evolution paths for future cellular network architecture, C-RAN has attracted high academic research interest. Meanwhile, because the native support of cooperative radio capability built into the C-RAN architecture, it also enables many advanced algorithms that were hard to implement in cellular networks, including Cooperative Multi-Point Transmission/Receiving, Network Coding, etc. In October 2011, Wireless World Research Forum 27 was hosted in Germany, when China Mobile was invited to give a C-RAN presentation. In August 2012, IEEE C-RAN 2012 workshop was hosted in Kunming, China. CRC Press published a book, "Green Communications: Theore

    Read more →
  • Bitmap index

    Bitmap index

    A bitmap index is a special kind of database index that uses bitmaps. Bitmap indexes have traditionally been considered to work well for low-cardinality columns, which have a modest number of distinct values, either absolutely, or relative to the number of records that contain the data. The extreme case of low cardinality is Boolean data (e.g., does a resident in a city have internet access?), which has two values, True and False. Bitmap indexes use bit arrays (commonly called bitmaps) and answer queries by performing bitwise logical operations on these bitmaps. Bitmap indexes have a significant space and performance advantage over other structures for query of such data. Their drawback is they are less efficient than the traditional B-tree indexes for columns whose data is frequently updated: consequently, they are more often employed in read-only systems that are specialized for fast query - e.g., data warehouses, and generally unsuitable for online transaction processing applications. Some researchers argue that bitmap indexes are also useful for moderate or even high-cardinality data (e.g., unique-valued data) which is accessed in a read-only manner, and queries access multiple bitmap-indexed columns using the AND, OR or XOR operators extensively. Bitmap indexes are also useful in data warehousing applications for joining a large fact table to smaller dimension tables such as those arranged in a star schema. == Example == Continuing the internet access example, a bitmap index may be logically viewed as follows: On the left, Identifier refers to the unique number assigned to each resident, HasInternet is the data to be indexed, the content of the bitmap index is shown as two columns under the heading bitmaps. Each column in the left illustration under the Bitmaps header is a bitmap in the bitmap index. In this case, there are two such bitmaps, one for "has internet" Yes and one for "has internet" No. It is easy to see that each bit in bitmap Y shows whether a particular row refers to a person who has internet access. This is the simplest form of bitmap index. Most columns will have more distinct values. For example, the sales amount is likely to have a much larger number of distinct values. Variations on the bitmap index can effectively index this data as well. We briefly review three such variations. Note: Many of the references cited here are reviewed at (John Wu (2007)). For those who might be interested in experimenting with some of the ideas mentioned here, many of them are implemented in open source software such as FastBit, the Lemur Bitmap Index C++ Library, the Roaring Bitmap Java library and the Apache Hive Data Warehouse system. == Compression == For historical reasons, bitmap compression and inverted list compression were developed as separate lines of research, and only later were recognized as solving essentially the same problem. Software can compress each bitmap in a bitmap index to save space. There has been considerable amount of work on this subject. Though there are exceptions such as Roaring bitmaps, Bitmap compression algorithms typically employ run-length encoding, such as the Byte-aligned Bitmap Code, the Word-Aligned Hybrid code, the Partitioned Word-Aligned Hybrid (PWAH) compression, the Position List Word Aligned Hybrid, the Compressed Adaptive Index (COMPAX), Enhanced Word-Aligned Hybrid (EWAH) and the COmpressed 'N' Composable Integer SEt (CONCISE). These compression methods require very little effort to compress and decompress. More importantly, bitmaps compressed with BBC, WAH, COMPAX, PLWAH, EWAH and CONCISE can directly participate in bitwise operations without decompression. This gives them considerable advantages over generic compression techniques such as LZ77. BBC compression and its derivatives are used in a commercial database management system. BBC is effective in both reducing index sizes and maintaining query performance. BBC encodes the bitmaps in bytes, while WAH encodes in words, better matching current CPUs. "On both synthetic data and real application data, the new word aligned schemes use only 50% more space, but perform logical operations on compressed data 12 times faster than BBC." PLWAH bitmaps were reported to take 50% of the storage space consumed by WAH bitmaps and offer up to 20% faster performance on logical operations. Similar considerations can be done for CONCISE and Enhanced Word-Aligned Hybrid. The performance of schemes such as BBC, WAH, PLWAH, EWAH, COMPAX and CONCISE is dependent on the order of the rows. A simple lexicographical sort can divide the index size by 9 and make indexes several times faster. The larger the table, the more important it is to sort the rows. Reshuffling techniques have also been proposed to achieve the same results of sorting when indexing streaming data. == Encoding == Basic bitmap indexes use one bitmap for each distinct value. It is possible to reduce the number of bitmaps used by using a different encoding method. For example, it is possible to encode C distinct values using log(C) bitmaps with binary encoding. This reduces the number of bitmaps, further saving space, but to answer any query, most of the bitmaps have to be accessed. This makes it potentially not as effective as scanning a vertical projection of the base data, also known as a materialized view or projection index. Finding the optimal encoding method that balances (arbitrary) query performance, index size and index maintenance remains a challenge. Without considering compression, Chan and Ioannidis analyzed a class of multi-component encoding methods and came to the conclusion that two-component encoding sits at the kink of the performance vs. index size curve and therefore represents the best trade-off between index size and query performance. == Binning == For high-cardinality columns, it is useful to bin the values, where each bin covers multiple values and build the bitmaps to represent the values in each bin. This approach reduces the number of bitmaps used regardless of encoding method. However, binned indexes can only answer some queries without examining the base data. For example, if a bin covers the range from 0.1 to 0.2, then when the user asks for all values less than 0.15, all rows that fall in the bin are possible hits and have to be checked to verify whether they are actually less than 0.15. The process of checking the base data is known as the candidate check. In most cases, the time used by the candidate check is significantly longer than the time needed to work with the bitmap index. Therefore, binned indexes exhibit irregular performance. They can be very fast for some queries, but much slower if the query does not exactly match a bin. == History == The concept of bitmap index was first introduced by Professor Israel Spiegler and Rafi Maayan in their research "Storage and Retrieval Considerations of Binary Data Bases", published in 1985. The first commercial database product to implement a bitmap index was Computer Corporation of America's Model 204. Patrick O'Neil published a paper about this implementation in 1987. This implementation is a hybrid between the basic bitmap index (without compression) and the list of Row Identifiers (RID-list). Overall, the index is organized as a B+tree. When the column cardinality is low, each leaf node of the B-tree would contain long list of RIDs. In this case, it requires less space to represent the RID-lists as bitmaps. Since each bitmap represents one distinct value, this is the basic bitmap index. As the column cardinality increases, each bitmap becomes sparse and it may take more disk space to store the bitmaps than to store the same content as RID-lists. In this case, it switches to use the RID-lists, which makes it a B+tree index. == In-memory bitmaps == One of the strongest reasons for using bitmap indexes is that the intermediate results produced from them are also bitmaps and can be efficiently reused in further operations to answer more complex queries. Many programming languages support this as a bit array data structure. For example, Java has the BitSet class and .NET have the BitArray class. Some database systems that do not offer persistent bitmap indexes use bitmaps internally to speed up query processing. For example, PostgreSQL versions 8.1 and later implement a "bitmap index scan" optimization to speed up arbitrarily complex logical operations between available indexes on a single table. For tables with many columns, the total number of distinct indexes to satisfy all possible queries (with equality filtering conditions on either of the fields) grows very fast, being defined by this formula: C n [ n 2 ] ≡ n ! ( n − [ n 2 ] ) ! [ n 2 ] ! {\displaystyle \mathbf {C} _{n}^{\left[{\frac {n}{2}}\right]}\equiv {\frac {n!}{\left(n-\left[{\frac {n}{2}}\right]\right)!\left[{\frac {n}{2}}\right]!}}} . A bitmap index scan combines expressions on different indexes, thus requiring only one index per column t

    Read more →
  • Data set (IBM mainframe)

    Data set (IBM mainframe)

    In the context of IBM mainframe computers in the IBM System/360 line and its successors, a data set (IBM preferred) or dataset is a computer file having a record organization. Use of this term began with, e.g., DOS/360 and OS/360, and is still used by their successors, including the current VSE and z/OS. Documentation for these systems historically preferred this term rather than file. A data set is typically stored on a direct access storage device (DASD) or magnetic tape, however unit record devices, such as punch card readers, card punches, line printers and page printers can provide input/output (I/O) for a data set (file). Data sets are not unstructured streams of bytes, but rather are organized in various logical record and block structures determined by the DSORG (data set organization), RECFM (record format), and other parameters. These parameters are specified at the time of the data set allocation (creation), for example with Job Control Language DD statements. Within a running program they are stored in the Data Control Block (DCB) or Access Control Block (ACB), which are data structures used to access data sets using access methods. Records in a data set may be fixed, variable, or “undefined” length. == Data set organization == For OS/360, the DCB's DSORG parameter specifies how the data set is organized. It may be CQ Queued Telecommunications Access Method (QTAM) in Message Control Program (MCP) CX Communications line group DA Basic Direct Access Method (BDAM) GS Graphics device for Graphics Access Method(GAM) IS Indexed Sequential Access Method (ISAM) MQ QTAM message queue in application PO Partitioned Organization PS Physical Sequential among others. Data sets on tape may only be DSORG=PS. The choice of organization depends on how the data is to be accessed, and in particular, how it is to be updated. Programmers utilize various access methods (such as QSAM or VSAM) in programs for reading and writing data sets. Access method depends on the given data set organization. == Record format (RECFM) == Regardless of organization, the physical structure of each record is essentially the same, and is uniform throughout the data set. This is specified in the DCB RECFM parameter. RECFM=F means that the records are of fixed length, specified via the LRECL parameter. RECFM=V specifies a variable-length record. V records when stored on media are prefixed by a Record Descriptor Word (RDW) containing the integer length of the record in bytes and flag bits. With RECFM=FB and RECFM=VB, multiple logical records are grouped together into a single physical block on tape or DASD. FB and VB are fixed-blocked, and variable-blocked, respectively. RECFM=U (undefined) is also variable length, but the length of the record is determined by the length of the block rather than by a control field. The BLKSIZE parameter specifies the maximum length of the block. RECFM=FBS could be also specified, meaning fixed-blocked standard, meaning all the blocks except the last one were required to be in full BLKSIZE length. RECFM=VBS, or variable-blocked spanned, means a logical record could be spanned across two or more blocks, with flags in the RDW indicating whether a record segment is continued into the next block and/or was continued from the previous one. This mechanism eliminates the need for using any "delimiter" byte value to separate records. Thus data can be of any type, including binary integers, floating-point, or characters, without introducing a false end-of-record condition. The data set is an abstraction of a collection of records, in contrast to files as unstructured streams of bytes. == Partitioned data set == A partitioned data set (PDS) is a data set containing multiple members, each of which holds a separate sub-data set, similar to a directory in other types of file systems. This type of data set is often used to hold load modules (old format bound executable programs), source program libraries (especially Assembler macro definitions), ISPF screen definitions, and Job Control Language. A PDS may be compared to a Zip file or COM Structured Storage. A Partitioned Data Set can only be allocated on a single volume and have a maximum size of 65,535 tracks. Besides members, a PDS contains also a directory. Each member can be accessed indirectly via the directory structure. Once a member is located, the data stored in that member are handled in the same manner as a PS (sequential) data set. Whenever a member is deleted, the space it occupied is unusable for storing other data. Likewise, if a member is re-written, it is stored in a new spot at the back of the PDS and leaves wasted “dead” space in the middle. The only way to recover “dead” space is to perform file compression. Compression, which is done using the IEBCOPY utility, moves all members to the front of the data space and leaves free usable space at the back. (Note that in modern parlance, this kind of operation might be called defragmentation or garbage collection; data compression nowadays refers to a different, more complicated concept.) PDS files can only reside on DASD, not on magnetic tape, in order to use the directory structure to access individual members. Partitioned data sets are most often used for storing multiple job control language files, utility control statements, and executable modules. An improvement of this scheme is a Partitioned Data Set Extended (PDSE or PDS/E, sometimes just libraries) introduced with DFSMSdfp for MVS/XA and MVS/ESA systems. A PDS/E library can store program objects or other types of members, but not both. BPAM cannot process a PDS/E containing program objects. PDS/E structure is similar to PDS and is used to store the same types of data. However, PDS/E files have a better directory structure which does not require pre-allocation of directory blocks when the PDS/E is defined (and therefore does not run out of directory blocks if not enough were specified). Also, PDS/E automatically stores members in such a way that compression operation is not needed to reclaim "dead" space. PDS/E files can only reside on DASD in order to use the directory structure to access individual members. == Generation Data Group == A Generation Data Group (GDG) is a group of non-VSAM data sets that are successive generations of historically-related data stored on an IBM mainframe (running OS/360 and its successors or DOS/360 and its successors). A GDG is usually cataloged. An individual member of the GDG collection is called a "Generation Data Set." The latter may be identified by an absolute number, ACCTG.OURGDG(1234), or a relative number: (-1) for the previous generation, (0) for the current one, and (+1) the next generation. A GDG specifies how many generations of a data set are to be kept and at what age a generation will be deleted. Whenever a new generation is created, the system checks whether one or more obsolete generations are to be deleted. The purpose of GDGs is to automate archival, using the command language JCL, the data set name given is generic. When DSN appears, the GDG data set appears along with the history number, where (0) is the most recent version (-1), (-2), ... are previous generations (+1) a new generation (see DD) Another use of GDGs is to be able to address all generations simultaneously within a JCL script without having to know the number of currently available generations. To do this, you have to omit the parentheses and the generation number in the JCL when specifying the dataset. === GDG JCL & features === Generation Data Groups are defined using either the BLDG statement of the IEHPROGM utility or the DEFINE GENERATIONGROUP statement of the newer IDCAMS utility, which allows setting various parameters. LIMIT(10) would limit the number of generations limit to 10. SCRATCH FOR (91) would retain each member, up to the limited#generations, at least 91 days. IDCAMS can also delete (and optionally uncatalog) a GDG. ==== Example ==== Creation of a standard GDG for five safety scopes, each at least 35 days old: Delete a standard GDG:

    Read more →