The Pwnie Awards are an annual awards ceremony that recognizes both excellence and incompetence in the field of information security, described by SecurityWeek as an event that "recognizes excellence and mocks incompetence in cybersecurity." Winners are selected by a committee of security industry professionals from nominations collected from the information security community. Nominees are announced yearly at Summercon, and the awards themselves are presented at the Black Hat Security Conference. == Origins == The name Pwnie Award is based on the word "pwn", which is hacker slang meaning to "compromise" or "control" based on the previous usage of the word "own" (and it is pronounced similarly). The name "The Pwnie Awards," pronounced as "Pony," is meant to sound like the Tony Awards, an awards ceremony for Broadway theater in New York City. == History == The Pwnie Awards were founded in 2007 by Alexander Sotirov and Dino Dai Zovi following discussions regarding Dino's discovery of a cross-platform QuickTime vulnerability (CVE-2007-2175) and Alexander's discovery of an ANI file processing vulnerability (CVE-2007-0038) in Internet Explorer. == Winners == === 2024 === Most Epic Fail: Crowdstrike for 2024 CrowdStrike incident Best Mobile Bug: Operation Triangulation Lamest Vendor Response: Xiaomi for obstructing Pwn2Own researchers from using their services Best Cryptographic Attack: GoFetch Best Desktop Bug: forcing realtime WebAudio playback in Chrome (CVE-2023-5996) Best Song: Touch Some Grass by UwU Underground Best Privilege Escalation: Windows Streaming Service UAF (CVE-2024-30089) by Valentina Palmiotti (chompie) Best Remote Code Execution: Microsoft Message Queuing (MSMQ) Remote Code Execution Vulnerability (CVE-2024-30080) Most Epic Achievement: Discovery and reverse engineering of the XZ Utils backdoor Most Innovative Research: Let the Cache Cache and Let the WebAssembly Assemble: Knocking’ on Chrome’s Shell by Edouard Bochin, Tao Yan, and Bo Qu Most Underhyped Research: See No Eval: Runtime Dynamic Code Execution in Objective-C === 2023 === Best Desktop Bug: CountExposure! by RyeLv(@b2ahex) Best Cryptographic Attack: Video-based cryptanalysis: Extracting Cryptographic Keys from Video Footage of a Device’s Power LED by Ben Nassi, Etay Iluz, Or Cohen, Ofek Vayner, Dudi Nassi, Boris Zadov, Yuval Elovici Best Song: Clickin’ Most Innovative Research: Inside Apple’s Lightning: Jtagging the iPhone for Fuzzing and Profit Most Under-Hyped Research: Activation Context Cache Poisoning Best Privilege Escalation Bug: URB Excalibur: Slicing Through the Gordian Knot of VMware VM Escapes Best Remote Code Execution Bug: ClamAV RCE Lamest Vendor Response: Three Lessons From Threema: Analysis of a Secure Messenger Most Epic Fail: “Holy fucking bingle, we have the no fly list,” Epic Achievement: Clement Lecigne: 0-days hunter world champion Lifetime Achievement Award: Mudge === 2022 === Lamest Vendor Response: Google's "TAG" response team for "unilaterally shutting down a counterterrorism operation." Epic Achievement: Yuki Chen’s Windows Server-Side RCE Bugs Most Epic Fail: HackerOne Employee Caught Stealing Vulnerability Reports for Personal Gains Best Desktop Bug: Pietro Borrello, Andreas Kogler, Martin Schwarzl, Moritz Lipp, Daniel Gruss, Michael Schwarz for Architecturally Leaking Data from the Microarchitecture Most Innovative Research: Pietro Borrello, Martin Schwarzl, Moritz Lipp, Daniel Gruss, Michael Schwarz for Custom Processing Unit: Tracing and Patching Intel Atom Microcode Best Cryptographic Attack: Hertzbleed: Turning Power Side-Channel Attacks Into Remote Timing Attacks on x86 by Yingchen Wang, Riccardo Paccagnella, Elizabeth Tang He, Hovav Shacham, Christopher Fletcher, David Kohlbrenner Best Remote Code Execution Bug: KunlunLab for Windows RPC Runtime Remote Code Execution (CVE-2022-26809) Best Privilege Escalation Bug: Qidan He of Dawnslab, for Mystique in the House: The Droid Vulnerability Chain That Owns All Your Userspace Best Mobile Bug: FORCEDENTRY Most Under-Hyped Research: Yannay Livneh for Spoofing IP with IPIP Best Song: Dialed Up by Project Mammoth === 2021 === Lamest Vendor Response: Cellebrite, for their response to Moxie, the creator of Signal, reverse-engineering their UFED and accompanying software and reporting a discovered exploit. Epic Achievement: Ilfak Guilfanov, in honor of IDA's 30th Anniversary. Best Privilege Escalation Bug: Baron Samedit of Qualys, for the discovery of a 10-year-old exploit in sudo. Best Song: The Ransomware Song by Forrest Brazeal Best Server-Side Bug: Orange Tsai, for his Microsoft Exchange Server ProxyLogon attack surface discoveries. Best Cryptographic Attack: The NSA for its disclosure of a bug in the verification of signatures in Windows which breaks the certificate trust chain. Most Innovative Research: Enes Göktaş, Kaveh Razavi, Georgios Portokalidis, Herbert Bos, and Cristiano Giuffrida at VUSec for their research on the "BlindSide" Attack. Most Epic Fail: Microsoft, for their failure to fix PrintNightmare. Best Client-Side Bug: Gunnar Alendal's discovery of a buffer overflow on the Samsung Galaxy S20's secure chip. Most Under-Hyped Research: The Qualys Research Team for 21Nails, 21 vulnerabilities in Exim, the Internet's most popular mail server. === 2020 === Best Server-Side Bug: BraveStarr (CVE-2020-10188) – A Fedora 31 netkit telnetd remote exploit (Ronald Huizer') Best Privilege Escalation Bug: checkm8 – A permanent unpatchable USB bootrom exploit for a billion iOS devices. (axi0mX) Epic Achievement: "Remotely Rooting Modern Android Devices" (Guang Gong) Best Cryptographic Attack: Zerologon vulnerability (Tom Tervoort, CVE-2020-1472) Best Client-Side Bug: RCE on Samsung Phones via MMS (CVE-2020-8899 and -16747), a zero click remote execution attack. (Mateusz Jurczyk) Most Under-Hyped Research: Vulnerabilities in System Management Mode (SMM) and Trusted Execution Technology (TXT) (CVE-2019-0151 and -0152) (Gabriel Negreira Barbosa, Rodrigo Rubira Branco, Joe Cihula) Most Innovative Research: TRRespass: When Memory Vendors Tell You Their Chips Are Rowhammer-free, They Are Not. (Pietro Frigo, Emanuele Vannacci, Hasan Hassan, Victor van der Veen, Onur Mutlu, Cristiano Giuffrida, Herbert Bos, Kaveh Razavi) Most Epic Fail: Microsoft; for the implementation of Elliptic-curve signatures which allowed attackers to generate private pairs for public keys of any signer, allowing HTTPS and signed binary spoofing. (CVE-2020-0601) Best Song: Powertrace by Rebekka Aigner, Daniel Gruss, Manuel Weber, Moritz Lipp, Patrick Radkohl, Andreas Kogler, Maria Eichlseder, ElTonno, tunefish, Yuki and Kater Lamest Vendor Response: Daniel J. Bernstein (CVE-2005-1513) === 2019 === Best Server-Side Bug: Orange Tsai and Meh Chang, for their SSL VPN research. Most Innovative Research: Vectorized Emulation Brandon Falk Best Cryptographic Attack: \m/ Dr4g0nbl00d \m/ Mathy Vanhoef, Eyal Ronen Lamest Vendor Response: Bitfi Most Over-hyped Bug: Allegations of Supermicro hardware backdoors, Bloomberg Most Under-hyped Bug: Thrangrycat, (Jatin Kataria, Red Balloon Security) === 2018 === Most Innovative Research: Spectre/Meltdown (Paul Kocher, Jann Horn, Anders Fogh, Daniel Genkin, Daniel Gruss, Werner Haas, Mike Hamburg, Moritz Lipp, Stefan Mangard, Thomas Prescher, Michael Schwarz, Yuval Yarom) Best Privilege Escalation Bug: Spectre/Meltdown (Paul Kocher, Jann Horn, Anders Fogh, Daniel Genkin, Daniel Gruss, Werner Haas, Mike Hamburg, Moritz Lipp, Stefan Mangard, Thomas Prescher, Michael Schwarz, Yuval Yarom) Lifetime Achievement: Michał Zalewski Best Cryptographic Attack: ROBOT - Return Of Bleichenbacher’s Oracle Threat Hanno Böck, Juraj Somorovsky, Craig Young Lamest Vendor Response: Bitfi hardware crypto-wallet, after the "unhackable" device was hacked to extract the keys required to steal coins and rooted to play Doom. === 2017 === Epic Achievement: Federico Bento for Finally getting TIOCSTI ioctl attack fixed Most Innovative Research: ASLR on the line Ben Gras, Kaveh Razavi, Erik Bosman, Herbert Bos, Cristiano Giuffrida Best Privilege Escalation Bug: DRAMMER Victor van der Veen, Yanick Fratantonio, Martina Lindorfer, Daniel Gruss, Clementine Maurice, Giovanni Vigna, Herbert Bos, Kaveh Razavi, Cristiano Giuffrida Best Cryptographic Attack: The first collision for full SHA-1 Marc Stevens, Elie Bursztein, Pierre Karpman, Ange Albertini, Yarik Markov Lamest Vendor Response: Lennart Poettering - for mishandling security vulnerabilities most spectacularly for multiple critical Systemd bugs Best Song: Hello (From the Other Side) - Manuel Weber, Michael Schwarz, Daniel Gruss, Moritz Lipp, Rebekka Aigner === 2016 === Most Innovative Research: Dedup Est Machina: Memory Deduplication as an Advanced Exploitation Vector Erik Bosman, Kaveh Razavi, Herbert Bos, Cristiano Giuffrida Lifetime Achievement: Peiter Zatko aka Mudge Best Cryptographic Attack: DROWN attack Nimrod Aviram et al. Best Song: Cyberlier - Katie Mous
Sketchpad
Sketchpad (a.k.a. Robot Draftsman) is a computer program written by Ivan Sutherland in 1963 in the course of his PhD thesis, for which he received the Turing Award in 1988, and the Kyoto Prize in 2012. It pioneered human–computer interaction (HCI), and is considered the ancestor of modern computer-aided design (CAD) programs and as a major breakthrough in the development of computer graphics in general. For example, Sketchpad inspired the graphical user interface (GUI) and object-oriented programming. Using the program, Sutherland showed that computer graphics could be used for both artistic and technical purposes and for demonstrating a novel method of human–computer interaction. == History == See History of the graphical user interface for a more detailed discussion of GUI development. == Software == Sketchpad was the earliest program ever to use a complete graphical user interface. The clever way the program organizes its geometric data pioneered the use of master (objects) and occurrences (instances) in computing and pointed forward to object-oriented programming. The main idea was to have master drawings which can be instantiated into many duplicates. When a master drawing is changed, then all instances change also. This was the first known form of an entity component system: for example instead of encapsulating points inside of a line object, the points are stored in a ring buffer as described in pages 48 to 52 of the paper, and the line only points to them. This allowed moving one point to alter all the shapes that use it in a single operation. The structures in Sketchpad were also able to store pointers to functions, to achieve a different behavior depending on the kind of object. In figure 3.8 of the paper, the "instances generic block" stores several "subroutine entries" which are pointers to functions: "display", "howbig" etc. This was an early form of virtual functions. Geometric constraints was another major invention in Sketchpad, letting a user easily constrain geometric properties in the drawing: for instance, the length of a line or the angle between two lines could be fixed. As a trade magazine said, clearly Sutherland "broke new ground in 3D computer modeling and visual simulation, the basis for computer graphics and CAD/CAM". Very few programs can be called precedents for his achievements. Patrick J. Hanratty is sometimes called the "father of CAD/CAM" and wrote PRONTO, a numerical control language at General Electric in 1957, and wrote CAD software while working for General Motors beginning in 1961. Sutherland wrote in his thesis that Bolt, Beranek and Newman had a "similar program" and T-Square was developed by Peter Samson and one or more fellow MIT students in 1962, both for the PDP-1. The Computer History Museum holds program listings for Sketchpad. == Hardware == Sketchpad ran on the MIT Lincoln Laboratory TX-2 (1958) computer at the Massachusetts Institute of Technology (MIT), which had 64k of 36-bit words. The user drew on the computer monitor screen with the recently invented light pen, which relayed information on its position by computing at what time the light from the scanning cathode-ray tube screen is detected. To configure the initial position of the light pen, the word INK was displayed on the screen, which, upon tapping, initialised the program with a white cross to continue keeping track of the pen's movement relative to its prior position. Of the 36 bits available to store each display spot in the display file, 20 gave the coordinates of that spot for the display system and the remaining 16 gave the address of the n-component element responsible for adding that spot to display. The TX-2 was an experimental machine and the hardware changed often (on Wednesdays, according to Sutherland). By 1975, the light pen and the cathode-ray tube with which it had been used had been removed. == Publications == The Sketchpad program was part and parcel of Sutherland's Ph.D. thesis at MIT and peripherally related to the Computer-Aided Design project at that time. Sketchpad: A Man-Machine Graphical Communication System.
Keith Youngin George II
Keith "Youngin" George II is a former mixtape DJ, music executive, manager, producer, and technology app director. He has collaborated with Maino, T-Pain, Nas and Soulja Boy, among others. He was instrumental in the launch of social media app and website, Kandiid in 2021 and served as Fliiks App Director of Regional Development. == Career == Keith Anthony George II was born in Upper Heyford, Oxfordshire, England. His father was in the Air Force which exposed him to different cultures and music. He graduated from Allen High School and attended San Antonio College. George's music career began in 2006 as a mixtape DJ working as DJ Youngin Beatz. He performed at various shows and worked with a variety of artists, managers, and music executives. In 2007, George released the mixtape, Untapped market Vol. 1 (Da Underdogz), which featured tracks from artists including Kanye West, Lil Wayne, 50 Cent, Yung Berg, and Nelly. In 2008, he began working with Def Jam executive Sarah Alminawi who was managing Maino at the time. George played a key role in the marketing and promotional success of Maino's single, Hi Hater, which peaked at #8 on Billboard's US Bubbling Under Hot 100 chart. In 2021, George was an advisor and infrastructure head at Kandiid, a social media app which won a W3 Award in 2022. In 2023, he became involved with Fliiks App as Director of Regional Development which earned a Telly Award, two Muse Awards, and a W3 Award in 2025. In 2025, George was a composer and producer on two singles on Sekou Andrews's album, Koumami; The Chosen One: ACT 1 (featuring Lion Babe) and Love Don't Care (featuring Jordin Sparks and Omari Hardwick). In 2025, he was awarded an Atlanta City Proclamation for Philanthropy and Community Leadership for his partnership with Women's International Grail, a nonprofit organization that assists women, single mothers, and low-income families. He also collaborates with local youth programs, creative networks, and minority-owned startups, providing access to mentorship and industry knowledge. == Awards ==
ViBe
ViBe is a background subtraction algorithm which has been presented at the IEEE ICASSP 2009 conference and was refined in later publications. More precisely, it is a software module for extracting background information from moving images. It has been developed by Oliver Barnich and Marc Van Droogenbroeck of the Montefiore Institute, University of Liège, Belgium. ViBe is patented: the patent covers various aspects such as stochastic replacement, spatial diffusion, and non-chronological handling. ViBe is written in the programming language C, and has been implemented on CPU, GPU and FPGA. == Technical description == Source: === Pixel model and classification process === Many advanced techniques are used to provide an estimate of the temporal probability density function (pdf) of a pixel x. ViBe's approach is different, as it imposes the influence of a value in the polychromatic space to be limited to the local neighborhood. In practice, ViBe does not estimate the pdf, but uses a set of previously observed sample values as a pixel model. To classify a value pt(x), it is compared to its closest values among the set of samples. === Model update: Sample values lifespan policy === ViBe ensures a smooth exponentially decaying lifespan for the sample values that constitute the pixel models. This makes ViBe able to successfully deal with concomitant events with a single model of a reasonable size for each pixel. This is achieved by choosing, randomly, which sample to replace when updating a pixel model. Once the sample to be discarded has been chosen, the new value replaces the discarded sample. The pixel model that would result from the update of a given pixel model with a given pixel sample cannot be predicted since the value to be discarded is chosen at random. === Model update: Spatial Consistency === To ensure the spatial consistency of the whole image model and handle practical situations such as small camera movements or slowly evolving background objects, ViBe uses a technique similar to that developed for the updating process in which it chooses at random and update a pixel model in the neighborhood of the current pixel. By denoting NG(x) and p(x) respectively the spatial neighborhood of a pixel x and its value, and assuming that it was decided to update the set of samples of x by inserting p(x), then ViBe also use this value p(x) to update the set of samples of one of the pixels in the neighborhood NG(x), chosen at random. As a result, ViBe is able to produce spatially coherent results directly without the use of any post-processing method. === Model initialization === Although the model could easily recover from any type of initialization, for example by choosing a set of random values, it is convenient to get an accurate background estimate as soon as possible. Ideally a segmentation algorithm would like to be able to segment the video sequences starting from the second frame, the first frame being used to initialize the model. Since no temporal information is available prior to the second frame, ViBe populates the pixel models with values found in the spatial neighborhood of each pixel; more precisely, it initializes the background model with values taken randomly in each pixel neighborhood of the first frame. The background estimate is therefore valid starting from the second frame of a video sequence.
Structured-light 3D scanner
A structured-light 3D scanner is a device used to capture the three-dimensional shape of an object by projecting light patterns, such as grids or stripes, onto its surface. The deformation of these patterns is recorded by cameras and processed using specialized algorithms to generate a detailed 3D model. Structured-light 3D scanning is widely employed in fields such as industrial design, quality control, cultural heritage preservation, augmented reality gaming, and medical imaging. Compared to laser-based 3D scanning, structured-light scanners use non-coherent light sources, such as LEDs or projectors, which enable faster data acquisition and eliminate potential safety concerns associated with lasers. However, the accuracy of structured-light scanning can be influenced by external factors, including ambient lighting conditions and the reflective properties of the scanned object. == Principle == Projecting a narrow band of light onto a three-dimensional surface creates a line of illumination that appears distorted when viewed from perspectives other than that of the projector. This distortion can be analyzed to reconstruct the geometry of the surface, a technique known as light sectioning. Projecting patterns composed of multiple stripes or arbitrary fringes simultaneously enables the acquisition of numerous data points at once, improving scanning speed. While various structured light projection techniques exist, parallel stripe patterns are among the most commonly used. By analyzing the displacement of these stripes, the three-dimensional coordinates of surface details can be accurately determined. === Generation of light patterns === Two major methods of stripe pattern generation have been established: Laser interference and projection. The laser interference method works with two wide planar laser beam fronts. Their interference results in regular, equidistant line patterns. Different pattern sizes can be obtained by changing the angle between these beams. The method allows for the exact and easy generation of very fine patterns with unlimited depth of field. Disadvantages are high cost of implementation, difficulties providing the ideal beam geometry, and laser typical effects like speckle noise and the possible self interference with beam parts reflected from objects. Typically, there is no means of modulating individual stripes, such as with Gray codes. The projection method uses incoherent light and basically works like a video projector. Patterns are usually generated by passing light through a digital spatial light modulator, typically based on one of the three currently most widespread digital projection technologies, transmissive liquid crystal, reflective liquid crystal on silicon (LCOS) or digital light processing (DLP; moving micro mirror) modulators, which have various comparative advantages and disadvantages for this application. Other methods of projection could be and have been used, however. Patterns generated by digital display projectors have small discontinuities due to the pixel boundaries in the displays. Sufficiently small boundaries however can practically be neglected as they are evened out by the slightest defocus. A typical measuring assembly consists of one projector and at least one camera. For many applications, two cameras on opposite sides of the projector have been established as useful. Invisible (or imperceptible) structured light uses structured light without interfering with other computer vision tasks for which the projected pattern will be confusing. Example methods include the use of infrared light or of extremely high framerates alternating between two exact opposite patterns. === Calibration === Geometric distortions by optics and perspective must be compensated by a calibration of the measuring equipment, using special calibration patterns and surfaces. A mathematical model is used for describing the imaging properties of projector and cameras. Essentially based on the simple geometric properties of a pinhole camera, the model also has to take into account the geometric distortions and optical aberration of projector and camera lenses. The parameters of the camera as well as its orientation in space can be determined by a series of calibration measurements, using photogrammetric bundle adjustment. === Analysis of stripe patterns === There are several depth cues contained in the observed stripe patterns. The displacement of any single stripe can directly be converted into 3D coordinates. For this purpose, the individual stripe has to be identified, which can for example be accomplished by tracing or counting stripes (pattern recognition method). Another common method projects alternating stripe patterns, resulting in binary Gray code sequences identifying the number of each individual stripe hitting the object. An important depth cue also results from the varying stripe widths along the object surface. Stripe width is a function of the steepness of a surface part, i.e. the first derivative of the elevation. Stripe frequency and phase deliver similar cues and can be analyzed by a Fourier transform. Finally, the wavelet transform has recently been discussed for the same purpose. In many practical implementations, series of measurements combining pattern recognition, Gray codes and Fourier transform are obtained for a complete and unambiguous reconstruction of shapes. Another method also belonging to the area of fringe projection has been demonstrated, utilizing the depth of field of the camera. It is also possible to use projected patterns primarily as a means of structure insertion into scenes, for an essentially photogrammetric acquisition. === Precision and range === The optical resolution of fringe projection methods depends on the width of the stripes used and their optical quality. It is also limited by the wavelength of light. An extreme reduction of stripe width proves inefficient due to limitations in depth of field, camera resolution and display resolution. Therefore, the phase shift method has been widely established: A number of at least 3, typically about 10 exposures are taken with slightly shifted stripes. The first theoretical deductions of this method relied on stripes with a sine wave shaped intensity modulation, but the methods work with "rectangular" modulated stripes, as delivered from LCD or DLP displays as well. By phase shifting, surface detail of e.g. 1/10 the stripe pitch can be resolved. Current optical stripe pattern profilometry hence allows for detail resolutions down to the wavelength of light, below 1 micrometer in practice or, with larger stripe patterns, to approx. 1/10 of the stripe width. Concerning level accuracy, interpolating over several pixels of the acquired camera image can yield a reliable height resolution and also accuracy, down to 1/50 pixel. Arbitrarily large objects can be measured with accordingly large stripe patterns and setups. Practical applications are documented involving objects several meters in size. Typical accuracy figures are: Planarity of a 2-foot (0.61 m) wide surface, to 10 micrometres (0.00039 in). Shape of a motor combustion chamber to 2 micrometres (7.9×10−5 in) (elevation), yielding a volume accuracy 10 times better than with volumetric dosing. Shape of an object 2 inches (51 mm) large, to about 1 micrometre (3.9×10−5 in) Radius of a blade edge of e.g. 10 micrometres (0.00039 in), to ±0.4 μm === Navigation === As the method can measure shapes from only one perspective at a time, complete 3D shapes have to be combined from different measurements in different angles. This can be accomplished by attaching marker points to the object and combining perspectives afterwards by matching these markers. The process can be automated, by mounting the object on a motorized turntable on robotic inspection cell, or CNC positioning device. Markers can as well be applied on a positioning device instead of the object itself. The 3D data gathered can be used to retrieve CAD (computer aided design) data and models from existing components (reverse engineering), hand formed samples or sculptures, natural objects or artifacts. === Challenges === As with all optical methods, reflective or transparent surfaces raise difficulties. Reflections cause light to be reflected either away from the camera or right into its optics. In both cases, the dynamic range of the camera can be exceeded. Transparent or semi-transparent surfaces also cause major difficulties. In these cases, coating the surfaces with a thin opaque lacquer just for measuring purposes is a common practice. A recent method handles highly reflective and specular objects by inserting a 1-dimensional diffuser between the light source (e.g., projector) and the object to be scanned. Alternative optical techniques have been proposed for handling perfectly transparent and specular objects. Double reflections and inter-reflections can cause the stripe pattern to be overlaid with unwanted ligh
Recruitee
Tellent Recruitee is a cloud-based applicant tracking system (ATS) for talent acquisition owned by Tellent. It is used by internal HR teams for processes including job postings, candidate sourcing, reporting, and applicant tracking. == History == Perry Oostdam and Pawel Smoczyk founded Recruitee after working on a mobile gaming startup. The Recruitee was launched in August 2015. In September 2015, it received a seed funding round with participation from investors Robert Pijselman and Luc Brandts. Merger In February 2021, Recruitee and the Finnish HR software provider Sympa merged their operations, backed by the growth equity firm Providence Strategic Growth (PSG). Acquisition In 2022, the group acquired the French company Javelo and the German company kiwiHR. The parent company was subsequently renamed as Tellent while Recruitee renamed as Tellent Recruitee and continues to operate as a product unit within the Tellent group. == Platform == Tellent Recruitee is a customizable recruitment software. It functions as an ATS and talent acquisition platform and includes tools to create and publish job listings, source candidates, manage recruitment agencies, and track applicants through customizable pipelines. The interface allows drag-and-drop organization of candidates. The platform also includes features for team collaboration, such as shared notes, task assignments, and candidate evaluations. It also has integrated scheduling tools and automated email communication. Tellent Recruitee also provides analytics and reports on hiring and career site metrics. The software allows for customization of career site pages and application forms. It supports integrations with other HR and productivity software, such as WhatsApp, and has various AI functionalities to support with manual recruitment tasks.
Language model benchmark
A language model benchmark is a standardized test designed to evaluate the performance of language models on various natural language processing tasks. These tests are intended for comparing different models' capabilities in areas such as language understanding, generation, and reasoning. Benchmarks generally consist of a dataset and corresponding evaluation metrics. The dataset provides text samples and annotations, while the metrics measure a model's performance on tasks like answering questions, text classification, and machine translation. These benchmarks are developed and maintained by academic institutions, research organizations, and industry players to track progress in the field. In addition to accuracy, the metrics can include throughput, energy efficiency, bias, trust, and sustainability. == Overview == === Types === Benchmarks may be described by the following adjectives, not mutually exclusive: Classical: These tasks are studied in natural language processing, even before the advent of deep learning. Examples include the Penn Treebank for testing syntactic and semantic parsing, as well as bilingual translation benchmarked by BLEU scores. Question answering: These tasks have a text question and a text answer, often multiple-choice. They can be open-book or closed-book. Open-book QA resembles reading comprehension questions, with relevant passages included as annotation in the question, in which the answer appears. Closed-book QA includes no relevant passages. Closed-book QA is also called open-domain question-answering. Before the era of large language models, open-book QA was more common, and understood as testing information retrieval methods. Closed-book QA became common since GPT-2 as a method to measure knowledge stored within model parameters. Omnibus: An omnibus benchmark combines many benchmarks, often previously published. It is intended as an all-in-one benchmarking solution. Reasoning: These tasks are usually in the question-answering format, but are intended to be more difficult than standard question answering. Multimodal: These tasks require processing not only text, but also other modalities, such as images and sound. Examples include OCR and transcription. Agency: These tasks are for a language-model–based software agent that operates a computer for a user, such as editing images, browsing the web, etc. Adversarial: A benchmark is "adversarial" if the items in the benchmark are picked specifically so that certain models do badly on them. Adversarial benchmarks are often constructed after state of the art (SOTA) models have saturated (achieved 100% performance) a benchmark, to renew the benchmark. A benchmark is "adversarial" only at a certain moment in time, since what is adversarial may cease to be adversarial as newer SOTA models appear. Public/Private: A benchmark might be partly or entirely private, meaning that some or all of the questions are not publicly available. The idea is that if a question is publicly available, then it might be used for training, which would be "training on the test set" and invalidate the result of the benchmark. Usually, only the guardians of the benchmark have access to the private subsets, and to score a model on such a benchmark, one must send the model weights, or provide API access, to the guardians. The boundary between a benchmark and a dataset is not sharp. Generally, a dataset contains three "splits": training, test, and validation. Both the test and validation splits are essentially benchmarks. In general, a benchmark is distinguished from a test/validation dataset in that a benchmark is typically intended to be used to measure the performance of many different models that are not trained specifically for doing well on the benchmark, while a test/validation set is intended to be used to measure the performance of models trained specifically on the corresponding training set. In other words, a benchmark may be thought of as a test/validation set without a corresponding training set. Conversely, certain benchmarks may be used as a training set, such as the English Gigaword or the One Billion Word Benchmark, which in modern language is just the negative log-likelihood loss on a pretraining set with 1 billion words. Indeed, the distinction between benchmark and dataset in language models became sharper after the rise of the pretraining paradigm, whereby a model is first trained on massive, unlabeled datasets to learn general language patterns, syntax, and knowledge (pretraining), and the base model is then adapted to specific, downstream tasks using smaller, labeled datasets (fine-tuning). === Lifecycle === Generally, the life cycle of a benchmark consists of the following steps: Inception: A benchmark is published. It can be simply given as a demonstration of the power of a new model (implicitly) that others then picked up as a benchmark, or as a benchmark that others are encouraged to use (explicitly). Growth: More papers and models use the benchmark, and the performance on the benchmark grows. Maturity, degeneration or deprecation: A benchmark may be saturated, after which researchers move on to other benchmarks. Progress on the benchmark may also be neglected as the field moves to focus on other benchmarks. Renewal: A saturated benchmark can be upgraded to make it no longer saturated, allowing further progress. === Construction === Like datasets, benchmarks are typically constructed by several methods, individually or in combination: Web scraping: Ready-made question-answer pairs may be scraped online, such as from websites that teach mathematics and programming. Conversion: Items may be constructed programmatically from scraped web content, such as by blanking out named entities from sentences, and asking the model to fill in the blank. This was used for making the CNN/Daily Mail Reading Comprehension Task. Crowd sourcing: Items may be constructed by paying people to write them, such as on Amazon Mechanical Turk. This was used for making the MCTest. === Evaluation === Generally, benchmarks are fully automated. This limits the questions that can be asked. For example, with mathematical questions, "proving a claim" would be difficult to automatically check, while "calculate an answer with a unique integer answer" would be automatically checkable. With programming tasks, the answer can generally be checked by running unit tests, with an upper limit on runtime. The benchmark scores are of the following kinds: For multiple choice or cloze questions, common scores are accuracy (frequency of correct answer), precision, recall, F1 score, etc. pass@n: The model is given n {\displaystyle n} attempts to solve each problem. If any attempt is correct, the model earns a point. The pass@n score is the model's average score over all problems. k@n: The model makes n {\displaystyle n} attempts to solve each problem, but only k {\displaystyle k} attempts out of them are selected for submission. If any submission is correct, the model earns a point. The k@n score is the model's average score over all problems. cons@n: The model is given n {\displaystyle n} attempts to solve each problem. If the most common answer is correct, the model earns a point. The cons@n score is the model's average score over all problems. Here "cons" stands for "consensus" or "majority voting". The pass@n score can be estimated more accurately by making N > n {\displaystyle N>n} attempts, and use the unbiased estimator 1 − ( N − c n ) ( N n ) {\displaystyle 1-{\frac {\binom {N-c}{n}}{\binom {N}{n}}}} , where c {\displaystyle c} is the number of correct attempts. For less well-formed tasks, where the output can be any sentence, there are the following commonly used scores including BLEU ROUGE, METEOR, NIST, word error rate, LEPOR, CIDEr, and SPICE. === Issues === error: Some benchmark answers may be wrong. ambiguity: Some benchmark questions may be ambiguously worded. subjective: Some benchmark questions may not have an objective answer at all. This problem generally prevents creative writing benchmarks. Similarly, this prevents benchmarking writing proofs in natural language, though benchmarking proofs in a formal language is possible. open-ended: Some benchmark questions may not have a single answer of a fixed size. This problem generally prevents programming benchmarks from using more natural tasks such as "write a program for X", and instead uses tasks such as "write a function that implements specification X". inter-annotator agreement: Some benchmark questions may be not fully objective, such that even people would not agree with 100% on what the answer should be. This is common in natural language processing tasks, such as syntactic annotation. shortcut: Some benchmark questions may be easily solved by an "unintended" shortcut. For example, in the SNLI benchmark, having a negative word like "not" in the second sentence is a strong signal for the "Contradiction" category, regardless of what the se