Model compression is a machine learning technique for reducing the size of trained models. Large models can achieve high accuracy, but often at the cost of significant resource requirements. Compression techniques aim to compress models without significant performance reduction. Smaller models require less storage space, and consume less memory and compute during inference. Compressed models enable deployment on resource-constrained devices such as smartphones, embedded systems, edge computing devices, and consumer electronics computers. Efficient inference is also valuable for large corporations that serve large model inference over an API, allowing them to reduce computational costs and improve response times for users. Model compression is not to be confused with knowledge distillation, in which a smaller "student" model is trained to imitate the input-output behavior of a larger "teacher" model (as opposed to using the "teacher"'s trained parameters or the "teacher"'s training targets). == Techniques == Several techniques are employed for model compression. === Pruning === Pruning sparsifies a large model by setting some parameters to exactly zero. This effectively reduces the number of parameters. This allows the use of sparse matrix operations, which are faster than dense matrix operations. Pruning criteria can be based on magnitudes of parameters, the statistical pattern of neural activations, Hessian values, etc. === Quantization === Quantization reduces the numerical precision of weights and activations. For example, instead of storing weights as 32-bit floating-point numbers, they can be represented using 8-bit integers. Low-precision parameters take up less space, and takes less compute to perform arithmetic with. It is also possible to quantize some parameters more aggressively than others, so for example, a less important parameter can have 8-bit precision while another, more important parameter, can have 16-bit precision. Inference with such models requires mixed-precision arithmetic. Quantized models can also be used during training (rather than after training). PyTorch implements automatic mixed-precision (AMP), which performs autocasting, gradient scaling, and loss scaling. === Low-rank factorization === Weight matrices can be approximated by low-rank matrices. Let W {\displaystyle W} be a weight matrix of shape m × n {\displaystyle m\times n} . A low-rank approximation is W ≈ U V T {\displaystyle W\approx UV^{T}} , where U {\displaystyle U} and V {\displaystyle V} are matrices of shapes m × k , n × k {\displaystyle m\times k,n\times k} . When k {\displaystyle k} is small, this both reduces the number of parameters needed to represent W {\displaystyle W} approximately, and accelerates matrix multiplication by W {\displaystyle W} . Low-rank approximations can be found by singular value decomposition (SVD). The choice of rank for each weight matrix is a hyperparameter, and jointly optimized as a mixed discrete-continuous optimization problem. The rank of weight matrices may also be pruned after training, taking into account the effect of activation functions like ReLU on the implicit rank of the weight matrices. == Training == Model compression may be decoupled from training, that is, a model is first trained without regard for how it might be compressed, then it is compressed. However, it may also be combined with training. The "train big, then compress" method trains a large model for a small number of training steps (less than it would be if it were trained to convergence), then heavily compress the model. It is found that at the same compute budget, this method results in a better model than lightly compressed, small models. In Deep Compression, the compression has three steps. First loop (pruning): prune all weights lower than a threshold, then finetune the network, then prune again, etc. Second loop (quantization): cluster weights, then enforce weight sharing among all weights in each cluster, then finetune the network, then cluster again, etc. Third step: Use Huffman coding to losslessly compress the model. The SqueezeNet paper reported that Deep Compression achieved a compression ratio of 35 on AlexNet, and a ratio of ~10 on SqueezeNets.
Scale space implementation
In the areas of computer vision, image analysis and signal processing, the notion of scale-space representation is used for processing measurement data at multiple scales, and specifically enhance or suppress image features over different ranges of scale (see the article on scale space). A special type of scale-space representation is provided by the Gaussian scale space, where the image data in N dimensions is subjected to smoothing by Gaussian convolution. Most of the theory for Gaussian scale space deals with continuous images, whereas one when implementing this theory will have to face the fact that most measurement data are discrete. Hence, the theoretical problem arises concerning how to discretize the continuous theory while either preserving or well approximating the desirable theoretical properties that lead to the choice of the Gaussian kernel (see the article on scale-space axioms). This article describes basic approaches for this that have been developed in the literature, see also for an in-depth treatment regarding the topic of approximating the Gaussian smoothing operation and the Gaussian derivative computations in scale-space theory, and for a complementary treatment regarding hybrid discretization methods. == Statement of the problem == The Gaussian scale-space representation of an N-dimensional continuous signal, f C ( x 1 , ⋯ , x N , t ) , {\displaystyle f_{C}\left(x_{1},\cdots ,x_{N},t\right),} is obtained by convolving fC with an N-dimensional Gaussian kernel: g N ( x 1 , ⋯ , x N , t ) . {\displaystyle g_{N}\left(x_{1},\cdots ,x_{N},t\right).} In other words: L ( x 1 , ⋯ , x N , t ) = ∫ u 1 = − ∞ ∞ ⋯ ∫ u N = − ∞ ∞ f C ( x 1 − u 1 , ⋯ , x N − u N , t ) ⋅ g N ( u 1 , ⋯ , u N , t ) d u 1 ⋯ d u N . {\displaystyle L\left(x_{1},\cdots ,x_{N},t\right)=\int _{u_{1}=-\infty }^{\infty }\cdots \int _{u_{N}=-\infty }^{\infty }f_{C}\left(x_{1}-u_{1},\cdots ,x_{N}-u_{N},t\right)\cdot g_{N}\left(u_{1},\cdots ,u_{N},t\right)\,du_{1}\cdots du_{N}.} However, for implementation, this definition is impractical, since it is continuous. When applying the scale space concept to a discrete signal fD, different approaches can be taken. This article is a brief summary of some of the most frequently used methods. == Separability == Using the separability property of the Gaussian kernel g N ( x 1 , … , x N , t ) = G ( x 1 , t ) ⋯ G ( x N , t ) {\displaystyle g_{N}\left(x_{1},\dots ,x_{N},t\right)=G\left(x_{1},t\right)\cdots G\left(x_{N},t\right)} the N-dimensional convolution operation can be decomposed into a set of separable smoothing steps with a one-dimensional Gaussian kernel G along each dimension L ( x 1 , ⋯ , x N , t ) = ∫ u 1 = − ∞ ∞ ⋯ ∫ u N = − ∞ ∞ f C ( x 1 − u 1 , ⋯ , x N − u N , t ) G ( u 1 , t ) d u 1 ⋯ G ( u N , t ) d u N , {\displaystyle L(x_{1},\cdots ,x_{N},t)=\int _{u_{1}=-\infty }^{\infty }\cdots \int _{u_{N}=-\infty }^{\infty }f_{C}(x_{1}-u_{1},\cdots ,x_{N}-u_{N},t)G(u_{1},t)\,du_{1}\cdots G(u_{N},t)\,du_{N},} where G ( x , t ) = 1 2 π t e − x 2 2 t {\displaystyle G(x,t)={\frac {1}{\sqrt {2\pi t}}}e^{-{\frac {x^{2}}{2t}}}} and the standard deviation of the Gaussian σ is related to the scale parameter t according to t = σ2. Separability will be assumed in all that follows, even when the kernel is not exactly Gaussian, since separation of the dimensions is the most practical way to implement multidimensional smoothing, especially at larger scales. Therefore, the rest of the article focuses on the one-dimensional case. == The sampled Gaussian kernel == When implementing the one-dimensional smoothing step in practice, the presumably simplest approach is to convolve the discrete signal fD with a sampled Gaussian kernel: L ( x , t ) = ∑ n = − ∞ ∞ f ( x − n ) G ( n , t ) {\displaystyle L(x,t)=\sum _{n=-\infty }^{\infty }f(x-n)\,G(n,t)} where G ( n , t ) = 1 2 π t e − n 2 2 t {\displaystyle G(n,t)={\frac {1}{\sqrt {2\pi t}}}e^{-{\frac {n^{2}}{2t}}}} (with t = σ2) which in turn is truncated at the ends to give a filter with finite impulse response L ( x , t ) = ∑ n = − M M f ( x − n ) G ( n , t ) {\displaystyle L(x,t)=\sum _{n=-M}^{M}f(x-n)\,G(n,t)} for M chosen sufficiently large (see error function) such that 2 ∫ M ∞ G ( u , t ) d u = 2 ∫ M t ∞ G ( v , 1 ) d v < ε . {\displaystyle 2\int _{M}^{\infty }G(u,t)\,du=2\int _{\frac {M}{\sqrt {t}}}^{\infty }G(v,1)\,dv<\varepsilon .} A common choice is to set M to a constant C times the standard deviation of the Gaussian kernel M = C σ + 1 = C t + 1 {\displaystyle M=C\sigma +1=C{\sqrt {t}}+1} where C is often chosen somewhere between 3 and 6. Using the sampled Gaussian kernel can, however, lead to implementation problems, in particular when computing higher-order derivatives at finer scales by applying sampled derivatives of Gaussian kernels. When accuracy and robustness are primary design criteria, alternative implementation approaches should therefore be considered. For small values of ε (10−6 to 10−8) the errors introduced by truncating the Gaussian are usually negligible. For larger values of ε, however, there are many better alternatives to a rectangular window function. For example, for a given number of points, a Hamming window, Blackman window, or Kaiser window will do less damage to the spectral and other properties of the Gaussian than a simple truncation will. Notwithstanding this, since the Gaussian kernel decreases rapidly at the tails, the main recommendation is still to use a sufficiently small value of ε such that the truncation effects are no longer important. == The discrete Gaussian kernel == A more refined approach is to convolve the original signal with the discrete Gaussian kernel T(n, t) L ( x , t ) = ∑ n = − ∞ ∞ f ( x − n ) T ( n , t ) {\displaystyle L(x,t)=\sum _{n=-\infty }^{\infty }f(x-n)\,T(n,t)} where T ( n , t ) = e − t I n ( t ) {\displaystyle T(n,t)=e^{-t}I_{n}(t)} and I n ( t ) {\displaystyle I_{n}(t)} denotes the modified Bessel functions of integer order, n. This is the discrete counterpart of the continuous Gaussian in that it is the solution to the discrete diffusion equation (discrete space, continuous time), just as the continuous Gaussian is the solution to the continuous diffusion equation. This filter can be truncated in the spatial domain as for the sampled Gaussian L ( x , t ) = ∑ n = − M M f ( x − n ) T ( n , t ) {\displaystyle L(x,t)=\sum _{n=-M}^{M}f(x-n)\,T(n,t)} or can be implemented in the Fourier domain using a closed-form expression for its discrete-time Fourier transform: T ^ ( θ , t ) = ∑ n = − ∞ ∞ T ( n , t ) e − i θ n = e t ( cos θ − 1 ) . {\displaystyle {\widehat {T}}(\theta ,t)=\sum _{n=-\infty }^{\infty }T(n,t)\,e^{-i\theta n}=e^{t(\cos \theta -1)}.} With this frequency-domain approach, the scale-space properties transfer exactly to the discrete domain, or with excellent approximation using periodic extension and a suitably long discrete Fourier transform to approximate the discrete-time Fourier transform of the signal being smoothed. Moreover, higher-order derivative approximations can be computed in a straightforward manner (and preserving scale-space properties) by applying small support central difference operators to the discrete scale space representation. As with the sampled Gaussian, a plain truncation of the infinite impulse response will in most cases be a sufficient approximation for small values of ε, while for larger values of ε it is better to use either a decomposition of the discrete Gaussian into a cascade of generalized binomial filters or alternatively to construct a finite approximate kernel by multiplying by a window function. If ε has been chosen too large such that effects of the truncation error begin to appear (for example as spurious extrema or spurious responses to higher-order derivative operators), then the options are to decrease the value of ε such that a larger finite kernel is used, with cutoff where the support is very small, or to use a tapered window. == Recursive filters == Since computational efficiency is often important, low-order recursive filters are often used for scale-space smoothing. For example, Young and van Vliet use a third-order recursive filter with one real pole and a pair of complex poles, applied forward and backward to make a sixth-order symmetric approximation to the Gaussian with low computational complexity for any smoothing scale. By relaxing a few of the axioms, Lindeberg concluded that good smoothing filters would be "normalized Pólya frequency sequences", a family of discrete kernels that includes all filters with real poles at 0 < Z < 1 and/or Z > 1, as well as with real zeros at Z < 0. For symmetry, which leads to approximate directional homogeneity, these filters must be further restricted to pairs of poles and zeros that lead to zero-phase filters. To match the transfer function curvature at zero frequency of the discrete Gaussian, which ensures an approximate semi-group property of additive t, two poles at Z = 1 + 2 t − ( 1 + 2 t ) 2 − 1 {\displaystyle
Cyber attribution
In the area of computer security, cyber attribution is an attribution of cybercrime, i.e., finding who perpetrated a cyberattack. Uncovering a perpetrator may give insights into various security issues, such as infiltration methods, communication channels, etc., and may help in enacting specific countermeasures. Cyber attribution is a costly endeavor requiring considerable resources and expertise in cyber forensic analysis. For governments and other major players dealing with cybercrime would require not only technical solutions, but legal and political ones as well, and for the latter ones cyber attribution is crucial. Attributing a cyberattack is difficult, and of limited interest to companies that are targeted by cyberattacks. In contrast, secret services often have a compelling interest in finding out whether a state is behind the attack. A further challenge in attribution of cyberattacks is the possibility of a false flag attack, where the actual perpetrator makes it appear that someone else caused the attack. Every stage of the attack may leave artifacts, such as entries in log files, that can be used to help determine the attacker's goals and identity. In the aftermath of an attack, investigators often begin by saving as many artifacts as they can find, and then try to determine the attacker.
Zardoz (computer security)
In computer security, the Security-Digest list, better known as the Zardoz list, was a semi-private full disclosure mailing list run by Neil Gorsuch from 1989 through 1991. It identified weaknesses in systems and gave directions on where to find them. It was a perennial target for computer hackers, who sought archives of the list for information on undisclosed software vulnerabilities. == Membership restrictions == Access to Zardoz was approved on a case-by-case basis by Gorsuch, principally by reference to the user account used to send subscription requests; requests were approved for root users, valid UUCP owners, or system administrators listed at the NIC. The openness of the list to users other than Unix system administrators was a regular topic of conversation, with participants expressing concern that vulnerabilities and exploitation details disclosed on the list were liable to spread to hackers. The circulation of Zardoz postings was an open secret among computer hackers, and mocked in a Phrack parody of an IRC channel populated by security experts. == Notable participants == Keith Bostic discussed BSD Sendmail vulnerabilities Chip Salzenberg discussed Peter Honeyman's posting of a UUCP worm, and shell script security Gene Spafford discussed VMS and Ultrix bugs, and relayed law enforcement enquiries about the Morris Worm Tom Christiansen discussed SUID shell scripts Chris Torek discussed devising exploits from general descriptions of vulnerabilities Henry Spencer discussed Unix security Brendan Kehoe discussed systems security Alec Muffett announced Crack, the Unix password cracker The majority of Zardoz participants were Unix systems administrators and C software developers. Neil Gorsuch and Gene Spafford were the most prolific contributors to the list.
Scenery generator
A scenery generator (or terrain generator) is a software used to create landscape images, 3D models, and animations. These programs often use procedural generation to generate the landscapes, or sometimes created and rendered by a 3D artist. These programs are often used in video games or movies. Basic elements of landscapes created by scenery generators include terrain, water, foliage, and clouds. The process for basic random generation uses a diamond square algorithm. == Common features == Most scenery generators can create basic heightmaps to simulate the variation of elevation in basic terrain. Common techniques include Simplex noise, fractals, or the diamond-square algorithm, which can generate 2-dimensional heightmaps. A version of scenery generator can be very simplistic. Using a diamond-square algorithm with some extra steps involving fractals, an algorithm for random generation of terrain can be made with only 120 lines of code. The program in example takes a grid and then divides the grid repeatedly. Each smaller grid is then split into squares and diamonds and the algorithm then makes the randomized terrain for each square and diamond. Most programs for creating landscapes also allow for adjustment and editing of the landscape. For example, World Creator allows for terrain sculpting, which uses a similar brush system as Photoshop, and allows for additional terrain enhancement with its procedural techniques such as erosion, sediments, and more. Other tools in the World Creator program include terrain stamping, which allows you to import elevation maps and use them as a base. The programs tend to also allow for additional placement of rocks, trees, etc. These can be done procedurally or by hand depending on the program. Typically the models used for the placement objects are the same as to lessen the amount of work that would be done if the user was to create a multitude of different trees. The terrain generated the computer does a generation of multifractals then integrates them until finally rendering them onto the screen. These techniques are typically done “on-the-fly” which typically for a 128 × 128 resolution terrain would mean 1.5 seconds on a CPU from the early 1990s. == Applications == Scenery generators are commonly used in movies, animations, 3D rendering, and video games. For example, Industrial Light & Magic used E-on Vue to create the fictional environments for Pirates of the Caribbean: Dead Man's Chest. In such live-action cases, a 3D model of the generated environment is rendered and blended with live-action footage. Scenery generated by the software may also be used to create completely computer-generated scenes. In the case of animated movies such as Kung Fu Panda, the raw generation is assisted by hand-painting to accentuate subtle details. Environmental elements not commonly associated with landscapes, such as ocean waves, have also been handled by the software. Scenery generation is used in most 3D based video-games. These typically use either custom or purchased engines that contain their own scenery generators. For some games they tend to use a procedurally generated terrain. These typically use a form of height mapping and use of Perlin noise. This will create a grid that with one point in a 2D coordinate will create the same heightmap as it is pseudorandom, meaning it will result in the same output with the same input. This can then easily be translated into the product 3D image. These can then be changed from the editor tools in most engines if the terrain will be custom built. With recent developments neural networks can be built to create or texture the terrain based on previously suggested artwork or heightmap data. These would be generated using algorithms that have been able to identify images and similarities between them. With the info the machine can take other heightmaps and render a very similar looking image to the style image. This can be used to create similar images in example a Studio Ghibli or Van Gogh art-style. == Software == Most game engines, whether custom or proprietary, will have terrain generation built in. Some terrain generator programs include, Terragen, which can create terrain, water, atmosphere and lighting; L3DT, which provides similar functions to Terragen, and has a 2048 × 2048 resolution limit; and World Creator, which can create terrain, and is fully GPU powered. === List of 3D terrain generation software ===
Soterml
SoTerML (Soil and Terrain Markup Language) is a XML-based markup language for storing and exchanging soil and terrain related data. SoTerML development is being done within The e-SoTer Platform. GEOSS plans a global Earth Observation System and, within this framework, the e-SOTER project addresses the felt need for a global soil and terrain database. The Centre for Geospatial Science (Currently Nottingham Geospatial Institute) at the University of Nottingham has initiated the development since January 2009. Further development and maintenance is currently handled in National Soil Resources Institute (NSRI) at Cranfield University, UK. The role of CGS is within the development of the e-SOTER dissemination platform, which is based on INSPIRE principles. The SoTerML development included: 1. Development of a data dictionary for nomenclatures and various data sources (data and metadata). 2. Development of an exchange format/procedures from the World Reference Base 2006.
Vulnerabilities Equities Process
The Vulnerabilities Equities Process (VEP) is a process used by the U.S. federal government to determine on a case-by-case basis how it should treat zero-day computer security vulnerabilities: whether to disclose them to the public to help improve general computer security, or to keep them secret for offensive use against the government's adversaries. The VEP was first developed during the period 2008–2009, but only became public in 2016, when the government released a redacted version of the VEP in response to a FOIA request by the Electronic Frontier Foundation. Following public pressure for greater transparency in the wake of the Shadow Brokers affair, the U.S. government made a more public disclosure of the VEP process in November 2017. == Participants == According to the VEP plan published in 2017, the Equities Review Board (ERB) is the primary forum for interagency deliberation and determinations concerning the VEP. The ERB meets monthly, but may also be convened sooner if an immediate need arises. The ERB consists of representatives from the following agencies: Office of Management and Budget Office of the Director of National Intelligence (including the Intelligence Community-Security Coordination Center) United States Department of the Treasury United States Department of State United States Department of Justice (including the Federal Bureau of Investigation and the National Cyber Investigative Joint Task Force) Department of Homeland Security (including the National Cybersecurity and Communications Integration Center and the United States Secret Service) United States Department of Energy United States Department of Defense (to include the National Security Agency, including Information Assurance and Signals Intelligence elements), United States Cyber Command, and DoD Cyber Crime Center) United States Department of Commerce Central Intelligence Agency The National Security Agency serves as the executive secretariat for the VEP. == Process == According to the November 2017 version of the VEP, the process is as follows: === Submission and notification === When an agency finds a vulnerability, it will notify the VEP secretariat as soon as is possible. The notification will include a description of the vulnerability and the vulnerable products or systems, together with the agency's recommendation to either disseminate or restrict the vulnerability information. The secretariat will then notify all participants of the submission within one business day, requesting them to respond if they have an relevant interest. === Equity and discussions === An agency expressing an interest must indicate whether it concurs with the original recommendation to disseminate or restrict within five business days. If it does not, it will hold discussions with the submitting agency and the VEP secretariat within seven business days to attempt to reach consensus. If no consensus is reached, the participants will suggest options for the Equities Review Board. === Determination to disseminate or restrict === Decisions whether to disclose or restrict a vulnerability should be made quickly, in full consultation with all concerned agencies, and in the overall best interest of the competing interests of the missions of the U.S. government. As far as possible, determinations should be based on rational, objective methodologies, taking into account factors such as prevalence, reliance, and severity. If the review board members cannot reach consensus, they will vote on a preliminary determination. If an agency with an equity disputes that decision, they may, by providing notice to the VEP secretariat, elect to contest the preliminary determination. If no agency contests a preliminary determination, it will be treated as a final decision. === Handling and follow-on actions === If vulnerability information is released, this will be done as quickly as possible, preferably within seven business days. Disclosure of vulnerabilities will be conducted according to guidelines agreed on by all members. The submitting agency is presumed to be most knowledgeable about the vulnerability and, as such, will be responsible for disseminating vulnerability information to the vendor. The submitting agency may elect to delegate dissemination responsibility to another agency on its behalf. The releasing agency will promptly provide a copy of the disclosed information to the VEP secretariat for record keeping. Additionally, the releasing agency is expected to follow up so the ERB can determine whether the vendor's action meets government requirements. If the vendor chooses not to address a vulnerability, or is not acting with urgency consistent with the risk of the vulnerability, the releasing agency will notify the secretariat, and the government may take other mitigation steps. == Criticism == The VEP process has been criticized for a number of deficiencies, including restriction by non-disclosure agreements, lack of risk ratings, special treatment for the NSA, and less than whole-hearted commitment to disclosure as the default option. == UK equivalent == British intelligence agencies—GCHQ in particular—follow a similar approach, also known as the Equities Process, to determine whether to disclose or retain security vulnerabilities. The Investigatory Powers Act 2016 was amended in 2022 to bring oversight of the operation of the process within the remit of the Investigatory Powers Commissioner. Details of the process were made public in 2018.