Proximal policy optimization (PPO) is a reinforcement learning (RL) algorithm for training an intelligent agent. Specifically, it is a policy gradient method, often used for deep RL when the policy network is very large. == History == The predecessor to PPO, Trust Region Policy Optimization (TRPO), was published in 2015. It addressed the instability issue of another algorithm, the Deep Q-Network (DQN), by using the trust region method to limit the KL divergence between the old and new policies. However, TRPO uses the Hessian matrix (a matrix of second derivatives) to enforce the trust region, but the Hessian is inefficient for large-scale problems. PPO was published in 2017. It was essentially an approximation of TRPO that does not require computing the Hessian. The KL divergence constraint was approximated by simply clipping the policy gradient. Since 2018, PPO was the default RL algorithm at OpenAI. PPO has been applied to many areas, such as controlling a robotic arm, beating professional players at Dota 2 (OpenAI Five), and playing Atari games. == TRPO == TRPO, the predecessor of PPO, is an on-policy algorithm. It can be used for environments with either discrete or continuous action spaces. The pseudocode is as follows: Input: initial policy parameters θ 0 {\textstyle \theta _{0}} , initial value function parameters ϕ 0 {\textstyle \phi _{0}} Hyperparameters: KL-divergence limit δ {\textstyle \delta } , backtracking coefficient α {\textstyle \alpha } , maximum number of backtracking steps K {\textstyle K} for k = 0 , 1 , 2 , … {\textstyle k=0,1,2,\ldots } do Collect set of trajectories D k = { τ i } {\textstyle {\mathcal {D}}_{k}=\left\{\tau _{i}\right\}} by running policy π k = π ( θ k ) {\textstyle \pi _{k}=\pi \left(\theta _{k}\right)} in the environment. Compute rewards-to-go R ^ t {\textstyle {\hat {R}}_{t}} . Compute advantage estimates, A ^ t {\textstyle {\hat {A}}_{t}} (using any method of advantage estimation) based on the current value function V ϕ k {\textstyle V_{\phi _{k}}} . Estimate policy gradient as g ^ k = 1 | D k | ∑ τ ∈ D k ∑ t = 0 T ∇ θ log π θ ( a t ∣ s t ) | θ k A ^ t {\displaystyle {\hat {g}}_{k}=\left.{\frac {1}{\left|{\mathcal {D}}_{k}\right|}}\sum _{\tau \in {\mathcal {D}}_{k}}\sum _{t=0}^{T}\nabla _{\theta }\log \pi _{\theta }\left(a_{t}\mid s_{t}\right)\right|_{\theta _{k}}{\hat {A}}_{t}} Use the conjugate gradient algorithm to compute x ^ k ≈ H ^ k − 1 g ^ k {\displaystyle {\hat {x}}_{k}\approx {\hat {H}}_{k}^{-1}{\hat {g}}_{k}} where H ^ k {\textstyle {\hat {H}}_{k}} is the Hessian of the sample average KL-divergence. Update the policy by backtracking line search with θ k + 1 = θ k + α j 2 δ x ^ k T H ^ k x ^ k x ^ k {\displaystyle \theta _{k+1}=\theta _{k}+\alpha ^{j}{\sqrt {\frac {2\delta }{{\hat {x}}_{k}^{T}{\hat {H}}_{k}{\hat {x}}_{k}}}}{\hat {x}}_{k}} where j ∈ { 0 , 1 , 2 , … K } {\textstyle j\in \{0,1,2,\ldots K\}} is the smallest value which improves the sample loss and satisfies the sample KL-divergence constraint. Fit value function by regression on mean-squared error: ϕ k + 1 = arg min ϕ 1 | D k | T ∑ τ ∈ D k ∑ t = 0 T ( V ϕ ( s t ) − R ^ t ) 2 {\displaystyle \phi _{k+1}=\arg \min _{\phi }{\frac {1}{\left|{\mathcal {D}}_{k}\right|T}}\sum _{\tau \in {\mathcal {D}}_{k}}\sum _{t=0}^{T}\left(V_{\phi }\left(s_{t}\right)-{\hat {R}}_{t}\right)^{2}} typically via some gradient descent algorithm. == PPO == The pseudocode is as follows: Input: initial policy parameters θ 0 {\textstyle \theta _{0}} , initial value function parameters ϕ 0 {\textstyle \phi _{0}} for k = 0 , 1 , 2 , … {\textstyle k=0,1,2,\ldots } do Collect set of trajectories D k = { τ i } {\textstyle {\mathcal {D}}_{k}=\left\{\tau _{i}\right\}} by running policy π k = π ( θ k ) {\textstyle \pi _{k}=\pi \left(\theta _{k}\right)} in the environment. Compute rewards-to-go R ^ t {\textstyle {\hat {R}}_{t}} . Compute advantage estimates, A ^ t {\textstyle {\hat {A}}_{t}} (using any method of advantage estimation) based on the current value function V ϕ k {\textstyle V_{\phi _{k}}} . Update the policy by maximizing the PPO-Clip objective: θ k + 1 = arg max θ 1 | D k | T ∑ τ ∈ D k ∑ t = 0 T min ( π θ ( a t ∣ s t ) π θ k ( a t ∣ s t ) A π θ k ( s t , a t ) , g ( ϵ , A π θ k ( s t , a t ) ) ) {\displaystyle \theta _{k+1}=\arg \max _{\theta }{\frac {1}{\left|{\mathcal {D}}_{k}\right|T}}\sum _{\tau \in {\mathcal {D}}_{k}}\sum _{t=0}^{T}\min \left({\frac {\pi _{\theta }\left(a_{t}\mid s_{t}\right)}{\pi _{\theta _{k}}\left(a_{t}\mid s_{t}\right)}}A^{\pi _{\theta _{k}}}\left(s_{t},a_{t}\right),\quad g\left(\epsilon ,A^{\pi _{\theta _{k}}}\left(s_{t},a_{t}\right)\right)\right)} typically via stochastic gradient ascent with Adam. Fit value function by regression on mean-squared error: ϕ k + 1 = arg min ϕ 1 | D k | T ∑ τ ∈ D k ∑ t = 0 T ( V ϕ ( s t ) − R ^ t ) 2 {\displaystyle \phi _{k+1}=\arg \min _{\phi }{\frac {1}{\left|{\mathcal {D}}_{k}\right|T}}\sum _{\tau \in {\mathcal {D}}_{k}}\sum _{t=0}^{T}\left(V_{\phi }\left(s_{t}\right)-{\hat {R}}_{t}\right)^{2}} typically via some gradient descent algorithm. Like all policy gradient methods, PPO is used for training an RL agent whose actions are determined by a differentiable policy function by gradient ascent. Intuitively, a policy gradient method takes small policy update steps, so the agent can reach higher and higher rewards in expectation. Policy gradient methods may be unstable: A step size that is too big may direct the policy in a suboptimal direction, thus having little possibility of recovery; a step size that is too small lowers the overall efficiency. To solve the instability, PPO implements a clip function that constrains the policy update of an agent from being too large, so that larger step sizes may be used without negatively affecting the gradient ascent process. === Basic concepts === To begin the PPO training process, the agent is set in an environment to perform actions based on its current input. In the early phase of training, the agent can freely explore solutions and keep track of the result. Later, with a certain amount of transition samples and policy updates, the agent will select an action to take by randomly sampling from the probability distribution P ( A | S ) {\displaystyle P(A|S)} generated by the policy network. The actions that are most likely to be beneficial will have the highest probability of being selected from the random sample. After an agent arrives at a different scenario (a new state) by acting, it is rewarded with a positive reward or a negative reward. The objective of an agent is to maximize the cumulative reward signal across sequences of states, known as episodes. === Policy gradient laws: the advantage function === The advantage function (denoted as A {\displaystyle A} ) is central to PPO, as it tries to answer the question of whether a specific action of the agent is better or worse than some other possible action in a given state. By definition, the advantage function is an estimate of the relative value for a selected action. If the output of this function is positive, it means that the action in question is better than the average return, so the possibilities of selecting that specific action will increase. The opposite is true for a negative advantage output. The advantage function can be defined as A = Q − V {\displaystyle A=Q-V} , where Q {\displaystyle Q} is the discounted sum of rewards (the total weighted reward for the completion of an episode) and V {\displaystyle V} is the baseline estimate. Since the advantage function is calculated after the completion of an episode, the program records the outcome of the episode. Therefore, calculating advantage is essentially an unsupervised learning problem. The baseline estimate comes from the value function that outputs the expected discounted sum of an episode starting from the current state. In the PPO algorithm, the baseline estimate will be noisy (with some variance), as it also uses a neural network, like the policy function itself. With Q {\displaystyle Q} and V {\displaystyle V} computed, the advantage function is calculated by subtracting the baseline estimate from the actual discounted return. If A > 0 {\displaystyle A>0} , the actual return of the action is better than the expected return from experience; if A < 0 {\displaystyle A<0} , the actual return is worse. === Ratio function === In PPO, the ratio function ( r t {\displaystyle r_{t}} ) calculates the probability of selecting action a {\displaystyle a} in state s {\displaystyle s} given the current policy network, divided by the previous probability under the old policy. In other words: If r t ( θ ) > 1 {\displaystyle r_{t}(\theta )>1} , where θ {\displaystyle \theta } are the policy network parameters, then selecting action a {\displaystyle a} in state s {\displaystyle s} is more likely based on the current policy than the previous policy. If 0 ≤ r t ( θ ) < 1 {\displaystyle 0\leq r_{t}(\theta )<1} , then selecting actio
Spotify Kids
Spotify Kids is a Swedish kid-friendly Music streaming service developed by Spotify. It offers curated content for children, including music, audiobooks, lullabies, and bedtime stories, while providing their parents with parental controls. The service is only available to subscribers to Spotify's Premium Family subscription plan. == Function == Spotify Kids is a Swedish Kid-friendly Music Streaming Service that allows children to browse Spotify with parental controls. Using the app, parents can view their children's listening history, block specific songs, and share playlists with their children. The app also includes sing-along songs, playlists designed for young children, and curated audiobooks, lullabies, and bedtime stories. Access is included in Spotify's Premium Family subscription plan, and is exclusive to subscribers to the plan. Users can configure the app for a specific age group upon first launch. The playlists on Spotify Kids are curated by groups including Discovery Kids, Nickelodeon, Universal Pictures, and The Walt Disney Company. All content on the Spotify Kids app is curated by editors. As of March 2021, there were roughly 8,000 songs available on the platform. The design of the Spotify Kids app is colorful, and user interface varies depending on the age group for which the app is configured. Spotify Kids is designed to comply with consent and data collection regulations for apps used by children. TechCrunch explains that it is "designed on a grand scale to drive subscriptions to Spotify's top-tier $14.99-per-month Premium Family Plan." == Release == After being beta tested in Ireland in October 2019, it was released as a beta across the United Kingdom on February 11, 2020. It was later released in Sweden, Denmark, Australia, New Zealand, Mexico, Argentina, and Brazil. On March 31, 2021, it was made available in France, Canada, and the United States.
ArcObjects
ArcObjects is a development environment of the ArcGIS family of applications. Using Visual Basic for Applications, C# or Java SDK for ArcGIS, it allows developers to extend these applications.ArcObjects is a library of Component Object Model (COM) components that build up the foundation of Esri's ArcGIS platform. ArcObjects is written primarily in the C++ programming language. Since ArcGIS is completely built on top of ArcObjects, the ArcGIS platform can be fully customized and extended by making use of its COM services and capabilities. This allows for easy extension of the ArcObjects data model with any programming language that is compatible with COM, such as Visual Basic, C#, Visual Basic.NET, Java and Python. COM enables components to be reused at a binary level, meaning developers do not require access to the source code of ArcObjects in order to extend the ArcGIS platform. For this reason, an ArcObjects programmer can make use of any type inside the ArcObjects system without knowing the implementation details of the type, only needing to know what the type is able to do. The ArcObjects data model is based on the COM standard, which makes it compatible with other COM objects and applications. This allows for easy integration and collaboration with other systems that are also based on the COM standard. The ArcGIS platform was built using ArcObjects types, such as classes, interfaces, and enumerations. ArcObjects use COM interfaces to organize and communicate properties and methods of its classes, ensuring compatibility with other COM-based objects and systems. When working with an ArcObjects COM class, its properties and methods are accessed solely through one of its implemented interfaces via the process of Query Interface (QI). Multiple interfaces are commonly available for classes in ArcObjects. For example, it is possible to query for additional interfaces implemented by an object after instantiation via the process of QI. Although only one interface can be used when instantiating an object, multiple interfaces are often available for classes in ArcObjects, allowing for greater flexibility and compatibility with other systems based on the COM standard.
Suno (platform)
Suno is a generative artificial intelligence music creation platform. It is designed to generate music that can include vocals and instrumentation. The platform was initially developed by Suno, Inc., of Cambridge, Massachusetts. Suno has been widely available since December 20, 2023, after the launch of a web application and a partnership with Microsoft, which included Suno as a plugin in Microsoft Copilot. The program operates by producing songs based on text or audio prompts provided by its users. Suno does not disclose the dataset used to train its artificial intelligence. == History == Suno, Inc., was founded by four people: Michael Shulman, Georg Kucsko, Martin Camacho, and Keenan Freyberg. They all worked for Kensho, an AI startup, before starting their own company in Cambridge, Massachusetts. In April 2023, Suno released their open-source text-to-speech and audio model called "Bark" on GitHub. On March 21, 2024, Suno released its V3 version for all users. The new version allowed users to create a limited number of four-minute songs using a free account. Users can pay for more features. In April 2024, a sentimental ballad was generated with Suno based on the text of the MIT License. In June 2024, a lawsuit, led by the Recording Industry Association of America, was filed against Suno and Udio alleging widespread infringement of copyrighted sound recordings. The lawsuit sought to bar the companies from training on copyrighted music, as well as damages of up to $150,000 per work from infringements that have already taken place. On July 1, 2024, a mobile app for Suno was released. On November 19, 2024, Suno upgraded its AI song model program to v4. In January 2025, Michael Shulman remarked on a podcast, "I think the majority of people don't enjoy the majority of the time they spend making music." In March 2025, one day after thousands of musicians including Thom Yorke and ABBA's Björn Ulvaeus signed a letter calling for Suno to stop training its model on copyrighted music, Timbaland endorsed Suno in a video on the company's website. In July 2025, Suno user imoliver signed a record deal with Hallwood Media, which became the first instance of a traditional music label signing an AI-based creator. Hallwood later signed with AI-artist Xania Monet for US$3 million. Monet's songs were generated by Suno AI by poet Telisha Jones. In November 2025, Suno agreed to a $500 million dollar lawsuit settlement, in which Suno would be allowed to train its models on Warner Music Group's music catalog, and WMG would control aspects of AI likeness, music, audio, software, copyrights, AI tools and music created by users on Suno. As part of the settlement, Suno also acquired the concert discovery platform Songkick from WMG. == Controversy == Suno, Inc., has been sued by the Recording Industry Association of America for copyright infringement, and thousands of musicians have signed a letter demanding that the company cease using copyrighted music in their training data. Suno does not disclose the dataset used to train its artificial intelligence.
Report generator
A report generator is a computer program whose purpose is to take data from a source such as a database, XML stream or a spreadsheet, and use it to produce a document in a format which satisfies a particular human readership. Report generation functionality is almost always present in database systems, where the source of the data is the database itself. It can also be argued that report generation is part of the purpose of a spreadsheet. Standalone report generators may work with multiple data sources and export reports to different document formats. Information systems theory specifies that information delivered to a target human reader must be timely, accurate and relevant. Report generation software targets the final requirement by making sure that the information delivered is presented in the way most readily understood by the target reader. == History == An early report writer was part of NOMAD developed in the 1970s. The evolution of reporting software has a rich history dating back to the mid-20th century, driven by the increasing need for businesses to efficiently analyze and present data. Initially, manual extraction and tabulation were commonplace, but the advent of computers in the 1960s marked a transformative phase with the emergence of basic reporting tools. The 1980s saw the widespread adoption of database management systems, laying the groundwork for more sophisticated reporting capabilities. Notable dedicated reporting software, such as Crystal Reports and BusinessObjects, gained prominence in the 1990s amidst the growing demand for business intelligence. The 21st century witnessed a paradigm shift towards web-based reporting solutions and the rise of self-service BI tools, empowering users to create reports independently. Presently, reporting software continues to evolve with a focus on data visualization, integration of artificial intelligence, and the imperative for real-time analytics in decision-making.
Object co-segmentation
In computer vision, object co-segmentation is a special case of image segmentation, which is defined as jointly segmenting semantically similar objects in multiple images or video frames. == Challenges == It is often challenging to extract segmentation masks of a target/object from a noisy collection of images or video frames, which involves object discovery coupled with segmentation. A noisy collection implies that the object/target is present sporadically in a set of images or the object/target disappears intermittently throughout the video of interest. Early methods typically involve mid-level representations such as object proposals. == Dynamic Markov networks-based methods == A joint object discover and co-segmentation method based on coupled dynamic Markov networks has been proposed recently, which claims significant improvements in robustness against irrelevant/noisy video frames. Unlike previous efforts which conveniently assumes the consistent presence of the target objects throughout the input video, this coupled dual dynamic Markov network based algorithm simultaneously carries out both the detection and segmentation tasks with two respective Markov networks jointly updated via belief propagation. Specifically, the Markov network responsible for segmentation is initialized with superpixels and provides information for its Markov counterpart responsible for the object detection task. Conversely, the Markov network responsible for detection builds the object proposal graph with inputs including the spatio-temporal segmentation tubes. == Graph cut-based methods == Graph cut optimization is a popular tool in computer vision, especially in earlier image segmentation applications. As an extension of regular graph cuts, multi-level hypergraph cut is proposed to account for more complex high order correspondences among video groups beyond typical pairwise correlations. With such hypergraph extension, multiple modalities of correspondences, including low-level appearance, saliency, coherent motion and high level features such as object regions, could be seamlessly incorporated in the hyperedge computation. In addition, as a core advantage over co-occurrence based approach, hypergraph implicitly retains more complex correspondences among its vertices, with the hyperedge weights conveniently computed by eigenvalue decomposition of Laplacian matrices. == CNN/LSTM-based methods == In action localization applications, object co-segmentation is also implemented as the segment-tube spatio-temporal detector. Inspired by the recent spatio-temporal action localization efforts with tubelets (sequences of bounding boxes), Le et al. present a new spatio-temporal action localization detector Segment-tube, which consists of sequences of per-frame segmentation masks. This Segment-tube detector can temporally pinpoint the starting/ending frame of each action category in the presence of preceding/subsequent interference actions in untrimmed videos. Simultaneously, the Segment-tube detector produces per-frame segmentation masks instead of bounding boxes, offering superior spatial accuracy to tubelets. This is achieved by alternating iterative optimization between temporal action localization and spatial action segmentation. The proposed segment-tube detector is illustrated in the flowchart on the right. The sample input is an untrimmed video containing all frames in a pair figure skating video, with only a portion of these frames belonging to a relevant category (e.g., the DeathSpirals). Initialized with saliency based image segmentation on individual frames, this method first performs temporal action localization step with a cascaded 3D CNN and LSTM, and pinpoints the starting frame and the ending frame of a target action with a coarse-to-fine strategy. Subsequently, the segment-tube detector refines per-frame spatial segmentation with graph cut by focusing on relevant frames identified by the temporal action localization step. The optimization alternates between the temporal action localization and spatial action segmentation in an iterative manner. Upon practical convergence, the final spatio-temporal action localization results are obtained in the format of a sequence of per-frame segmentation masks (bottom row in the flowchart) with precise starting/ending frames.
JBoss Tools
JBoss Tools is a set of Eclipse plugins and features designed to help JBoss and JavaEE developers develop applications. It is an umbrella project for the JBoss developed plugins that will make it into JBoss Developer Studio. == Modules == JBoss Tools includes the following modules: Visual Page Editor (VPE). The visual editor contributed by Exadel supports visual editing of HTML and JSF (JSP and Facelets) pages. VPE also includes visual support for JSF component libraries including JBoss RichFaces. Seam Tools. Includes support for (for example) seam-gen, RichFaces VE integration, Seam related code completion and refactoring. Hibernate Tools. Supporting mapping files, annotations and JPA with reverse engineering, code completion, project wizards, refactoring, interactive HQL/JPA-QL/Criteria execution and more. In short a merger of Hibernate Tools and Exadel ORM features. JBoss AS Tools. Easy start, stop and debug of JBoss AS 4+ servers from within Eclipse. Also includes features for packaging and deployment of any type of Eclipse project. Drools IDE. Rules file editing, Rete View, working memory debugging/inspection and more. jBPM Tools. jBPM workflow editing, deployment, etc. JBossWS Tools. Inspecting, invoking, developing and functional/load/compliance testing of web services over HTTP, base tooling provided by soapUI with the addition of JBossWS specific features/support. JBoss ESB Tools. The structured xml editor for the jboss-esb.xml file used in JBoss ESB. Birt Tools. Hibernate and Seam extensions for Eclipse BIRT. Portal Tools. JBoss Tools supports the JSR-168 Portlet Specification (Portlet 1.0), JSR-286 Portlet Specification (Portlet 2.0) and works with PortletBridge for supporting Portlets in JSF/Seam applications. To enable these features, add the JBoss Portlet facet to a new or an existing web project. Core/General Tools. To reduce the UI clutter, most of the "configure project" menu items move into the Configure menu introduced in Eclipse 3.5 instead of always having a static JBoss Tools menu entry show up even in projects unrelated to JBoss Tools. Smooks Tools. The editor for Smooks configuration files. JBoss ESB Tools. The ESB project Wizard, which creates a project that can be deployed as an .esb archive to a JBoss AS-based server with JBoss ESB installed. JMX Tools. JMX Tools allows establishing multiple JMX connections and provides views for exploring the JMX tree and execute operations directly from Eclipse. The JMX Tools replaces the JMX node previously available in the JBoss Server View. JST/JSF Tools. RichFaces Support, Code Assists, Web XML/JSP/XHTML Editors, CSS Style Editing, web.xml validation, Faceleted taglib in taglib.xml is supported with XSD schema location. Project Examples. The experimental feature called Project Example wizard aims to allow users to download example projects from a remote site and have them working out-of-the-box. AS/Project Archives Tools. To deploy projects compressed, configurable in the server editor. If enabled, all projects deployed to that server will be compressed instead of in an exploded folder. Maven Tools. The optional integration with m2eclipse to provide Maven support for projects created by JBoss Tools and to some extent core WTP projects. BPEL Tools. A BPEL Editor based on the Eclipse BPEL project has been added to JBoss Tools. This means that users can create, edit and deploy BPEL artifacts for the Riftsaw BPEL Runtime. CDI (JSR-299) Tools. Support of the Contexts and Dependency Injection annotations; it works on any Eclipse Java project (via the Configure menu with CDI enabled).