lerobot/robot-learning-tutorial
508
1---2title: "Robot Learning: A Tutorial"3authors:4 - name: "Francesco Capuano"5 affiliations: [1, 2]6 - name: "Caroline Pascal"7 affiliations: [2]8 - name: "Adil Zouitine"9 affiliations: [2]10 - name: "Thomas Wolf"11 affiliations: [2]12 - name: "Michel Aractingi"13 affiliations: [2]14affiliations:15 - name: "University of Oxford"16 - name: "Hugging Face"17published: "Oct 15, 2025"18tableOfContentsAutoCollapse: true19---20 21import MultiImage from '../components/MultiImage.astro';22import Image from '../components/Image.astro';23import Quote from '../components/Quote.astro';24import ch2_planar_manipulator_free from './assets/image/figures/ch2/ch2-planar-manipulator-free.png';25import ch2_planar_manipulator_floor from './assets/image/figures/ch2/ch2-planar-manipulator-floor.png';26import ch2_planar_manipulator_floor_shelf from './assets/image/figures/ch2/ch2-planar-manipulator-floor-shelf.png';27import ch1_lerobot_figure1 from './assets/image/figures/ch1/ch1-lerobot-figure1.png';28import ch2_approaches from './assets/image/figures/ch2/ch2-approaches.png';29import ch2_platforms from './assets/image/figures/ch2/ch2-platforms.png';30import ch2_cost_accessibility from './assets/image/figures/ch2/ch2-cost-accessibility.png';31import ch2_so100_to_planar_manipulator from './assets/image/figures/ch2/ch2-so100-to-planar-manipulator.png';32import ch2_planar_manipulator_floor_box from './assets/image/figures/ch2/ch2-planar-manipulator-floor-box.png';33import ch2_classical_limitations from './assets/image/figures/ch2/ch2-classical-limitations.png';34import ch3_learning_benefits from './assets/image/figures/ch3/ch3-learning-benefits.png';35import ch3_learning_atlas from './assets/image/figures/ch3/ch3-learning-atlas.png';36import ch3_rl_examples from './assets/image/figures/ch3/ch3-rl-examples.png';37import ch3_agent_env from './assets/image/figures/ch3/ch3-agent-env.png';38import ch3_rl_algorithms_atlas from './assets/image/figures/ch3/ch3-rl-algorithms-atlas.png';39import ch3_duck_sim_vs_real from './assets/image/figures/ch3/ch3-duck-sim-vs-real.png';40import ch3_many_ducks from './assets/image/figures/ch3/ch3-many-ducks.png';41import ch3_hil_serl_examples from './assets/image/figures/ch3/ch3-hil-serl-examples.png';42import ch3_hil_serl_architecture from './assets/image/figures/ch3/ch3-hil-serl-architecture.png';43import ch4_bc_trajectories from './assets/image/figures/ch4/ch4-bc-trajectories.png';44import ch4_observation_action_mapping from './assets/image/figures/ch4/ch4-observation-action-mapping.png';45import ch4_issues_with_bc from './assets/image/figures/ch4/ch4-issues-with-bc.png';46import ch4_task_effect_on_pairs from './assets/image/figures/ch4/ch4-task-effect-on-pairs.png';47import ch4_latent_variable_model from './assets/image/figures/ch4/ch4-latent-variable-model.png';48import ch4_many_latents from './assets/image/figures/ch4/ch4-many-latents.png';49import ch4_diffusion_robot_actions from './assets/image/figures/ch4/ch4-diffusion-robot-actions.png';50import ch4_action_vs_observation_distribution from './assets/image/figures/ch4/ch4-action-vs-observation-distribution.png';51import ch4_normalizing_flows from './assets/image/figures/ch4/ch4-normalizing-flows.png';52import ch4_diffusion_vs_flowmatching from './assets/image/figures/ch4/ch4-diffusion-vs-flowmatching.png';53import ch4_act_encoder from './assets/image/figures/ch4/ch4-act-encoder.png';54import ch4_act_decoder from './assets/image/figures/ch4/ch4-act-decoder.png';55import ch4_act from './assets/image/figures/ch4/ch4-act.png';56import ch4_diffusion_policy from './assets/image/figures/ch4/ch4-diffusion-policy.png';57import ch4_async_inference from './assets/image/figures/ch4/ch4-async-inference.png';58import ch4_queues from './assets/image/figures/ch4/ch4-queues.png';59import ch5_ml_vs_robotics_foundation from './assets/image/figures/ch5/ch5-ml-vs-robotics-foundation.png';60import ch5_generalist_policies_timeline from './assets/image/figures/ch5/ch5-generalist-policies-timeline.png';61import ch5_trends from './assets/image/figures/ch5/ch5-trends.png';62import ch5_pi0 from './assets/image/figures/ch5/ch5-pi0.png';63import ch5_pi0_sampling_timesteps from './assets/image/figures/ch5/ch5-pi0-sampling-timesteps.png';64import ch5_smolvla from './assets/image/figures/ch5/ch5-smolvla.png';65 66 67## Foreword68 69Robotics is an inherently multidisciplinary field, which is witnessing unprecedented advancements since its inception in the 1960s. Yet, more than sixty years after the debut of Unimate, robots have still not fully integrated into the rich, unstructured, and dynamic world we humans inhabit. Over the decades, numerous disciplines have shown immense promise in tackling the challenges of creating autonomous robotic systems. This tutorial takes a clear stance in the debate on whether modern Machine Learning can play a pivotal role in the development of autonomous robots: we believe this to be the case.70 71Nonetheless, we also hold that the wealth of research from both academia and industry in classical robotics over the past six decades is, simply put, too valuable to be cast aside in favor of purely learning-based methods. However, the interplay between classical robotics and modern machine learning is still in its nascent stages, and the path to integration yet to be clearly defined. In turn our goal here is to present what we consider to be the most relevant approaches within robot learning today, while warmly extending an invite to collaborate to expand the breadth of this work! Start contributing today [here](https://github.com/fracapuano/robot-learning-tutorial).72 73This tutorial...74 75- Does *not* aim to be a comprehensive guide to general field of robotics, manipulation or underactuated systems: @sicilianoSpringerHandbookRobotics2016 and @tedrakeRoboticManipulationPerception, @tedrakeUnderactuatedRoboticsAlgorithms do this better than we ever could.76 77- Does *not* aim to be an introduction to statistical or deep learning: @shalev-shwartzUnderstandingMachineLearning2014 and @prince2023understanding cover these subjects better than we ever could.78 79- Does *not* aim to be a deep dive into Reinforcement Learning, Diffusion Models, or Flow Matching: invaluable works such as @suttonReinforcementLearningIntroduction2018, @nakkiranStepbyStepDiffusionElementary2024, and @lipmanFlowMatchingGuide2024 do this better than we ever could.80 81Instead, our goal here is to provide an intuitive explanation as per why these disparate ideas have converged to form the exciting field of modern robot learning, driving the unprecedented progress we see today. In this spirit, we follow the adage: "a jack of all trades is a master of none, *but oftentimes better than a master of one*."82 83We sincerely hope this tutorial serves as a valuable starting point for your journey into robot learning.84 85## Introduction86 87<Image88 src={ch1_lerobot_figure1}89 zoomable90 downloadable91 alt="Figure"92 caption={'lerobot is the open-source library for end-to-end robotics developed by Hugging Face. The library is vertically integrated on the entire robotics stack, supporting low-level control of real-world robot devices, advanced data and inference optimizations, as well as SOTA robot learning methods with simple implementations in pure Pytorch.'}/>93 94Autonomous robotics holds the premise of relieving humans from repetitive, tiring or dangerous manual tasks. Consequently, the field of robotics has been widely studied since its first inception in the 1950s. Lately, advancements in Machine Learning (ML) have sparked the development of a relatively new class of methods used to tackle robotics problems, leveraging large amounts of data and computation rather than human expertise and modeling skills to develop autonomous systems.95 96The frontier of robotics research is indeed increasingly moving away from classical model-based control paradigm, embracing the advancements made in ML, aiming to unlock (1) monolithic perception-to-action control pipelines and (2) multi-modal data-driven feature extraction strategies, together with (3) reduced reliance on precise models of the world and (4) a better positioning to benefit from the growing availability of open robotics data. While central problems in manipulation, locomotion and whole-body control demand knowledge of rigid-body dynamics, contact modeling, planning under uncertainty, recent results seem to indicate learning can prove just as effective as explicit modeling, sparking interest in the field of *robot learning*. This interest can be largely justified considering the significant challenges related to deriving accurate models of robot-environment interactions.97 98Moreover, since end-to-end learning on ever-growing collections of text and image data has historically been at the core of the development of *foundation models* capable of semantic reasoning across multiple modalities (images, text, audio, etc.), deriving robotics methods grounded in learning appears particularly consequential, especially as the number of openly available datasets continues to grow.99 100Robotics is, at its core, an inherently multidisciplinary field, requiring a wide range of expertise in both *software* and *hardware*. The integration of learning-based techniques further broadens this spectrum of skills, raising the bar for both research and practical applications. `lerobot` is an open-source library designed to integrate end-to-end with the entire robotics stack. With a strong focus on accessible, real-world robots <mark>(1) `lerobot` supports many, openly available, robotic platforms</mark> for manipulation, locomotion and even whole-body control. `lerobot`also implements a <mark>(2) unified, low-level approach to reading/writing robot configurations</mark> to extend support for other robot platforms with relatively low effort. The library introduces `LeRobotDataset`, <mark>(3) a native robotics dataset’s format</mark> currently being used by the community to efficiently record and share datasets. `lerobot` also supports many state-of-the-art (SOTA) algorithms in robot learning--mainly based on Reinforcement Learning (RL) and Behavioral Cloning (BC) techniques--with efficient implementations in Pytorch, and extended support to experimentation and experiments tracking. Lastly, `lerobot` defines a custom, optimized inference stack for robotic policies decoupling action planning from action execution, proving effective in guaranteeing more adaptability at runtime.101 102This tutorial serves the double purpose of providing useful references for the Science behind--and practical use of--common robot learning techniques. To this aim, we strike to provide a rigorous yet concise overview of the core concepts behind the techniques presented, paired with practical examples of how to use such techniques concretely, with code examples in `lerobot`, for researchers and practitioners interested in the field of robot learning. This tutorial is structured as follows:103 104- Section <a href="#classical" data-reference-type="ref" data-reference="classical">[classical]</a> reviews classical robotics foundations, introducing the limitations of dynamics-based approaches to robotics.105 106- Section <a href="#learning-rl" data-reference-type="ref" data-reference="learning-rl">[learning-rl]</a> elaborates on the limitations of dynamics-based methods, and introduce RL as a practical approach to solve robotics problems, considering its upsides and potential limitations.107 108- Section <a href="#learning-imitation" data-reference-type="ref" data-reference="learning-imitation">[learning-imitation]</a> further describes robot learning techniques that aim at solving single-tasks learning, leveraging BC techniques to autonomously reproduce specific expert demonstrations.109 110- Section <a href="#learning-foundation" data-reference-type="ref" data-reference="learning-foundation">[learning-foundation]</a> presents recent contributions on developing generalist models for robotics applications, by learning from large corpora of multi-task multi-robot data (*robotics foundation models*).111 112Our goal with this tutorial is to provide an intuitive explanation of the reasons various disparate ideas from Machine Learning (ML) have converged and are powering the current evolution of Robotics, driving the unprecedented progress we see today. We complement our presentation of the most common and recent approaches in robot learning with practical code implementations using `lerobot`, and start here by presenting the dataset format introduced with `lerobot`.113 114### `LeRobotDataset`115 116`LeRobotDataset` is one of the most impactful features of `lerobot`, developed in keeping with the observation that robotics data is increasingly central in robot learning. Thus, `lerobot` defines a standardized dataset format designed to address the specific needs of robot learning research, providing a unified and convenient access to robotics data across modalities, including sensorimotor readings, multiple camera feeds and teleoperation status. `LeRobotDataset` also accommodates for storing general information regarding the data being collected, including textual descriptions of the task being performed by the teleoperator, the kind of robot used, and relevant measurement specifics like the frames per second at which the recording of both image and robot state’s streams are proceeding.117 118In this, `LeRobotDataset` provides a unified interface for handling multi-modal, time-series data, and it is designed to seamlessly integrate with the PyTorch and Hugging Face ecosystems. `LeRobotDataset` can be easily extended by users and it is highly customizable by users, and it already supports openly available data coming from a variety of embodiments supported in `lerobot`, ranging from manipulator platforms like the SO-100 arm and ALOHA-2 setup, to real-world humanoid arm and hands, as well as entirely simulation-based datasets, and self-driving cars. This dataset format is built to be both efficient for training and flexible enough to accommodate the diverse data types encountered in robotics, while promoting reproducibility and ease of use for users.119 120#### The dataset class design121 122A core design choice behind `LeRobotDataset` is separating the underlying data storage from the user-facing API. This allows for efficient storage while presenting the data in an intuitive, ready-to-use format.123 124Datasets are always organized into three main components:125 126- **Tabular Data**: Low-dimensional, high-frequency data such as joint states, and actions are stored in efficient memory-mapped files, and typically offloaded to the more mature `datasets` library by Hugging Face, providing fast with limited memory consumption.127 128- **Visual Data**: To handle large volumes of camera data, frames are concatenated and encoded into MP4 files. Frames from the same episode are always grouped together into the same video, and multiple videos are grouped together by camera. To reduce stress on the file system, groups of videos for the same camera view are also broke into multiple sub-directories, after a given threshold number.129 130- **Metadata** A collection of JSON files which describes the dataset’s structure in terms of its metadata, serving as the relational counterpart to both the tabular and visual dimensions of data. Metadata include the different feature schema, frame rates, normalization statistics, and episode boundaries.131 132For scalability, and to support datasets with potentially millions of trajectories (resulting in hundreds of millions or billions of individual camera frames), we merge data from different episodes into the same high-level structure. Concretely, this means that any given tabular collection and video will not typically contain information about one episode only, but rather a concatenation of the information available in multiple episodes. This keeps the pressure on the file system limited, both locally and on remote storage providers like Hugging Face, though at the expense of leveraging more heavily relational-like, metadata parts of the dataset, which are used to reconstruct information such as at which position, in a given file, an episode starts or ends. An example struture for a given `LeRobotDataset` would appear as follows:133 134- `meta/info.json`: This metadata is a central metadata file. It contains the complete dataset schema, defining all features (e.g., `observation.state`, `action`), their shapes, and data types. It also stores crucial information like the dataset’s frames-per-second (`fps`), `lerobot`’s version at the time of capture, and the path templates used to locate data and video files.135 136- `meta/stats.json`: This file stores aggregated statistics (mean, std, min, max) for each feature across the entire dataset, used for data normalization for most policy models and accessible externally via `dataset.meta.stats`.137 138- `meta/tasks.jsonl`: This file contains the mapping from natural language task descriptions to integer task indices, which are useful for task-conditioned policy training.139 140- `meta/episodes/*` This directory contains metadata about each individual episode, such as its length, the corresponding task, and pointers to where its data is stored in the dataset’s files. For scalability, this information is stored in files rather than a single large JSON file.141 142- `data/*`: Contains the core frame-by-frame tabular data, using parquet files to allow for fast, memory-mapped access. To improve performance and handle large datasets, data from multiple episodes are concatenated into larger files. These files are organized into chunked subdirectories to keep the size of directories manageable. A single file typically contains data for more than one single episode.143 144- `videos/*`: Contains the MP4 video files for all visual observation streams. Similar to the `data/` directory, the video footage from multiple episodes is concatenated into single MP4 files. This strategy significantly reduces the number of files in the dataset, which is more efficient for modern filesystems.145 146### Code Example: Batching a (Streaming) Dataset147 148This section provides an overview of how to access datasets hosted on Hugging Face using the `LeRobotDataset` class. Every dataset on the Hugging Face Hub containing the three main pillars presented above (Tabular, Visual and relational Metadata), and can be assessed with a single instruction.149 150In practice, most reinforcement learning (RL) and behavioral cloning (BC) algorithms tend to operate on stack of observation and actions. For the sake of brevity, we will refer to joint spaces, and camera frames with the single term of *frame*. For instance, RL algorithms may use a history of previous frames $o_{t-H_o:t}$ to mitigate partial observability, and BC algorithms are in practice trained to regress chunks of multiple actions ($a_{t+t+H_a}$) rather than single controls. To accommodate for these specifics of robot learning training, `LeRobotDataset` provides a native windowing operation, whereby users can define the *seconds* of a given window (before and after) around any given frame, by using the `delta_timestemps` functionality. Unavailable frames are opportunely padded, and a padding mask is also returned to filter out the padded frames. Notably, this all happens within the `LeRobotDataset`, and is entirely transparent to higher level wrappers commonly used in training ML models such as `torch.utils.data.DataLoader`.151 152Conveniently, by using `LeRobotDataset` with a Pytorch `DataLoader` one can automatically collate the individual sample dictionaries from the dataset into a single dictionary of batched tensors for downstream training or inference. `LeRobotDataset` also natively supports streaming mode for datasets. Users can stream data of a large dataset hosted on the Hugging Face Hub, with a one-line change in their implementation. Streaming datasets supports high-performance batch processing (ca. 80-100 it/s, varying on connectivity) and high levels of frames randomization, key features for practical BC algorithms which otherwise may be slow or operating on highly non-i.i.d. data. This feature is designed to improve on accessibility so that large datasets can be processed by users without requiring large amounts of memory and storage.153 154<div class="pbox">155 156Batching a (Streaming) Dataset 157[https://github.com/fracapuano/robot-learning-tutorial/blob/main/snippets/ch1/01_datasets.py](https://github.com/fracapuano/robot-learning-tutorial/blob/main/snippets/ch1/01_datasets.py)158 159```python160import torch161from lerobot.datasets.lerobot_dataset import LeRobotDataset162from lerobot.datasets.streaming_dataset import StreamingLeRobotDataset163 164delta_timestamps = {165 "observation.images.wrist_camera": [-0.2, -0.1, 0.0] # 0.2, and 0.1 seconds *before* each frame166}167 168# Optionally, use StreamingLeRobotDataset to avoid downloading the dataset169dataset = LeRobotDataset(170 "lerobot/svla_so101_pickplace",171 delta_timestamps=delta_timestamps172)173 174# Streams frames from the Hugging Face Hub without loading into memory175streaming_dataset = StreamingLeRobotDataset(176 "lerobot/svla_so101_pickplace",177 delta_timestamps=delta_timestamps178)179 180# Get the 100th frame in the dataset by 181sample = dataset[100]182print(sample)183# {184# 'observation.state': tensor([...]), 185# 'action': tensor([...]), 186# 'observation.images.wrist_camera': tensor([3, C, H, W]), for delta timesteps187# ...188# }189 190batch_size=16191# wrap the dataset in a DataLoader to use process it batches for training purposes192data_loader = torch.utils.data.DataLoader(193 dataset,194 batch_size=batch_size195)196 197# Iterate over the DataLoader in a training loop198num_epochs = 1199device = "cuda" if torch.cuda.is_available() else "cpu"200 201for epoch in range(num_epochs):202 for batch in data_loader:203 # Move data to the appropriate device (e.g., GPU)204 observations = batch["observation.state"].to(device)205 actions = batch["action"].to(device)206 images = batch["observation.images.wrist_camera"].to(device)207 208 # Next, you can do amazing_model.forward(batch)209 ...210```211 212</div>213 214### Code Example: Collecting Data215 216<span id="paragraph-collecting-data" style="position: absolute;"></span>217<div class="pbox">218 219Record a Dataset 220[https://github.com/fracapuano/robot-learning-tutorial/blob/main/snippets/ch1/02_record_data.py](https://github.com/fracapuano/robot-learning-tutorial/blob/main/snippets/ch1/02_record_data.py)221 222```python223import torch224from lerobot.datasets.lerobot_dataset import LeRobotDataset225from lerobot.datasets.streaming_dataset import StreamingLeRobotDataset226 227delta_timestamps = {228 "observation.images.wrist_camera": [-0.2, -0.1, 0.0] # 0.2, and 0.1 seconds *before* each frame229}230 231# Optionally, use StreamingLeRobotDataset to avoid downloading the dataset232dataset = LeRobotDataset(233 "lerobot/svla_so101_pickplace",234 delta_timestamps=delta_timestamps235)236 237# Streams frames from the Hugging Face Hub without loading into memory238streaming_dataset = StreamingLeRobotDataset(239 "lerobot/svla_so101_pickplace",240 delta_timestamps=delta_timestamps241)242 243# Get the 100th frame in the dataset by 244sample = dataset[100]245print(sample)246# {247# 'observation.state': tensor([...]), 248# 'action': tensor([...]), 249# 'observation.images.wrist_camera': tensor([3, C, H, W]), for delta timesteps250# ...251# }252 253batch_size=16254# wrap the dataset in a DataLoader to use process it batches for training purposes255data_loader = torch.utils.data.DataLoader(256 dataset,257 batch_size=batch_size258)259 260# Iterate over the DataLoader in a training loop261num_epochs = 1262device = "cuda" if torch.cuda.is_available() else "cpu"263 264for epoch in range(num_epochs):265 for batch in data_loader:266 # Move data to the appropriate device (e.g., GPU)267 observations = batch["observation.state"].to(device)268 actions = batch["action"].to(device)269 images = batch["observation.images.wrist_camera"].to(device)270 271 # Next, you can do amazing_model.forward(batch)272 ...273```274 275</div>276 277## Classical Robotics278 279<span id="classical" style="position: absolute;"></span>280<Quote source="Sun Tzu">281 Know your enemy282</Quote>283<div class="callout">284 285TL;DR Learning-based approaches to robotics are motivated by the need to (1) generalize across tasks and embodiments (2) reduce dependency on human expertise (3) leverage historical trends on the production of data--all traditionally overlooked by dynamics-based techniques.286 287</div>288 289### Explicit and Implicit Models290 291<Image292 src={ch2_approaches}293 zoomable294 downloadable295 alt="Figure"296 caption={'Overview of methods to generate motion (clearly non-exhausitve, see @bekrisStateRobotMotion2024). The different methods can be grouped based on whether they explicitly (dynamics-based) or implicitly (learning-based) model robot-environment interactions.'}/>297 298Robotics is concerned with producing artificial motion in the physical world in useful, reliable and safe fashion. Thus, robotics is an inherently multi-disciplinar domain: producing autonomous motion in the physical world requires, to the very least, interfacing different software (motion planners) and hardware (motion executioners) components. Further, knowledge of mechanical, electrical, and software engineering, as well as rigid-body mechanics and control theory have therefore proven quintessential in robotics since the field first developed in the 1950s. More recently, Machine Learning (ML) has also proved effective in robotics, complementing these more traditional disciplines @connellRobotLearning1993. As a direct consequence of its multi-disciplinar nature, robotics has developed as a rather wide array of methods, all concerned with the main purpose of <mark>producing artificial motion in the physical world</mark>.299 300Methods to produce robotics motion range from traditional *explicit* models--<mark>dynamics-based</mark>[^1] methods, leveraging precise descriptions of the mechanics of robots’ rigid bodies and their interactions with eventual obstacles in the environment--to *implicit* models--<mark>learning-based</mark> methods, treating artificial motion as a statistical pattern to learn given multiple sensorimotor readings @agrawalComputationalSensorimotorLearning, @bekrisStateRobotMotion2024. A variety of methods have been developed between these two extrema. For instance, @hansenTemporalDifferenceLearning2022 show how learning-based systems can benefit from information on the physics of problems, complementing a traditional learning method such as Temporal Difference (TD)-learning @suttonReinforcementLearningIntroduction2018 with Model-Predictive Control (MPC). Conversely, as explicit models may be relying on assumptions proving overly simplistic--or even unrealistic--in practice, learning can prove effective to improve modeling of complex phenomena or complement perception @mccormacSemanticFusionDense3D2016. Such examples aim at demonstrating the richness of approaches to robotics, and Figure <a href="#generating-motion-atlas" data-reference-type="ref" data-reference="generating-motion-atlas">[generating-motion-atlas]</a> graphically illustrates some of the most relevant techniques. Such a list is clearly far from being exhaustive, and we refer to @bekrisStateRobotMotion2024 for a more comprehensive overview of both general and application-specific methods for motion generation. In this section, we wish to introduce the inherent benefits of <mark>learning-based approaches to robotics</mark>--the core focus on this tutorial.301 302### Different Types of Motion303 304<Image305 src={ch2_platforms}306 zoomable307 downloadable308 alt="Figure"309 caption={'Different kinds of motions are achieved with potentially very different robotic platforms. From left to right, top to bottom: ViperX, SO-100, Boston Dynamics’ Spot, Open-Duck, 1X’s NEO, Boston Dynamics’ Atlas. This is an example list of robotic platforms and is (very) far from being exhaustive.'}/>310 311In the vast majority of instances, robotics deals with producing motion via actuating joints connecting nearly entirely-rigid links. A key distinction between focus areas in robotics is based on whether the generated motion modifies (1) the absolute state of the environment (via dexterity), (2) the relative state of the robot with respect to its environment (exercising mobility skills), or (3) a combination of the two (Figure <a href="#robotics-platforms-atlas" data-reference-type="ref" data-reference="robotics-platforms-atlas">[robotics-platforms-atlas]</a>).312 313Effects such as (1) are typically achieved *through* the robot, i.e. generating motion to perform an action inducing a desirable modification, effectively *manipulating* the environment (manipulation). Motions like (2) may result in changes in the robot’s physical location within its environment. Generally, modifications to a robot’s location within its environment may be considered instances of the general *locomotion* problem, further specified as *wheeled* or *legged* locomotion based on whenever a robot makes use of wheels or leg(s) to move in the environment. Lastly, an increased level of dynamism in the robot-environment interactions can be obtained combining (1) and (2), thus designing systems capable to interact with *and* move within their environment. This category is problems is typically termed *mobile manipulation*, and is characterized by a typically much larger set of control variables compared to either locomotion or manipulation alone.314 315The traditional body of work developed since the very inception of robotics is increasingly complemented by learning-based approaches. ML has indeed proven particularly transformative across the entire robotics stack, first empowering planning-based techniques with improved state estimation used for traditional planning @tangPerceptionNavigationAutonomous2023 and then end-to-end replacing controllers, effectively yielding perception-to-action methods @koberReinforcementLearningRobotics. Work in producing robots capable of navigating a diverse set of terrains demonstrated the premise of both dynamics and learning-based approaches for locomotion @griffinWalkingStabilizationUsing2017, @jiDribbleBotDynamicLegged2023, @leeLearningQuadrupedalLocomotion2020, @margolisRapidLocomotionReinforcement2022, and recent works on whole-body control indicated the premise of learning-based approaches to generate rich motion on complex robots, including humanoids @zhangWoCoCoLearningWholeBody2024, @bjorckGR00TN1Open2025. Manipulation has also been widely studied, particularly considering its relevance for many impactful use-cases ranging from high-risk applications for humans @fujitaDevelopmentRobotsNuclear2020, @alizadehComprehensiveSurveySpace2024 to manufacturing @sannemanStateIndustrialRobotics2020. While explicit models have proven fundamental in achieving important milestones towards the development of modern robotics, recent works leveraging implicit models proved particularly promising in surpassing scalability and applicability challenges via learning @koberReinforcementLearningRobotics.316 317### Example: Planar Manipulation318 319Robot manipulators typically consist of a series of links and joints, articulated in a chain finally connected to an *end-effector*. Actuated joints are considered responsible for generating motion of the links, while the end effector is instead used to perform specific actions at the target location (e.g., grasping/releasing objects via closing/opening a gripper end-effector, using a specialized tool like a screwdriver, etc.).320 321Recently, the development of low-cost manipulators like the ALOHA @zhaoLearningFineGrainedBimanual2023 ALOHA-2 @aldacoALOHA2Enhanced and SO-100/SO-101 @knightStandardOpenSO100 platforms significantly lowered the barrier to entry to robotics, considering the increased accessibility of these robots compared to more traditional platforms like the Franka Emika Panda arm (Figure <a href="#robotic-platforms-costs" data-reference-type="ref" data-reference="robotic-platforms-costs">[robotic-platforms-costs]</a>).322 323<Image324 src={ch2_cost_accessibility}325 zoomable326 downloadable327 alt="Figure"328 caption={'Cheaper, more accessible robots are starting to rival traditional platforms like the Panda arm platforms in adoption in resource-constrained scenarios. The SO-100, in particular, has a cost in the 100s of Euros, and can be entirely 3D-printed in hours, while the industrially-manufactured Panda arm costs tens of thousands of Euros and is not openly available.'}/>329 330Deriving an intuition as per why learning-based approaches are gaining popularity in the robotics community requires briefly analyzing traditional approaches for manipulation, leveraging tools like forward and inverse kinematics (FK, IK) and control theory. Providing a detailed overview of these methods falls (well) out of the scope of this tutorial, and we refer the reader to works including @sicilianoSpringerHandbookRobotics2016, @lynchModernRoboticsMechanics2017, @tedrakeRoboticManipulationPerception, @tedrakeUnderactuatedRoboticsAlgorithms for a much more comprehensive description of these techniques. Here, we mostly wish to highlight the benefits of ML over these traditional techniques331 332<Image333 src={ch2_so100_to_planar_manipulator}334 zoomable335 downloadable336 alt="Figure"337 caption={'The SO-100 arm is a 6-dof manipulator arm. Preventing some of its joints (shoulder pane, wrist flex and wrist roll) from actuating, it can be represented as a traditional 2-dof planar manipulator (the gripper joint in the end-effector is not considered towards the count of the degrees of freedom used to produce motion).'}/>338 339Consider the (simple) case where a SO-100 is restrained from actuating (1) the shoulder pane and (2) the wrist flex and roll motors. This effectively reduces the degrees of freedom of the SO-100 from the original 5+1 (5 joints + 1 gripper) to 2+1 (shoulder lift, elbow flex + gripper). As the end-effector does not impact motion in this model, the SO-100 is effectively reduced to the planar manipulator robot presented in Figure <a href="#make-so100-planar-manipulator" data-reference-type="ref" data-reference="make-so100-planar-manipulator">[make-so100-planar-manipulator]</a>, where spheres represent actuators, and solid lines indicate length-$l$ links from the base of the SO-100 to the end-effector (*ee*).340 341Further, let us make the simplifying assumption that actuators can produce rotations up to $2 \pi$ radians. In practice, this is seldom the case due to movement obstructions caused by the robot body itself (for instance, the shoulder lift cannot produce counter-clockwise movement due to the presence of the robot’s base used to secure the SO-100 to its support and host the robot bus), but we will introduce movement obstruction at a later stage.342 343All these simplifying assumptions leave us with the planar manipulator of Figure <a href="#planar-manipulation-simple" data-reference-type="ref" data-reference="planar-manipulation-simple">[planar-manipulation-simple]</a>, free of moving its end-effector by controlling the angles $\theta_1$ and $\theta_2$, jointly referred to as the robot’s *configuration*, and indicated with $q = [\theta_1, \theta_2 ] \in [-\pi, +\pi]^2$. The axis attached to the joints indicate the associated reference frame, whereas circular arrows indicate the maximal feasible rotation allowed at each joint. In this tutorial, we do not cover topics related to spatial algebra, and we instead refer the reader to and for excellent explanations of the mechanics and theoretical foundations of producing motion on rigid bodies.344 345<MultiImage346 images={[347 {348 src: ch2_planar_manipulator_free,349 alt: "Free to move",350 caption: "Free to move",351 id: "planar-manipulation-simple"352 },353 {354 src: ch2_planar_manipulator_floor,355 alt: "Constrained by the surface",356 caption: "Constrained by the surface",357 id: "planar-manipulator-floor"358 },359 {360 src: ch2_planar_manipulator_floor_shelf,361 alt: "Constrained by surface and (fixed) obstacle",362 caption: "Constrained by surface and (fixed) obstacle",363 id: "planar-manipulator-floor-shelf"364 }365 ]}366 layout="3-column"367 zoomable368 downloadable369 caption="Planar, 2-dof schematic representation of the SO-100 manipulator under diverse deployment settings. From left to right: completely free of moving; constrained by the presence of the surface; constrained by the surface and presence of obstacles. Circular arrows around each joint indicate the maximal rotation feasible at that joint."370/>371 372Considering the (toy) example presented in Figure <a href="#planar-manipulation-simple" data-reference-type="ref" data-reference="planar-manipulation-simple">[planar-manipulation-simple]</a>, then we can analytically write the end-effector’s position $p \in \mathbb R^2$ as a function of the robot’s configuration, $p = p(q), p: \mathcal Q \mapsto \mathbb R^2$. In particular, we have: $p(q) = \begin{pmatrix} p_x(\theta_1, \theta_2) \\ p_y(\theta_1, \theta_2) \end{pmatrix} = \begin{pmatrix} l \cos(\theta_1) + l \cos(\theta_1 + \theta_2) \\ l \sin(\theta_1) + l \sin(\theta_1 + \theta_2) \end{pmatrix} \in S^{n=2}_{l_1+l_2} = \{ p(q) \in \mathbb R^2: \Vert p(q) \Vert_2^2 \leq (2l)^2, \ \forall q \in \mathcal Q \}$373 374Deriving the end-effector’s *pose*--position *and* orientation--in some $m$-dimensional space $\boldsymbol{p} \in \mathcal{P} \subset \mathbb{R}^{m}$ starting from the configuration ${\textnormal{q}}\in \mathcal Q \subset \mathbb R^n$ of a $n$-joints robot is referred to as *forward kinematics* (FK), whereas identifying the configuration corresponding to any given target pose is termed *inverse kinematics* (IK). In that, FK is used to map a robot configuration into the corresponding end-effector pose, whereas IK is used to reconstruct the configuration(s) given an end-effector pose.375 376In the simplified case here considered (for which $\boldsymbol{p} \equiv p$, as the orientation of the end-effector is disregarded for simplicity), one can solve the problem of controlling the end-effector’s location to reach a goal position $p^*$ by solving analytically for $q: p(q) = f_{\text{FK}}(q) = p^*$. However, in the general case, one might not be able to solve this problem analytically, and can typically resort to iterative optimization methods comparing candidate solutions using a loss function (in the simplest case, $\Vert p(q) - p^* \Vert_2^2$ is a natural candidate), yielding:377 378<span id="ik_problem" style="position: absolute;">379</span>380 381``` math382\begin{align}383\min_{q \in \mathcal Q} \Vert p(q) - p^* \Vert_2^2 \, .384 385\end{align}386```387 388Exact analytical solutions to IK are even less appealing when one considers the presence of obstacles in the robot’s workspace, resulting in constraints on the possible values of $q \in \mathcal Q \subseteq [-\pi, +\pi]^n \subset \mathbb R^n$ in the general case of $n$-links robots.389 390For instance, the robot in Figure <a href="#planar-manipulator-floor" data-reference-type="ref" data-reference="planar-manipulator-floor">[planar-manipulator-floor]</a> is (very naturally) obstacled by the presence of the surface upon which it rests: $\theta_1$ can now exclusively vary within $[0, \pi]$, while possible variations in $\theta_2$ depend on $\theta_1$ (when $\theta_1 \to 0$ or $\theta_1 \to \pi$, further downwards movements are restricted). Even for a simplified kinematic model, developing techniques to solve eq. <a href="#ik_problem" data-reference-type="ref" data-reference="ik_problem">[ik_problem]</a> is in general non-trivial in the presence of constraints, particularly considering that the feasible set of solutions $\mathcal Q$ may change across problems. Figure <a href="#planar-manipulator-floor-shelf" data-reference-type="ref" data-reference="planar-manipulator-floor-shelf">[planar-manipulator-floor-shelf]</a> provides an example of how the environment influences the feasible set considered, with a new set of constraints deriving from the position of a new obstacle.391 392However, IK--solving eq. <a href="#ik_problem" data-reference-type="ref" data-reference="ik_problem">[ik_problem]</a> for a feasible $q$--only proves useful in determining information regarding the robot’s configuration in the goal pose, and crucially does not provide information on the *trajectory* to follow over time to reach a target pose. Expert-defined trajectories obviate to this problem providing a length-$K$ succession of goal poses $\tau_K = [p^*_0, p^*_1, \dots p^*_K]$ for tracking. In practice, trajectories can also be obtained automatically through *motion planning* algorithms, thus avoiding expensive trajectory definition from human experts. However, tracking $\tau_K$ via IK can prove prohibitively expensive, as tracking would require $K$ resolutions of eq. <a href="#ik_problem" data-reference-type="ref" data-reference="ik_problem">[ik_problem]</a> (one for each target pose). *Differential* inverse kinematics (diff-IK) complements IK via closed-form solution of a variant of eq. <a href="#ik_problem" data-reference-type="ref" data-reference="ik_problem">[ik_problem]</a>. Let $J(q)$ denote the Jacobian matrix of (partial) derivatives of the FK-function $f_\text{FK}- \mathcal Q \mapsto \mathcal P$, such that $J(q) = \frac{\partial f_{FK}(q)}{\partial q }$. Then, one can apply the chain rule to any $p(q) = f_{\text{FK}}(q)$, deriving $\dot p = J(q) \dot q$, and thus finally relating variations in the robot configurations to variations in pose, thereby providing a platform for control.393 394Given a desired end-effector trajectory $\dot {p}^*(t)$ (1) indicating anchor regions in space and (2) how much time to spend in each region, diff-IK finds $\dot q(t)$ solving for joints’ *velocities* instead of *configurations*,395<span id="reg_ik_velocity" style="position: absolute;">396</span>397 398``` math399\begin{align}400\dot q(t) = \arg\min_\nu \; \lVert J(q(t)) \nu - \dot {p}^*(t) \rVert_2^2401 402\end{align}403```404 405Unlike eq. <a href="#ik_problem" data-reference-type="ref" data-reference="ik_problem">[ik_problem]</a>, solving for $\dot q$ is much less dependent on the environment (typically, variations in velocity are constrained by physical limits on the actuators). Conveniently, eq. <a href="#reg_ik_velocity" data-reference-type="ref" data-reference="reg_ik_velocity">[reg_ik_velocity]</a> also often admits the closed-form solution $\dot q = J(q)^+ \dot {p}^*$, where $J^+(q)$ denotes the Moore-Penrose pseudo-inverse of $J(q)$. Finally, discrete-time joint configurations $q$ can be reconstructed from joint velocities $\dot q$ using forward-integration on the continuous-time joint velocity , $q_{t+1} = q_t + \Delta t\,\dot q_t$ for a given $\Delta t$, resulting in tracking via diff-IK.406 407Following trajectories with diff-IK is a valid option in well-controlled and static environments (e.g., industrial manipulators in controlled manufacturing settings), and relies on the ability to define a set of target velocities to track $[\dot {p}^*_0, \dot {p}^*_1, \dots, \dot {p}^*_k ]$--an error-prone task largely requiring human expertise. Furthermore, diff-IK relies on the ability to (1) access $J(q) \, \forall q \in \mathcal Q$ and (2) compute its pseudo-inverse at every iteration of a given control cycle--a challenging assumption in highly dynamical settings, or for complex kinematic chains.408 409#### Adding Feedback Loops410 411While very effective when a goal trajectory has been well specified, the performance of diff-IK can degrade significantly in the presence of modeling/tracking errors, or in the presence of non-modeled dynamics in the environment.412 413<div class="wrapfigure">414 415r0.3 <Image416 src={ch2_planar_manipulator_floor_box}417 zoomable418 downloadable419 alt="image"420/>421 422<span id="planar-manipulator-box-velocity" style="position: absolute;"></span>423</div>424 425One such case is presented in Figure <a href="#planar-manipulator-box-velocity" data-reference-type="ref" data-reference="planar-manipulator-box-velocity">[planar-manipulator-box-velocity]</a>, where another rigid body other than the manipulator is moving in the environment along the horizontal axis, with velocity $\dot x_B$. Accounting analytically for the presence of this disturbance--for instance, to prevent the midpoint of the link from ever colliding with the object--requires access to $\dot x_B$ at least, to derive the equation characterizing the motion of the environment.426 427Less predictable disturbances however (e.g., $\dot x_B \leftarrow \dot x_B + {\varepsilon}, {\varepsilon}\sim N(0,1)$) may prove challenging to model analytically, and one could attain the same result of preventing link-object collision by adding a condition on the distance between the midpoint of $l$ and $x_B$, enforced through a feedback loop on the position of the robot and object at each control cycle.428 429To mitigate the effect of modeling errors, sensing noise and other disturbances, classical pipelines indeed do augment diff-IK with feedback control looping back quantities of interest. In practice, following a trajectory with a closed feedback loop might consist in backwarding the error between the target and measured pose, $\Delta p = p^*- p(q)$, hereby modifying the control applied to $\dot q = J(q)^+ (\dot {p}^*+ k_p \Delta p )$, with $k_p$ defined as the (proportional) gain.430 431More advanced techniques for control consisting in feedback linearization, PID control, Linear Quatratic Regulator (LQR) or Model-Predictive Control (MPC) can be employed to stabilize tracking and reject moderate perturbations, and we refer to for in-detail explanation of these concepts, or for a simple, intuitive example in the case of a point-mass system. Nonetheless, feedback control presents its challenges as well: tuning gains remains laborious and system-specific. Further, manipulation tasks present intermittent contacts inducing hybrid dynamics (mode switches) and discontinuities in the Jacobian, challenging the stability guarantees of the controller and thus often necessitating rather conservative gains and substantial hand-tuning.432 433We point the interested reader to , , and for extended coverage of FK, IK, diff-IK and control for (diff-)IK.434 435### Limitations of Dynamics-based Robotics436 437Despite the last 60+ years of robotics research, autonomous robots are still largely incapable of performing tasks at human-level performance in the physical world generalizing across (1) robot embodiments (different manipulators, different locomotion platforms, etc.) and (2) tasks (tying shoe-laces, manipulating a diverse set of objects). While essential in the early development of robotics, the aforementioned methods require significant human expertise to be used in practice, and are typically specific to a particular applicative problem.438 439<Image440 src={ch2_classical_limitations}441 zoomable442 downloadable443 alt="Figure"444 caption={'Dynamics-based approaches to robotics suffer from several limitations: (1) orchestrating multiple components poses integration challenges; (2) the need to develop custom processing pipelines for the sensing modalities and tasks considered hinders scalability; (3) simplified analytical models of physical phenomena (here friction at the gripper; credits to @antonovaReinforcementLearningPivoting2017) limit real-world performance. Lastly, (4) dynamics-based methods overlook trends in the availability and growth of robotics data.'}/>445 446Dynamics-based robotics pipelines have historically been <mark>developed sequentially, engineering the different blocks</mark> now within most architectures for specific purposes. That is, sensing, state estimation, mapping, planning, (diff-)IK, and low-level control have been traditionally developed as distinct modules with fixed interfaces. Pipelining these specific modules proved error-prone, and brittleness emerges--alongside compounding errors--whenever changes incur (e.g., changes in lighting for sensing, occlusion/failure of sensors, control failures). Adapting such a stack to new tasks or robotic platforms often entails re-specifying objectives, constraints, and heuristics at multiple stages, incurring significant engineering overhead.447 448Moreover, classical planners operate on compact, assumed-sufficient state representations; extending them to reason directly over raw, heterogeneous and noisy data streams is non-trivial. This results in a <mark>limited scalability to multimodal data and multitask settings</mark>, as incorporating high-dimensional perceptual inputs (RGB, depth, tactile, audio) traditionally required extensive engineering efforts to extract meaningful features for control. Also, the large number of tasks, coupled with the adoption of *per-task* planners, goal parameterizations, and safety constraints, results in an explosion in design and validation options, with little opportunity to reuse solutions across tasks.449 450Setting aside integration and scalability challenges: developing accurate modeling of contact, friction, and compliance for complicated systems remains difficult. Rigid-body approximations are often insufficient in the presence of deformable objects, and <mark>relying on approximated models hinders real-world applicability</mark> of the methods developed. In the case of complex, time-dependent and/or non-linear dynamics, even moderate mismatches in parameters, unmodeled evolutions, or grasp-induced couplings can qualitatively affect the observed dynamics.451 452Lastly, dynamics-based methods (naturally) overlook the rather recent <mark>increase in availability of openly-available robotics datasets</mark>. The curation of academic datasets by large centralized groups of human experts in robotics @oneillOpenXEmbodimentRobotic2025, @khazatskyDROIDLargeScaleInTheWild2025 is now increasingly complemented by a <mark>growing number of robotics datasets contributed in a decentralized fashion</mark> by individuals with varied expertise. If not tangentially, dynamics-based approaches are not posed to maximally benefit from this trend, which holds the premise of allowing generalization in the space of tasks and embodiments, like data was the cornerstone for advancements in vision @alayracFlamingoVisualLanguage2022 and natural-language understanding @brownLanguageModelsAre2020.453 454Taken together, these limitations (Figure <a href="#classical-limitations" data-reference-type="ref" data-reference="classical-limitations">[classical-limitations]</a>) motivate the exploration of learning-based approaches that can (1) integrate perception and control more tightly, (2) adapt across tasks and embodiments with reduced expert modeling interventions and (3) scale gracefully in performance as more robotics data becomes available.455 456## Robot (Reinforcement) Learning457 458<span id="learning-rl" style="position: absolute;"></span>459<Quote source="Richard Sutton">460 Approximate the solution, not the problem461</Quote>462<div class="callout">463 464TL;DR The need for expensive, high-fidelity simulators can be obviated learning from real-world data, using sample-efficient algorithms that can safely train directly on hardware.465 466</div>467<Image468 src={ch3_learning_benefits}469 zoomable470 downloadable471 alt="Figure"472 caption={'Learning-based robotics streamlines perception-to-action by learning a (1) unified high-level controller capable to take (2) high-dimensional, unstructured sensorimotor information. Learning (3) does not require a dynamics model and instead focuses on interaction data, and (4) empirically correlates with the scale of the data used.'}/>473 474Learning-based techniques for robotics naturally address the limitations presented in Section <a href="#classical" data-reference-type="ref" data-reference="classical">[classical]</a> (Figure <a href="#robot-learning-upsides" data-reference-type="ref" data-reference="robot-learning-upsides">[robot-learning-upsides]</a>). In particular, learning-based techniques typically rely on monolithich prediction-to-action pipelines (*visuomotor policies*) which do directly map sensorimotor inputs to predicted actions, streamlining control policies by removing the need to interface multiple components. Mapping sensory inputs to actions also makes it possible to incorporate diverse input modalities, leveraging the automatic feature extraction capabilities of modern learning systems. Moreover, learning-based approaches can, in principle, bypass explicit modeling altogether and instead rely solely on interaction data--an advantage that proves transformative when dynamics are difficult to model or entirely unknown. Lastly, learning for robotics (*robot learning*) is naturally well posed to leverage the growing amount of robotics data openly available, just as computer vision and natural language processing did historically benefit from large-scale corpora of data, in great part overlooked by dynamics-based approaches.475 476Being a field at its relative nascent stages, no prevalent technique(s) proves distinctly better than any other in the domain of robot learning. Still, two major classes of methods gained prominence- <mark>Reinforcement Learning (RL)</mark> and <mark>Behavioral Cloning (BC)</mark> (Figure <a href="#robot-learning-atlas" data-reference-type="ref" data-reference="robot-learning-atlas">[robot-learning-atlas]</a>). In this section, we provide a conceptual overview of applications of RL to robotics, as well as introduce practical examples of how to use RL within `lerobot`. We then introduce the major limitations RL suffers from, to introduce BC techniques in Section <a href="#learning-imitation" data-reference-type="ref" data-reference="learning-imitation">[learning-imitation]</a> and Section sec-learning-foundation.477 478<div class="wrapfigure">479 480r0.3 <Image481 src={ch3_learning_atlas}482 zoomable483 downloadable484 alt="image"485/>486 487<span id="robot-learning-atlas" style="position: absolute;"></span>488</div>489 490In Figure <a href="#robot-learning-atlas" data-reference-type="ref" data-reference="robot-learning-atlas">[robot-learning-atlas]</a> we deliberately include generalist robot models @blackp0VisionLanguageActionFlow2024, @shukorSmolVLAVisionLanguageActionModel2025 alongside task-specific BC methods. While significantly different in spirit--*generalist* models are language-conditioned and use instructions to generate motion valid across many tasks, while *task-specific* models are typically not language-conditioned and used to perform a single task--*foundation* models are still largely trained to reproduce trajectories contained in a (large) training set of input demonstrations. Thus, we argue generalist policies can indeed be grouped alongside other task-specific BC methods, as they both leverage similar training data and schemas. Figure <a href="#robot-learning-atlas" data-reference-type="ref" data-reference="robot-learning-atlas">[robot-learning-atlas]</a> illustrates this categorization graphically, explicitly listing all the robot learning policies currently available in `lerobot`- Action Chunking with Transformers (ACT) @zhaoLearningFineGrainedBimanual2023, Diffusion Policy @chiDiffusionPolicyVisuomotor2024, Vector-Quantized Behavior Transformer (VQ-BeT) @leeBehaviorGenerationLatent2024, $\pi_0$ @blackp0VisionLanguageActionFlow2024, SmolVLA @shukorSmolVLAVisionLanguageActionModel2025, Human-in-the-loop Sample-efficient RL (HIL-SERL) @luoPreciseDexterousRobotic2024 and TD-MPC @hansenTemporalDifferenceLearning2022.491 492<Image493 src={ch3_rl_examples}494 zoomable495 downloadable496 alt="Figure"497 caption={'Examples of two different robotics tasks performed using RL. In the manipulation task (A) an agent learns to reach for a yellow plastic block in its environment, and to put it inside of a box. In the locomotion task (B) an agent learns to move its center of mass sideways without falling.'}/>498 499Applications of RL to robotics have been studied long enough that the relationship between these two disciplines has been compared to that of physics and matematics @koberReinforcementLearningRobotics. Indeed, due to their inherently interactive and sequential nature, robotics control problems can be directly cast as RL problems. Figure <a href="#robotics-with-rl-examples" data-reference-type="ref" data-reference="robotics-with-rl-examples">[robotics-with-rl-examples]</a> presents two of such cases. Reaching for an object to then move it somewhere else in the scene is a sequential problem where over time the controller needs to adjust the position of the robot arm based on the current configuration and the (possibly varying) position of the object. Figure <a href="#robotics-with-rl-examples" data-reference-type="ref" data-reference="robotics-with-rl-examples">[robotics-with-rl-examples]</a> also shows an example of a locomotion problem, where sequentiality is inherent in the problem formulation- while sliding to the side, the controller needs to keep adjusting to the robot’s to avoid failure (falling).500 501### A (Concise) Introduction to RL502 503The RL framework @suttonReinforcementLearningIntroduction2018, which we briefly introduce here, has often been used to tackle robotics problems @koberReinforcementLearningRobotics. RL is a subfield within ML fundamentally concerned with the development of autonomous systems (*agents*) capable to *continuously behave* in an evolving environment, developing (ideally, well-performing) control strategies (*policies*). Crucially for robotics, RL agents improve through trial and error, bypassing explicit models of the problem dynamics in favor of interaction data. In RL, this feedback loop between actions and outcomes (Figure <a href="#rl-most-famous-pic" data-reference-type="ref" data-reference="rl-most-famous-pic">[rl-most-famous-pic]</a>) is established through the agent sensing a scalar quantity (*reward*) measuring how desirable a given *transition* is for the accomplishment of its goal.504 505<Image506 src={ch3_agent_env}507 zoomable508 downloadable509 alt="Figure"510 caption={'Agent-Environment interaction diagram (image credits to @suttonReinforcementLearningIntroduction2018).'}/>511 512Formally, interactions between an agent and its environment are typically modeled via a Markov Decision Process (MDP) @bellmanMarkovianDecisionProcess1957. Representing robotics problems via MDPs offers several advantages, including (1) incorporating uncertainty through MDP’s inherently stochastic formulation and (2) providing a theoretically-sound framework for learning *without* an explicit model of the environment dynamics. While accommodating a continuous time formulation too, MDPs are typically considered in discrete time in RL, assuming interactions to atomically take place at discrete *timestep* $t=0,1,2,3, \dots, T$. MDPs allowing for an unbounded number of interactions ($T \to + \infty$) are termed *infinite-horizon*, and opposed to *finite-horizon* MDPs in which $T$ is finite. Unless diversely specified, we will only be referring to discrete-time finite-horizon (*episodic*) MDPs.513 514Formally, a lenght-$T$ Markov Decision Process (MDP) is a tuple $\mathcal M = \langle \mathcal S, \mathcal A, \mathcal D, r, \gamma, \rho, T \rangle$, where:515 516- $\mathcal S$ is the *state space*; $s_t\in \mathcal S$ denotes the (possibly non-directly observable) environment state at time $t$. In robotics, states often comprise robot configuration and velocities ($q_t, \dot q_t$), and can also accomodate sensor readings such as camera or audio streams.517 518- $\mathcal A$ is the *action space*; $a_t\in \mathcal A$ may represent joint torques, joint velocities, or even end-effector commands at timestep $t$. In general, actions correspond to commands intervenings on the configuration of the robot.519 520- $\mathcal D$ represents the (possibly non-deterministic) environment dynamics, with $\mathcal D: \mathcal S\times \mathcal A\times \mathcal S\mapsto [0, 1]$, $\mathcal D\, (s_t, a_t, s_{t+1})= \mathbb P (s_{t+1}\vert s_t, a_t)$. For instance, for a planar manipulator dynamics could be considered deterministic when the environment is fully described (Figure <a href="#planar-manipulation-simple" data-reference-type="ref" data-reference="planar-manipulation-simple">[planar-manipulation-simple]</a>), and stochastic when unmodeled disturbances depending on non-observable parameters intervene (Figure <a href="#planar-manipulator-box-velocity" data-reference-type="ref" data-reference="planar-manipulator-box-velocity">[planar-manipulator-box-velocity]</a>).521 522- $r- \mathcal S\times \mathcal A\times \mathcal S\to \mathbb R$ is the *reward function*, weighing the transition $(s_t, a_t, s_{t+1})$ in the context of the achievement of an arbitrary goal. For instance, a simple reward function for quickly moving along the $x$ axis (Figure <a href="#robotics-with-rl-examples" data-reference-type="ref" data-reference="robotics-with-rl-examples">[robotics-with-rl-examples]</a>) could be based on the absolute position of the robot along the $x$ axis ($p_{x_t}$), present negative penalties for falling over (measured from $p_{z_t}$) and a introduce bonuses $\dot p_{x_t}$ for speed, $r (s_t, a_t, s_{t+1})\equiv r(s_t) = p_{x_t} \cdot \dot p_{x_t} - \tfrac{1}{p_{z_t}}$.523 524Lastly, $\gamma \in [0,1)$ represent the discount factor regulating preference for immediate versus long-term reward (with an effective horizon equal to $\tfrac{1}{1-\gamma}$), and $\rho$ is the distribution over $\mathcal S$ for the MDP’s *initial*, $s_0 \sim \rho$.525 526Therefore, a length-$T$ *trajectory* is the (random) sequence527``` math528\htmlId{trajectory_definition}{\tau = (s_0, a_0, r_0, s_1, a_1, r_1, \dots, s_{T-1}, a_{T-1}, r_{T-1}, s_T),}529```530with per-step rewards defined as $r_t = r (s_t, a_t, s_{t+1})$ for ease of notation. Interestingly, assuming both the environment dynamics and conditional distribution over actions given states--i.e., the *policy*--to be *Markovian*:531<span id="dynamics_markovian" style="position: absolute;">532</span>533 534``` math535\begin{align}536\mathbb P(s_{t+1}\vert s_t, a_t, s_{t-1}, a_{t-1}, \dots s_0, a_0 ) &= \mathbb P (s_{t+1}\vert s_t, a_t) \\537\mathbb P(a_t\vert s_t, a_{t-1}, s_{t-1}, s_0, a_0) &= \mathbb P(a_t\vert s_t), 538\end{align}539```540 541the probability of observing a given trajectory $\tau$ factorizes into:542 543``` math544\htmlId{traj_prob}{\mathbb P(\tau) = \mathbb P (s_0) \prod_{t=0}^{T-1} \mathbb P (s_{t+1}\vert s_t, a_t)\ \mathbb P(a_t\vert s_t).}545```546 547Policies $\mathbb P(a_t\vert s_t)$ are typically indicated as $\pi(a_t\vert s_t)$, often parametrized via $\theta$, yielding $\pi_\theta (a_t\vert s_t)$, and are traine by optimizing the (discounted) *return* associated to a given $\tau$, i.e. the (random) sum of measured rewards over an arbitrary trajectory,548``` math549G(\tau) = \sum_{t=0}^{T-1} \gamma^{t} r_t.550```551In that, agents seek to learn control strategies (*policies*, $\pi_\theta$) maximizing the expected return $\mathbb E_{\tau \sim \pi_\theta} G(\tau)$. For a given dynamics $\mathcal D$--i.e., for a given problem--taking the expectation over the (possibly random) trajectories resulting from acting according to a certain policy provides a direct, goal-conditioned ordering in the space of all the possible policies $\Pi$, yielding the (maximization) target $J : \Pi \mapsto \mathbb R$552<span id="RL-j-function" style="position: absolute;">553</span>554 555``` math556\begin{align}557 J(\pi_\theta) &= \mathbb E_{\tau \sim \mathbb P_{\theta; \mathcal D}} [G(\tau)], \\558 \mathbb P_{\theta; \mathcal D} (\tau) &= \rho \prod_{t=0}^{T-1} \mathcal D (s_t, a_t, s_{t+1})\ \pi_\theta (a_t\vert s_t).559\end{align}560```561 562Crucially, in the RL framework the agent is assumed to only *observe* the environment dynamics and not to intervene on them, and thus eq. <a href="#RL-j-function" data-reference-type="ref" data-reference="RL-j-function">[RL-j-function]</a> varies exclusively with the policy followed. In turn, MDPs naturally provide a framework to optimize over the space of the possible behaviors an agent might enact ($\pi \in \Pi$), searching for the *optimal policy* $\pi^* = \arg \max_{\theta} J(\pi_\theta)$, where $\theta$ is the parametrization adopted by the policy set $\Pi: \pi_\theta \in \Pi, \ \forall \theta$. Besides providing a target for policy search, $G(\tau)$ can also be used to discriminate between states $s_t$ and $s_t, a_t$ pairs. Given any state $s \in \mathcal S$--e.g., given a configuration $q$ of a robot--the *state-value* function563 564``` math565V_\pi(s) = \mathbb E_{\tau \sim \pi} [G(\tau) \big \vert s_0 = s]566```567can be used to discriminate between desirable and undesirable state in terms of long-term (discounted) reward maximization, under a given policy $\pi$. Similarily, the *state-action* value function also conditions the cumulative discounted reward on selecting action $a$ when in $s$, and thereafter act according to $\pi$,568``` math569Q_\pi(s,a) = \mathbb E_{\tau \sim \pi} [G (\tau) \big \vert s_0 = s, a_0=a].570```571Importantly, value functions are interrelated:572<span id="q-as-v" style="position: absolute;">573</span>574 575``` math576\begin{align}577Q_\pi(s_t, a_t) &= \mathbb{E}_{s_{t+1}\sim \mathbb P(\bullet \vert s_t, a_t)} [r_t + \gamma V_\pi(s_{t+1})] \\578V_\pi(s_t) &= \mathbb E_{a_t\sim \pi(\bullet \vert s_t)} [Q_\pi (s_t, a_t)],579 580\end{align}581```582inducing an ordering over states and state-action pairs under $\pi$, and value functions are thus central to most RL algorithms. A variety of algorithms have been developed in RL attempting to find (approximate) solutions to the problem of maximizing cumulative reward (we report some in Figure <a href="#rl-algos-atlas" data-reference-type="ref" data-reference="rl-algos-atlas">[rl-algos-atlas]</a>).583 584<Image585 src={ch3_rl_algorithms_atlas}586 zoomable587 downloadable588 alt="Figure"589 caption={'Popular RL algorithms. See @SpinningUp2018 for a complete list of citations.'}/>590 591Popular approaches to continuous state and action space--such as those studied within robotics--include , and . Across manipulation @akkayaSolvingRubiksCube2019 and locomotion problems @leeLearningQuadrupedalLocomotion2020, RL proved extremely effective in providing a platform to (1) leverage a unified, streamlined perception-to-action pipeline, (2) natively integrate propioperception with multi-modal high-dimensional sensory streams (3) disregard a description of the environment dynamics, by focusing on observed interaction data rather than modeling, and (4) anchor policies in the experience collected and stored in datasets. For a more complete survey of applications of RL to robotics, we refer the reader to @koberReinforcementLearningRobotics, @tangDeepReinforcementLearning2025.592 593### Real-world RL for Robotics594 595Streamlined end-to-end control pipelines, data-driven feature extraction and a disregard for explicit modeling in favor of interaction data are all features of RL for robotics. However, RL still suffers from limitations concerning safety and learning efficiency, particularly pressing for real-world robotics applications.596 597First, especially early in training, <mark>actions are typically explorative, and thus may be erractic</mark>. On physical systems, untrained policies may command high velocities, self-collisiding configurations, or torques exceeding joint limits, leading to wear and potential hardware damage. Mitigating these risks requires external safeguards (e.g., watchdogs, safety monitors, emergency stops), often incuring in a high degree of human supervision. Further, in the typical episodic setting considered in most robotics problems, experimentation is substantially slowed down by the need to manually reset the environment over the course of training, a time-consuming and error-prone process. Second, learning efficiently remains problematic in RL, <mark>limiting the applicability of RL in real-world robotics due to consequently prohibitive timescales of training</mark>. Even strong algorithms such as SAC @haarnojaSoftActorCriticOffPolicy2018 typically require a large numbers of transitions $\{ (s_t, a_t, r_t, s_{t+1})\}_{t=1}^N$. On real-world hardware, generating this data is time-consuming.598 599<Image600 src={ch3_duck_sim_vs_real}601 zoomable602 downloadable603 alt="Figure"604 caption={'Simulated (left) vs. real-world (right) OpenDuck. Discrepancies in the simulation dynamics (reality gap) pose risks to policy transfer.'}/>605 606Training RL policies in simulation @tobinDomainRandomizationTransferring2017 addresses both issues, eliminating physical risk and dramatically increasing throughput. Yet, simulators require significant modeling effort, and rely on assumptions (simplified physical modeling, instantaneous actuation, static environmental conditions, etc.) limiting the possibilities to transfer the policies learned in simulation, due the discrepancy between real and simulated environments (*reality gap*, Figure <a href="#synthetic-vs-real-duck" data-reference-type="ref" data-reference="synthetic-vs-real-duck">[synthetic-vs-real-duck]</a>). *Domain randomization* @tobinDomainRandomizationTransferring2017 (DR) is a popular technique to overcome the reality gap, and consists in randomizing the parameters of the simulated environment during training, aiming at inducing robustness to specific disturbances. In this, DR is typically employed to increase the diversity of scenarios over the course of training, improving on the performace sim-to-real transferred policies @akkayaSolvingRubiksCube2019, @antonovaReinforcementLearningPivoting2017, @jiDribbleBotDynamicLegged2023. In practice, DR is performed training in simulation on simulated dynamics $\mathcal D$, further parametrized as $\mathcal D \equiv \mathcal D_\xi$, with a *dynamics* (random) vector $\xi$ drawn an arbitrary distribution, $\xi \sim \Xi$. For instance, one could decide to randomize the friction coefficient of the surface in a locomotion task (Figure <a href="#ducks-on-terrains" data-reference-type="ref" data-reference="ducks-on-terrains">[ducks-on-terrains]</a>), or the center of mass of an object for a manipulation task. Over the course of training--typically at each episode’s reset--a new $\xi$ is drawn, and used to specify the environment’s dynamics for that episode.607 608<Image609 src={ch3_many_ducks}610 zoomable611 downloadable612 alt="Figure"613 caption={'The same locomotion task can be carried out in different (simulated) domains (exemplified by the difference in terrains) at training time, resulting to increased robustness over diverse environment dynamics.'}/>614 615While effective in transfering policies across the reality gap in real-world robotics @tobinDomainRandomizationTransferring2017, @akkayaSolvingRubiksCube2019, @jiDribbleBotDynamicLegged2023, @tiboniDomainRandomizationEntropy2024, DR often requires extensive manual engineering. First, identifying which parameters to randomize--i.e., the *support* $\text{supp} (\Xi)$ of $\Xi$--is an inherently task specific process. When locomoting over different terrains, choosing to randomize the friction coefficient is a reasonable choice, yet not completely resolutive as other factors (lightning conditions, external temperature, joints’ fatigue, etc.) may prove just as important in practice, making selecting these parameters yet another source of brittlness.616 617Selecting the dynamics distribution $\Xi$ is also non-trivial. On the one hand, distributions with low entropy might risk to cause failure at transfer time, due to the limited robustness induced over the course of training. On the other hand, excessive randomization may cause over-regularization and hinder performance @margolisRapidLocomotionReinforcement2022. Consequently, the research community investigated approaches to automatically select the randomization distribution $\Xi$, using signals from the training process or tuning it to reproduce observed real-world trajectories. @akkayaSolvingRubiksCube2019 use a parametric uniform distribution $\mathcal U(a, b)$ as $\Xi$, widening the bounds $a, b$ as training progresses and the agent’s performance improves (AutoDR). While effective, AutoDR requires significant tuning--the bounds are widened by a fixed, pre-specified amount $\Delta$ along--and may disregard data when performance *does not* improve after a distribution update @tiboniDomainRandomizationEntropy2024. @tiboniDomainRandomizationEntropy2024 propose a similar method to AutoDR (DORAEMON) to evolve $\Xi$ based on the training signal, but with the key difference of explicitly maximizing the entropy of a parametric Beta distribution--inherently more flexible than uniform distributions--with learned updates instead of fixed $\Delta$. In this, DORAEMON proves particularly effective at dynamically increasing the entropy levels of the training distribution by employing an outer-loop max-entropy objective, tackled under performance constraints in the inner-loop RL problem. Other approaches to automatically perform DR consist in specifically tuning $\Xi$ to align as much as possible the simulation and real-world domains. For instance, @chebotarClosingSimtorealLoop2019 interleave in-simulation policy training with repeated real-world policy rollouts used to adjust $\Xi$ based on real-world data, while @tiboniDROPOSimtoRealTransfer2023 leverage a single, pre-collected set of real-world trajectories and tune $\Xi$ under a simple likelihood objective.618 619While DR has shown promise, it does not address the main limitation that, even under the assumption that an ideal distribution $\Xi$ was available, many robotics problems <mark>cannot be simulated with high-enough fidelity under practical computational constraints</mark>. Simulating contact-rich manipulation of possibly deformable or soft materials--i.e., *folding a piece of clothing*--can prove time-intensive, limiting the benefits of in-simulation training.620 621A perhaps more foundamental limitation of RL for robotics is the general unavailability of complicated tasks’ *dense* reward function, the design of which is essentially based on human expertise, ingenuity and trial-and-error. In practice, *sparse* reward functions can be used to conclude whether one specific goal has been attained--*has this t-shirt been correctly folded?*--but unfortunately incur in more challenging learning. As a result, despite notable successes, deploying RL directly on real-world robots at scale remains challenging.622 623To make the most of (1) the growing number of openly available datasets and (2) relatively inexpensive robots like the SO-100, RL could (1) be anchored in already-collected trajectories--limiting erratic and dangerous exploration--and (2) train in the real-world directly--bypassing the aforementioned issues with low-fidelity simulations. In such a context, sample-efficient learning is also paramount, as training on the real-world is inherently time-bottlenecked.624 625Off-policy algorithms like Soft Actor-Critic (SAC) @haarnojaSoftActorCriticOffPolicy2018 tend to be more sample efficient then their on-policy counterpart @schulmanProximalPolicyOptimization2017, due to the presence a *replay buffer* used over the course of training. Other than allowing to re-use past transitions $(s_t, a_t, r_t, s_{t+1})$, the replay buffer can also accomodate for the injection of previously-collected data in the training process @ballEfficientOnlineReinforcement2023. Using expert demonstrations to guide learning together with learned rewards, RL can be effectively carried out in the real-world @luoSERLSoftwareSuite2025. Interestingly, when complemented with in-training human interventions, real-world RL agents have been shown to learn policies with near-perfect success rates on challenging manipulation tasks in 1-2 hours @luoPreciseDexterousRobotic2024.626 627##### Sample-efficient RL628 629In an MDP, the optimal policy $\pi^*$ can be derived from its associated $Q$-function, $Q^* \equiv Q_{\pi^*}$, and in particular the optimal action(s) $\mu(s_t)$ can be selected maximizing the optimal $Q$-function over the action space,630``` math631\mu(s_t) = \max_{a_t\in \mathcal A} Q^*(s_t, a_t).632```633Interestingly, the $Q^*$-function satisfies a recursive relationship (*Bellman equation*) based on a very natural intuition [^2]:634 635> \[...\] If the optimal value $Q^*(s_{t+1}, a_{t+1})$ of the \[state\] $s_{t+1}$ was known for all possible actions $a_{t+1}$, then the optimal strategy is to select the action $a_{t+1}$ maximizing the expected value of $r_t + \gamma Q^*(s_{t+1}, a_{t+1})$636> ``` math637> Q^*(s_t, a_t) = \mathbb E_{s_{t+1} \sim \mathbb P(\bullet \vert s_t, a_t)} [r_t + \gamma \max_{a_{t+1} \in \mathcal A} Q^*(s_{t+1}, a_{t+1}) \big\vert s_t, a_t]638> ```639 640In turn, the optimal $Q$-function is guaranteed to be self-consistent by definition. *Value-iteration* methods exploit this relationship (and/or its state-value counterpart, $V^*(s_t)$ ) by iteratively updating an initial estimate of $Q^*$, $Q_k$ using the Bellman equation as update rule (*Q-learning*):641``` math642Q_{i+1}(s_t, a_t) \leftarrow \mathbb E_{s_{t+1} \sim \mathbb P(\bullet \vert s_t, a_t)} [r_t + \gamma \max_{a_{t+1} \in \mathcal A} Q_i (s_{t+1}, a_{t+1}) \big\vert s_t, a_t], \quad i=0,1,2,\dots,K643```644Then, one can derive the (ideally, near-optimal) policy by explicitly maximizing over the action space the final (ideally, near-optimal) estimate $Q_K \approx Q^*$ at each timestep. Indeed, one can show that under certain assumptions on the MDP considered, $Q_K \to Q^* \, \text{as } K \to \infty$.645 646Effective in its early applications to small-scale discrete problems, vanilla Q-learning was found complicated to scale to large $\mathcal S\times \mathcal A$ problems, in which storing $Q : \mathcal S\times \mathcal A\mapsto \mathbb R$ alone might result prohibitive. Also, vanilla Q-learning is not directly usable for *continuous*, unstructured state-action space MPDs, such as those considered in robotics. In their seminal work on *Deep Q-Learning* (DQN), @mnihPlayingAtariDeep2013 propose learning Q-values using deep convolutional neural networks, thereby accomodating for large and even unstructured *state* spaces. DQN parametrizes the Q-function using a neural network with parameters $\theta$, updating the parameters by sequentially minimizing the expected squared temporal-difference error (TD-error, $\delta_i$):647<span id="dqn-loss" style="position: absolute;">648</span>649 650``` math651\begin{align}652\mathcal L(\theta_i) &= \mathbb E_{(s_t, a_t) \sim \chi(\bullet)} 653 \big[ 654 (\underbrace{y_i - Q_{\theta_i}(s_t, a_t)}_{\delta_i})^2 655 \big], \\656 y_i &= \mathbb E_{s_{t+1} \sim \mathbb P(\bullet \vert s_t, a_t)} \big[ r_t + \gamma \max_{a_t\in \mathcal A} Q_{\theta_{i-1}} (s_{t+1}, a_{t+1}) \big], 657\end{align}658```659where $\chi$ represents a behavior distribution over state-action pairs. Crucially, $\chi$ can in principle be different from the policy being followed, effectively allowing to reuse prior data stored in a *replay buffer* $D$ in the form of $(s_t, a_t, r_t, s_{t+1})$ transitions, used to form the TD-target $y_i$, TD-error $\delta_i$ and loss function eq. <a href="#dqn-loss" data-reference-type="ref" data-reference="dqn-loss">[dqn-loss]</a> via Monte-Carlo (MC) estimates.660 661While effective in handling large, unstructured state spaces for discrete action-space problems, DQN’s application to continous control problems proved challenging. Indeed, in the case of high-capacity function approximators such as neural networks, solving $\max_{a_t \in \mathcal A} Q_\theta(s_t, a_t)$ at each timestep is simply unfeasible due to the (1) continous nature of the action space ($\mathcal A\subset \mathbb R^n$ for some $n$) and (2) impossibility to express the policy with a cheap (ideally, even closed-form) formulation, so that $\max Q_\theta$ could be solved analytically. @pmlr-v32-silver14 tackle these fundamental challenges by using a *deterministic* function of the state $s_t$ as policy, $\mu_\phi(s_t) = a_t$, parametrized by $\phi$. Thus, policies can be iteratively refined updating $\phi$ along the direction:662``` math663\htmlId{deterministic-pg}{d_\phi = \mathbb E_{s_t \sim \mathbb P (\bullet)} [\nabla_\phi Q(s_t, a_t)\vert_{a_t = \mu_\phi(s_t)}] = \mathbb E_{s_t \sim \mathbb P(\bullet)} [\nabla_{a_t} Q(s_t, a_t) \vert_{a_t = \mu_\phi(s_t)} \cdot \nabla_\phi \mu(s_t)]}664```665 666Provably, eq. <a href="#deterministic-pg" data-reference-type="ref" data-reference="deterministic-pg">[deterministic-pg]</a> is the *deterministic policy gradient* (DPG) of the policy $\mu_\phi$ @pmlr-v32-silver14, so that updates $\phi_{k+1}\leftarrow \phi_k + \alpha d_\phi$ are guaranteed to increase the (deterministic) cumulative discounted reward, $J(\mu_\phi)$. @lillicrapContinuousControlDeep2019a extended DPG to the case of (1) high-dimensional unstructured observations and (2) continuous action spaces, introducing Deep Deterministic Policy Gradient (DDPG), an important algorithm in RL and its applications to robotics. DDPG adopts a modified TD-target compared to eq. <a href="#TD-target" data-reference-type="ref" data-reference="TD-target">[TD-target]</a>, by maintaining a policy network used to select actions, yielding667 668``` math669\htmlId{TD-target-ddpg}{y_i = \mathbb E_{s_{t+1} \sim \mathbb P(\bullet \vert s_t, a_t)} \big[ r_t + \gamma Q_{\theta_{i-1}} (s_{t+1}, \mu_\phi(s_{t+1})) \big] .}670```671Similarily to DQN, DDPG also employs the same replay buffer mechanism, reusing past transitions over training for increased sample efficiency and estimate the loss function via MC-estimates.672 673Soft Actor-Critic (SAC) @haarnojaSoftActorCriticOffPolicy2018 is a derivation of DDPG in the max-entropy (MaxEnt) RL framework, in which RL agents are tasked with <mark>maximizing the discounted cumulative reward, while acting as randomly as possible</mark>. MaxEnt RL @haarnojaReinforcementLearningDeep2017b has proven particularly robust thanks to the development of diverse behaviors, incentivized by its entropy-regularization formulation. In that, MaxEnt revisits the RL objective $J (\pi)$ to specifically account for the policy entropy $\mathcal H(\pi (\bullet \vert s_t))$,674<span id="J-soft" style="position: absolute;">675</span>676 677``` math678\begin{align}679 J(\pi) &= \sum_{t=0}^T \mathbb{E}_{(s_t, a_t) \sim \chi} [r_t + \alpha \mathcal H(\pi (\bullet \vert s_t))].680 681\end{align}682```683 684This modified objective results in the *soft* TD-target:685 686``` math687\htmlId{soft-td-target}{y_i = \mathbb E_{s_{t+1} \sim \mathbb P( \bullet \vert s_t, a_t)} [r_t + \gamma \left( Q_{\theta_{i-1}} (s_{t+1}, a_{t+1}) - \alpha \log \pi_\phi(a_{t+1} \vert s_{t+1}) \right)], \quad a_{t+1} \sim \pi_\phi(\bullet \vert s_t)}688```689Similarily to DDPG, SAC also maintains an explicit policy, trained under the same MaxEnt framework for the maximization of eq. <a href="#J-soft" data-reference-type="ref" data-reference="J-soft">[J-soft]</a>, updated using-690``` math691\htmlId{sac-policy-update}{\pi_{k+1} \leftarrow \arg\min_{\pi^\prime \in \Pi} \text{D}_{\text{KL}}\left(\pi^\prime (\bullet \vert s_t) \bigg\Vert \frac{\exp(Q_{\pi_k}(s_t, \bullet))}{Z_{\pi_k}(s_t)} \right)}692```693The update rule provided in eq. <a href="#sac-policy-update" data-reference-type="ref" data-reference="sac-policy-update">[sac-policy-update]</a> optimizes the policy while projecting it on a set $\Pi$ of tractable distributions (e.g., Gaussians, @haarnojaReinforcementLearningDeep2017b).694 695##### Sample-efficient, data-driven RL696 697Sampling $(s_t, a_t, r_t, s_{t+1})$ from the replay buffer $D$ conveniently allows to approximate expectations for TD-target and TD-error through Monte-Carlo (MC) estimates. The replay buffer $D$ also proves extremely useful in maintaining a history of previous transitions and using it for training, improving on sample efficiency. Furthermore, it also naturally provides an entry point to inject offline trajectories recorded by a human demonstrator into the training process.698 699Reinforcement Learning with Prior Data (RLPD) @ballEfficientOnlineReinforcement2023 is an Offline-to-Online RL algorithm leveraging prior data to effectively accelerate the training of a SAC agent. Unlike previous works on Offline-to-Online RL, RLPD avoids any pre-training and instead only uses the available offline data $D_\text{offline}$ to improve online-learning from scratch. During each training step, transitions from both the offline and online replay buffers are sampled in equal proportions, and used in the underlying SAC routine. Together with other implementation details (using LayerNorm layers to prevent value overestimation, and the use of ensembles techniques to form the TD-target), RLPD proves a particularly simple yet effective approach to use $D_\text{offline}$ for Offline-to-Online RL.700 701##### Sample-efficient, data-driven, real-world RL702 703Despite the possibility to leverage offline data for learning, the effectiveness of real-world RL training is still limited by the need to define a task-specific, hard-to-define reward function. Further, even assuming to have access to a well-defined reward function, typical robotics pipelines rely on augmenting propioperceptive inputs with camera streams, and thus even well-defined rewards would need to be defined starting from unstructured observation--a challenging assumption in practice. In their technical report, @luoSERLSoftwareSuite2025 empirically address the needs (1) to define a reward function and (2) to use it starting from unstructured, image observations. In particular, introduces a suite of tools streamlining training of *reward classifiers* $c$, as well as jointly learn forward-backward controllers to speed up real-world RL.704 705Reward classifiers are particularly useful in treating complex, dynamic tasks--e.g., folding a t-shirt--for which a precise reward formulation is arbitrarily complex to obtain, or that do require significant shaping and are more easily learned directly from demonstrations of success ($e^+$) or failure ($e^-$) states, rather than from a precise formulation of $r_t$, with a natural target for the reward classifier being $r(s) = \log c(e^+ \ vert s )$. Furthermore, @luoSERLSoftwareSuite2025 demonstrate the benefits of learning separate (1) *forward* and (2) *backward* controllers--parametrized by separate policies--where (1) the former learns to execute a task to completion and (2) the latter learns to reset the environment to its initial state from terminal states, thereby aiding training in real-world episodic settings.706 707Lastly, in order to improve on the robustness of their approach to different goals while maintaing practical scalability, @luoSERLSoftwareSuite2025 introduced a modified state and action space, expressing proprioperceptive configurations $q$ and actions $\dot q$ in the frame of the end-effector pose at $t=0$. Randomizing the initial pose of the end-effector ($s_0$), @luoSERLSoftwareSuite2025 achieved a similar result to that of manually randomizing the environment at every timestep, but with the benefit of maintaining the environment in the same condition across multiple training episodes, achieving higher scalability of their method thanks to the increased practicality of their approach.708 709<Image710 src={ch3_hil_serl_examples}711 zoomable712 downloadable713 alt="Figure"714 caption={'(A) HIL-SERL allows for real-world training of high performance RL agents by building on top advancements presented by of SAC, RLPD and SERL. (B) Example of human intervention during a HIL-SERL training process on a real-world SO-100.'}/>715 716Building on off-policy deep Q-learning with replay buffers, entropy regularization for better exploration, expert demonstrations to guide learning, and a series of tools and recommendations for real-world training using reward classifiers (Figure <a href="#hil-serl-blocks" data-reference-type="ref" data-reference="hil-serl-blocks">[hil-serl-blocks]</a>), @luoPreciseDexterousRobotic2024 introduce human interactions during training, learning near-optimal policies in challenging real-world manipulation tasks in 1-2 hours.717 718Human-in-the-Loop, Sample Efficient Robot reinforcement Learning (HIL-SERL) @luoPreciseDexterousRobotic2024 augments offline-to-online RL with targeted human corrections during training, and employs prior data to (1) train a reward classifier and (2) bootstrap RL training on expert trajectories. While offline demonstrations provide the initial dataset seeding learning and constraining early exploration, interactive, online corrections allow a human supervisor to intervene on failure modes and supply targeted interventions, greatly aiding the learning process @luoPreciseDexterousRobotic2024. Crucially, human intervention data is stored in *both* the offline and online replay buffers, differently from the autonomous transitions generated at training time and stored in the online buffer only. In turn, given an intervention timestep $k \in (0, T)$, length-$K$ human intervention data $\{ s^{\text{human}}_k, a^{\text{human}}_k, r^{\text{human}}_k, s^{\text{human}}_{k+1},\}_{k=1}^K$ is more likely to be sampled than the data generated online during training, providing stronger supervision to the agent while still allowing for autonomous learning. Empirically, HIL-SERL attains near-perfect success rates (99%+) on diverse manipulation tasks within 1-2 hours of training @luoPreciseDexterousRobotic2024, underscoring how offline datasets with online RL can markedly improve stability and data efficiency, and ultimately even allow real-world RL-training.719 720#### Code Example- Real-world RL721 722<Image723 src={ch3_hil_serl_architecture}724 zoomable725 downloadable726 alt="Figure"727 caption={'HIL-SERL is a SOTA RL algorithm for training control policies directly in the real-world. Its implementation in lerobot relies on a decoupled actor-learner architecture, communicating over processes (and possibly networks) with queues used to share (1) transitions (s t , a t , r t , s t + 1) and (2) parameters θ .'}/>728 729This example shows how to use the HIL-SERL implementation supported by `lerobot`. This code example is organized into four parts: we first show how to train a reward classifier from a custom set of demonstrations, then define the `Actor` and `Learner` components, and finally, we bring them together in a complete script showing how to use HIL-SERL in practice.730 731At a higher level, the HIL-SERL architecture (Figure <a href="#ch3-hil-serl-architecture" data-reference-type="ref" data-reference="ch3-hil-serl-architecture">[ch3-hil-serl-architecture]</a>) relies on two main components:732 733- An `Actor`, running a frozen policy network used to interact with the environment and obtain observations. Observations are used to both condition the frozen actor in selecting the action to enact, and to form $(s_t, a_t, r_t, s_{t+1})$ transitions that are shared with the `Learner`. Rewards are inferred using a custom, learned reward classifier trained on a dataset of offline demonstrations.734 735- A `Learner`, used to optimize the policy’s parameters $\theta$ for maximum expected return. The learner samples batches of offline data from online and offline buffers in equal proportion @ballEfficientOnlineReinforcement2023, and shares updated parameters with the `Actor`.736 737The HIL-SERL architecture presented in this example can be exclusively run locally, but the implementation in `lerobot` also allows the `Actor` and `Learner` to run on two separate machines connected by the network.738 739<div class="pbox">740 741Training a Reward Classifier 742[https://github.com/fracapuano/robot-learning-tutorial/blob/main/snippets/ch3/01_reward_classifier.py](https://github.com/fracapuano/robot-learning-tutorial/blob/main/snippets/ch3/01_reward_classifier.py)743 744```python745import torch746from lerobot.datasets.lerobot_dataset import LeRobotDataset747from lerobot.datasets.streaming_dataset import StreamingLeRobotDataset748 749delta_timestamps = {750 "observation.images.wrist_camera": [-0.2, -0.1, 0.0] # 0.2, and 0.1 seconds *before* each frame751}752 753# Optionally, use StreamingLeRobotDataset to avoid downloading the dataset754dataset = LeRobotDataset(755 "lerobot/svla_so101_pickplace",756 delta_timestamps=delta_timestamps757)758 759# Streams frames from the Hugging Face Hub without loading into memory760streaming_dataset = StreamingLeRobotDataset(761 "lerobot/svla_so101_pickplace",762 delta_timestamps=delta_timestamps763)764 765# Get the 100th frame in the dataset by 766sample = dataset[100]767print(sample)768# {769# 'observation.state': tensor([...]), 770# 'action': tensor([...]), 771# 'observation.images.wrist_camera': tensor([3, C, H, W]), for delta timesteps772# ...773# }774 775batch_size=16776# wrap the dataset in a DataLoader to use process it batches for training purposes777data_loader = torch.utils.data.DataLoader(778 dataset,779 batch_size=batch_size780)781 782# Iterate over the DataLoader in a training loop783num_epochs = 1784device = "cuda" if torch.cuda.is_available() else "cpu"785 786for epoch in range(num_epochs):787 for batch in data_loader:788 # Move data to the appropriate device (e.g., GPU)789 observations = batch["observation.state"].to(device)790 actions = batch["action"].to(device)791 images = batch["observation.images.wrist_camera"].to(device)792 793 # Next, you can do amazing_model.forward(batch)794 ...795```796 797</div>798<div class="pbox">799 800Defining the `Actor` 801[https://github.com/fracapuano/robot-learning-tutorial/blob/main/snippets/ch3/02_actor.py](https://github.com/fracapuano/robot-learning-tutorial/blob/main/snippets/ch3/02_actor.py)802 803```python804import torch805from lerobot.datasets.lerobot_dataset import LeRobotDataset806from lerobot.datasets.streaming_dataset import StreamingLeRobotDataset807 808delta_timestamps = {809 "observation.images.wrist_camera": [-0.2, -0.1, 0.0] # 0.2, and 0.1 seconds *before* each frame810}811 812# Optionally, use StreamingLeRobotDataset to avoid downloading the dataset813dataset = LeRobotDataset(814 "lerobot/svla_so101_pickplace",815 delta_timestamps=delta_timestamps816)817 818# Streams frames from the Hugging Face Hub without loading into memory819streaming_dataset = StreamingLeRobotDataset(820 "lerobot/svla_so101_pickplace",821 delta_timestamps=delta_timestamps822)823 824# Get the 100th frame in the dataset by 825sample = dataset[100]826print(sample)827# {828# 'observation.state': tensor([...]), 829# 'action': tensor([...]), 830# 'observation.images.wrist_camera': tensor([3, C, H, W]), for delta timesteps831# ...832# }833 834batch_size=16835# wrap the dataset in a DataLoader to use process it batches for training purposes836data_loader = torch.utils.data.DataLoader(837 dataset,838 batch_size=batch_size839)840 841# Iterate over the DataLoader in a training loop842num_epochs = 1843device = "cuda" if torch.cuda.is_available() else "cpu"844 845for epoch in range(num_epochs):846 for batch in data_loader:847 # Move data to the appropriate device (e.g., GPU)848 observations = batch["observation.state"].to(device)849 actions = batch["action"].to(device)850 images = batch["observation.images.wrist_camera"].to(device)851 852 # Next, you can do amazing_model.forward(batch)853 ...854```855 856</div>857<div class="pbox">858 859Defining the `Learner` 860[https://github.com/fracapuano/robot-learning-tutorial/blob/main/snippets/ch3/03_learner.py](https://github.com/fracapuano/robot-learning-tutorial/blob/main/snippets/ch3/03_learner.py)861 862```python863import torch864from lerobot.datasets.lerobot_dataset import LeRobotDataset865from lerobot.datasets.streaming_dataset import StreamingLeRobotDataset866 867delta_timestamps = {868 "observation.images.wrist_camera": [-0.2, -0.1, 0.0] # 0.2, and 0.1 seconds *before* each frame869}870 871# Optionally, use StreamingLeRobotDataset to avoid downloading the dataset872dataset = LeRobotDataset(873 "lerobot/svla_so101_pickplace",874 delta_timestamps=delta_timestamps875)876 877# Streams frames from the Hugging Face Hub without loading into memory878streaming_dataset = StreamingLeRobotDataset(879 "lerobot/svla_so101_pickplace",880 delta_timestamps=delta_timestamps881)882 883# Get the 100th frame in the dataset by 884sample = dataset[100]885print(sample)886# {887# 'observation.state': tensor([...]), 888# 'action': tensor([...]), 889# 'observation.images.wrist_camera': tensor([3, C, H, W]), for delta timesteps890# ...891# }892 893batch_size=16894# wrap the dataset in a DataLoader to use process it batches for training purposes895data_loader = torch.utils.data.DataLoader(896 dataset,897 batch_size=batch_size898)899 900# Iterate over the DataLoader in a training loop901num_epochs = 1902device = "cuda" if torch.cuda.is_available() else "cpu"903 904for epoch in range(num_epochs):905 for batch in data_loader:906 # Move data to the appropriate device (e.g., GPU)907 observations = batch["observation.state"].to(device)908 actions = batch["action"].to(device)909 images = batch["observation.images.wrist_camera"].to(device)910 911 # Next, you can do amazing_model.forward(batch)912 ...913```914 915</div>916<div class="pbox">917 918Using HIL-SERL 919[https://github.com/fracapuano/robot-learning-tutorial/blob/main/snippets/ch3/04_hil_serl.py](https://github.com/fracapuano/robot-learning-tutorial/blob/main/snippets/ch3/04_hil_serl.py)920 921```python922import torch923from lerobot.datasets.lerobot_dataset import LeRobotDataset924from lerobot.datasets.streaming_dataset import StreamingLeRobotDataset925 926delta_timestamps = {927 "observation.images.wrist_camera": [-0.2, -0.1, 0.0] # 0.2, and 0.1 seconds *before* each frame928}929 930# Optionally, use StreamingLeRobotDataset to avoid downloading the dataset931dataset = LeRobotDataset(932 "lerobot/svla_so101_pickplace",933 delta_timestamps=delta_timestamps934)935 936# Streams frames from the Hugging Face Hub without loading into memory937streaming_dataset = StreamingLeRobotDataset(938 "lerobot/svla_so101_pickplace",939 delta_timestamps=delta_timestamps940)941 942# Get the 100th frame in the dataset by 943sample = dataset[100]944print(sample)945# {946# 'observation.state': tensor([...]), 947# 'action': tensor([...]), 948# 'observation.images.wrist_camera': tensor([3, C, H, W]), for delta timesteps949# ...950# }951 952batch_size=16953# wrap the dataset in a DataLoader to use process it batches for training purposes954data_loader = torch.utils.data.DataLoader(955 dataset,956 batch_size=batch_size957)958 959# Iterate over the DataLoader in a training loop960num_epochs = 1961device = "cuda" if torch.cuda.is_available() else "cpu"962 963for epoch in range(num_epochs):964 for batch in data_loader:965 # Move data to the appropriate device (e.g., GPU)966 observations = batch["observation.state"].to(device)967 actions = batch["action"].to(device)968 images = batch["observation.images.wrist_camera"].to(device)969 970 # Next, you can do amazing_model.forward(batch)971 ...972```973 974</div>975 976#### Limitations of RL in Real-World Robotics: Simulators and Reward Design977 978Despite the advancements in real-world RL training, training RL agents for real-world tasks still suffers from the following limitations:979 980- In those instances where real-world training experience is prohibitively expensive to gather (e.g., Tokamak control @degraveMagneticControlTokamak2022, Autonomous Stratospehere Navigation @bellemareAutonomousNavigationStratospheric2020)in-simulation training is often the only viable option. However, high-fidelity simulators for real-world problems can be difficult to build and maintain, especially for contact-rich manipulation and tasks involving deformable or soft materials.981 982- Reward design is a fundamental source of brittleness in real-world RL pipelines. While shaping dense rewards is often necessary to guide exploration in long-horizon tasks, the process is error-prone and heavily reliant on human expertise and intuition. Poorly tuned terms can lead to specification gaming or convergence to local optima, making reward shaping a critical challenge for applying RL in practice. Sparse rewards that only signal successful trajectories can avoid these pitfalls but typically result in much slower learning due to reduced supervision.983 984Advances in learning to act from potentially large corpora of human demonstrations via Behavioral Cloning (BC) address both of these concerns. Although suffering from an inherent suboptimality--imitation learning can at most match the performance level of the demonstrator--learning to reproduce expert demonstrations via BC has proven increasingly competitive and practical, bypassing the need for simulated environments and hard-to-define reward functions.985 986## Robot (Imitation) Learning987 988<span id="learning-imitation" style="position: absolute;"></span>989<Quote source="Norbert Wiener">990 The best material model for a cat is another, or preferably the same cat991</Quote>992<div class="callout">993 994TL;DR Behavioral Cloning provides a natural platform to learn from real-world interactions without the need to design any reward function, and generative models prove more effective than point-wise policies at dealing with multimodal demonstration datasets.995 996</div>997<Image998 src={ch4_bc_trajectories}999 zoomable1000 downloadable1001 alt="Figure"1002 caption={'(A) Average (with standard deviation) evolution of the actuation levels over the first 5 recorded episodes in lerobot/svla_so101_pickplace. Proprioperceptive states provide invaluable to determine the robot’s state during an episode. (B) Camera frames are also recorded alongside measurements on the robot’s state, capturing information about the robot’s interaction with its environment.'}/>1003 1004Learning from human demonstrations provides a pragmatic alternative to the RL pipeline discussed in Section <a href="#learning-rl" data-reference-type="ref" data-reference="learning-rl">[learning-rl]</a>. Indeed, especially in real-world robotics, online exploration is typically <mark>costly and potentially unsafe</mark>, and designing (dense) reward signals is a <mark>brittle and task-specific</mark> process. Further, even success detection itself often requires bespoke instrumentation, while episodic training demands reliable resets--all factors complicating training RL algorithms on hardware at scale. Behavioral Cloning (BC) sidesteps these constraints by <mark>casting control an imitation learning problem</mark>, leveraging previously collected expert demonstrations to anchor the learned autonomous behavior. Most notably, by *learning-to-imitate*, autonomous systems naturally adhere to the objectives, preferences, and success criteria implicitly encoded in the data, which reduces early-stage exploratory failures and obviates hand-crafted reward shaping altogether.1005 1006Formally, let $\mathcal D = \{ \tau^{(i)} \}_{i=1}^N$ be a set of expert trajectories, with $\tau^{(i)} = \{(o_t^{(i)}, a_t^{(i)})\}_{t=0}^{T_i}$ representing the $i$-th length-$T_i$ trajectory in $\mathcal D$, $o_t \in \mathcal O$ denoting observations (e.g., images and proprioception altogether), and $a_t \in \mathcal A$ the expert actions. Typically, observations $o \in \mathcal O$ consist of both image and proprioperceptive information, while actions $a \in \mathcal A$ represent control specifications for the robot to execute, e.g. a joint configuration. Note that differently from Section <a href="#learning-rl" data-reference-type="ref" data-reference="learning-rl">[learning-rl]</a>, in the imitation learning context $\mathcal D$ denotes an offline dataset collecting $N$ length-$T_i$ reward-free (expert) human trajectories $\tau^{(i)}$, and *not* the environment dynamics. Similarily, in this section $\tau^{(i)}$ represent a length-$T_i$ trajectory of observation-action pairs, which crucially *omits entirely any reward* information. Figure <a href="#ch4-bc-trajectories" data-reference-type="ref" data-reference="ch4-bc-trajectories">[ch4-bc-trajectories]</a> graphically shows trajectories in terms of the average evolution of the actuation on the 6 joints of a teleoperated SO-100 manipulator. Notice how proprioperceptive states are captured jointly with camera frames over the course of the recorded episodes, providing a unified high-frame rate collection of both image and joint teleoperation data. Figure <a href="#ch4-observation-action-mapping" data-reference-type="ref" data-reference="ch4-observation-action-mapping">[ch4-observation-action-mapping]</a> shows $(o_t, a_t)$-pairs for the same dataset, with the actions performed by the human expert illustrated alongside the corresponding observation. In principle, (expert) trajectories $\tau^{(i)}$ can have different lengths since demonstrations might exhibit multi-modal strategies to attain the same goal, resulting in multiple, different behaviors.1007 1008<Image1009 src={ch4_observation_action_mapping}1010 zoomable1011 downloadable1012 alt="Figure"1013 caption={'Sample observations and action pairs over the course of a given trajectory recorded in lerobot/svla_so101_pickplace. Observations, comprising of both proprioperceptive and visual information, are recorded alongside the configuration of a second, leader robot controlled by a human expert, providing complete information for regressing actions given observations.'}/>1014 1015Behavioral Cloning (BC) @pomerleauALVINNAutonomousLand1988 aims at producing synthetic behaviors by learning the mapping from observations to actions, and in its most natural formulation can be effectively tackled as a *supevised* learning problem, consisting of learning the (deterministic) mapping $f: \mathcal O\mapsto \mathcal A, \ a_t = f(o_t)$ by solving1016``` math1017\htmlId{loss-minimization-SL}{\min_{f} \mathbb{E}_{(o_t, a_t) \sim p(\bullet)} \mathcal L(a_t, f(o_t)),}1018```1019given an arbitrary risk function $\mathcal L: \mathcal A \times \mathcal A \mapsto \mathbb{R}, \ \mathcal L (a, a^\prime)$.1020 1021Typically, the expert’s joint observation-action distribution $p: \mathcal O\times \mathcal A\mapsto [0,1]$ is assumed to be unknown, in keeping with a classic Supervised Learning (SL) framework[^3]. However, differently from standard SL assumptions, the samples collected in $\mathcal D$--realizations of the underlying $p$--are *not* i.i.d., as expert demonstrations are collected *sequentially* in the form of trajectories. In practice, this aspect can be partially mitigated by considering pairs in a non-sequential order--*shuffling* the samples in $\mathcal D$--so that the expected risk under $p$ can be approximated using MC estimates, although these estimates may in general be less accurate. Another strategy to mitigate the impact of regressing over non-i.i.d. samples relies on the possibility of interleaving BC and data collection , aggregating multiple datasets iteratively. However, because we only consider the case where a single offline dataset $\mathcal D$ of trajectories is available and no more data can be collected, DAgger falls out of our scope.1022 1023Despite the inherent challenges of learning from non-i.i.d. data, the BC formulation presents several operational advantages in robotics. First, training happens offline and naturally accomodates for expert, demonstration data, hereby severily limiting exploration risks by preventing the robot from performing dangerous actions altogether, by anchoring action in imitation. Second, reward design is entirely unnecessary in BC, as demonstrations already reflect human intent. The absence of rewards also prevents the risk of misalignment and specification gaming (*reward hacking*), otherwise inherent in purely reward-based RL @heessEmergenceLocomotionBehaviours2017. Third, because expert trajectories encode terminal conditions, success detection and resets are implicit in the dataset. Finally, empirical evidence suggests the performance of BC scales naturally with growing corpora of demonstrations collected across tasks, embodiments, and environments. Nonetheless, BC can, in principle, only reproduce behaviors that are at best as good as those of the demonstrator, and therefore offers no remedy for the suboptimal decisions that humans may enact. This limitation is particularly problematic in sequential decision-making tasks where expert demonstrations are scarce---either because data collection is costly or because human performance is inherently suboptimal. Yet, many robotics applications still benefit from relatively inexpensive pipelines for collecting high-quality human-generated trajectories, justifying the use of BC in such settings.1024 1025<Image1026 src={ch4_issues_with_bc}1027 zoomable1028 downloadable1029 alt="Figure"1030 caption={'Point-wise policies suffer from limitations due to (A) covariate shifts and (B) poor approximation of multimodal demonstrations. (A) Small errors may drive the policy out of distribution, incuring in a vicious circle ultimately resulting in failure. (B) Both modes of reaching for a target object in the scene--either left or right-first--are equally as good and thus equally as likely to be present in a dataset of human demonstrations, ultimately resulting in multimodal demonstrations.'}/>1031 1032While conceptually elegant, *point-estimate policies* $f : \mathcal O\mapsto \mathcal A$ learned by solving eq. <a href="#loss-minimization-SL" data-reference-type="ref" data-reference="loss-minimization-SL">[loss-minimization-SL]</a> have been observed to suffer from (1) compounding errors @rossReductionImitationLearning2011 and (2) poor fit to multimodal distributions @florenceImplicitBehavioralCloning2022, @keGraspingChopsticksCombating2020. Figure <a href="#ch4-issues-with-bc" data-reference-type="ref" data-reference="ch4-issues-with-bc">[ch4-issues-with-bc]</a> illustrates these two key issues related to learning *explicit policies* @florenceImplicitBehavioralCloning2022. Besides sequentiality in $\mathcal D$, compounding errors due to *covariate shift* may also prove catastrophic, as even small $\epsilon$-prediction errors $0 < \Vert \mu(o_t) - a_t \Vert \leq \epsilon$ can quickly drive the policy into out-of-distribution states, incuring in less confident generations and thus compounding errors (Figure <a href="#ch4-issues-with-bc" data-reference-type="ref" data-reference="ch4-issues-with-bc">[ch4-issues-with-bc]</a>, left). Moreover, point-estimate policies typically fail to learn *multimodal* targets, which are very common in human demonstrations solving real-world robotics problems, as multiple trajectories can be equally as good towards the accomplishment of a goal (e.g., symmetric grasps, Figure <a href="#ch4-issues-with-bc" data-reference-type="ref" data-reference="ch4-issues-with-bc">[ch4-issues-with-bc]</a>, right). In particular, unimodal regressors tend to average across modes, yielding indecisive or even unsafe commands @florenceImplicitBehavioralCloning2022. To address poor multimodal fitting, @florenceImplicitBehavioralCloning2022 propose learning the *generative model* $p(o, a)$ underlying the samples in $\mathcal D$, rather than explicitly learning a prediction function $f- a = f(o)$.1033 1034### A (Concise) Introduction to Generative Models1035 1036Generative Models (GMs) aim to learn the stochastic process underlying the very generation of the data collected, and typically do so by fitting a probability distribution that approximates the unknown *data distribution*, $p$. In keeping with the GM literature, $p(x) \leftarrow \mathbb P(x), x \sim p$. In the case of BC, the unknown data distribution $p$ may represent the expert’s joint distribution over $(o, a)$-pairs. Thus, given a finite set of $N$ pairs $\mathcal D = \{ (o,a)_i \}_{i=0}^N$ available as an imitation learning target (and thus assumed to be i.i.d.), GMs seek to learn a *parametric* distribution $p_\theta(o,a)$ such that (1) new samples $(o,a) \sim p_\theta(\bullet)$ resemble those stored in $\mathcal D$, and (2) high likelihood is assigned to the *observed* regions of the *unobservable* $p$. Likelihood-based learning provides a principled training objective to achieve both goals, and it is thus extensively used in GMs @prince2023understanding.1037 1038#### Variational Auto-Encoders1039 1040<Image1041 src={ch4_task_effect_on_pairs}1042 zoomable1043 downloadable1044 alt="Figure"1045 caption={'Intuitively, latent variable in a single latent model may contain information regarding the task being performed, which directly results in the likelihood of the same observation-action pair being different for two different tasks. When (A) picking a block the likelihood of a wide gripper’s opening should be higher than narrower one, while it should be the opposite when (B) pushing the block.'}/>1046 1047A common inductive bias used in GM posits samples $(o,a)$ are influenced from an unobservable latent variable $z \in Z$, resulting in:1048``` math1049\htmlId{BC-latent-variable}{p (o,a) = \int_{\operatorname{supp}({Z})} p(o,a \vert z) p(z)}1050```1051Intuitively, in the case of observation-action pairs $(o, a)$ for a robotics application, $z$ could be interpreted as some high level representation of the underlying task being performed by the human demonstrator. In such case, treating $p(o,a)$ as a marginalization over $\operatorname{supp}({Z})$ of the complete joint distribution $p(o,a,z)$ natively captures the effect different tasks have on the likelihood of observation-action pairs. Figure <a href="#ch4-task-effect-on-pairs" data-reference-type="ref" data-reference="ch4-task-effect-on-pairs">[ch4-task-effect-on-pairs]</a> graphically illustrates this concept in the case of a (A) picking and (B) pushing task, for which, nearing the target object, the likelihood of actions resulting in opening the gripper--the higher $q_6$, the wider the gripper’s opening--should intuitively be (A) high or (B) low, depending on the task performed. While the latent space $Z$ typically has a much richer structure than the set of all actual tasks performed, eq. <a href="#BC-latent-variable" data-reference-type="ref" data-reference="BC-latent-variable">[BC-latent-variable]</a> still provides a solid framework to learn joint distribution conditioned on unobservable yet relevant factors. Figure <a href="#ch4-latent-variable-model" data-reference-type="ref" data-reference="ch4-latent-variable-model">[ch4-latent-variable-model]</a> represents this latent-variable framework in the context of a robotics application- the true, $z$-conditioned generative process assigns *likelihood* $p((o,a) \vert z)$ to the single $(o,a)$-pair. Using Bayes’ theorem, one can reconstruct the *posterior* distribution on $\operatorname{supp}({Z})$, $q_\theta(z \vert o,a)$ from the likelihood $p_\theta(o,a \vert z)$, *prior* $p_\theta(z)$ and *evidence* $p_\theta(o,a)$. VAEs approximate the latent variable model presented in eq. <a href="#BC-latent-variable" data-reference-type="ref" data-reference="BC-latent-variable">[BC-latent-variable]</a> using an *approximate posterior* $q_\phi(z \vert o,a)$ while regressing parameters for a parametric likelihood, $p_\theta(o,a \vert z)$ (Figure <a href="#ch4-latent-variable-model" data-reference-type="ref" data-reference="ch4-latent-variable-model">[ch4-latent-variable-model]</a>).1052 1053<Image1054 src={ch4_latent_variable_model}1055 zoomable1056 downloadable1057 alt="Figure"1058 caption={'(A) The latent variable model in a robotics application regulates influence between observed ( o, a) variables and an unobservable latent variable. (B) VAEs approximate exact latent variable models by means of variational inference.'}/>1059 1060Given a dataset $\mathcal D$ consisting of $N$ i.i.d. observation-action pairs, the log-likelihood of all datapoints under $\theta$ (in Bayesian terms, the *evidence* $p_\theta(\mathcal D)$) can be written as:1061<span id="evidence-definition-1" style="position: absolute;">1062</span>1063 1064``` math1065\begin{align}1066 \log p_\theta(\mathcal D) &= \log \sum_{i=0}^N p_\theta ((o,a)_i) \\1067 &= \log \sum_{i=0}^N \int_{\operatorname{supp}({Z})} p_\theta((o,a)_i \vert z) p(z) \\1068 &= \log \sum_{i=0}^N \int_{\operatorname{supp}({Z})} \frac{q_\theta(z \vert (o,a)_i)}{q_\theta(z \vert (o,a)_i)} \cdot p_\theta((o,a)_i \vert z) p(z) \\1069 &= \log \sum_{i=0}^N \mathbb E_{z \sim q_\theta(\bullet \vert (o,a)_i)} [\frac{p(z)}{q_\theta(z \vert (o,a)_i)} \cdot p_\theta((o,a)_i \vert z)], 1070\end{align}1071```1072where we used eq. <a href="#BC-latent-variable" data-reference-type="ref" data-reference="BC-latent-variable">[BC-latent-variable]</a> in eq. <a href="#evidence-definition-1" data-reference-type="ref" data-reference="evidence-definition-1">[evidence-definition-1]</a>, multiplied by $1 = \frac{q_\theta(z \vert (o,a)_i)}{q_\theta(z \vert (o,a)_i)}$ in eq. <a href="#evidence-definition-2" data-reference-type="ref" data-reference="evidence-definition-2">[evidence-definition-2]</a>, and used the definition of expected value in eq. <a href="#evidence-definition" data-reference-type="ref" data-reference="evidence-definition">[evidence-definition]</a>.1073 1074In the special case where one assumes distributions to be tractable, $p_\theta (\mathcal D)$ is typically tractable too, and $\max_\theta \log p_\theta(\mathcal D)$ provides a natural target for (point-wise) infering the unknown parameters $\theta$ of the generative model. Unfortunately, eq. <a href="#evidence-definition" data-reference-type="ref" data-reference="evidence-definition">[evidence-definition]</a> is rarely tractable when the distribution $p$ is modeled with approximators such as neural networks, especially for high-dimensional, unstructured data.1075 1076In their seminal work on Variational Auto-Encoders (VAEs), @kingma2013auto present two major contributions to learn complex latent-variable GMs from unstructured data, proposing (1) a tractable, variational lower-bound to eq. <a href="#evidence-definition" data-reference-type="ref" data-reference="evidence-definition">[evidence-definition]</a> as an optimization target to jointly learn likelihood and posterior and (2) using high-capacity function approximators to model the likelihood $p_\theta(o,a\vert z)$ and (approximate) posterior distribution $q_\phi(z \vert o,a) \approx q_\theta(z \vert o,a)$.1077 1078In particular, the lower bound on eq. <a href="#evidence-definition" data-reference-type="ref" data-reference="evidence-definition">[evidence-definition]</a> (Evidence LOwer Bound, *ELBO*) can be derived from eq. <a href="#evidence-definition" data-reference-type="ref" data-reference="evidence-definition">[evidence-definition]</a> applying Jensen’s inequality--$\log \mathbb{E}[\bullet] \geq \mathbb{E} [\log (\bullet)]$--yielding:1079<span id="ELBO-intractable" style="position: absolute;">1080</span>1081 1082``` math1083\begin{align}1084 \log p_\theta(\mathcal D) &\geq \sum_{i=0}^{N} \left(1085 \mathbb{E}_{z \sim q_\theta(\bullet \vert (o,a)_i)} \big[ \log p_\theta((o,a)_i \vert z) \big]1086 + \mathbb{E}_{z \sim q_\theta(\bullet \vert (o,a)_i)} [\log \left( \frac{p(z)}{q_\theta(z \vert (o,a)_i)} \right)]1087 \right) \\1088 &= \sum_{i=0}^{N} \left(1089 \mathbb{E}_{z \sim q_\theta(\bullet \vert (o,a)_i)} \big[ \log p_\theta((o,a)_i \vert z) \big]1090 - \text{D}_{\text{KL}}\big[ q_\theta(z \vert (o,a)_i) \Vert p(z) \big]1091 \right) 1092\end{align}1093```1094 1095The true, generally intractable, posterior $q_\theta (z \vert o,a)$ prevents computing both the expectation and KL divergence terms in eq. <a href="#ELBO-intractable" data-reference-type="ref" data-reference="ELBO-intractable">[ELBO-intractable]</a>, and therefore @kingma2013auto propose deriving the ELBO using an *approximate* posterior $q_\phi(z \vert o,a)$, resulting in the final, tractable, ELBO objective,1096<span id="ELBO" style="position: absolute;">1097</span>1098 1099``` math1100\begin{align}1101\text{ELBO}_{\mathcal D}(\theta, \phi) = \sum_{i=0}^{N} \left(1102 \mathbb{E}_{z \sim q_\phi(\bullet \vert (o,a)_i)} \big[ \log p_\theta((o,a)_i \vert z) \big]1103 - \text{D}_{\text{KL}}\big[ q_\phi(z \vert (o,a)_i) \Vert p(z) \big]1104 \right)1105 1106\end{align}1107```1108From Jensen’s inequality, maximizing ELBO results in maximizing the log-likelihood of the data too, thus providing a natural, tractable optimization target. Indeed, expectations can be estimated using MC estimates from the learned distributions in eq. <a href="#ELBO" data-reference-type="ref" data-reference="ELBO">[ELBO]</a>, while the KL-divergence term can typically be computed in closed-form (1) modeling $q_\phi$ as a Gaussian $q_\phi(z \vert o,a) = \mathcal N\big(\mu_\phi(o,a), \Sigma_\phi(o,a) \big)$ with learned mean vector $\mu_\phi(o,a)$ and learned variance-covariance matrix $\Sigma_\phi(o,a)$ and (2) imposing a standard Gaussian prior on the latent space, $p(z) = \mathcal N(\mathbf{0}, \mathbf{I})$.1109 1110An intuitive explanation of the learning dynamics of VAEs can be given considering the equivalent case of *minimizing the negative ELBO*, which admits the particularly interpretable factorization (considering, without loss of generality, only one $(o,a) \sim \mathcal D$):1111<span id="VAE-min-neg-ELBO" style="position: absolute;">1112</span>1113 1114``` math1115\begin{align}1116\min_{\theta, \phi} - \text{ELBO}_{\mathcal (o,a) \sim \mathcal D}(\theta, \phi) &= \min_{\theta, \phi}\mathbf{L^{\text{rec}}}(\theta) + \mathbf{L^{\text{reg}}}(\phi), \\1117\mathbf{L^{\text{rec}}}(\theta) &= \mathbb{E}_{z \sim q_\phi(\bullet \vert o,a}) \big[ \log p_\theta(o,a \vert z) \big] \\1118\mathbf{L^{\text{reg}}}(\phi) &= \text{D}_{\text{KL}}\big[ q_\phi(z \vert o,a) \Vert p(z) \big]. 1119\end{align}1120```1121 1122For any given $(o,a)$ pair, the expected value term in eq. <a href="#VAE-Lrec" data-reference-type="ref" data-reference="VAE-Lrec">[VAE-Lrec]</a> is typically computed via MC estimates, resulting in1123 1124``` math1125-\mathbb{E}_{z \sim q_\phi(\bullet \vert o,a)} \big[ \log p_\theta(o,a \vert z) \big] = \mathbf{L^{\text{rec}}} \approx - \frac{1}{n} \sum_{i=0}^n \log p_\theta(o,a \vert z_i).1126```1127Assuming $p_\theta(o,a \vert z)$ to be parametrized with an isotropic Gaussian distribution with mean $\mu_\theta (z) \in \mathbb R^d$ and variance $\sigma^2$, the log-likelihood thus simplifies to-1128``` math1129\log p(o,a \vert z_i) = -\frac{1}{2\sigma^{2}} \big \Vert (o,a)-\mu_\theta(z_i) \big\Vert_2^2 -\frac{d}{2}\log(2\pi \sigma^{2}) \implies \mathbf{L^\text{rec}} \approx \frac {1}{n} \sum_{i=0}^n \big\Vert (o,a) - \mu_\theta(z_i) \big \Vert^2_21130```1131In practice, it is common to approximate the learned likelihood $p_\theta(o,a \vert z)$ with a parametric distribution (e.g., Gaussian) whose parameters are given by a learned coefficient vector derived from $\mu_\theta(z), \ z \sim p(\bullet)$. Under this formulation, learning a VAE amounts to (1) *reconstructing* the examples in $\mathcal{D}$ by minimizing (1) the reconstruction loss $\mathbf{L^{\text{rec}}}$--a standard *supervised learning* objective for regression--while (2) *regularizing* the latent representation by minimizing $\mathbf{L^{\text{reg}}}$. The latter enforces information compression, since with the common prior choice $p(z) = \mathcal{N}(\mathbf{0}, \mathbf{I})$ in eq. <a href="#VAE-Lreg" data-reference-type="ref" data-reference="VAE-Lreg">[VAE-Lreg]</a>, the regularizer constrains the posterior and thereby limits the expressivity of $q_\phi(z \vert o,a)$.1132 1133#### Diffusion Models1134 1135VAEs approximate probability distributions via a *single* latent variable model, assuming the underlying unknown distribution can be factored according to eq. <a href="#BC-latent-variable" data-reference-type="ref" data-reference="BC-latent-variable">[BC-latent-variable]</a>, and solve the variational-inference problem of jointly learning the likelihood $p_\theta$ and (approximate) posterior $q_\phi$ for such model. In that, the unknown data distribution $p(o,a)$ is effectively approximated via $\int_Z p(z) p_\theta(o,a \vert z)$, and the underlying generative process reproduced by (1) sampling a latent variable and (2) learning to decode it into a high-likelihood sample under the (unknown) $p(o,a)$. Diffusion Models (DMs) @hoDenoisingDiffusionProbabilistic2020 are another class of GMs which treat the similar problem of approximating an underlying unknown data distribution--*variational inference*--by *partially* extending VAEs to the case where *multiple* latent variables influence each other and the generative process underlying $o,a$ itself. In particular, DMs posit the generative process can be decomposed to a series of piece-wise (Markovian) interactions between (latent) variables (Figure <a href="#ch4-many-latents" data-reference-type="ref" data-reference="ch4-many-latents">[ch4-many-latents]</a>), resulting in1136<span id="BC-multi-latent-model-1" style="position- absolute;">1137</span>1138 1139``` math1140\begin{align}1141 p(\underbrace{o,a}_{= z_0}) &= \int_{\operatorname{supp}({Z_0})} \int_{\operatorname{supp}({Z_1})} \ldots \int_{\operatorname{supp}({Z_T})} p(z_0, z_1, \dots z_T) \\ 1142 p(z_0, z_1, \dots z_T) &= p(z_T) \prod_{t=1}^{T} p(z_{t-1} \vert z_t), 1143\end{align}1144```1145where we explicitly showed the marginalization over the multiple latents in eq. <a href="#BC-multi-latent-model-1" data-reference-type="ref" data-reference="BC-multi-latent-model-1">[BC-multi-latent-model-1]</a>, and used the law of conditional probability and Markov property in eq. <a href="#BC-multi-latent-model-2" data-reference-type="ref" data-reference="BC-multi-latent-model-2">[BC-multi-latent-model-2]</a>. Also, for ease of notation, we will refer to observation-action pairs $o,a$ as $z_0$.1146 1147<Image1148 src={ch4_many_latents}1149 zoomable1150 downloadable1151 alt="Figure"1152 caption={'HMLV models posit the data generation process is influenced by a stack of Markov-dependent latent variables, with samples from the posterior distribution being progressively higher up in the hierarchy.'}/>1153 1154Similar to VAEs, it is generally not possible to assign an *exact* interpretation to the latent variables. Nevertheless, a reasonable application-driven intuition is that Hierarchical Markov Latent Variable (HMLV) models, by capturing hierarchical and decoupled interactions among latent variables, can reflect the different resolutions at which conditioning factors intervene. For example, in a robotics setting, one might naturally distinguish between high-level trajectory planning (higher up in the hierarchy, $t \to T$) and fine-grained motion adjustments (closer to empirical observations, $t \to 0$). In that, HMLV models thus provide a framework to perform variational inference via multiple, sequential sampling steps from different higher level distributions instead of approximating the generative process with a single-latent variable model. DMs are a particular instantiation of HMLV models for which the posterior is fixed to $q( z_t \vert z_{t-1}) = \mathcal N(z_t \sqrt{1-\beta_t}, \beta_t \mathbf{I})$, for a given $\beta_t \in \mathbb R^+$. In practice, $\beta_t$ is used to iteratively reduce the signal-to-noise ratio along the latents’ hierarchy, similarily to how a diffusion process influences the information of a physical system.1155 1156Just like VAEs, DMs attemp to learn to reproduce an underlying data distribution $p (o,a)$ given a collection of i.i.d. samples approximating the model posited to have generated the data in the first place (eq. <a href="#BC-multi-latent-model-1" data-reference-type="ref" data-reference="BC-multi-latent-model-1">[BC-multi-latent-model-1]</a>). Similarily to VAEs, DMs approximate the process of sampling from the unknown $p(o,a)$ by (1) sampling from an easy-to-sample distribution (e.g., Gaussian) and (2) learning to reconstruct high-likelihood samples under the unknown distribution. However, in stark contrast with VAEs, the easy-to-sample distribution contains *no mutual information* regarding the data distribution $p(o,a)$. Crucially, as no information from the sample $(o,a)$ (denoted as $z_0 \equiv (o,a)$ for simplicity of notation) is assumed to be propagated throughout the chain of latents, the posterior $q(z_t \vert z_{t-1})$ assumes a relatively amicable structure in DMs, reducing complexity. The *true* likelihood $p(z_{t-1} \vert z_t)$ is instead typically approximated using the parametrization $p_\theta (z_{t-1} \vert z_t)$. In that, the information contained in the unknwon data distribution is *reconstructed* via a process in which samples from a fixed distribution are iteratively turned into (ideally) high-likelihood samples under $p(o,a)$--a process referred to as *denoising*.1157 1158Under such model, we can express the log-likelihood of an arbitrary sample $z_0$ as:1159<span id="diffusion-1" style="position: absolute;">1160</span>1161 1162``` math1163\begin{align}1164 \log p_\theta (z_0) &= \log \int_{\operatorname{supp}({Z_1}) \times \operatorname{supp}({Z_2}) \times \dots \times \operatorname{supp}({Z_T})} p_\theta(\underbrace{z_0, z_1, z_2, \dots z_T}_{z_{0:T}}) \\1165 &= \log \int_{\operatorname{supp}({Z_{1:T}})} \frac{p_\theta(z_{0:T}) \cdot q(z_{1:T} \vert z_0)}{q(z_{1:T} \vert z_0)} \\1166 &= \log \mathbb{E}_{z_{1:T} \sim q(\bullet \vert z_0)} \bigg[ \frac{p_\theta(z_{0:T})}{q(z_{1:T} \vert z_0)} \bigg] \\1167 &\geq \mathbb{E}_{z_{1:T} \sim q(\bullet \vert z_0)} \bigg[ \log \frac{p_\theta(z_{0:T})}{q(z_{1:T} \vert z_0)} \bigg] \\1168 &= \mathbb{E}_{z_{1:T} \sim q(\bullet \vert z_0)} \bigg[ \log \frac{p(z_T) \prod_{t=1}^{T} p_\theta (z_{t-1} \vert z_t)}{\prod_{t=1}^T q(z_t \vert z_{t-1})} \bigg] \\1169 &= \mathbb{E}_{z_{1:T} \sim q(\bullet \vert z_0)} \bigg[ \log \frac{p(z_T) \cdot p_\theta (z_0 \vert z_1) \prod_{t=2}^{T} p_\theta (z_{t-1} \vert z_t)}{q(z_T \vert z_{T-1}) \prod_{t=1}^{T-1} q(z_t \vert z_{t-1})} \bigg] \\1170 &= \mathbb{E}_{z_{1:T} \sim q(\bullet \vert z_0)} \bigg[ \log \frac{p(z_T) \cdot p_\theta (z_0 \vert z_1) \prod_{t=1}^{T-1} p_\theta (z_{t} \vert z_{t+1})}{q(z_T \vert z_{T-1}) \prod_{t=1}^{T-1} q(z_t \vert z_{t-1})} \bigg] \\1171 &= 1172 \mathbb{E}_{z_{1:T} \sim q(\bullet \vert z_0)} \bigg[ \log \frac{p(z_T) \cdot p_\theta (z_0 \vert z_1)}{q(z_t \vert z_{t-1})} \bigg] + 1173 \mathbb{E}_{z_{1:T} \sim q(\bullet \vert z_0)} \bigg[ \log \prod_{t=1}^{T-1} \frac{p_\theta (z_{t} \vert z_{t+1})}{q(z_t \vert z_{t-1})}\bigg]1174 \\1175 &=1176 \mathbb{E}_{z_{1:T} \sim q(\bullet \vert z_0)} \big[ \log p_\theta (z_0 \vert z_1) \big] + 1177 \mathbb{E}_{z_{1:T} \sim q(\bullet \vert z_0)} \bigg[ \log \frac{p (z_T)}{q(z_T \vert z_{T-1})} \bigg] +1178 \sum_{t=1}^{T-1} \mathbb{E}_{z_{1:T} \sim q(\bullet \vert z_0)} \bigg[ \log \frac{p_\theta (z_{t} \vert z_{t+1})}{q(z_t \vert z_{t-1})}\bigg]1179 \\1180 &= 1181 \mathbb{E}_{z_1 \sim q(\bullet \vert z_0)} \big[ \log p_\theta (z_0 \vert z_1) \big] + 1182 \mathbb{E}_{z_{T-1:T} \sim q(\bullet \vert z_0)} \bigg[ \log \frac{p (z_T)}{q(z_T \vert z_{T-1})} \bigg] +1183 \sum_{t=1}^{T-1} \mathbb{E}_{z_{t-1:t+1} \sim q(\bullet \vert z_0)} \bigg[ \log \frac{p_\theta (z_{t} \vert z_{t+1})}{q(z_t \vert z_{t-1})}\bigg]1184 \\1185 &= \mathbb{E}_{z_1 \sim q(\bullet \vert z_0)} \log p_\theta (z_0 \vert z_1) - \mathbb{E}_{z_{T-1} \sim q(\bullet \vert z_0)} \big[ \text{D}_{\text{KL}}(q(z_T \vert z_{T-1}) \Vert p(z_T) ) \big] \\1186 &- \sum_{t=1}^{T-1} \mathbb{E}_{(z_{t-1}, z_{t+1}) \sim q(\bullet \vert z_0)} \big[ \text{D}_{\text{KL}}(q(z_t \vert z_{t-1}) \Vert p_\theta(z_t \vert z_{t+1}) ) \big], \notag1187\end{align}1188```1189where we: used eq. <a href="#BC-multi-latent-model-1" data-reference-type="ref" data-reference="BC-multi-latent-model-1">[BC-multi-latent-model-1]</a> and multiplied by $1 = \tfrac{q(z_{1-T} \vert z_0)}{q(z_{1:T} \vert z_0)}$ in eq. <a href="#diffusion-1" data-reference-type="ref" data-reference="diffusion-1">[diffusion-1]</a>; used Jensen’s inequality in eq. <a href="#diffusion-jensen" data-reference-type="ref" data-reference="diffusion-jensen">[diffusion-jensen]</a>; used the law of conditional probability for both numerator and denominator in eq. <a href="#diffusion-2" data-reference-type="ref" data-reference="diffusion-2">[diffusion-2]</a>; stepped forward and backward the products in the numerator and denominator products in eq. <a href="#diffusion-3" data-reference-type="ref" data-reference="diffusion-3">[diffusion-3]</a>, respectively; reindexed the product terms in eq. <a href="#diffusion-4" data-reference-type="ref" data-reference="diffusion-4">[diffusion-4]</a>; removed out-of-expectation variables in eq. <a href="#diffusion-expectation-indices" data-reference-type="ref" data-reference="diffusion-expectation-indices">[diffusion-expectation-indices]</a>; used the defintion of KL-divergence in eq. <a href="#diffusion-likelihood" data-reference-type="ref" data-reference="diffusion-likelihood">[diffusion-likelihood]</a>. In turn, eq. <a href="#diffusion-likelihood" data-reference-type="ref" data-reference="diffusion-likelihood">[diffusion-likelihood]</a> provides an optimization target to *learn* $p_\theta$ solving $\max_\theta \log p_\theta (\mathcal D)$.1190 1191In their seminal work on using DMs for variational inference, @hoDenoisingDiffusionProbabilistic2020 introduce major contributions regarding solving $\min_\theta -\log p_\theta(z_0)$. In particular, @hoDenoisingDiffusionProbabilistic2020 exclusively adopt a *fixed, isotropic Gaussian posterior* in the form of $q(z_t \vert z_{t-1}) = \mathcal{N}(\sqrt{1-\beta_t}z_{t-1}, \beta_t \mathbf I)$. The choice of adopting Gaussians has profound implications on the generative process modeled. Indeed, under the (mild) assumption that the variance is sufficiently small $\beta_t \leq \eta, \eta \in \mathbb R^+$, @sohnLearningStructuredOutput2015 proved that the likelihood $p(z_{t-1} \vert z_t)$ is Gaussian as well, which allows for the particularly convenient parametrization of the approximate likelihood $p_\theta (z_{t-1} \vert z_t) = \mathcal N(\mu_\theta(z_t, t), \Sigma_\theta(z_t,t)), \ t \in [1,T]$, as well as for closed-form tractability of the KL-divergence terms in eq. <a href="#diffusion-likelihood" data-reference-type="ref" data-reference="diffusion-likelihood">[diffusion-likelihood]</a>. Further, the posterior’s structure also enables the analytical description of the distribution of the $t$-th latent variable, $q(z_t \vert z_0) = \mathcal N (\sqrt{\bar{\alpha}_t}z_0, (1-\bar{\alpha}_t) \mathbf{I})$, with $\alpha_t = 1-\beta_t, \ \bar \alpha_t = \prod_{k=1}^t \alpha_k$, conveniently preventing iterative posterior sampling simplifying computing eq. <a href="#diffusion-likelihood" data-reference-type="ref" data-reference="diffusion-likelihood">[diffusion-likelihood]</a>. It follows-1192<span id="diffusion-likelihood-gradient" style="position: absolute;">1193</span>1194 1195``` math1196\begin{align}1197 \nabla_\theta \log p_\theta (z_0) = \mathbb E_{z_1 \sim q(\bullet \vert z_0)} \nabla_\theta \log p_\theta (z_0 \vert z_1) - \sum_{t=1}^{T-1} \mathbb E_{z_{t-1}, z_{t+1} \sim q(\bullet \vert z_0)} \nabla_\theta \text{D}_{\text{KL}}(q(z_t \vert z_{t-1}) \Vert p_\theta(z_t \vert z_{t+1}), 1198\end{align}1199```1200where the former term is equivalent to the reconstruction term in eq. <a href="#VAE-min-neg-ELBO" data-reference-type="ref" data-reference="VAE-min-neg-ELBO">[VAE-min-neg-ELBO]</a> and the latter term can be obtained in closed form.