knu-physical-ai/fr3-action-space-case-study

FR3 Action-Space Case Study

A small, deliberately-instructive teleoperation dataset. Every frame carries both the follower's realized joint trajectory and the GELLO leader's command, so you can see directly why the choice between them decides whether a deployed policy moves smoothly or stutters.

<a class="flex" href="https://huggingface.co/spaces/lerobot/visualize_dataset?path=knu-physical-ai/fr3-action-space-case-study"> <img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/visualize-this-dataset-xl.svg"/> <img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/visualize-this-dataset-xl-dark.svg"/> </a>

한 줄 요약. action을 리더 명령이 아니라 팔로워의 실현 궤적으로 기록하면, 정책이 원리적으로 로봇보다 앞설 수 없게 되어 매 replan마다 목표가 뒤로 밀리고 동작이 1.3~2.0배 느려지며 주기적으로 멈칫거린다. 이 데이터셋은 두 신호를 한 파일에 나란히 담아 그 차이를 눈으로 확인할 수 있게 만든 교육용 자료다.

Why this dataset exists

It is not meant for training a competent policy — 4 episodes is far too few. It exists to make one design decision concrete and measurable:

Should the recorded action be what the teleoperator commanded, or what the robot actually did?

The ACT/ALOHA paper answers this explicitly, and the reason is force:

"It is important to use the leader joint positions instead of the follower's, because the amount of force applied is implicitly defined by the difference between them, through the low-level PID controller." — Zhao et al., *Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware*

Our collection pipeline had drifted from that convention. This dataset is the record of what that costs, measured on real hardware.

Contents

Features

### ⚠️ This dataset is a frozen record of a convention that has since been fixed action here was produced by the collector before the fix described below. In the current collector, the joint_absolute action space records the leader's command — i.e. what this dataset stores in observation.commanded_state. The old behaviour was not renamed or kept alongside the new one; it was replaced, so that exactly one convention exists going forward. This dataset is deliberately kept as-is: it is the evidence, not a template. Do not use its action field as a model for new collection.

The critical relation, verified on every episode:

action[t] == observation.state[t+1]          exact, max |diff| = 0.000000
action[t] vs observation.commanded_state[t]  max |diff| = 0.144 … 0.253 rad

So action is the achieved trajectory, and observation.commanded_state is the command. The gripper column is the exception — there action[:,7] == commanded_gripper exactly. Within a single array, joints are achieved values while the gripper is a command. Watch for this when writing a training loop.

Lesson 1 — the action leads the state, but not nearly enough

A common first reaction is "if `action[t] = state[t+1]`, doesn't the action already lead?" It does — by exactly one tick. The problem is how small that lead is. Real values, episode 0, joint 1:

At t=4 the follower sits at −0.191 while the operator is commanding +0.013 — a lead of 0.204 rad. The recorded action leads by only 0.021 rad, one tick of already-slowed motion. The 0.2 rad of lead that was actually dragging the arm forward lives only in commanded_state.

The follower here runs behind a critically-damped reference filter (ωn = 10 rad/s, saturated at v_max = 1.0 rad/s, a_max = 4.0 rad/s²), and trails the leader by roughly 4 ticks (200 ms); |cmd − state| is median 0.006, max 0.28 rad.

Lesson 2 — double filtering

Collection:  leader u ──filter H──> follower q          recorded action ≈ q
Deployment:  policy outputs q ──filter H──> motion ≈ H·q = H²·u

What was the filter's output at collection becomes the filter's input at deployment. Simulating the real filter on this data (model validated against the recorded trajectory to median 1–2 mrad):

Note the last row: double filtering alone does not cause oscillation.

Lesson 3 — the stutter, and why it is always backward

Chunk-boundary discontinuity in action-chunked policies is a known phenomenon, usually described as periodic motion stuttering, and the standard mitigation is ACT's temporal ensembling (query every timestep, exponentially-weighted average over overlapping chunks).

What is specific to this action-space choice is that the discontinuity is systematically backward. Because the policy only ever emits "one tick past the current observation", it can never command a catch-up. Closed-loop simulation with a perfect policy — one that reproduces the demonstration exactly, so all learning error is removed:

Every single boundary regresses. When a chunk ends the arm is ~3.5 steps (174 ms) behind the commanded frontier; the next chunk, anchored to that lagged observation, re-issues about 2.5 steps the previous chunk had already commanded.

Two consequences worth internalizing:

1. 1.Shortening the chunk makes it worse. The regression (~2 steps) is set by filter lag and is nearly independent of horizon length, so a shorter chunk just pays that fixed cost more often. H=5 runs at half demonstration speed.
2. 2.Temporal ensembling would hide it, not fix it. Averaging overlapping chunks smooths the jump, but every chunk is still anchored to a lagged observation, so the slowdown remains.

The inference gap is a minor factor: at the measured 33–66 ms, setting it to zero changes nothing. Only past ~120 ms does it start to amplify the effect.

What to do instead — and what was actually done

Record the command, not the achieved state — i.e. train on observation.commanded_state. Replaying the leader command through the same filter reproduces the demonstration with median error 0.0020 rad at 100 % wall-clock: no double filtering, no regression, no slowdown.

This is now the collector's behaviour. gello/libero_format.py's joint_absolute action space was changed in place to emit the GELLO leader command; the old realized-trajectory behaviour was deleted rather than kept as a second option, so there is only one convention to reason about. Datasets collected after that change have action == observation.commanded_state.

The cost is that the leader signal is rougher — jerk RMS 44.8 vs 5.6 rad/s³ (8×), and 1.4× the high-frequency power. The reference filter removes it downstream, so it is not a safety issue, but whether a policy can fit it cleanly should be verified.

On action_ee

action_ee is an OSCPOSE-style EE-frame delta derived offline from the leader command via FR3 forward kinematics: `[(pcmd − pachieved)/0.05, axisangle(Rcmd · Rachievedᵀ)/0.5, gripper]`. Because it encodes the command minus achieved difference, it preserves the operator's lead and does not suffer the "cannot lead" failure above.

Caveat, and it matters: with ACTION_POS_MAX = 0.05 m, 1.8–16.7 % of frames saturate in these episodes — highest exactly where contact makes the command/achieved gap largest, which is precisely the force information the field is trying to preserve. Raise the constant to ≥0.1 m before relying on this field.

The FK used to derive it reproduces libfranka's own O_T_EE to a residual of 3.5 × 10⁻⁵ mm, recovering the standard Franka Hand flange→EE transform ([0, 0, 103.4] mm, −45° about Z).

Suggested exercises

1. 1.Plot observation.state[:,0], action[:,0], observation.commanded_state[:,0] on one axis. The one-tick lead of action and the far larger lead of commanded_state are immediately visible.
2. 2.Compute action[t] − observation.state[t+1] (expect exactly 0) and commanded_state[t] − observation.state[t] (expect median ≈0.006, max ≈0.28 rad).
3. 3.Implement the reference filter (2nd-order critically damped, ωn=10, saturations above), feed it action and then commanded_state, and compare the resulting trajectories against observation.state.
4. 4.Simulate chunked execution at horizons 15 / 10 / 5 and reproduce the efficiency table.

Limitations

• —4 episodes. Not a training set. Do not benchmark on it.
• —Single task, single scene layout.
• —action uses the deprecated convention on purpose — that is the point of the dataset.
• —Collected with the reference-filter parameters above; the specific numbers (2-tick lag, ~2-step regression) depend on those gains, though the mechanism does not.

Citation and references

• —Zhao, Kumar, Levine, Finn. *Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware* (ACT / ALOHA) — the leader-vs-follower action convention and temporal ensembling.
• —Action Chunking with Transformer (ACT) — overview
• —*SEAM: Smooth Execution of Action-Chunked Motion for Vision-Language-Action Policies*
• —Black et al. *Real-Time Execution of Action Chunking Flow Policies* (Physical Intelligence)

Collected with GELLO on an FR3 via libfranka; converted with LeRobot v3.0.