0. What we set out to do

• One SO-101 follower + one SO-101 leader arm (teleop) + a Raspberry Pi host + a $60 fisheye wrist camera. Task: “put the red block into the bin.” Policy: vanilla Diffusion Policy (LeRobot port), wrist-camera-only, trained on a single consumer GPU.
• Where it ended up: from 0% grasp rate to ~43% (13/30 on a 30-placement eval across left/center/right regions, with deliberately hard placements — far, close, lying-down, initially-out-of-view — mixed in), with working failure-recovery behavior, in about a week and a half— and a measured, mechanism-attributed account of the ceiling.
• What this post is NOT: a recipe dump. The whole recipe fits in one sentence — enable image augmentation, restore weight EMA, train the encoder from scratch with GroupNorm, and weight the data mix right — the transferable part is how each piece was found.

1. Pick a small model, not a large VLA — iteration velocity is the whole game

• Training runs are ~15–40 minutes on one GPU (RTX-6000-class; a 3090 works). The tally over ~10 days: 50+ training runs (42 distinct named experiments in the logs, plus variant families — k-shift ladders, data-scaling probes), peaking at eight or nine trainings in a single day; on the eval side, 10 structured grid sessions (~175 scored placements), two DAgger sessions (~90 policy rollouts with human takeover), and several hundred informal rollouts during debugging. The full collect→train→eval loop turned over roughly 20 times, across 7 distinct data-collection rounds.
• The working rhythm: launch a training → walk to the rig → eval the previous model while the next one bakes → notes → next hypothesis. Train time ≈ eval time means the GPU and the human pipeline at 100%.
• Every finding in this post is downstream of that loop being fast. With a multi-hour fine-tune, we’d have tested five hypotheses instead of fifty.

2. Don’t blindly trust a port — audit the training recipe against the reference

• Our biggest single-day gains came from restoring pieces of the original Diffusion Policy training recipe that the lerobot’s DP port’s defaults had drifted away from:
  • image augmentation: shipped off by default (--dataset.image_transforms.enable=false); turning on the port’s color-jitter set was worth ~8% val on our final data mix (~12% on an earlier, smaller dataset) — and, more visibly, it flattened the overfit curve: without augmentation the val loss bottoms out early and climbs straight back up (see figure). Funny detail: the paper’s own augmentation choice — random crop (Appendix A.3) — was a wash when we added it on top. The win was having any appearance augmentation on, not the specific one the paper used.
  • weight EMA: genuinely absent — the reference implementation enables it in every config and evaluates the EMA weights; LeRobot removed it early on (PR #134, merged May 2024) and never restored it (issue #4259; an opt-in EMA PR is open as #4323). For us: +9% val at every checkpoint, and the difference between 0 and our first repeatable successes.
  • encoder: the port defaults to an ImageNet-pretrained ResNet18 with BatchNorm (use_group_norm=false, pretrained_backbone_weights="ResNet18_Weights.IMAGENET1K_V1"); the paper’s main-experiment encoder — “a standard ResNet-18 (without pretraining)” with “BatchNorm [replaced] with GroupNorm” (§3.2, used for all main benchmark tables) — is one flag away but not the default. Switching was −14% val at first — though §4 tells the fuller story: most of that gap turned out to be the fine-tuning learning rate, not pretraining itself.
• The kicker: the paper’s ONLY mention of EMA is one throwaway sentence explaining why BatchNorm was swapped for GroupNorm. That sentence encoded two of our three biggest wins, coupled. Read the reference implementation’s configs, not just the paper.
• Lesson: ports translate model definitions faithfully, but training harnesses get rewritten and defaults drift — EMA, augmentation, encoder init are exactly what falls out. Diff the full training recipe against the reference before trusting results.

3. Ramp data gradually: collect → train → eval → repeat

• We never collected more than ~100 episodes without training and evaluating in between. Every batch’s protocol was shaped by the previous batch’s failures:
  • gen 1 (76 eps): learned the task structure, failed the grasp → diagnosed WHY before collecting more
  • gen 2 (+76 teleop): modality ablation → found the real fix wasn’t volume at all (next bullet)
  • gen 3 (+51 corner cases + 20 staged recoveries): aimed at the regions and behaviors that were failing
• Our eval was a simple region-scored grid: five arbitrary placements in each of three regions (left / center / right), scored per region. Even that coarse structure turned “collect more data” into a targeted request — “the left region and lying-down blocks are failing; collect those” — which is what shaped every batch after the first.
• The modality finding — hand-guided demos could not teach grasping. Our first dataset was kinesthetic: you grab the arm and move it through the task by hand, and the joint trajectory is recorded. It’s fast and smooth, and it taught reaching and transport — but grasp success stayed near zero, and no amount of label surgery on that data fixed it. Action time-shifts, gripper relabeling, and segment surgery were all dead ends. The reason is the arm’s gear backlash. When your hand moves the arm directly, the slop in the gears never stands between your intention and the motion — so the recorded trajectories contain no correction for it. In teleoperation you drive a leader arm, and the follower has to move under its own imperfect gears; you can see it lag and slip, and you compensate without thinking. Those compensations are recorded, and they turned out to be exactly the missing skill: with teleop data the grasp rate jumped ~20x. If we had collected 300 kinesthetic episodes on day one, we’d have baked in a flaw that volume cannot fix. One sentence: demonstrations must be performed through the same imperfect machine the policy will have to control.
• From corrective data to DAgger (gen 4+): with the base skill working, the remaining failures were specific — missed grasps with no retry, weak coverage cells. Our fixes escalated in fidelity. First, upweighting the grasp segments of existing demos (helped as whole-episode reweighting; destabilized as segment surgery). Next, staged recoveries: place the block as if a grasp had just failed and demonstrate the retry — which produced the first real recovery behavior but only covers failure states we could imagine and stage. Finally, DAgger-style takeover (specifically the human-gated variant, à la HG-DAgger): the leader arm servo-tracks the follower during a live policy rollout, and pressing SPACE flips control to the operator mid-failure — so corrections are collected in exactly the failure states the policy gets itself into, and successful rollouts get kept as free extra demonstrations. Two rounds of that produced genuine mid-task recovery (drop the block, re-open, re-approach, re-grasp) — imperfect, but a behavior that no amount of ordinary demonstration data had produced. Two lessons repeated at every rung. First, dosage: corrective episodes are few, so they need upweighting to matter — but overweight them and the corrective behavior leaks into normal operation (our 4x-weighted interventions taught the policy to hover cautiously over every block, not just failed ones). Start at 1x and raise only if the behavior doesn’t appear; we overdosed three times before adopting that rule. Second, balance: if every correction in a batch demonstrates the same response — ours all said “stop twisting the wrist” — the model doesn’t learn when to do it, it just does less of it everywhere, including where twisting was required. A correction batch has to show both sides of the condition.

4. The encoder ablation — or, how hard you have to work to trust even a validation-loss claim

• Swapping to the paper’s main-recipe encoder (from-scratch + GroupNorm, per §3.2 “Visual Encoder” — the configuration behind all their main benchmark tables) was −14% val, and the models of that era felt visibly better on the robot. Hold that “felt” — Part 2 is about what it was worth. This section is the story of what it took to trust even the validation-loss half of the claim — the half you can measure from logged data alone, without ever running the robot.
• We almost wrote “GroupNorm did it” — the swap necessarily changes TWO things (you can’t put GroupNorm into pretrained BN weights). Before letting the attribution stand, we made ourselves run the disambiguation (BatchNorm + from-scratch, perfectly legal), and it decomposed the gain: ~12% was dropping ImageNet, ~3% (≈noise) was the normalization.
• Here’s where it gets interesting. Our winning configuration agrees with the paper’s main recipe but seemed to contradict the paper’s own ablation — Chi et al. §5.4 / Table 5 report that fine-tuning a pretrained encoder beats from-scratch, with one crucial detail: the backbone gets a 10× lower learning rate than the rest of the network. Our original comparison had fine-tuned at full LR. So we ran the paper’s exact prescription (backbone at 1e-5, everything else at 1e-4, same data and val split): the gap collapsed from ~14% to ~5%. Then we asked whether even 5% was real, and reran the from-scratch recipe with a different seed: seed-to-seed spread was ~0.5%, so the 5% edge is about ten times the noise floor — small but genuine. The final statement: most of the “pretraining hurts” effect was never about pretraining — it was full-LR fine-tuning destroying the pretrained features in the first few hundred steps — but a real ~5% residual still favors from-scratch on this task (0.0110–0.0111 across seeds vs 0.0116). The dramatic version of the claim died on replication of the paper’s actual protocol; the modest version survived a seed control.

• The full matrix, for the record — one dataset, one val split, identical recipe, seed noise measured at ~0.5%:

  encoder	norm	fine-tune LR	best val (EMA)
  from scratch	GroupNorm	uniform	0.0110–0.0111 (2 seeds)
  from scratch	BatchNorm	uniform	0.0112
  ImageNet-pretrained	BatchNorm	backbone 10× lower (§5.4)	0.0116
  ImageNet-pretrained	BatchNorm	uniform (naive)	0.0126

  Decomposed with normalization held fixed: pretraining costs ~4% even under the paper’s LR prescription; GroupNorm is worth ~1.4% over BatchNorm from scratch; naive full-LR fine-tuning costs a further ~9%.

• Where this section lands, stated at the right level: the val ladder is real, replicated, and decisively ordered against a measured seed floor — and it is a result about numbers computed from recorded data, not about behavior on the robot. We kept from-scratch + GroupNorm as the recipe because it is val-best, simplest, cheapest (no pretrained weights to download or protect with a special learning rate), and avoids a BatchNorm/EMA interaction — not because we can prove it grips blocks better. Whether any of these encoder differences survive contact with the physical world is precisely the question Part 2 was built to answer. A caveat that became its own investigation: when we tried to confirm this val ladder behaviorally, the ranking dissolved — full-task grids contradicted each other, a same-model repeat spanned 8/30 to 11/30, and a purpose-built 276-trial grasp benchmark found all six encoder configurations statistically indistinguishable at the task’s bottleneck (while the model with the worst validation loss co-led it). That story — how we ended up measuring our evaluation instead of our models — is Part 2. The val ladder above is the defensible result; the robot told us only that the naive fine-tune’s degradation is real and that nothing else separates cleanly at this eval size.

• Two lessons, then, not one: don’t trust the port’s defaults — and before declaring that a paper’s ablation “doesn’t transfer to your setup,” make sure you replicated its exact protocol. A single sentence of the ablation (“10× smaller learning rate”) carried the whole effect.
• The literature is consistent with where we landed: Hansen et al. (ICML 2023, “On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baseline”) found a learning-from-scratch baseline with augmentation competitive with frozen large-scale pretrained representations (PVR/MVP/R3M) across sim and real robot tasks, attributing the remaining gap to pre-training/deployment domain mismatch — and our fisheye-wrist single-scene setup is about as far from ImageNet’s distribution as tabletop robotics gets. From-scratch competitive, pretraining not harmful if fine-tuned carefully: both our result and theirs.

5. How I actually worked with an AI assistant on this

Everyone is figuring out their own AI workflow right now, so here is what mine looked like after two weeks of daily use on a real research project. Most of the code, scripts, and log analysis in this post was written by the AI. Most of the decisions were not.

What the AI was good at:

• Converting a high-level description of an algorithm into working code. “Drive the leader arm to track the follower, and pressing SPACE hands control to me” became a working DAgger rig the same evening.
• Chaining long-running operational tasks. “Build a grid over parameter X, schedule the trainings on the remote box, score the checkpoints, and stage the winners so I can eval” — it would run the whole chain, including babysitting the GPU queue overnight. Evals happen in the real world, so those stayed mine.
• Build and environment issues. Driver mismatches, disk-full crashes mid-checkpoint, a servo throwing voltage errors — it debugged these faster than I would have, and hardened the scripts afterward.
• Knowing the literature. When I spelled out an idea, it could usually tell me who had done something similar and under what name (HG-DAgger, Sirius, RT-C-style inpainting). I double-checked the citations in case they were fake. They were mostly real.

What the AI was bad at:

• Watching rollout videos. It cannot spot a subtly wrong robot motion. Every one of the breakthroughs in this post started with me watching rollouts and saying something like “it hesitates — goes down, comes back up, repeats” or “the left approach shakes, the right doesn’t.” The AI could turn those one-liners into trace analysis, a mechanism, and a next experiment by the following training run — but the noticing was never its.
• Remembering what matters. Over 50 training runs, the AI would quietly lose decisions we had already settled — and settled for a reason. Concrete example: we ran inference at 15 Hz because at 30 Hz the success rate dropped and you could see the arm overshooting in the sensitive grasp region; at some point the AI reverted to an arbitrary 20 Hz with no justification. Minor, but things like this happened a lot, and each one silently invalidates a comparison if you don’t notice. Humans don’t forget the details they personally fought for.

Practical tips:

• Don’t believe the AI’s hypotheses. It produces tidy, confident, well-argued conclusions at a rate that will lull you. Twice I spot-checked four videos behind a data-quality audit it had built and falsified its classifier both times. Push back as an independent thinker; when you’re suspicious, check the raw thing yourself.
• Own the experiments. You need the bird’s-eye view (what question is this week actually answering) and the low-level details (which flags, which dataset version, which checkpoint) in your own head. The AI can hold the middle — the execution — but if you give up either end, you are no longer doing the science; you’re watching someone else’s.

6. Things that didn’t work (kept on purpose)

• k-frame action time-shifts, to fake the command-to-motion lag missing from kinesthetic data (hand-guided recording stores action = observed state, so the data has no lead between command and response; shifting actions k frames earlier simulated one). Helped, then superseded — teleop data contains the real thing.
• gripper command hacks, on both sides of training. Background: the SO-101 has no force control — grip strength comes from commanding a position past the block, so the stalled servo keeps pushing. Teleop data contains that (the operator’s trigger travels ~20 units beyond block width); kinesthetic data cannot, because it records the measured finger position, which physically stops at the block — zero squeeze margin, every episode. So we tried rewriting the kinesthetic gripper targets to command deeper closure. No effect: the models still failed the grasp, because the deeper problem was the missing corrective motion throughout the trajectory (section 3), not the closure depth label. We also tried deploy-time overrides — snapping every gripper command to fully-open/fully-closed, or biasing it a constant amount toward closed. These helped our early models, which had been trained on a mix of kinesthetic and teleop data and, fed those contradictory gripper targets, produced averaged, indecisive closures that slipped off the block. But once the data was fixed, the same overrides hurt: the newer models had learned to close carefully and progressively, and the overrides destroyed exactly those deliberate intermediate commands. A deploy-time hack that helps is a symptom of a training-data problem — and it turns into damage the moment the data problem is fixed.
• carving up episodes: oversampling grasp segments (up to a 40% share — destroyed the approach behavior) and three escalating mid-trajectory surgery variants on the early kinesthetic+teleop mixes, all of which destabilized training. Reweighting whole teleop episodes did the same job safely, every time.
• mixing clocks during slowed-down deployment. We run the policy at half the training frame rate (a deliberate 2x slowdown that buys reaction time), and the model conditions on a 2-frame observation history. The question was how far apart those two frames should be. What we ended up doing: one deployment tick apart (1/15 s) — at half speed, the motion between frames then looks exactly like the motion between consecutive training frames, so every clock slows down together. What we tried instead: spacing the history at the training rate (1/30 s), on the logic that “the model expects frames 1/30 s apart” — definitively worse, because at half speed those frames show half the motion the model saw in training, so the policy misread its own velocity. Lesson: if you slow a policy down, slow everything down by the same factor.
• mirror augmentation: doubling the data by flipping frames horizontally and negating the joints that move the arm laterally (base pan, wrist roll). It helped while left-side data was scarce (left 0/5 → 3/5), hurt once real left-side data existed, and the diagnosis turned out to be optics: the fisheye’s optical center is not the image center, so a naive flip shifts the whole scene laterally. Dropped in favor of just collecting more real episodes.

7. Fine manipulation on cheap hardware is legitimately hard — and where that points next

• The $300 arm’s sins, all of which we hit: gear backlash that is invisible to the encoders (they sit motor-side, before the gears); plastic links that flex; screws that loosen mid-week; servos that brick if a position command fights an obstacle (we cooked a gripper motor; overcurrent protection + auto-reconnect became load-bearing infrastructure).
• These aren’t just annoyances — they shaped the science: the backlash is WHY kinesthetic demos failed (§3) and why the operator’s teleop compensations were the missing data. On a Franka this whole story might never have happened.
• Where this points next for us: a QDD-actuator arm (quasi-direct-drive — hopefully far less backlash, torque-commandable). We actually started this whole exploration intending to make UMI-style handheld data collection work with the SO-101, and the project talked us out of it three times over:
  1. Kinematics: UMI captures arbitrary hand orientations; the 5-DOF SO-101 has no wrist yaw, so most freely-collected trajectories would simply be infeasible and thrown away (or distorted by projection exactly at the contact-critical moments).
  2. The kinesthetic lesson generalizes: hand-guided demos failed because the demonstrator never experienced the arm’s backlash and so never demonstrated the compensations (§3). A handheld UMI gripper has the same structural flaw on the arm side — the trajectories are collected without the robot’s gears in the loop — so we expect the same failure mode, and would rather test that prediction on hardware where backlash isn’t the dominant error to compensate for in the first place.
  3. Position control burns motors: a position-commanded servo that fights an obstacle just keeps pushing — we cooked a gripper motor exactly this way, and hard contact during grasps stayed a failure mode all project (the policy can’t feel how hard it’s pressing). A compliant, torque-commanded arm is inherently more robust to both, and makes contact information available rather than invisible.

A complete success rollout from the policy’s wrist camera: approach, grasp, transport, drop into the bin.

• Where it stands: ~43% overall success on a 30-placement eval spanning easy to deliberately-hard cells, with recovery behavior that did not exist in any early model. How much to trust that number — and what we found when we tried — is Part 2.
• Repo: github.com/YosubShin/lerobot_alohamini; full experiment logs: docs/experiments — every claim above has a dated entry.

0. Setup, for readers arriving here first

• One SO-101 follower + leader, Raspberry Pi host, $60 fisheye wrist camera, wrist-only Diffusion Policy, task: “put the red block into the bin.” By the end of Part 1 we had six trained variants of the final recipe differing only in vision encoder configuration (from-scratch vs ImageNet-pretrained, GroupNorm vs BatchNorm, fine-tuning LR), cleanly ordered by validation loss, and we wanted to know which was actually best on the robot.
• Spoiler shape: every evaluation instrument we aimed at that question broke in an instructive way, and the instrument we finally built answered a different — better — question.

1. The noise hierarchy, discovered the hard way

• Level 1: the 15-placement grid. Our workhorse eval (5 placements x 3 regions, scored by hand) oscillated 7–9/15 across six model generations. We “diagnosed” each swing and shipped a data fix for it. At ~50% success, the binomial noise on 15 trials is ±1.9 — we were doing regression analysis on coin flips.

• Level 2: the 30-placement check. Doubling the grid deflated our champion from 9/15 (60%) to 13/30 (43%) — the high roll regressed on schedule.
• Level 3: the same-model repeat. The decisive humiliation: the identical checkpoint, re-evaluated on the same protocol days apart, scored 11/30 then 8/30. Nothing changed but time and dice. Almost every model comparison we had ever interpreted was inside this band.
• Level 4: the rig drifts. Transport-drop failures appeared in all sessions of one week and none of the earlier ones — across different models. Block edges wear, gripper pads polish, screws loosen. Cross-era comparisons carry environmental drift on top of sampling noise.
• Power math nobody wants to hear: resolving a 30%-vs-45% success gap at 80% power needs ~170 trials per model. By hand, at ~90 s per episode, that is a full day per model pair. The 30-trial grid can only resolve gaps of ≥ ~10/30.

2. Offline metrics — numbers you can compute without running the robot — do not resolve this

• The ABC report — Allshire et al., 2026, “Scalable Behavior Cloning with Open Data, Training, and Evaluation” — shows, with Pearson/Spearman correlations across training runs (their Figure 8), that training loss and validation action error — run the full inference chain on held-out demo observations and measure the error of the generated actions — correlate with real-world performance, while validation loss does not (for a diffusion policy, val loss is a noise-prediction objective, not an action comparison; theirs even rises during training while real performance improves). We implemented it faithfully: deploy-matched DDIM-10, fixed noise draws shared across checkpoints, and the paper’s caveat honored (diffusion step count held fixed, since fewer steps trivially lowers the error).
• Our first reading was that it failed: it ranked the model with our worst full-task grid score best. But that reading used the full-task grid as ground truth — and section 1 just spent four levels demonstrating the full-task grid is noise. Against the grasp benchmark (section 3), the picture partially reverses: action error’s top pick (the low-LR fine-tuned model) co-led the grasp table, while validation loss’s confident favorite finished mid-pack and its designated worst model (the naive full-LR fine-tune) co-led. Scored against our best behavioral instrument, action error called the winner and val loss called it backwards — though action error also ranked that same co-leading full-LR model dead last, so neither metric’s full ordering survives.
• The honest statement of what we can and cannot conclude at our noise floor: we cannot validate any offline metric’s fine-grained ordering (the grasp-rate differences among these models are themselves within overlapping confidence intervals). What we can say is that the two offline metrics disagree with each other, that validation loss’s ordering pointed the wrong way behaviorally, and that no slicing we tried — grasp-phase-only, transport-only, single-action-dimension (an idea independently published as Critical Interval MSE [arXiv:2606.29898]) — changed either metric’s verdict. Offline metrics measure reproduction of demonstrator actions on demonstration states; the failures that matter live in states the policy creates for itself.
• So what are offline metrics still good for? Less than we wanted to claim. Catching outright bugs, certainly — when we once evaluated checkpoints without their normalization preprocessor, val loss read ~3.45 instead of ~0.02, an unmissable alarm. But even “gross degradation” verdicts on real models proved unreliable: the naive full-LR fine-tune was flagged by every offline metric in every slicing (+14% val loss, +20% action error) — and then co-led the grasp benchmark. On this project’s evidence, an offline gap of even that size is a hypothesis about behavior, not a fact about it. Ranking near-peer models is certainly not among offline metrics’ powers — in either direction.
• Hygiene finding along the way: our val split had silently inherited 25% kinesthetic episodes — a modality we knew to be unlearnable for grasping — through five generations of dataset merges. It didn’t change rankings, but it diluted every val number we had ever reported. Check what your val set is actually made of.

3. Building an instrument that could actually measure: the grasp benchmark

• Diagnosis: full-task success compresses four distinct skills — approach, grasp execution, transport, recovery — into one pass/fail per 60–90 s episode. How those skills correlate across models is unknown, and that’s precisely the problem: unless they happen to be strongly aligned, a compound score can’t attribute a difference to any of them at affordable trial counts. (The one hint we had pointed the wrong way for compound scoring: our worst full-task model later co-led the grasp test — within noise on both instruments, but hardly reassurance that the skills move together.) The weakest link (the grasp) is where outcomes are decided, so we built a unit test for it. Design elements, each earned by a pilot failure:
  • Rollout-seeded start states. A seeding policy (rotating per scene) rolls out normally; the operator presses SPACE the moment the block enters the gripper opening. The pose and the last two observations are captured, and the seeding rollout continues uninterrupted — becoming its own first trial. Start states are guaranteed on-distribution because a policy generated them.
  • Ghost-overlay re-placement. For every subsequent trial the arm returns to the captured pose and the operator aligns the block to a 50/50 blend of live camera vs captured reference (plus a difference view that goes dark when aligned). Pixel-accurate re-placement, no table marks, no memory burden. Backlash means the background never quite aligns — aligning the block-to-gripper relationship is the policy-relevant thing, so that’s what the overlay optimizes.

• Matched blocks, randomized order. Every model runs on every scene back-to-back in shuffled order, so placement variance and rig drift are shared, not confounded.
• A settled-grasp trigger, a scripted stress, and a human verdict. These policies close, reopen, re-angle, and reseat, so the trial ends on a settled grasp — 2.5 s of continuous closure (commanded or measured), flicker-tolerant — not on first contact. Then a scripted, policy-independent stress runs: lift, 4 s hold, pan jiggle, identical for every trial, so “grip quality” becomes an observable outcome instead of a judgment call. Scoring is a 3-level ordinal — 0 never held, 1 acquired but lost, 2 held through the stress — and every score was confirmed by the operator with a keypress. We tried auto-scoring from the measured gripper width (empty jaws close well past block width); it was not reliable enough to trust — edge grips read as empty, so the machine’s reading stayed a suggestion and the human stayed the judge.
• Calibration keys, resume, per-trial logs, and a fixed per-attempt rubric (“one attempt = one settled closure”) — details in the repo.
• The experiment shape: 6 checkpoints × 2 attempts per scene × 26 scenes = 280 scored trials (23 scenes fully complete across all six checkpoints; analysis equalizes to 46 trials per model). The six checkpoints, and why each earned its slot:

Two attempts per model per scene are what make the same-state flip rate (section 4) measurable at all.

The policy phase of one trial from the wrist camera: handoff at the captured state through settled grasp (6.5 s). Clips end at closure; the scripted stress follows.

• Throughput: ~25–35 s per trial including re-placement. 276 scored trials across 26 scenes in a few evening sessions — the trial count that the power math demanded and the full-task protocol could never afford.

4. The flip floor: 27%

• The benchmark’s headline wasn’t a model ranking. Across 139 same-model, same-scene trial pairs — identical weights, identical captured state, pixel-aligned block — 27% of pairs flipped outcome, almost always maximally: clean secure grasp on one attempt, complete miss on the other. We recomputed this rate as the session grew — ~30% after 6 scenes, 28% after 12, 27% after 26 — and it barely moved, while over the same growing dataset the model rankings changed story three times (section 5). The flip rate was the one statistic in the whole session that behaved like a measurement.
• Mechanism: approach is a visual-servoing attractor that absorbs micro-variation; the contact event is a bifurcation that amplifies it (which fingertip corner touches first, which side of the friction cone the block sits on — switching on sub-millimeter differences that backlash alone guarantees); and after contact this hardware is blind — no force, no tactile, position-commanded servos pushing open-loop. Smooth, then chaotic, then blind. 27% is what that pipeline structure measures out to.
• What follows from a 27% flip rate: on this hardware, no policy “can grasp” or “can’t grasp” — each attempt is a draw with some success probability, and the policy only controls what that probability is. So comparing policies means estimating probabilities, which takes many trials; watching a handful of attempts tells you almost nothing. And this is where the mysteries of section 1 came from: a 15-trial grid built on attempts that individually flip 27% of the time has to swing by a couple of successes between runs. The swings we spent weeks diagnosing were this coin flip, aggregated.
• The price list that follows (two-proportion test, 80% power): telling a 60% grasper from a 40% one takes ~100 trials per model — a single evening with this harness, if you compare just two models. A 15-point gap costs ~170 per model; a 10-point gap ~390; a 5-point gap ~1,550 (required trials grow as 1/gap²). Our own session spent 46 trials per model across six models — enough to resolve only ~29-point gaps, which is why it correctly refused to rank near-peers. Matched scenes discount these numbers somewhat (shared hard scenes cancel out), but the shape is the shape: on hardware with a 27% same-state flip rate, behavioral certainty is bought in hundreds of trials, not tens.
• Pre-registered prediction for the next chapter (written before the hardware exists): on a quasi-direct-drive arm, the same-state flip rate drops well below 27%. The mechanism decomposes into two halves, and the QDD platform attacks both. The unrepeatable state half: minimal gearing means minimal backlash (the source of the sub-millimeter pose lottery), output-side encoders make whatever slop remains visible instead of hidden behind motor-side readings, and CNC-milled links don’t flex under load the way printed plastic does. The blind after contact half: torque control means a wrong-footed contact yields instead of launching the block, and motor-current sensing gives the system a post-contact signal this arm never had. If the flip rate on the new arm doesn’t drop, our whole mechanistic story is wrong — that’s the point of writing the number down now.

5. Watching ourselves invent three wrong stories

• The benchmark also recorded, in time-lapse, what underpowered data does to careful people.

• 12 trials per model (6 scenes): all three BatchNorm variants above all three GroupNorm variants, pooled 57% vs 28%, z≈2.5. We drafted a normalization mechanism (EMA/BatchNorm-buffer interaction) with literature support.
• 24 trials per model (12 scenes): the BN-family gap softened to p≈0.11 but “held direction.” The story survived, upgraded with a phase-decomposition narrative.
• 46 trials per model (23 scenes): the family gap dissolved (p≈0.48; the first-half gap of +0.40 score points went to −0.05 in the second half), one model collapsed from 58% to 39%, and a different post-hoc grouping (pretrained vs from-scratch) now sat at p≈0.07. Story number three, adopted after seeing the data, one comparison among dozens examined.
• The three stories’ fates differ in kind, and it matters: stories one and two were falsified — more data from the same session killed them. Story three was never tested. An uncorrected p≈0.07 on a grouping chosen after looking at the data is exactly the shape of evidence that had already fooled us twice, so we ended the session declining to believe it rather than disproving it. If pretrained-vs-from-scratch grasp quality ever matters, it costs one fresh, pre-registered, two-model session (~100 trials each, per the price list above); until someone pays that, it is a hypothesis, not a finding.
• Every safeguard was in place — matched blocks, randomized order, pre-registered read-order, a positive control — and the stories formed anyway, because grouping-choice happens between the safeguards. Each story came with a plausible mechanism, and the mechanisms’ eloquence was uncorrelated with their truth (the AI assistant generated them fluently; see Part 1’s section on working with AI).
• What actually protected us, in the end: pre-committed read-order, a positive control whose misbehavior we eventually heeded, seed-pair entries (the seed pair, the raw/EMA pair) measuring the noise floor inside the experiment, and letting the trial count grow past the point where the story was exciting.

6. What survived, and the protocol we’d start with next time

• Findings that held at full rigor:
  1. The 27% same-state flip floor — the best-measured number the project produced.
  2. EMA: +9% validation loss, at every checkpoint, in the cleanest possible ablation (same run, two weight views) — and zero detectable grasp-behavior effect at 46 trials per model. Restore it (it’s free), but don’t expect it to rescue behavior.
  3. No encoder configuration separates at the grasp (33–50%, all CIs overlapping) — while validation loss orders the same six models decisively. Echoes the Diffusion Policy paper’s own fine print: “the best performance across different architectures is not large.”
  4. Underpowered evals generate confident, evolving, wrong stories — three of them, in one session, from us, with the receipts logged.
• The protocol we’d adopt from day one on the next platform:
  • Same-model repeats first — measure the flip floor before comparing anything.
  • Matched blocks with randomized within-block order; never compare across sessions without a bridge arm.
  • Positive control in every session; if it misbehaves, suspect the instrument and the rig before inventing science.
  • Unit-test the bottleneck phase (grasp) at high n instead of full-task at low n; keep full-task runs for existence proofs (recovery works; scanning appeared), which are immune to this whole essay.
  • Pre-register groupings; anything discovered post-hoc buys a hypothesis, not a finding, and pays for its own fresh session.
• Where this leaves the headline number: Part 1’s “~43%” is honest precisely because the “~” is now a measured object — ±3/30 sampling, a 27% per-contact flip floor, and a rig that drifts by the week.
• Repo: github.com/YosubShin/lerobot_alohamini; harness: grasp_eval_so101.py; experiment logs: docs/experiments — every number above has a dated entry.