ReconVLA を超え：言語注意力掩蔽による付注なし視覚 Grounding

注目の付与付注を言語駆動の注意力マスクに置き換えることで、ロボットの感知を付注なしとし、推論を最大 5 倍に高速化しました。そこに至った過程を振り返ります。 お前からロボットアームがテーブルを挟んで向かい合っている状況を想像してください。お前は言う："黒いボウルを棚に置け。"アームは動く。しかしそれはボウルへ向かうわけではない。空気に浮遊し、迷う。そしてそれは誤ったものを取り込む。外からは、これは単なる調整の失敗に見えるかもしれない。しかし内側から見ると、それはロボットの世界をどのように感知するかという根本的な問題である。 ロボットは言語について混乱していたわけではない。言葉は完璧に理解していた。失敗の原因は視覚にあった。その認識システムは、テーブル、壁、棚のハンドル、ボウル、ボウル側のコップに至るまで、シーンの全体をほぼ均等に注意力を分散させていた。指示で実際に命名されたその一つのオブジェクトに注意力を集中的に集中させる信頼できる仕組みを持ってはいなかった。この散漫な認識が、現代のロボティクスにおけるほとんどの操作の失敗の根本原因である。 最近の論文「ReconVLA」がこれを解決しようとした。私はこの論文を注意深く読み、その仮説を検証し、それを実装および拡張する意味について深く考え、数ヶ月に及ぶ時間を費やした。私が得たのは、いくつかの点では感銘を受けると同時に、真に驚惧を覚えるものだった。この記事は、その探求の物語、そしてそれに回答として設計したアーキテクチャである。 ReconVLA が正しかった点 ReconVLA behind には、優雅な核心的な洞察がある。外部オブジェクト検出モジュール（これはラベルされた境界ボックスを必要とする）を追加したり、動作予測前に境界ボックストークンを生成したり（これは出力形式を変更する）のではなく、ReconVLA は視覚再構築を純粋に内部的な監視シグナルとして使用する。 それはどのように機能するか。モデルは、操作対象に対応する入力画像内の"注目領域"を特定する。その後、バックボーンの内部的な視覚トークンのみを使用して、その注目領域を再構築する拡散トランスフォーマーヘッドをトレーニングする。論理は簡単である：もしバックボーンがターゲットオブジェクトの形状と正確な位置をエンコードしていないなら、注目領域を再構築できない。再構築のタスクは、バックボーンに幾何学的に精密で空間的に構造化された表現を開発させる勾配圧力を生み出す。 再構築のタスクはバックボーンをターゲットオブジェクトの形状と位置をエンコードせよと強制する。それがボウルの場所を知っていないなら、ボウルを再構築できない。 推論時、再構築は起こらない。改善されたバックボーンは単に良い動作予測を生み出すのみである。外部モジュールなし、追加の出力形式なし、目に見える縫合なし。ReconVLA は LIBERO-Spatial、LIBERO-Long、および CALVIN ベンチマークで、OpenVLA や RT-2 スタイルのベースラインを超えている。彼らが可視化する注意力マップは、真に焦点を絞った感知を示している。これが本当の進歩である。 では、問題はどこにあるのか？ 私が発見したギャップ 論文を注意深く読み、それら reproduce, extend, trust するための何を必要とすることかを考える後、私は 3 つの本質的な問題を特定した。 ギャップ 1：注目領域は隠された作業を行っている 再構築対象として使用されている注目領域は、ロボット眼追跡またはトレーニングデータ内の注注に由来する。この論文では、これらが BridgeData V2、LIBERO、および CALVIN という 3 つのデータソース全体にわたってどのように得られているかが、完全に特定されていない。注目領域が論理的に導出されている場合（例えば、指示で命名されたオブジェクトを取り囲む境界ボックスを描くこと）に、この方法は埋め込まれた円形な依存関係を持つ。 再構築目標は、動作を誘導する同じ言語指示から計算される。モデルは短絡を学ぶことができる：シーンに対する真幾何学的理解の開発ではなく、言語的ヒントに注意力を向ける。どちらの場合も良いベンチマークの数値を得られるし、その違いを区別する手段も存在しない。 重要なのは、この論文に注目領域の再構築とランダムなパッチ領域の再構築を比較するアブレーション実験がないという点である。この単一の欠落した実験は、我々にパフォーマンスの改善を注目領域に帰因できないことを意味する。

Replacing gaze annotations with language-driven attention masking makes robot perception annotation-free and up to 5x faster at inference. Here is how I got there. Picture a robot arm sitting across a table from you. You say: "Put the black bowl in the drawer." The arm moves. But not toward the bowl. It hovers. It hesitates. Then it grabs the wrong thing. From the outside this looks like a minor coordination failure. From the inside, it is a fundamental problem with how the robot perceives the world. The robot was not confused about language. It understood the words perfectly. The failure was visual. Its perception system was distributing attention more or less equally across the entire scene: the table, the wall, the drawer handle, the bowl, the cup beside the bowl. It had no reliable mechanism to concentrate attention on the one object the instruction actually named. This scattered perception is the root cause of most manipulation failures in modern robotics. A recent paper called ReconVLA attempted to solve this. I spent a significant stretch of time reading it carefully, stress-testing its assumptions, and thinking about what it would mean to implement and extend it. What I found impressed me in some ways and genuinely troubled me in others. This post is the story of that investigation, and the architecture I designed in response. What ReconVLA Got Right The core insight behind ReconVLA is elegant. Instead of adding an external object detection module (which requires labelled bounding boxes) or generating bounding box tokens before action prediction (which changes the output format), ReconVLA uses visual reconstruction as a purely internal supervisory signal. Here is how it works. The model identifies a "gaze region" in the input image corresponding to the manipulation target. It then trains a diffusion transformer head to reconstruct that gaze region using only the backbone's internal visual tokens. The logic is clean: if the backbone does not encode the shape and precise position of the target object, it cannot reconstruct the gaze region. The reconstruction task creates a gradient pressure that forces the backbone to develop geometrically precise, spatially structured representations. The reconstruction task forces the backbone to encode the shape and position of the target object. If it does not know where the bowl is, it cannot reconstruct the bowl region. At inference, no reconstruction happens. The improved backbone simply produces better action predictions. No external module, no extra output format, no visible seams. ReconVLA outperforms OpenVLA and RT-2 style baselines on LIBERO-Spatial, LIBERO-Long, and CALVIN benchmarks. The attention maps they visualise show genuinely more focused perception. This is real progress. So where is the problem? Where I Found the Gaps After reading the paper closely and thinking through what it would take to reproduce, extend, and trust these results, I identified three substantive issues. Gap 1: The gaze region is doing hidden work The gaze regions used as reconstruction targets come from robot eye-tracking or annotation in the training data. The paper does not fully specify how these are obtained across all three data sources: BridgeData V2, LIBERO, and CALVIN. If the gaze regions are derived heuristically (for example, a bounding box drawn around the object named in the instruction), then there is a circular dependency buried in the method. The reconstruction target is computed from the same language instruction that guides the action. The model could learn to shortcut: attend to language cues rather than developing genuine geometric understanding of the scene. You would get good benchmark numbers either way, and you would have no way to tell the difference. Critically, there is no ablation in the paper comparing reconstruction of gaze regions against reconstruction of random patch regions. This single missing experiment means we cannot attribute the performance improvement to gaze-specific grounding versus the simpler hypothesis that any auxiliary reconstruction task would help. Without it, we do not know what the method is actually learning. Gap 2: The diffusion transformer adds overhead they never measured Diffusion models require T iterative denoising steps per forward pass. In robot manipulation, inference latency directly determines control frequency. If your model runs at 1 Hz, it cannot close a control loop that needs 10 Hz. ReconVLA does not report any inference latency benchmarks. For a robotics paper, this is a significant omission. Diffusion Policy, for comparison, explicitly benchmarks latency and shows diffusion-based policies typically operating at 1 to 2 Hz due to iterative denoising. ReconVLA provides no comparable numbers. Gap 3: Evaluation scope is narrower than the generalisation claims LIBERO and CALVIN are simulation benchmarks. Real-world results are limited to qualitative demonstrations on a single robot arm. The pretraining dataset overlaps with evaluation environments, which raises data leakage concerns. CALVIN evaluates long-horizon tasks with a fixed language vocabulary, which does not test open-vocabulary instruction following: the core promise of VLA models. Taken together, the generalisation claims exceed what the evaluation design can actually support. The Architecture I Designed: LA-ReconVLA The research question I set myself: can we replace gaze-region supervision with language-driven attention masking, deriving reconstruction targets that are semantically grounded in the task instruction, while replacing the diffusion transformer with a computationally efficient MAE decoder? The two problems addressed simultaneously: annotation dependency and inference overhead. How It Works, Step by Step 1. Extract cross-attention maps from the backbone Using PaliGemma-3B as the backbone, I extract cross-attention scores between language tokens and image patch tokens from the last 3 transformer layers. These are aggregated across all language tokens and attention heads to produce a single saliency map A over the 196 patch positions (a 14x14 grid for a 224x224 image). The aggregation uses the last 3 layers specifically to reduce noise from the frozen earlier layers. 2. Apply attention-guided masking Select the top 49 patches from the saliency map: the top 25% of the image by cross-attention score. These patches are semantically grounded in the instruction because they come directly from the backbone's own language understanding. The word "bowl" in the instruction produces high attention weights on patches containing bowl-like features. The binary mask M produced by this process is the reconstruction target. 3. Single-pass MAE decoder reconstruction A 4-layer transformer decoder (hidden dimension 256, 8 attention heads) receives unmasked patch tokens from the backbone and learnable mask tokens at masked positions. It reconstructs pixel values at masked positions in a single forward pass. Reconstruction loss is pixel MSE over the masked region. For spatial grounding, coarse reconstruction at correct locations suffices. The geometry matters more than photorealism. 4. Joint training with action prediction The total loss combines action prediction and reconstruction with a weighting hyperparameter. Action prediction uses cross-entropy over discretised action bins (7 degrees of freedom x 256 bins per DoF). Lambda defaults to 0.5 with ablations planned at 0.1 and 1.0. Why This Should Work: The Theoretical Reasoning I want to be honest that this is a hypothesis until the experiments say otherwise. But the theoretical grounding is solid across four independent arguments. Self-supervised learning tells us this will help Masked Autoencoder (MAE) research established that masking semantically meaningful regions produces stronger visual representations than masking random patches or using contrastive objectives. By masking specifically the patches the language model attends to when processing the instruction, we create the hardest and most informative prediction problem we can construct without external labels. The backbone has to predict task-relevant content or fail at reconstruction. Information bottleneck creates the right pressure Masking high-attention patches and requiring their reconstruction creates an information bottleneck. The backbone must retain spatial information in its latent representations that it would otherwise be free to compress away. This regularisation pressure pushes the backbone toward encoding geometric structure as a side effect of minimising reconstruction loss. Direct gradients are better than multi-step gradients In diffusion models, gradients flow through T denoising timesteps before reaching the encoder. Each step introduces noise into the gradient signal. The MAE decoder provides direct, single-step gradients back to the backbone. Theoretically, this produces more stable and efficient training. Attention-guided masking creates a self-reinforcing loop Using attention maps as masking targets creates a productive feedback cycle. The attention map determines what is masked. The reconstruction loss improves backbone features. Better backbone features produce sharper, more semantically coherent attention maps in the next forward pass. The system's grounding quality should improve during training as a natural consequence of the architecture. // Total training objective L_total = L_action + lambda * L_recon // Where: L_action = CrossEntropy(action_bins) // 7 DoF x 256 bins L_recon = MSE(decoder_output, original_pixels) // masked patches only lambda = 0.5 // ablations: 0.1, 0.5, 1.0 The Experiments I am Running I designed four experimental conditions on LIBERO-Spatial, training on 3 tasks x 50 demonstrations, running on a single T4 GPU. The ablation in Condition 2 is the experiment I care about most. If random masking performs as well as attention-guided masking, it means the performance gain comes from the auxiliary task structure, not from language grounding. If attention-guided masking wins, it validates the core hypothesis. This is precisely the ablation that was missing from ReconVLA. On Accessibility and Reproducibility One thing that struck me about ReconVLA's experimental setup: it requires 8 A100 80GB GPUs and 2 million training samples. That is a real barrier. Most academic groups cannot reproduce it, let alone extend it. Scientific iteration requires accessibility. LA-ReconVLA is designed to run on a single T4 (Google Colab). The architectural choices that make this possible are not compromises: the MAE decoder is lighter than a diffusion transformer by design, PaliGemma-3B is smaller and partially frozen to reduce gradient computation, and the training pipeline avoids the large pretraining dataset requirement by relying on the backbone's pretrained language understanding instead. What Comes Next The experiments are running. Part 2 of this work will share full quantitative results across all four conditions, latency benchmarks against ReconVLA, attention visualisations comparing AOS scores, and an honest analysis of where the method falls short. There is a known limitation worth naming now: LA-ReconVLA assumes cross-attention maps are extractable from the backbone. Architectures without explicit cross-attention require adaptation, for example falling back to self-attention over image tokens. I have documented this in the design and will report on it during implementation. Real-robot validation is deferred to future work. For now, this is simulation-only. If you work on VLA models, robotic manipulation, or self-supervised visual representation learning, I would genuinely like to hear from you. The hypothesis space here is large and I do not think one architecture will be the final answer. But I do think eliminating the gaze annotation dependency and the diffusion overhead is the right direction, and I think the ablation design will tell us something we did not know before. This is an ongoing independent research experiment. Results, code, and full experimental logs will be published once the implementation phase is complete. Vision-Language ModelsRobot Manipulation Self-Supervised-Learning MAELIBERO-Benchmark Open-Source AI