Ashok Elluswamy ICCV 2025 WDFM-AD 演讲：中英文对照学习版

Ashok opens by saying he leads the Tesla AI team and is presenting recent work from the team. He frames the talk around Tesla’s progress in autonomy, including robotaxi service in Austin and the Bay Area, factory-to-customer autonomous delivery, and vehicles moving themselves from production lines to loading docks.

He emphasizes that these are production vehicles using production cameras and onboard computers, adapting to real-world driving conditions.

Ashok 开场说明自己负责 Tesla AI 团队，这次分享的是团队近期成果。他先用几个自动驾驶进展建立背景：Austin 和湾区的 Robotaxi、车辆从 Austin 工厂自动交付到客户家、以及工厂内车辆从产线自动开到装车区。

重点是：这些不是实验车，而是量产车、量产摄像头、量产车载计算机，在真实道路条件下运行。

The first technical point is the shift to a single large end-to-end neural network. The model takes pixels and other vehicle/sensor context as input and produces the next driving action as output.

Rather than explicitly passing through separate perception labels such as vehicles, lane boundaries, or road edges, those concepts can exist implicitly inside the network or be trained as auxiliary outputs. The main objective is the action the car should take.

第一个技术重点是：Tesla 切换到一个大型端到端神经网络。模型输入像素、车辆状态和其他传感器/上下文信息，直接输出下一步驾驶动作。

它不再把“车辆、车道边界、道路边缘”等显式感知结果作为硬接口逐级传下去；这些概念可以隐含在网络内部，也可以作为辅助任务训练。真正核心的输出是车辆应该采取的动作。

Ashok argues that modular autonomy stacks are hard to align with human preferences. Smoothness, comfort, speed, and safety are contextual and difficult to encode as fixed rules or cost functions.

The interface between perception and planning can also be lossy: uncertainty and subtle visual evidence may not survive when converted into discrete labels. A single homogeneous neural network also gives more predictable latency than a branching rule-heavy system.

Ashok 认为模块化自动驾驶很难和人的偏好对齐。平顺、舒适、速度、安全之间没有一个固定公式，不同速度、道路和场景下的权衡都不同，很难用规则或代价函数写死。

感知到规划之间的接口也会丢信息：不确定性、细微视觉线索在变成离散标签时可能丢失。相比之下，单一同构神经网络还能带来更可预测的实时延迟。

He uses several examples to show why explicit rules fail: moving into the oncoming lane to avoid a puddle when visibility is good; waiting for animals crossing; distinguishing stationary animals from animals that are about to cross; reversing and going around when appropriate.

The claim is that these decisions are obvious to a human looking at the full scene, but very hard to encode with hand-written conditions.

他用几个长尾场景说明显式规则的困难：为了避开水坑，在视野充分且无来车时短暂借对向车道；等待动物过路；判断动物只是停在路边还是准备横穿；必要时倒车绕行。

这些选择对看完整场景的人类很直观，但如果要靠手写条件和代价函数表达，就会非常脆弱。

The first major difficulty is dimensionality. Tesla vehicles have many high-resolution cameras, long temporal context, navigation route, speed, and kinematic information. A naive model would have to map an enormous input context into only a few action outputs.

Tesla’s advantage is fleet data. The challenge is not merely collecting data, but refining massive daily driving volume into the right distribution: rare cases, errors, and scenarios that cover the driving spectrum without storing everything.

第一个大难题是维度灾难。Tesla 车辆有多路高分辨率摄像头、长时间上下文、导航路线、速度和运动学信息。朴素地建模，就是把极其庞大的输入压缩成少数几个动作输出。

Tesla 的优势是车队数据。但关键不是“数据越多越好”，而是把巨量日常驾驶数据筛成真正有用的分布：罕见场景、失败案例、覆盖完整驾驶谱系的样本，而不是无差别存下所有数据。

Ashok shows a highway example where the system brakes before a spinning vehicle hits the barrier and rebounds. The point is that the model is not reacting only to first-order motion; it anticipates a second-order outcome.

This illustrates why large-scale data matters: a less intelligent system might wait until the danger is obvious, while the end-to-end system can act early enough to be both safe and smooth.

Ashok 展示了一个高速场景：前车失控打滑，Tesla 在它撞上护栏并反弹回来之前就开始制动。重点是模型不是只看一阶运动，而是在预测二阶后果。

这说明大规模数据的价值：较弱系统可能等到危险已经显性化才反应，而端到端系统可以更早判断异常，从而既安全又平顺。

He anticipates the question: if the system is end-to-end, how do engineers debug it? His answer is that the same model can be prompted or trained to predict many auxiliary outputs: 3D occupancy, objects, traffic lights, traffic signs, road boundaries, and even language explanations.

The only output that truly matters in the car is the control action, but auxiliary predictions help the model learn and help engineers inspect what it understands.

他主动回应一个问题：如果系统是端到端的，工程师怎么调试？他的回答是：同一个模型可以被要求预测许多辅助输出，比如 3D 占用、目标、红绿灯、交通标志、道路边界，甚至语言解释。

在车上真正决定行为的是控制动作，但这些辅助预测能帮助模型学习，也能帮助工程师检查模型到底理解了什么。

Ashok discusses Gaussian splatting. Traditional approaches struggle when vehicle data comes mostly from forward driving with limited baseline; views near training cameras may look good, while novel views break down.

Tesla’s variant generates more stable 3D structure and semantics from the same limited vehicle camera views. Such representations can be used to inspect whether the system is safely avoiding obstacles or understanding scene geometry.

Ashok 讨论了 Gaussian splatting。传统方法在车辆数据上容易受限，因为车辆通常向前行驶，视角基线不足；接近训练视角的画面可以很好，但换到新视角就容易崩。

Tesla 的变体能从同样有限的车载摄像头视角中生成更稳定的 3D 结构和语义。这类表征可以用来检查系统是否安全避障、是否理解场景几何。

The model can also be interacted with through natural language: it can point to relevant things and explain why it made a decision. Ashok stresses that this is not required for every real-time control step.

When necessary, the model can spend more computation, generate reasoning tokens, and produce an action consistent with that reasoning. When detailed reasoning is unnecessary, it can directly produce control actions to keep latency low.

模型也可以通过自然语言交互：它可以指出相关对象，并解释为什么做出某个决策。Ashok 强调，实时驾驶并不需要每一步都这样做。

在必要时，模型可以花更多计算，生成 reasoning tokens，再输出与推理一致的动作；如果不需要详细推理，就直接输出控制动作，以保持低延迟。

The final major problem is evaluation. A model can look strong in open-loop metrics while still driving poorly when placed in the closed-loop world. Random fleet samples are also insufficient because they overrepresent boring highway driving.

Tesla needs balanced evaluation sets that cover rare and important cases. This makes the data engine important not only for training, but also for evaluation.

最后一个大问题是评估。一个模型在开环指标上看起来很好，放到闭环真实驾驶里却可能表现不佳。随机抽取车队数据也不够，因为大部分只是普通高速巡航。

Tesla 需要覆盖长尾和关键场景的均衡评估集。因此，数据引擎不仅服务训练，也服务评估。

Ashok introduces a learned world simulator. Given past states and actions, it synthesizes future states conditioned on actions. The training data is cheap because it does not require perfect driving; even poor driving can teach the simulator how the world responds.

The simulator generates all eight camera streams at once and remains consistent across views, vehicles, traffic lights, and long rollouts. It can be coupled with the policy network to simulate closed-loop driving.

Ashok 引入神经世界模拟器：给定过去状态和动作，模型生成由动作条件化的未来状态。它的数据很便宜，因为不需要完美驾驶；即使是糟糕驾驶，也能教模拟器世界如何响应动作。

模拟器可以同时生成八路摄像头视频，并在视角、车辆、红绿灯和长时间 rollout 上保持一致。它可以和 policy 网络连接起来，做闭环驾驶仿真。

The simulator can replay old issues and test newer policies against them. For example, if an old model came too close to a pedestrian, engineers can rerun a newer model in the same scenario and see whether it offsets earlier.

It can also create counterfactual cases, such as making another vehicle cut across the path while keeping the rest of the scene consistent. This enables synthetic corner-case evaluation.

模拟器可以重放旧问题，并用新策略重新测试。例如，如果旧模型离行人太近，工程师可以在同一场景里运行新模型，看它是否更早偏移避让。

它也能制造反事实场景，比如让某辆车突然切入路径，同时保持场景其他部分一致。这就能用于合成长尾和对抗性评估。

With reduced test-time compute, the generated world can run close to real time. A user can steer, brake, and navigate inside the neural network generated video world.

This makes the simulator useful for evaluation and potentially for closed-loop reinforcement learning: the car can be allowed to drive in generated worlds for long periods while checking for collisions or unsafe behavior.

如果降低测试时计算量，生成世界可以接近实时运行。用户可以在神经网络生成的视频世界中转向、刹车和导航。

这让模拟器既能用于评估，也可能用于闭环强化学习：让车在生成世界里长时间行驶，检查是否碰撞或出现不安全行为。

Ashok then turns to scaling the service: making autonomy better, deploying it across Tesla’s fleet, and supporting future vehicles such as Cybercab. He presents the same neural network approach as scalable across platforms, locations, and weather.

The larger claim is that self-driving is one instance of robotics. The same technology can transfer to other robot bodies, including Optimus.

Ashok 接着谈规模化：让自动驾驶服务更好，扩展到 Tesla 车队，并支持 Cybercab 这样的未来车型。他认为同一神经网络路线可以跨车型、地域和天气扩展。

更大的主张是：自动驾驶只是机器人问题的一个实例。同样的技术可以迁移到其他机器人身体，包括 Optimus。

He shows that video generation and action-conditioned prediction can also work for Optimus in a factory setting. Different robot actions lead to different generated futures, using the same broader neural network approach with additional Optimus data.

The closing message is that Tesla is focused on robotics at scale and invites people working in AI, robotics, and computer vision to join.

他展示了视频生成和动作条件预测也可以用于工厂里的 Optimus。不同机器人动作会导向不同生成未来；整体仍是同一类神经网络方法，只是加入 Optimus 数据。

结尾信息是：Tesla 正集中投入大规模机器人技术，并邀请从事 AI、机器人和计算机视觉的人加入。