Robotics and AI: From LLMs to VLA Models

Text models think. Vision models see. How does AI act? Embodied AI—intelligence in the physical world.

Why Robotics + AI Is Hard

ChatGPT wrong → user sees bad text
Self-driving wrong → collision risk
Robot wrong → broken objects, injury risk

Extra constraints:
  real-time control (milliseconds)
  noisy sensors
  every environment differs
  safety bar extremely high

Three Generations

Gen 1: Rules + Classical CV (pre-2020)

camera → YOLO detection → rule engine → scripted motion

Works on factory lines; breaks in new environments.

Gen 2: Reinforcement Learning (2020–2023)

simulation → RL → policy → real robot

Examples: OpenAI Dactyl, Google QT-Opt.

Problem: sim-to-real gap—sim policies often fail in reality.

Gen 3: VLA Models (2024–2026)

Vision + Language → Action

Transfer LLM world knowledge to control:

Human: "Move the red cup on the table next to the sink"

VLA:
  1. see cup and sink
  2. parse spatial relation "next to"
  3. plan arm trajectory

Major VLA Models

Model	Org	Notes	Status
RT-2	Google DeepMind	early VLA, PaLM + control	research
OpenVLA	Stanford et al.	open 7B/34B, fine-tunable	open source
π0 (Pi-Zero)	Physical Intelligence	general robot foundation	2024
Helix	Figure AI	humanoid Figure 01	commercial
Optimus AI	Tesla	in-house for Optimus	commercial

RT-2 Idea (Google, 2023)

Tokenize actions:

Classic: image → CNN → continuous action vector
RT-2:    image + instruction → LLM → action token sequence

Example tokens: "move gripper to (0.3, 0.5, 0.2), close gripper"

Benefit: reuse LLM reasoning—"empty cup → dishwasher"

OpenVLA

7B open VLA on Open X-Embodiment (22 robots, 1M+ trajectories). Best reproducible starting point for researchers.

Real Cases

Figure 01 + OpenAI (2024)

Human: "I'm hungry—anything edible on the table?"
Figure: GPT-4 reasons (apple vs energy bar)
        VLA picks apple, hands it over
        GPT-4 speaks explanation

LLM planning + VLA control layered stack.

Tesla Optimus

Vision-heavy end-to-end from fleet camera data—not LLM-centric reasoning. Huge data appetite; generalization TBD.

1X NEO (Home)

Humanoid for households: sim pretrain → home fine-tune; safety-focused hardware; VLA + teleop data collection.

Current Bottlenecks

Bottleneck	Now	Outlook
Data	10× less robot data than text	sim + teleop scale
Sim-to-real	gap remains	better domain randomization
Generalization	new objects/scenes drop performance	foundation model + few-shot
Latency	VLA 100–500ms	smaller models, custom silicon
Safety	immature standards	regulation catching up
Cost	humanoids $20K–$150K	volume → ~$5K?

For AI Engineers (2026)

Short term:
  - VLA hot in research/startups; consumer far
  - warehouses/factories use non-VLA mature stacks
  - watch OpenVLA / RT-2 open ecosystem

Medium term (2027–2028):
  - early adopter home robots possible
  - foundation VLA + scene fine-tune standard

Long term:
  - general robot + general AI = physical AGI form?

Model Stack for Robots

Text models    → planning and reasoning
Vision models  → eyes
Audio models   → ears and speech
Multimodal     → unified brain
VLA            → cerebellum (motor control)

Full stack = brain + cerebellum + senses

References

Summary

Robot AI shifts from programming + classical CV toward VLA foundation models. 2026 is early—active research, nascent products, clear bottlenecks. Understanding VLA and sim-to-real is how you track the next wave.