Native Multimodal: Unified Model Architecture Today and Tomorrow

Fundamentals #07 introduced multimodal concepts. This article goes deeper on architecture—how models unify modalities and what that means for engineering.

Read the fundamentals multimodal intro first if you’re new to the topic.

Three Generations of Multimodal Architecture

Gen 1: Pipelines (2022–2023)

image → vision model → caption → LLM → answer
audio → ASR → text → LLM → answer

Problems: information loss, triple latency/cost, no true cross-modal reasoning.

Examples: early GPT-4V stack, LLaVA-style pipelines.

Gen 2: Fused Vision Tokens (2023–2024)

image → vision encoder → vision tokens ─┐
                                        ├→ unified Transformer → output
text  → text tokenizer → text tokens ──┘

Vision and language tokens processed together—direct cross-modal links.

Examples: GPT-4V, Claude 3, Gemini 1.5, LLaVA-1.5.

Still: separate encoders per modality.

Gen 3: Native Unified (2024–2026)

any modality → unified tokenizer → unified Transformer → any modality out

Goal: any-to-any—text, image, audio, video in and out.

Examples: GPT-4o, Gemini 2.0, Meta Chameleon, Google Unified-IO 2.

GPT-4o Design

Released May 2024 (“o” = omni):

In:  text, images, audio (realtime)
Out: text, images, audio (realtime)
Voice latency: ~320ms (near human reaction time)

Architecture:
  - end-to-end—not ASR + LLM + TTS glue
  - audio as tokens, not forced transcription
  - single model all modalities

Realtime API Impact

Classic voice stack:
  ASR (~300ms) + LLM (~500ms) + TTS (~300ms) ≈ 1100ms (noticeable lag)

GPT-4o Realtime:
  unified model ~320ms → audio out

Game-changer for voice support, assistants, education.

Gemini 2.0 Long Video

Context: ~1M tokens (~1 hour video / 700 pages)
Native video tokens—not frame captions only
Can generate images (Imagen integration)

Google AI Studio example: upload 30-minute demo video → timestamped feature summary → answer “how did export work at 15:00?” → generate docs.

Requires gen 2/3 architectures.

Training Unified Models

Pairs: (image, caption), (audio, transcript), (video, summary), (text, text), (image, image)
All → unified token sequence → next-token prediction

Challenges:

Modality alignment—1 image ≈ 1000 text tokens of information density
Data mix—modalities skew final capabilities
Compute—multimodal training ~5–10× text-only cost

When to Use Multimodal in Production

Scenario	Native multimodal?	Alternative
Screenshot Q&A	✅ GPT-4V / Claude	OCR + text LLM (weaker)
Live voice chat	✅ GPT-4o Realtime	ASR+LLM+TTS (slower)
Video analysis	✅ Gemini 2.0	frame-by-frame GPT-4V (costly)
Document OCR	⚠️ overkill	PaddleOCR, Tesseract
Pure text	❌ wasteful	text-only model cheaper/faster

Rule: no multimodal input → don’t pay for multimodal models.

Robotics Angle

Vision → camera
Audio  → microphone
Touch  → sensors
Action → motors

Unified multimodal model = robot "brain"
  all sensors → one model → action decisions

Leads to VLA models—see robotics article.

References

Summary

Multimodal architecture moves from stitching to unification. GPT-4o and Gemini 2.0 lead today. Engineer for need-based adoption—multimodal when inputs require it.