Beyond Local Learning in Video Autoregression
Video overview Download
In autoregressive video generation, each chunk becomes context for what follows. Self Forcing trains on the model’s own generated history, but detaches historical KV caches. This prevents later predictions from guiding how earlier context is generated.
Connected Self Forcing restores selected cross-chunk paths through Shortcut Gradient Replay, recovering feedback through historical KV states and the computations that produced them. It does so without retaining the full rollout graph or adding parameters. The autoregressive inference process stays the same.
To stream long videos while maintaining visual quality and temporal consistency, Self Forcing mitigates exposure bias through self-rollout training on self-generated histories with key-value (KV) caching. To keep memory manageable, it detaches historical caches, preserving forward dependencies between chunks but severing the backward gradient paths.
We introduce Connected Self Forcing, a training framework that reconnects gradient paths across autoregressive chunks, allowing feedback from later predictions to guide how earlier context is generated. These connections go beyond historical KV-writing: gradients pass through generated latents into the computations that produced them, linking the generation of earlier context to its use in later predictions. To make this connected training memory-efficient, we develop shortcut gradient replay, which recovers cross-chunk gradients without retaining the full rollout computation graph.
Integrated with distribution matching distillation, Connected Self Forcing trains historical chunks according to both their direct supervision and their contribution to subsequent generation. Experiments on autoregressive video generation show improvements in long-horizon visual quality and temporal consistency, without changing the inference procedure.
Each prompt is shown with SF, SGF, and CSF at the original playback speed. Use the shared controls to play, pause, or seek all three videos together.
Chunk-wise · Causal CD initialization
Full prompt: A stylish woman walks down a Tokyo street filled with warm glowing neon and animated city signage. She wears a black leather jacket, a long red dress, and black boots, and carries a black purse. She wears sunglasses and red lipstick. She walks confidently and casually. The street is damp and reflective, creating a mirror effect of the colorful lights. Many pedestrians walk about.
Chunk-wise · Causal CD initialization
Full prompt: A Samoyed and a Golden Retriever dog are playfully romping through a futuristic neon city at night. The neon lights emitted from the nearby buildings glistens off of their fur.
Chunk-wise · Teacher Forcing initialization
Full prompt: A young man at his 20s is sitting on a piece of cloud in the sky, reading a book.
Frame-wise · Causal CD initialization
Full prompt: A beautiful homemade video showing the people of Lagos, Nigeria in the year 2056. Shot with a mobile phone camera.
Chunk-wise · Causal CD initialization
Full prompt: a toy robot wearing purple overalls and cowboy boots taking a pleasant stroll in Mumbai India during a winter storm
Chunk-wise · Causal CD initialization
Full prompt: A steaming cup of tea in a cold room, with tendrils of steam rising and dissipating in the air above it.
Chunk-wise · Teacher Forcing initialization
Full prompt: A low-angle shot of a child reaching out to catch falling snowflakes, with a backdrop of tall evergreen trees.
Chunk-wise · Teacher Forcing initialization
Full prompt: An older man playing piano, lit from the side.
Frame-wise · Teacher Forcing initialization
Full prompt: A couple sits at a peaceful lakeside picnic, occasionally reaching into a basket for food, while the gentle ripples on the lake reflect the shifting colors of the sky.