AIAny

Multi-Scale Context Aggregation by Dilated Convolutions

Introduced dilated (atrous) convolutions, which expand a filter's receptive field exponentially with no loss of resolution and no extra parameters — the trick that let dense-prediction networks see wide context while keeping per-pixel detail.

Introduction

Classification networks throw away spatial resolution on purpose — pooling and strided convolutions are how a net trades "where" for "what." Semantic segmentation needs both at once, and for years the field fought this with clumsy fixes: downsample to gather context, then upsample and hope the boundaries survive. This paper's quiet insight was that the resolution loss was never necessary in the first place.

Key Findings
  • Receptive field without resolution loss. Inserting zeros between filter taps (dilation) widens what each neuron sees, so context grows exponentially across layers while the feature map keeps its full size — no pooling, no upsampling artifacts.
  • No extra parameters or compute per layer. A dilated 3x3 filter still has nine weights; only its reach changes. Aggregating multi-scale context becomes essentially free.
  • A drop-in context module. The aggregation block plugs into existing segmentation front-ends and improves dense-prediction accuracy without retraining the whole pipeline from scratch.
  • Stripping the classification baggage. The authors also show that the adapted classification net carries vestigial structure that hurts dense prediction, and simplifying it helps.
How It Works

Dilated convolution generalizes the standard kernel with a spacing factor: dilation 1 is ordinary convolution, dilation 2 skips every other pixel, and so on. Stacking layers with exponentially increasing dilation lets the receptive field double each step, covering an entire image region in a logarithmic number of layers — all at native resolution.

Great Fit / When to Skip

Read it if you work on any pixel-dense task — segmentation, depth, or generative models where the dilation idea later resurfaced (WaveNet, DeepLab). Look elsewhere if you want a turnkey modern segmentation recipe: this is the mechanism, not the architecture, and transformer-based segmenters now offer different tradeoffs. Its lasting value is the concept, which quietly became standard equipment.

Information

  • Websitear5iv.labs.arxiv.org
  • OrganizationsPrinceton University, Intel Labs
  • AuthorsFisher Yu, Vladlen Koltun
  • Published date2015/11/23

More Items

Learns generalizable World Action Models for robotic manipulation by scaling causal egocentric video pretraining and grounding learned dynamics with heterogeneous robot trajectories. Key features: a three-stage curriculum (video pretraining, video-action mid-training with a unified action representation, and target-robot specialization) and a Slow–Fast dual-system for 30 Hz real-time action prediction.

Develops a vision-language foundation model for autonomous driving that unifies 3D BEV perception, visual question answering, and motion planning without changing the pretrained VLM architecture. Key elements include an external BEV perception head for 3D detection and occupancy, a Planning Expert using flow-matching for trajectory prediction, and a staged training recipe combining driving and general VLM data.

Converts posed indoor RGB(-D) video into editable, simulation-ready 3D scene graphs by parsing multi-view evidence into per-object bundles, generating complete object assets from that evidence, and placing them with GizmoAct, a VLM policy that refines 9-DoF poses through closed-loop GUI actions.