One-sentence Summary

A multi-scale deep network combining a coarse global depth prediction from the entire image with a local refinement network, using a scale-invariant error to focus on depth relations rather than absolute scale, proposed by researchers from New York University and Facebook AI Research, achieves state-of-the-art single-image depth estimation on NYU Depth and KITTI and produces sharp depth boundaries without superpixelation.

Key Contributions

• A two-component deep network architecture first estimates a coarse global depth map from the entire image, then refines it locally to improve boundary precision.
• A scale-invariant loss explicitly penalizes errors in relative depth relationships between pixel locations, reducing the impact of global scale ambiguity during training.
• The method achieves state-of-the-art depth estimation on NYU Depth and KITTI and captures detailed depth boundaries without requiring superpixel post-processing.

Introduction

Estimating depth from a single image is essential for recovering scene geometry, with direct applications in 3D modeling, robotics, and scene understanding. While stereo depth estimation can rely on recoverable correspondences, monocular prediction must resolve inherently ambiguous cues such as perspective, object size, and scene layout, making it an ill-posed problem. Prior monocular approaches depend on handcrafted features, restrictive alignment assumptions, or expensive nearest-neighbor retrieval at runtime, which limits accuracy and efficiency. The authors introduce a two-stage convolutional neural network that first captures global scene structure and then refines it with local detail, trained with a loss that accounts for both pointwise depth error and pairwise depth relationships. This design achieves state-of-the-art depth estimates on NYU Depth and KITTI benchmarks while running in real time on unseen images.

Dataset

The authors build their training and test sets from two well-known depth estimation benchmarks, NYU Depth and KITTI. Processing includes temporal synchronization, spatial alignment, masking of invalid regions, and removal of near-duplicate frames. Both datasets are balanced per scene and then shuffled with duplicates to create the final training lists for a two-stage (coarse-fine) network.

• NYU Depth v2

• Source: 464 indoor video sequences captured with a Microsoft Kinect. The official split (249 train / 215 test) is used.

• Resolution: RGB frames are downsampled from 640×480 to 320×240.

• Temporal alignment: Each depth frame is paired with the closest RGB frame in time; frames where one RGB matches multiple depths are discarded.

• Masking: Pixels with no depth value are masked. Additionally, depths equal to the minimum or maximum value in each image are masked to remove artifacts from windows, open doors, and specular surfaces.

• Training set size: 120K unique images, balanced to 1200 per scene, then shuffled with duplicates into a list of 220K images. Evaluation uses the 694-image NYU Depth v2 test set (with filled-in depths).

• Usage: The coarse network is trained for 2 million samples (batch size 32) on this list; the fine network is then trained for 1.5 million samples using the frozen coarse network’s predictions.

• KITTI

• Source: 56 outdoor driving scenes from the “city,” “residential,” and “road” categories of the raw data, split into 28 scenes for training and 28 for testing.

• Resolution: RGB images (originally 1224×368) are downsampled to half.

• Depth alignment: Ground-truth depth from a rotating LIDAR is irregularly sampled. Per-pixel conflicts are resolved by selecting the depth recorded closest to the RGB capture time.

• Cropping: Depth is only available in the lower image portion. The whole image is fed to the coarse network for context; the fine network sees only the bottom crop corresponding to the target area.

• Filtering: Shots where the car is stationary (acceleration below a threshold) are removed to avoid duplicates. Both left and right camera views are used as independent samples.

• Scene balancing: 800 images per scene are selected, giving 20K unique training images. These are balanced per scene and shuffled with duplicates into a 40K-image list.

• Usage: The coarse network is trained for 1.5 million samples, then the fine network for 1 million samples, using the same learning-rate ratios as NYU Depth.

Method

The authors leverage a network architecture made of two component stacks to predict scene depth. A coarse-scale network first predicts the depth at a global level, which is then refined within local regions by a fine-scale network. Both stacks are applied to the original input, but the coarse network output is also passed to the fine network as additional first-layer image features. This allows the local network to edit the global prediction to incorporate finer-scale details.

Refer to the framework diagram:

The global, coarse-scale network is designed to predict the overall depth map structure using a global view of the scene. Its upper layers are fully connected, containing the entire image in their field of view, while lower and middle layers combine information through max-pooling operations. This design integrates a global understanding of the full scene, which is necessary to utilize cues such as vanishing points and object locations. The network contains five feature extraction layers of convolution and max-pooling, followed by two fully connected layers. The final output is at 1/4-resolution compared to the input. Rather than relying on hardcoded upsampling, the authors allow the top full layer to learn templates over a larger area, effectively letting the network learn its own upsampling based on the features.

As shown in the figure below, the weight vectors from the coarse output layer demonstrate learned depth patterns:

After taking a global perspective, the authors make local refinements using a second, fine-scale network. This component edits the coarse prediction to align with local details such as object and wall edges. The fine-scale network stack consists of convolutional layers only, along with one pooling stage for the first layer edge features. The coarse output is fed in as an additional low-level feature map and concatenated with the output of the first fine-scale layer. Subsequent layers maintain this size using zero-padded convolutions. All hidden units use rectified linear activations, except for the final linear layer which predicts the target depth. The authors train the coarse network first against ground-truth targets, then train the fine-scale network keeping the coarse-scale output fixed.

To address the fundamental ambiguity of global scale in depth prediction, the authors introduce a scale-invariant error metric. For a predicted depth map $y$y and ground truth $y^*$y∗, the scale-invariant mean squared error in log space is defined as: $D(y, y^*) = \frac{1}{n} \sum_{i=1}^n (\log y_i - \log y_i^* + \alpha(y, y^*))^2$D(y,y∗)=n1​∑i=1n​(logyi​−logyi∗​+α(y,y∗))2 where $\alpha(y, y^*) = \frac{1}{n} \sum_i (\log y_i^* - \log y_i)$α(y,y∗)=n1​∑i​(logyi∗​−logyi​) is the value that minimizes the error. This metric compares relationships between pairs of pixels, ensuring that depth differences in the prediction match those in the ground truth.

Inspired by this metric, the per-sample training loss is set to: $L(y, y^*) = \frac{1}{n} \sum_i d_i^2 - \frac{\lambda}{n^2} \left(\sum_i d_i\right)^2$L(y,y∗)=n1​∑i​di2​−n2λ​(∑i​di​)2 where $d_i = \log y_i - \log y_i^*$di​=logyi​−logyi∗​ and $\lambda = 0.5$λ=0.5. This balances elementwise $l_2$l2​ error with the scale-invariant error. During training, missing values in target depth maps are masked out, and the loss is evaluated only on valid points.

The authors augment the training data with random online transformations, including scaling, rotation, translation, color adjustment, and horizontal flips. Scaling and translation do not preserve world-space geometry, but the authors correct for scaling by dividing depth values by the scale factor. They found that the extra data from translations benefited the network despite slight warping. At test time, a single center crop at scale 1.0 is used with no rotation or color transforms.

Experiment

the paper train a coarse-to-fine depth prediction network on raw NYU Depth v2 and KITTI datasets, leveraging their large size and natural depth gaps without preprocessing. Comparisons against Make3D and published methods show the model achieves the best performance on all standard metrics, with average relative gains of 35% on NYU and 31% on KITTI. The fine-scale network consistently sharpens depth boundaries in qualitative outputs, although it does not improve numeric error scores; using a scale-invariant loss provides slight visual improvements but no metric gain. The results validate the model's ability to learn accurate depth structure from challenging, unfilled raw data in both indoor and outdoor driving scenarios.

The architecture processes images through a coarse network that downsamples to a 1x1 bottleneck, then upsamples to a quarter of the input resolution. A fine-scale network of four layers refines predictions at this same quarter-resolution, sharpening depth boundaries without improving error metrics. The system outperforms baselines on NYU Depth and KITTI, with relative gains of 35% and 31% respectively, while the fine-scale refinement adds visible but non-numerical enhancements. The coarse pathway compresses the input to a 1x1 representation before upsampling to 1/4 resolution, where the fine network operates. Fine-scale layers do not improve error metrics but produce sharper transitions, especially near object boundaries and road edges. On NYU Depth, the method achieves a 35% relative gain over the runner-up, and on KITTI a 31% gain, benefiting from raw training data. Make3D's assumption of horizontal alignment degrades performance on NYU but helps on KITTI; the proposed method still surpasses it on both datasets. Using a scale-invariant loss provides no numeric gains but yields slight qualitative improvements on NYU Depth.

The coarse network outperforms all baselines on NYUDepth, cutting absolute relative difference to 0.228 and lifting δ<1.25 accuracy from 0.447 (Make3D) to 0.618. The fine-scale addition yields only marginal quantitative changes but sharpens depth boundaries qualitatively. Scale-invariant analysis shows that correcting the mean log depth eliminates a large portion of the error, indicating the model already captures scene structure well. The coarse network achieves 0.283 log RMSE and δ<1.25 accuracy of 0.618, substantially better than Make3D (0.409, 0.447) and Ladicky et al. (0.542). Applying a scale oracle to the proposed system lowers log RMSE from 0.28 to 0.22, revealing that global scale prediction accounts for much of the remaining error while relative depth relationships are largely correct.

On the KITTI driving dataset, the proposed coarse network delivers large, consistent gains over the Make3D baseline, lowering absolute relative error, squared relative error, and both linear and log RMSE. Incorporating the fine-scale network brings only marginal metric improvements but visibly sharpens depth boundaries near road edges, an effect constrained by sensor misalignment in the training data. The coarse network reduces the absolute relative difference from 0.280 (Make3D) to 0.194, and pushes the fraction of accurate pixels under delta < 1.25 from 0.601 to 0.679. Adding the fine-scale refinement drops log RMSE from 0.273 to just 0.270, confirming that the extra stage brings almost no quantitative benefit on the KITTI metrics. Qualitative outputs show sharper transitions with the fine-scale network, especially along the road edge, but the improvement is limited by uncorrected alignment between the rotating scanner depth maps and the input images.

The evaluation tests a two-stage architecture on NYU Depth and KITTI, using a coarse network that downsamples to a 1x1 bottleneck then upsamples, with an optional fine-scale network applied at quarter-resolution. The coarse network yields large quantitative gains over baselines, while the fine-scale stage produces sharper depth boundaries but no meaningful improvement in error metrics. Qualitative enhancements are visible near object edges, though scale-invariant analysis reveals that global scale prediction and sensor misalignment account for much of the remaining error.