One-sentence Summary

The Visual Geometry Group at the University of Oxford and Meta AI propose VGGT, a feed-forward transformer that directly infers camera parameters, point maps, depth maps, and 3D point tracks from as few as one to hundreds of views, outperforming specialized single-task models without requiring post-processing visual geometry optimization.

Key Contributions

• VGGT is a feed-forward neural network that directly infers camera intrinsics and extrinsics, point maps, depth maps, and 3D point tracks from one, a few, or hundreds of scene images. The model processes images in under one second and unifies camera parameter estimation, multi-view depth estimation, dense point cloud reconstruction, and 3D point tracking.

• VGGT produces directly usable 3D predictions that are competitive with or better than state-of-the-art methods relying on slow post-processing visual geometry optimization. When combined with bundle adjustment, it achieves state-of-the-art results across the evaluated 3D tasks, often substantially improving quality over task-specialized methods.

• Pretrained VGGT features serve as an effective backbone for downstream tasks, improving non-rigid point tracking and feed-forward novel view synthesis. When coupled with existing point trackers, these features yield state-of-the-art tracking performance.

Introduction

The authors address 3D reconstruction, a core problem for robotics, autonomous systems, and image-based rendering, where the goal is to estimate camera parameters, depth, dense geometry, and 3D tracks from images. Earlier systems rely heavily on visual geometry and iterative optimization, such as Bundle Adjustment, which improves accuracy but adds complexity and runtime cost. Recent learning-based methods reduce dependence on geometry, yet prior work often remains limited: some methods still require expensive post-optimization, while others support only pairwise reconstruction or handle a single 3D task. The authors leverage a large transformer-based feed-forward network, VGGT, that predicts multiple 3D attributes directly from one, several, or hundreds of views in seconds, showing that these predictions are highly usable and can reach state-of-the-art quality, especially when combined with lightweight optimization.

Method

The authors present VGGT as a transformer based system that maps a set of RGB images to multiple 3D related outputs. Given an image sequence $(I_i)_{i=1}^{N}$(Ii​)i=1N​ observing the same scene, the model is defined as

where each frame is associated with camera parameters $\mathbf{g}_i$gi​, a depth map $D_i$Di​, a point map $P_i$Pi​, and dense tracking features $T_i$Ti​. The camera representation is expressed as $\mathbf{g}=[\mathbf{q},\mathbf{t},\mathbf{f}]$g=[q,t,f], combining a rotation quaternion, a translation vector, and field of view parameters. The point maps and depth maps are expressed in the coordinate frame of the first image, which serves as the global reference frame. The architecture is designed to be permutation equivariant for all frames except the first one, allowing flexible input ordering.

At the input stage, each image is patchified by a DINO encoder into a set of tokens $\mathbf{t}_i^I\in\mathbb{R}^{K\times C}$tiI​∈RK×C. The tokens from all frames are combined and passed through a transformer backbone that alternates between frame-wise self-attention and global self-attention. Frame-wise attention operates only within each image, while global attention operates across tokens from all images. This alternating mechanism allows the model to preserve per-frame structure while also aggregating cross-frame geometric information. The backbone uses multiple layers of this alternating attention and does not rely on cross-attention layers.

For prediction, each frame's image tokens are augmented with a camera token and register tokens. The first frame uses a distinct set of learnable tokens so the transformer can recognize the reference camera. The output camera tokens are processed by a camera head that predicts camera parameters. The output image tokens are converted into dense feature maps using a DPT style head, followed by convolutional layers that produce depth maps and point maps. The same dense head also outputs tracking features, uncertainty maps, and point map uncertainty. The tracking module is implemented using a CoTracker2 style head. It samples the query point feature from the query frame feature map, correlates it with feature maps from the other frames, and predicts 2D locations across the image set. This tracking process does not require temporal ordering and can operate on unordered collections of images.

The method also emphasizes over-complete supervision. Although depth, point maps, and cameras are related through geometric constraints, training VGGT to predict all of them explicitly improves performance. During inference, the authors observe that combining independently estimated depth and cameras can yield more accurate 3D points than using a single specialized point map branch. This design turns the model into a unified 3D prediction system rather than a narrow single-task estimator.

The same backbone can be adapted to downstream tasks. For feed-forward novel view synthesis, source images are tokenized by DINO, while target viewpoints are represented by Plücker ray images encoded into tokens. These tokens are concatenated and processed by the alternating-attention transformer, and a DPT head regresses the RGB values for the target views. Notably, the model does not receive camera parameters for the source images, yet it can synthesize new views by learning geometry from data. Qualitative examples of this adaptation are shown as follows.

For dynamic point tracking, the authors replace the feature extractor in CoTracker2 with the pretrained VGGT backbone. The adapted model uses VGGT to produce tracking features for each frame, and the remaining CoTracker2 modules use those features to predict 2D point trajectories. This transfer allows the pretrained unordered image features to be reused for sequential tracking tasks and improves tracking quality without requiring the original backbone.

Experiment

The experiments evaluate VGGT on camera pose estimation, multi-view depth, point map reconstruction, image matching, and downstream tasks, comparing its feed forward predictions against state of the art optimization based and specialized baselines. Across in domain and unseen datasets, the model achieves strong accuracy while remaining fast, and ablations show that alternating attention and joint multi task supervision are important for accurate 3D reasoning. Finetuning results further indicate that the pretrained features generalize to novel view synthesis and dynamic point tracking, confirming the flexibility of the learned representation.

On RealEstate10K, which is unseen, and CO3Dv2, learned methods generally achieve higher relative pose accuracy than the SfM baselines, with MASt3R and VGGSfM v2 among the strongest listed approaches. The reported VGGT feedforward model outperforms these alternatives on both datasets, and bundle adjustment can further improve it while keeping runtime low. A concurrent fast method has the shortest listed runtime but lower pose accuracy than the strongest baselines. Learned feed-forward methods provide a clear accuracy gain over traditional SfM baselines on both unseen and CO3D evaluation. Predicted point and depth maps let VGGT combine with bundle adjustment for improved accuracy without the high cost of traditional iterative BA.

On the DTU dense multi-view stereo benchmark, methods using known ground-truth cameras achieve low error, while DUSt3R without camera knowledge performs much worse on accuracy and overall Chamfer distance. The reported camera-free method substantially lowers the Overall score from 1.741 to 0.382 and becomes comparable to known-camera methods. The paper attributes this gain to multi-image training that learns multiview triangulation directly. Known-camera methods generally achieve much lower error than DUSt3R. The camera-free method greatly improves over DUSt3R on accuracy and Overall. It approaches the error levels of methods that use ground-truth cameras.

On ETH3D, the proposed feed-forward method achieves better point cloud alignment than DUSt3R and MASt3R while being substantially faster than their global alignment procedures. The depth-and-camera decoding produces the strongest accuracy, completeness, and overall Chamfer distance results. These results support decomposing point map estimation into depth and camera prediction. Feed-forward prediction outperforms optimization-based global alignment baselines while requiring far less time. Constructing point clouds from depth and camera heads is more accurate than using the point map head directly.

The tracking head delivers the strongest relative pose accuracy on the ScanNet-1500 two-view matching benchmark, surpassing the prior state-of-the-art matching method Roma despite not being specialized for this setting. It maintains better AUC across stricter and looser pose error thresholds, indicating improved matching quality from query keypoints to correspondences in the second view. The method outperforms Roma across all reported AUC thresholds. The improvement is consistent at both stricter and looser pose accuracy measures. Strong two-view correspondence quality is reflected in better essential matrix and relative pose estimation.

The ablation results show that using all three training losses gives the best point map performance on accuracy, completeness, and overall error. Removing camera loss causes the largest drop, while track loss also helps noticeably and depth loss has the smallest effect. The compared architectures are evaluated under a fair setting with the same parameter count, and cross attention is noted to increase runtime as the number of frames grows. All losses together produce the lowest accuracy, completeness, and overall error Camera loss is the most important auxiliary objective for point map accuracy Depth loss provides only a marginal improvement compared with camera and track loss Cross attention increases computational cost when many input frames are used

Across RealEstate10K, CO3Dv2, DTU, ETH3D, and ScanNet-1500, the evaluated feed-forward method demonstrates better relative pose accuracy than SfM baselines, approaches camera-aware dense reconstruction performance, improves point cloud alignment with less alignment cost, and achieves stronger two-view matching than specialized methods. Ablations indicate that combining depth, camera, and track losses yields the best geometry, with camera loss most critical, while cross attention adds runtime as frame count grows. Overall, predicted point and depth maps enable efficient bundle adjustment and accurate reconstruction and matching.