One-sentence Summary

Researchers from the University of Toronto, Vector Institute, and Carnegie Mellon University present the first physically based neural inverse rendering system for multi-viewpoint videos of propagating light, employing a time-resolved neural radiance caching extension to capture direct and indirect light transport, which enables state-of-the-art $3\text{D}$3D reconstruction, view synthesis, direct/indirect decomposition, and multi-view time-resolved relighting from flash lidar measurements.

Key Contributions

• The paper introduces a time-resolved extension of neural radiance caching, which replaces recursive evaluation of the rendering equation with cache lookups to enable physically based inverse rendering from multi-viewpoint videos of propagating light.
• The method captures both direct and indirect light transport, and when applied to captured flash lidar measurements, it achieves state-of-the-art 3D reconstruction accuracy in scenes with strong indirect light.
• The approach enables novel capabilities, including view synthesis of propagating light, automatic decomposition of captured measurements into direct and indirect components, and multi-view time-resolved relighting of captured scenes.

Introduction

Ultrafast imaging systems such as lidar illuminate a scene with short light pulses and measure the backscattered echoes to reconstruct 3D geometry, making them central to autonomous driving, augmented reality, and remote sensing. However, conventional lidar pipelines keep only direct light transport (light that reflects once back to the sensor) and discard multiply scattered indirect light, because modeling it requires expensive recursive inverse rendering. Prior multi-view lidar reconstruction methods either operate on point clouds or direct-only time-resolved signals, and existing neural radiance field (NeRF) approaches for propagating light cannot recover accurate geometry or re-render scenes under novel illumination because they lack a physically based model. Differentiable path tracing handles indirect light but is computationally prohibitive and brittle to noise and initialization in captured multi-view experiments.

The authors propose a neural inverse rendering method that takes multi-viewpoint, time-resolved measurements from a flash lidar system and reconstructs both direct and indirect light transport. They use volume rendering for geometry and a physically based appearance model backed by a time-resolved radiance cache that stores incoming radiance at any point from any direction, avoiding recursive integration of the rendering equation. They also capture a new calibrated dataset of multi-view flash lidar measurements and demonstrate state-of-the-art geometry reconstruction under strong indirect lighting, plus rendering of propagating light from novel viewpoints and under novel illumination conditions.

Dataset

Dataset Composition and Sources

The authors construct two distinct evaluation datasets to validate their approach: a simulated dataset and a captured dataset.

Simulated Dataset

• Generated using a modified Mitsuba 2 renderer to produce multi-view transient data.
• Includes three small-scale, object-centric synthetic scenes: Cornell box, pots, and peppers.
• Includes one room-scale scene: kitchen.
• The simulated data is used to compare against two baselines: T-NeRF, which only models direct light, and FWP++, which models both direct and indirect radiance but lacks a physically accurate rendering model.

Captured Dataset

• Built with a hardware setup similar to that of Malik et al., capturing multi-viewpoint flash lidar data.
• A key difference: the light source moves with the camera's viewpoint rather than remaining stationary with respect to the scene.
• Three scenes are captured by the authors: globe, house, and spheres.
• The statue scene from Malik et al. is also included to demonstrate that the method handles stationary light sources when needed.
• Additional capture and calibration details are provided in the supplement.

How the Data Is Used

• For simulated experiments, the authors integrate lidar scans from novel viewpoints, recover surface normals, and render individual lidar frames.
• For captured experiments, the authors perform novel view synthesis of lidar measurements and report quantitative results in Table 1.
• In captured experiments, depth and normal metrics are omitted because no ground truth reference is available.

Key Findings from Data Usage

• T-NeRF fails on simulated scenes with strong indirect light, producing floating artifacts and inaccurate novel views.
• FWP++ recovers more accurate results but overfits by using mirror copies of the scene or incorrect depths to explain specular reflections and diffuse inter-reflections.
• In captured data, FWP++ shows slight improvements in view synthesis, likely because it uses a less constrained model, while the authors' physically grounded approach is more sensitive to calibration errors and model mismatch.

Method

The authors propose a method to model and invert time-resolved light transport, combining a physically-based renderer with a neural radiance cache to recover scene geometry and material properties from flash lidar measurements. As shown in the framework diagram below:

The rendering pipeline begins by casting a primary ray from the lidar sensor into the scene. For each point along this ray, the outgoing radiance is computed using a time-resolved rendering equation that incorporates the time of flight $\tau$τ. The incident radiance is decomposed into a direct component, representing light traveling directly from the pulsed source, and an indirect component, which is evaluated using the radiance cache to avoid the exponential computational cost of recursive evaluation. The final lidar measurement is obtained by volume rendering the outgoing radiance along the primary ray.

To efficiently model the indirect lighting, the method employs a time-resolved radiance cache parameterized by neural networks. This cache utilizes multi-resolution hash encodings to learn position-dependent appearance features and density for scene geometry. The cache distinguishes between direct and indirect outgoing radiance. The direct component utilizes a neural network to learn the BRDF, while the indirect component employs a split-sum approximation. This approximation separates the integration into an integrated BRDF and an integrated incident radiance, both predicted by separate neural networks conditioned on the light source position. Volume rendering along secondary rays cast from points on the primary ray yields the cached incident radiance.

The system is optimized through inverse rendering by minimizing the difference between the rendered incident radiance and the captured lidar measurements. The loss function includes a weighted photometric term that applies a tonemapping-like weighting to penalize errors more heavily in darker regions. Additionally, a radiometric prior ensures consistency between the radiance cache and the full physically-based model for both direct and indirect components. Further regularization is applied to tie predicted normals to analytic density gradients and to enforce smoothness on geometry and material parameters.

Experiment

The experiments evaluate the system on view synthesis of time-resolved and integrated lidar measurements, plus geometry reconstruction, against T-NeRF and FWP++ baselines on simulated and captured scenes. T-NeRF fails under indirect light due to its direct-only model, while FWP++ captures indirect radiance but overfits using mirror copies or incorrect depths. The physically grounded approach recovers more accurate geometry, especially where indirect light dominates, though on captured data FWP++ shows slight view synthesis gains likely from its less constrained model and calibration sensitivity. Additional results demonstrate relighting from novel light sources, training from CW-ToF or intensity-only supervision, and recovery of material parameters.

The proposed method achieves the best geometry recovery on simulated data, with the lowest normal MAE and depth L1 error, and the highest transient IOU. On captured data, it performs comparably to the FWP++ baseline for view synthesis but slightly trails it in PSNR, likely due to its more physically constrained model. Both the proposed method and FWP++ substantially outperform T-NeRF across all metrics on both simulated and real scenes. On simulated scenes, the proposed method recovers normals and depth with markedly lower errors than both baselines, and achieves the highest transient IOU. The proposed method and FWP++ yield similar view-synthesis quality on captured data, while T-NeRF lags far behind in all metrics. On captured scenes, FWP++ has a slight PSNR advantage over the proposed method, though their LPIPS and SSIM scores are nearly identical.

The experiments evaluate geometry recovery and view synthesis on simulated and captured scenes, comparing the proposed method against FWP++ and T-NeRF. On simulated data, the proposed method achieves the lowest normal and depth errors and the highest transient IOU, outperforming both baselines. On captured data, it delivers view synthesis quality comparable to FWP++, with nearly identical LPIPS and SSIM, though FWP++ holds a slight PSNR advantage; both substantially surpass T-NeRF across all metrics.