One-sentence Summary

Researchers from Johns Hopkins University propose a synchronization-free, single-shot depth-sensing platform for laparoscopic surgery that uses an LED-illuminated binary mask and a vector-quantized variational autoencoder prior coupled with a U-Net depth head to reconstruct dense depth from a single frame, achieving a mean absolute error of 3.70 mm and a processing rate of 26.0 Hz on a phantom dataset.

Key Contributions

• A compact LED-illuminated binary-pattern projection module replaces the DMD projector, fits within a dual-channel laparoscope, and enables synchronization-free single-shot structured light acquisition.
• A learned depth-reconstruction pipeline uses a VQ-VAE to encode the pattern-distorted image into a discrete latent representation and a U-Net decoder to predict depth directly, eliminating a separate mask-prediction branch.
• The training protocol decouples ground truth from conventional FPP by using an independent Zivid structured-light camera reprojected into the endoscopic frame; on a 722-image phantom dataset, the method achieves 3.70 mm MAE, δ=1.1 accuracy of 0.962, and stable 26.0 Hz inference on an NVIDIA A100 GPU.

Introduction

Autonomous laparoscopic robotic surgery demands dense, real-time metric depth maps for both visualization and closed-loop tool control. Prior learning-based monocular endoscopy approaches recover depth only up to an unknown scale, while conventional fringe projection profilometry (FPP) systems require multi-shot acquisition, bulky DMD projectors, and precise hardware synchronization, all of which complicate clinical translation. The authors present a single-shot structured light endoscopic camera that replaces the DMD projector with a compact LED-illuminated passive binary mask, and they introduce a VQ-VAE-regularized depth reconstruction network that operates synchronization-free at the camera frame rate. Their training protocol decouples ground truth from the legacy FPP algorithm by using an independent Zivid structured-light camera, achieving a mean absolute error of 3.70 mm and stable 26 Hz inference on phantom data.

Dataset

The authors construct a dedicated phantom dataset to train a single-shot depth estimation network for laparoscopic imaging. The data is collected with a custom system that pairs a CMOS endoscopic camera with a laterally mounted Zivid structured-light camera, while an intestinal phantom moves along the viewing axis on a motorized linear stage.

• Dataset composition and sources

  • The dataset contains 722 paired acquisitions.
  • Each pair consists of a grayscale SSLE image (captured under binary-pattern illumination through the laparoscope) and a dense reference depth map from the Zivid camera.
  • An automated program coordinates the endoscopic camera, the Zivid camera, and the linear stage to guarantee repeatable one-to-one pairing and to sample different working distances and surface poses.

• Subset split and usage

  • The 722 acquisitions are divided into training, validation, and held-out test subsets.
  • The training set optimizes the network parameters.
  • The validation set is used for model selection and hyperparameter monitoring.
  • All reported quantitative metrics are computed exclusively on the held-out test set.

• Reference-depth generation and processing

  • The Zivid depth map is reprojected into the SSLE coordinate frame through a one-time stereo calibration that provides the intrinsic matrices and the rigid transform between the two cameras.
  • The reprojection chain back-projects each Zivid pixel to a 3D point, transforms it into the SSLE frame, and projects it onto the SSLE image plane.
  • A z-buffer retains the nearest depth where multiple Zivid rays fall onto the same SSLE pixel.
  • The procedure produces a dense reference depth map and a binary validity mask that marks pixels with a valid reprojected depth inside the SSLE image bounds.

• Training pair format

  • A training sample is a triplet: the processed SSLE grayscale image, the reprojected SSLE depth map, and the validity mask.
  • The reconstruction loss and evaluation metrics are computed only at pixels where the mask equals 1, decoupling the supervision target from the single-shot algorithm and anchoring it to an external structured-light measurement.

Method

The authors design a single-shot structured light endoscopy (SSLE) platform that projects a static binary pattern through the illumination channel of a dual-channel laparoscope and reconstructs dense depth maps from a single camera frame using a latent-space deep network. The overall pipeline spans a custom optomechanical integration, automated data acquisition with cross-camera depth supervision, and a VQ-VAE–based depth prediction architecture.

The mechanical integration mounts a passive binary mask, an LED source, and relay optics inside a compact sealed enclosure that threads directly onto the illumination port of a laparoscope. Because the mask is static and the LED is DC-driven, no synchronization between projection and imaging is required; every captured frame carries a valid pattern.

The cutaway view shows the LED, relay, and mask arranged in a volume comparable to a standard light-guide connector, while the imaging channel of the endoscope remains unobstructed. The assembled laparoscope can be mounted on the camera arm of a UR5-style robotic platform without interfering with the tool-arm articulation envelopes.

The optical configuration relays the binary pattern from the LED-illuminated mask through lens L₁ and a pinhole into the illumination channel of the dual-channel laparoscope. The imaging channel collects the pattern-distorted scene and directs it via a folding prism onto a monochrome FLIR camera.

The bench-top realization places all components on an optical breadboard. A laterally mounted Zivid 2⁺ M60 structured-light depth camera is used only during data collection to provide independent ground-truth depth; at deployment the Zivid camera is removed and the endoscope operates as a stand-alone single-shot sensor. A representative endoscopic image of an intestinal phantom under binary-pattern illumination confirms that the projected stripes are clearly visible.

To ensure the pattern remains usable across typical laparoscopic working distances, the authors characterize the binary-pattern visibility on an inclined depth-of-field target.

At working distances of 7 cm and 10 cm, the projected stripes remain visually resolvable. Extracted intensity profiles show repeated peak-to-valley modulation with Michelson contrasts of 0.58 and 0.44, respectively, confirming sufficient pattern contrast for single-shot depth reconstruction.

Training data are collected with an automated acquisition program that coordinates the CMOS endoscopic camera, the Zivid camera, and a motorized linear stage. An intestinal phantom is translated along the endoscope viewing direction to sample different working distances and surface poses. At each position, a grayscale SSLE image under binary-pattern illumination is captured together with a Zivid RGB image, point cloud, and depth map. A common frame index preserves pairing between the SSLE input and the reference depth.

For every SSLE frame, the Zivid depth map is reprojected into the SSLE image frame using a one-time stereo calibration. Each valid Zivid pixel is back-projected to a 3-D point in the Zivid frame, transformed by the rigid-body transform (R, t) into the SSLE frame, and projected onto the SSLE image plane. A z-buffer retains the nearest depth where multiple Zivid rays map to the same SSLE pixel, producing a dense reference depth map D_SSLE and a binary validity mask M. This reprojection decouples the training target from the reconstruction algorithm and provides pixel-aligned supervision.

The depth-reconstruction network replaces a prior two-branch U-Net design with a single-branch pipeline that incorporates a vector-quantized variational autoencoder (VQ-VAE) as a latent-space prior.

A 256×256 grayscale fringe image x is processed through three sequential components. First, a VQ-VAE image encoder maps x to a continuous latent tensor z_in ∈ ℝ^{64×128×128}, then independently quantizes each spatial position by nearest-neighbor lookup in a learned codebook of K_img = 512 entries to produce the discrete latent representation \hat{z}_in. Second, a four-level multi-resolution residual U-Net (MRD U-Net) operates directly on \hat{z}_in, successively halving spatial resolution through max-pooling and doubling channel width, then reconstructing via bilinear upsampling with skip connections and a multi-level residual branch. The output is a latent depth representation \hat{z}_depth ∈ ℝ^{64×128×128}. Third, a VQ-VAE depth decoder mirrors the image encoder in reverse, using a codebook of K_depth = 1024 entries and a transposed-convolution upsampling step, followed by a tanh activation and linear rescaling to produce the final dense depth map \hat{D} ∈ ℝ^{1×256×256}. Operating in the quantized latent space eliminates the need for explicit foreground segmentation and provides a compact reconstruction pathway with stable inference timing.

The network is trained with a total loss that combines a spatial reconstruction term and a latent-space matching term:

The spatial loss is a weighted sum of an ℓ₁ loss and a structural similarity (SSIM) loss evaluated only over valid pixels indicated by M. The latent loss is the ℓ₁ distance between the standardized reference-depth latent representation z_d (obtained from a pretrained depth VQ-VAE encoder) and the predicted depth latent representation \hat{z}_d. The weight β(t) follows a linear ramp schedule that gradually increases during training, encouraging alignment with the learned quantized structural prior as training stabilizes.

Separate VQ-VAE models are pretrained for the input-image domain and the reference-depth domain using the taming-transformers framework. The MRD U-Net depth-prediction network is then trained for 161 epochs with the Adam optimizer, a cosine annealing learning-rate schedule, and an effective batch size of 4. All images and depth maps are resized to 256×256. Evaluation uses standard depth metrics—MAE, AbsRel, and accuracy at thresholds δ=1.1 and δ=1.1²—computed exclusively on the held-out test set over the Zivid validity mask.

Experiment

The evaluation uses a Zivid-referenced phantom dataset with 722 paired SSLE–Zivid acquisitions, comparing the proposed VQ-VAE + U-Net pipeline against a dual U-Net baseline and off-the-shelf monocular depth models. The proposed model achieves lower average error and its raw unmasked predictions already approximate the valid foreground, eliminating the need for an explicit mask branch, while inference runs at a stable 26 Hz. Remaining reconstruction errors are partly attributable to cross-camera reprojection mismatch from the Zivid supervision, and both task-specific models substantially outperform the monocular baselines, underscoring the importance of task-specific supervision for calibrated depth recovery in endoscopic structured-light settings.

The VQ-VAE + U-Net model achieved lower average depth error than the dual U-Net baseline on the Zivid-referenced phantom dataset, reducing MAE by about 14% and AbsRel by 42%, while retaining comparable accuracy at the stricter threshold. Public monocular depth models produced substantially higher errors, and the proposed model learned to implicitly represent the foreground without requiring an explicit mask branch. The proposed model reduced MAE from 4.30 mm to 3.70 mm and AbsRel from 0.0560 to 0.0326 compared to the MaskNet + DepthNet baseline, with only a marginal drop in δ=1.1 accuracy (0.962 vs. 0.965). Off-the-shelf monocular models (MiDaS DPT-Large and Depth Anything) yielded MAE above 7.8 mm, more than double the error of the proposed method. The raw unmasked predictions of the proposed model closely matched the phantom foreground, showing that the network learned an implicit segmentation without a separate mask prediction branch. Both the proposed and baseline models reported higher MAE than previously published FPP-supervised results because the Zivid cross-camera reprojection introduces a floor of misalignment error that cannot be compensated by the network.

The evaluation compares a VQ-VAE plus U-Net architecture against a dual U-Net baseline and public monocular depth models on a Zivid-referenced phantom dataset. The proposed model reduces average depth error substantially over the baseline, achieving lower mean absolute error and absolute relative error while maintaining comparable strict-accuracy, and it learns to implicitly represent the foreground without an explicit mask branch. Off-the-shelf monocular models produce more than double the error of the proposed method. All models exhibit higher error than previously published fringe projection profilometry-supervised results due to a floor of cross-camera misalignment introduced by the Zivid reprojection.