One-sentence Summary

Stony Brook University proposes Poppy, a training-free framework that refines frozen monocular normal estimators by optimizing per-pixel offsets using single-shot polarization measurements via a differentiable rendering layer, reducing mean angular error by 23\%$--$26\% on synthetic and 6\%$--$16\% on real data across seven benchmarks without retraining.

Key Contributions

• Poppy is a training-free framework that refines surface normals from any frozen RGB normal estimator by optimizing per-pixel image and normal offsets against single-shot polarization measurements at test time. It targets reflective, textureless, and dark surfaces without polarimetric training data or backbone retraining.

• The method couples a learned reflectance decomposition with a differentiable normal-to-Stokes rendering layer, so refined normals predict diffuse and specular polarization and penalize mismatches with observed signals. A learned specular radiance map helps resolve diffuse/specular ambiguity in single-view polarization.

• Across seven benchmarks and diffusion, flow, and feed-forward backbones, Poppy reduces mean angular error by 23 to 26 percent on synthetic data and 6 to 16 percent on real captures. The results show plug and play test-time guidance of frozen RGB-based normal estimators using physics-based polarization cues.

Introduction

Monocular surface normal estimation recovers per-pixel 3D orientation from a single RGB image, which is essential for robotics, augmented reality, scene understanding, and 3D reconstruction. Existing RGB-only methods often fail on reflective, textureless, and dark surfaces because these pixels have ambiguous or weak shading cues, while polarization-based approaches either suffer from intrinsic normal ambiguities or require expensive paired training data. The authors introduce Poppy, a training-free framework that uses linear polarization captured at test time to guide any frozen RGB normal estimator. By optimizing learnable per-pixel image and normal offsets plus a specular radiance map through a differentiable polarization rendering layer, the method resolves global scene orientation and diffuse-specular ambiguity, improving normals on challenging materials without retraining the backbone.

Method

The authors leverage a frozen monocular RGB normal estimation network and formulate polarization-aware surface normal recovery as an inference time guidance problem. The goal is to preserve the strong geometric priors learned by modern RGB normal estimators while enforcing consistency with polarization measurements captured by a linear polarization camera. Rather than retraining the backbone network, the method optimizes a small set of test time parameters that guide the network predictions toward physically valid Stokes measurements.

Given polarization images captured at angles $0^\circ$0∘, $45^\circ$45∘, $90^\circ$90∘, and $135^\circ$135∘, the pipeline first computes the observed Stokes vector

$$\mathbf{S} = [S_0, S_1, S_2]^{\top}.$$S=[S0​,S1​,S2​]⊤.

The unpolarized intensity $S_0$S0​ is used as the RGB input $x$x. A pretrained RGB normal estimator $f$f predicts a base normal field

$$\hat{n}_{\text{base}} = f(x).$$n^base​=f(x).

To compare this prediction with the observed polarization data, the authors introduce a differentiable rendering model that converts a candidate surface normal and specular radiance $L_s$Ls​ into a predicted Stokes vector

$$\hat{\mathbf{S}} = \mathcal{F}(\hat{n}, L_s).$$S^=F(n^,Ls​).

If only the specular radiance $L_s$Ls​ is optimized, the method can fail when the frozen backbone produces an incorrect normal estimate. In such cases, the residual $|\mathbf{S} - \hat{\mathbf{S}}|$∣S−S^∣ may persist because the error originates from the normal itself, or the radiance can overfit to the wrong geometry. As shown in the figure below:

To address this failure mode, the authors apply learnable offsets before and after the frozen backbone. The refined normal at iteration $t$t is written as

$$\hat{n}_t = f(x + O_x) + O_n,$$n^t​=f(x+Ox​)+On​,

where $O_x \in \mathbb{R}^{H \times W \times 3}$Ox​∈RH×W×3 is a per-pixel input offset and $O_n \in \mathbb{R}^{H \times W \times 3}$On​∈RH×W×3 is a per-pixel normal offset. The backbone weights remain fixed throughout optimization.

The input offset $O_x$Ox​ provides global guidance. Because a small perturbation in the RGB input can influence a broad region of the output normal map, modifying $O_x$Ox​ allows the optimization to steer the entire predicted geometry toward a polarization-consistent solution. This behavior is related to the input-to-normal sensitivity of the frozen network. The normal offset $O_n$On​ provides local guidance. It compensates for the tendency of RGB normal estimators to produce overly smooth predictions and helps recover high-frequency surface details that the backbone may miss.

During guidance, the trainable parameters are

$$\Theta = \{L_s, O_x, O_n\}.$$Θ={Ls​,Ox​,On​}.

At each iteration, the backbone processes the offset input $x + O_x$x+Ox​, producing a base normal prediction. The normal offset is then added to obtain the refined normal $\hat{n}_t$n^t​. Given $\hat{n}_t$n^t​ and the current specular radiance $L_s$Ls​, the differentiable renderer produces the predicted Stokes vector

$$\hat{\mathbf{S}}(\hat{n}_t, L_s).$$S^(n^t​,Ls​).

The method minimizes the discrepancy between the observed Stokes vector and the rendered Stokes vector:

$$\mathcal{L} = \sum_{p} M(p) \sum_{i=0}^{2} \left| S_i(p) - \widehat{S}_i(p) \right|,$$L=p∑​M(p)i=0∑2​​Si​(p)−Si​(p)​,

where $p$p denotes a pixel. The validity mask $M(p)$M(p) restricts the loss to physically meaningful measurements. A pixel is retained if it has sufficient unpolarized intensity,

$$S_0 > 0.01,$$S0​>0.01,

is not saturated,

$$S_0 < 1,$$S0​<1,

and satisfies the physical Stokes constraint

$$S_1^2 + S_2^2 \leq S_0^2.$$S12​+S22​≤S02​.

The optimization uses a staged schedule. In the initial phase, only $L_s$Ls​ and $O_x$Ox​ are updated. This allows the specular radiance to converge and the global geometry to become plausibly consistent with the observed polarization field. After a warm-up phase, the normal offset $O_n$On​ is introduced, enabling optimization of local high-frequency corrections. At that point, all three parameter groups, $L_s$Ls​, $O_x$Ox​, and $O_n$On​, are updated until the final iteration $T$T. Each parameter group can use its own learning rate.

The parameters are updated by gradient descent:

$$\Theta^{(k+1)} = \Theta^{(k)} - \lambda_{\Theta} \nabla_{\Theta} \mathcal{L}.$$Θ(k+1)=Θ(k)−λΘ​∇Θ​L.

After optimization, the final normal is obtained by normalizing the guided prediction:

$$\hat{n} = \frac{f(x + O_x) + O_n}{\left\| f(x + O_x) + O_n \right\|_2}.$$n^=∥f(x+Ox​)+On​∥2​f(x+Ox​)+On​​.

The learned specular radiance and the refined normal together produce a Stokes prediction that better matches the observed Stokes vector, while the frozen RGB backbone continues to contribute its learned geometric priors. The method assumes orthographic projection and neglects viewing directions, an approximation that introduces little error unless the scene is captured at very close range.

Experiment

Experiments evaluate polarization-guided normal estimation on synthetic and real benchmarks across three backbones, comparing against monocular normal baselines under textureless, specular, and low-SNR conditions. Polarization guidance consistently improves surface orientation and fine detail, with the image offset driving broad geometric correction and the normal offset recovering sharper local structure, while remaining stable under noise and varied reflectance. The refined normals also support appearance decomposition and better downstream mesh reconstruction.

Adding polarization guidance to RGB normal estimators consistently lowers mean and root mean square angular error while increasing thresholded accuracy on both real and synthetic benchmarks. The improvements are especially strong for low angular error tolerance, indicating that the method corrects global surface orientation and recovers fine local geometric details. The guided model also behaves conservatively when polarization cues are noisy, converging toward the unguided prediction instead of amplifying unreliable measurements. Polarization guidance reduces mean absolute and root mean square normal error across the evaluated RGB backbones. Thresholded accuracy improves most at the tighter angular error threshold, especially on synthetic data. Joint optimization of image and normal offsets helps correct broad geometry first and then refine high-frequency surface structures. The method remains useful under material variation and becomes conservative as polarization noise increases. The evaluated guidance generally outperforms backbone baselines, with degradation attributed to unmodeled perspective distortion in one case.

Poppy variants using polarization guidance reduce MAE relative to unguided backbone predictions on both synthetic and real-world datasets. Joint optimization generally matches or slightly improves over image-only guidance, suggesting that refining normal offsets helps recover more accurate geometry. The gains remain visible across real datasets, although challenging perspective effects can weaken the advantage of guidance. Both image-only and joint Poppy lower error compared with no-guidance baselines for the shown backbones. Joint Poppy often gives the best MAE, indicating that normal-offset refinement improves fine geometric detail. The benefit persists on real-world inputs, but perspective distortion can reduce guidance effectiveness on difficult cases.

Experiments evaluate polarization-guided RGB normal estimation, including Poppy variants, on synthetic and real benchmarks against unguided backbones, testing angular accuracy, material variation, noise robustness, and optimization modes. Polarization guidance consistently improves global surface orientation and fine geometric detail, with joint image and normal-offset refinement helping correct broad geometry first and then recover higher-frequency structures. The method remains conservative when polarization cues are noisy, although perspective distortion can reduce its advantage in difficult real-world cases.