One-sentence Summary

Researchers at Reality Labs, Meta propose a differentiable rasterizer gradient method leveraging novel micro-edges to handle visibility discontinuities, simplifying gradient computation without forward-pass modifications, preserving image integrity for masks, depth, and normals, and enabling geometry intersections, as demonstrated in dynamic human head reconstruction.

Key Contributions

• Introduces a simplified approximation strategy for computing gradients at visibility discontinuities in rasterization-based differentiable renderers, distilling prior edge-sampling and analytical antialiasing ideas into a fast and straightforward solution with accurate gradients.
• Introduces a micro-edges concept that treats rasterized images as outcomes of a differentiable continuous process aligned with discrete-pixel rasterization, eliminating the need for rendering approximations or forward-pass modifications and enabling application to masks, depth, and normals images where filtering is prohibitive.
• Explicitly handles self-penetrating geometry and geometry intersections with virtually no additional overhead, and demonstrates enhanced mesh reconstruction fidelity across diverse inverse rendering scenarios, including fully automatic mouth interior fitting in dynamic human head scene reconstruction.

Introduction

Differentiable rendering is a core enabler for 3D object reconstruction, pose estimation, and emerging text-to-3D generative models, where mesh-based rasterization remains attractive for its high performance on opaque surfaces. However, rasterization depends on discrete operations for visibility and coverage, which create severe challenges for gradient computation, especially at triangle boundaries, occlusion edges, and geometry intersections. Prior approaches approximate soft membership through analytic antialiasing, Monte Carlo sampling, or falloff functions, yet these methods exhibit approximation errors that fail at intersections, and some alter the rendered image or require costly preprocessing to handle interpenetrating geometry. The authors introduce a micro-edge formulation that interprets the rasterized image as the result of a continuous process, producing simple, fast, and accurate gradient estimates without modifying the forward pass. This enables robust optimization of segmentation masks, depth, and normal maps, and unlocks complex applications such as accurate inner mouth fitting in dynamic facial performance capture, where occlusions and deformations are significant.

Method

The authors introduce EdgeGrad, a differentiable rasterization framework designed to compute gradients of pixel values with respect to mesh vertex positions, including visibility changes such as occlusion and disocclusion. Rather than approximating rasterization through continuous antialiasing representations, the method relies on a direct approximation where all geometric boundaries are decomposed into strictly vertical or horizontal micro-edges. These micro-edges are positioned precisely between pixel centers, ensuring that boundaries never intersect pixels and that each pixel is either fully covered or completely uncovered. This construction effectively treats the rasterization process as if an antialiasing filter were applied, rendering the filter an identity operation while preserving the discrete-pixel output.

Refer to the framework diagram for the micro-edge construction and the corresponding intensity variation across adjacent pixels.

The gradient computation is formulated as a local operation analyzing one edge between two neighboring pixels at a time. For a horizontal pixel pair separated by an edge at position $p_{AB}$pAB​, the intensities $I_A$IA​ and $I_B$IB​ are modeled using Heaviside step functions to capture the discontinuity. Because the edge lies exactly on the pixel boundary, a C1 discontinuity arises at $p_{AB} = 0$pAB​=0, making the intensity values non-differentiable at that exact point. The authors circumvent this by computing the one-sided limits of the derivatives from the left and right, and then averaging them. This yields the gradient of the loss function with respect to the edge position as $\frac{\partial L}{\partial p_{AB}} = \frac{1}{2} \left(\frac{\partial L}{\partial I_A} + \frac{\partial L}{\partial I_B}\right) (I_A - I_B) + \Omega$∂pAB​∂L​=21​(∂IA​∂L​+∂IB​∂L​)(IA​−IB​)+Ω, where $\Omega$Ω represents the gradient contributions from the smooth interior regions of the primitives.

To propagate these gradients back to the scene parameters, specifically the triangle vertex positions, the edge gradients must be scattered to the adjacent fragments. The backward propagation pipeline traces the loss gradient and distributes it across the relevant fragments based on how edge movement influences fragment positions. Refer to the gradient computation flow diagram for the backward propagation process.

The computation of the fragment gradients requires multiplying the edge gradient by the partial derivative $\frac{\partial p_{AB}}{\partial r}$∂r∂pAB​​, which relates the movement of the edge to the movement of the fragment. The value of this derivative depends heavily on the geometric relationship between the primitives forming the edge. The framework employs a classification heuristic that examines each neighboring pixel pair by comparing their triangle IDs and testing pixel centers against the triangles of the adjacent pixels. Refer to the edge classification schematic for the decision logic applied to different geometric configurations.

The classification categorizes edges into four distinct types. If the triangle IDs match, no edge exists and gradients are not scattered. When the IDs differ, the method tests whether pixel centers lie inside the adjacent triangle. If both centers are outside their respective adjacent triangles, the primitives are classified as adjacent, sharing a common boundary without discontinuity, and gradients are not scattered. If one center is inside the other's triangle, the configuration is identified as overhanging, where the gradient is scattered exclusively to the foreground primitive. If both centers are inside each other's triangles, the primitives intersect.

For intersecting primitives, the derivative $\frac{\partial p}{\partial r}$∂r∂p​ is derived by analyzing the intersection line in the x-z plane, where the z-axis is perpendicular to the image plane. The micro-edge formulation simplifies this complex 3D interaction into a 2D problem. Refer to the geometry intersection derivation diagram for the x-z plane analysis.

The authors compute the derivative by fixing one fragment and varying the other, then switching roles, utilizing the normals of the fixed and varying primitives. Once the fragment gradients are computed for all edge types, they are accumulated via the backward function of the barycentric interpolator module. This gathers the per-fragment gradients into the final vertex gradients, completing the differentiable rasterization pipeline without requiring additional connectivity data structures or modifying the initial rasterized image.

Experiment

The experiments validate an edge gradient rasterization approach, showing accurate forward and backward gradients even with intersecting geometry, outperforming nvdifrast and matching raytracing at lower computational cost. Scene reconstruction on the Blender dataset yields improved mesh detail, and runtime is faster across image sizes. The method is further applied to dynamic head scene fitting, where correct edge gradients enable fully automatic reconstruction of complex mouth interior geometry, including teeth, tongue, and lips, with segmentation masks helping anchor these regions.

The proposed EdgeGrad method consistently achieves higher PSNR and SSIM than the nvdiffrast baseline across all tested scenes with complex geometries and non-Lambertian materials. It also shows notable gains over a continuous-only baseline that ignores discontinuity gradients, and its variant without intersection handling performs slightly worse but still above nvdiffrast. The results indicate that accurate edge gradients improve reconstruction quality, particularly for scenes with intricate geometry. EdgeGrad outperforms nvdiffrast in PSNR on every scene, with the largest gains on Chair, Hotdog, and Ficus. SSIM is also improved by EdgeGrad for most scenes, especially on Chair, Ship, Ficus, and Drums. The continuous-only baseline yields much lower scores, underscoring the importance of handling discontinuity gradients.

EdgeGrad consistently outperforms the nvdiffrast baseline in both PSNR and SSIM across all tested scenes with complex geometries and non-Lambertian materials, with the largest gains on scenes like Chair, Hotdog, and Ficus. A continuous-only baseline that ignores discontinuity gradients performs much worse, highlighting the importance of accurate edge gradients for reconstruction quality. A variant without intersection handling still surpasses nvdiffrast but slightly trails full EdgeGrad, confirming that precise discontinuity gradients are key, especially for intricate geometry.