One-sentence Summary

Imperial College London proposes Heat Kernel Textures (HKTex), a UV-free texture representation that replaces Gaussians with anisotropic heat kernels intrinsically defined on triangular mesh surfaces via discrete Riemannian geometry, eliminating UV unwrapping issues such as seams and distortions while reducing memory footprint and integrating with physically based renderers.

Key Contributions

• Heat Kernel Textures (HKTex) introduces an intrinsic texture representation for triangular meshes that replaces UV-mapped textures with anisotropic heat kernels defined through a Riemannian Laplace-Beltrami operator, eliminating seams, distortions, wasted UV space, vertex duplication, and varying resolution.
• The method provides surface-native optimization and adaptive densification for heat kernels, integrated with a differentiable physically based renderer, allowing kernel positions and surface appearance to be optimized from existing textures or multi-view images while remaining aligned with the mesh geometry.
• The spectral heat-kernel formulation is reported to run about one order of magnitude faster than sparse linear-system heat diffusion simulation and to scale to high-resolution meshes, enabling compression of existing UV textures and direct albedo reconstruction from multi-view images.

Introduction

Rendering and 3D content pipelines still rely heavily on UV-mapped textures, which are widely used in games, virtual and augmented reality, and visual effects but introduce seams, distorted resolution, and redundant vertex storage. Prior attempts using neural fields or point clouds reduce manual unwrapping yet often require large memory, topology-dependent structures, or many primitives, and 3D Gaussian Splatting fails to conform to surfaces because its primitives live in ambient Euclidean space. The authors introduce Heat Kernel Textures, a mesh intrinsic texture representation that replaces UVs and Euclidean Gaussians with anisotropic heat kernels evaluated directly on the surface through Riemannian optimization and manifold-aware density control. Their method supports differentiable physically based rendering, texture compression, and appearance reconstruction from multi-view images without UV maps.

Dataset

• The authors use a curated subset of Objaverse, a scraped collection of online 3D assets, for their experiments.

• The final dataset contains about 3,000 high quality models after filtering.

• Source filtering: objects must have LVIS category labels, valid textures, and at least one community like and view. Point clouds and animated models are removed.

• Computational filtering: meshes are limited to a maximum of 60,000 vertices.

• Geometric filtering for HKTex compatibility: meshes are excluded if they have multiple disconnected components, non manifold geometry, or complex internal structures. Complex internal structures are defined as having more than 20 intersections on any ray cast from a regular grid on the object bounding box.

• Mathematical filtering: models are rejected if eigendecomposition of the Laplace Beltrami operator fails, because the method relies on its eigenfunctions.

• For multi-view image fitting, meshes are rendered using albedo as the sole texture map.

• The paper does not describe explicit train/validation/test splits, mixture ratios, cropping rules, or additional metadata schemas beyond the filtering criteria above.

Method

The proposed HKTex model represents surface texture as an intrinsic, mesh continuous set of heat kernels located directly on a manifold mesh. Rather than assigning colors to texture coordinates or vertex attributes, the method treats the texture as a collection of $S$S optimizable kernels. Each kernel is parameterized as

$$\kappa = \left(p^{\star}, \theta, \eta, \tau, \varsigma, \mathbf{c}\right),$$κ=(p⋆,θ,η,τ,ς,c),

where $p^{\star}$p⋆ is the heat source position on the surface, $\theta$θ and $\eta$η control the directional and anisotropic diffusion behavior, $\tau$τ controls the spatial extent of the kernel, $\varsigma$ς controls border sharpness, and $\mathbf{c}$c is the kernel color. All kernel parameters are optimized jointly.

For a query point $\boldsymbol{p}$p on the surface, the heat kernel is evaluated using the continuous spectral formulation

$$h_t \big(\boldsymbol{p}, \boldsymbol{p}^{\star} \mid \theta, \eta\big) = \sum_{k=1}^{K} e^{-t\lambda_k^{\eta,\theta}} \varphi_k^{\eta,\theta}(\boldsymbol{p}) \varphi_k^{\eta,\theta}(\boldsymbol{p}^{\star}) m^{\star},$$ht​(p,p⋆∣θ,η)=k=1∑K​e−tλkη,θ​φkη,θ​(p)φkη,θ​(p⋆)m⋆,

where $\lambda_k^{\eta,\theta}$λkη,θ​ and $\varphi_k^{\eta,\theta}$φkη,θ​ are eigenvalues and eigenfunctions associated with the anisotropic diffusion operator, $t$t is the diffusion time, and $m^{\star}$m⋆ is the source mass. Because diffusion spreads energy away from the source, larger kernels tend to lose peak intensity. To keep optimization stable, each kernel is normalized so that its value at the source location is one:

$$\bar{h}_t \big(\boldsymbol{p} \mid \boldsymbol{p}^{\star}, \theta, \eta\big) = \frac{ h_t \big(\boldsymbol{p}, \boldsymbol{p}^{\star} \mid \theta, \eta\big) }{ h_t \big(\boldsymbol{p}^{\star}, \boldsymbol{p}^{\star} \mid \theta, \eta\big) }.$$hˉt​(p∣p⋆,θ,η)=ht​(p⋆,p⋆∣θ,η)ht​(p,p⋆∣θ,η)​.

Since truncated spectral kernels can have weak distant support and can produce ringing artifacts, the model applies a biharmonic distance Gaussian weight. The squared biharmonic distance between two surface points is computed from isotropic eigenproperties:

$$d_{\mathrm{BH}}(\boldsymbol{p}, \boldsymbol{p}^{\star})^2 = \sum_{k=1}^{K} \frac{ \left( \varphi_k^{\mathrm{iso}}(\boldsymbol{p}) - \varphi_k^{\mathrm{iso}}(\boldsymbol{p}^{\star}) \right)^2 }{ \left(\lambda_k^{\mathrm{iso}}\right)^2 }.$$dBH​(p,p⋆)2=k=1∑K​(λkiso​)2(φkiso​(p)−φkiso​(p⋆))2​.

The normalized kernel is then multiplied by a Gaussian decay term:

$$\tilde{h}_t \big(\boldsymbol{p} \mid \boldsymbol{p}^{\star}, \theta, \eta\big) = \bar{h}_t \big(\boldsymbol{p} \mid \boldsymbol{p}^{\star}, \theta, \eta\big) \cdot \exp\left( -\frac{ d_{\mathrm{BH}}(\boldsymbol{p}, \boldsymbol{p}^{\star})^2 }{ 2\sigma_{\mathrm{BH}}^2 } \right).$$h~t​(p∣p⋆,θ,η)=hˉt​(p∣p⋆,θ,η)⋅exp(−2σBH2​dBH​(p,p⋆)2​).

This produces a locally supported geodesic kernel whose response fades with distance from its source. The model does not rely on adjusting the diffusion time to control kernel scale. Instead, it applies a soft step filtering function that gives explicit control over both extent and sharpness:

$$\alpha(\boldsymbol{p} \mid \boldsymbol{p}^{\star}, \theta, \eta, \tau, \varsigma) = \frac{ \xi\big(\tilde{h}_t(\boldsymbol{p} \mid \boldsymbol{p}^{\star}, \theta, \eta)\big) - \xi(0) }{ \xi(1) - \xi(0) }, \quad \xi(x) = \sigma\big(\varsigma \cdot (x-\tau)\big),$$α(p∣p⋆,θ,η,τ,ς)=ξ(1)−ξ(0)ξ(h~t​(p∣p⋆,θ,η))−ξ(0)​,ξ(x)=σ(ς⋅(x−τ)),

where $\sigma(\cdot)$σ(⋅) is a sigmoid function. This filtering step keeps kernel values in a bounded range while allowing kernels to remain soft and geodesic or become sharper and more localized.

The final color at a surface point is formed by aggregating contributions from the strongest kernels covering that point. For each surface location $\boldsymbol{p}$p, the model identifies the top contributing kernels and forms an alpha weighted partition of unity over them. The resulting color is added as a residual to an optimizable global base color:

$$C(\boldsymbol{p}) = \mathbf{c}_{\mathrm{base}} + \frac{ \sum_{s \in \mathcal{K}(\boldsymbol{p})} \alpha_s(\boldsymbol{p}) \mathbf{c}_s }{ \sum_{s \in \mathcal{K}(\boldsymbol{p})} \alpha_s(\boldsymbol{p}) },$$C(p)=cbase​+∑s∈K(p)​αs​(p)∑s∈K(p)​αs​(p)cs​​,

where $\mathcal{K}(\boldsymbol{p})$K(p) denotes the selected local subset of kernels. This design gives controlled blending between overlapping kernels, while regions not covered by any kernel fall back naturally to the base color.

A key computational challenge is that each kernel may have a different anisotropy and orientation, and evaluating heat kernels directly for every kernel configuration would require constructing and decomposing a unique anisotropic Laplace Beltrami operator during optimization. To avoid this cost, the authors precompute spectral bases over a discretized parameter space of angles and anisotropies. Anisotropy values are mapped into a logarithmic space, $s=\ln(1+\eta)$s=ln(1+η), so that large anisotropies do not dominate the grid density. Each grid node stores precomputed eigenvalues and eigenvectors. For a continuous query with parameters $(\theta,\eta)$(θ,η), the model locates the surrounding grid cell and performs bilinear interpolation:

$$\boldsymbol{\Phi}_{\eta,\theta}^{\mathrm{q}} = \sum_{(a,b) \in \mathcal{C}} w_{a,b} \boldsymbol{\Phi}_{a,b}^{\ddagger}, \quad \boldsymbol{\Lambda}_{\eta,\theta}^{\mathrm{q}} = \sum_{(a,b) \in \mathcal{C}} w_{a,b} \boldsymbol{\Lambda}_{a,b}^{\ddagger}.$$Φη,θq​=(a,b)∈C∑​wa,b​Φa,b‡​,Λη,θq​=(a,b)∈C∑​wa,b​Λa,b‡​.

The local coordinates $\Delta\theta$Δθ and $\Delta s$Δs determine the interpolation weights from the four neighboring grid corners.

Because independently decomposed eigenbases may differ by permutations, rotations, and sign flips, the method includes an alignment procedure before interpolation. The permutation ambiguity is resolved by comparing each basis against a reference basis using a mass weighted cross correlation matrix and the Hungarian algorithm. The remaining orthogonal ambiguity is addressed by solving a mass weighted orthogonal Procrustes problem through singular value decomposition. Finally, sign corrections are applied using diagonal sign entries derived from the aligned bases. This yields a consistent spectral dictionary that can be interpolated reliably.

The model also decouples kernel positions from mesh vertices. Precomputed spectral quantities are initially defined only at vertices, but kernels and query points can exist anywhere on the surface. To evaluate a vertex defined spectral quantity $\mathbf{A}$A at an arbitrary surface location $\boldsymbol{p}$p, the method uses barycentric interpolation inside the containing triangle, written compactly as $b^p(\mathbf{A})$bp(A). This makes the representation continuous and allows heat sources to move off vertices.

Direct evaluation of every kernel at every target point would be too expensive. The method reduces this cost using the locality imposed by biharmonic weighting. A spectral embedding is defined as

$$\mathbf{Z} = \bar{\Phi}^{\mathrm{iso}} \left(\Lambda^{\mathrm{iso}}\right)^{-1},$$Z=Φˉiso(Λiso)−1,

where Euclidean distance in this embedding space is used as a proxy for biharmonic distance on the manifold. Kernel sources and target points are mapped into this embedding, and a nearest neighbor search selects only a small local set of relevant sources for each target point. Instead of evaluating all kernels globally, the system retrieves the interpolated spectral bases required only by the selected local kernels. The kernel values are then computed for those nearby sources, reducing the per point evaluation cost to a local complexity proportional to the number of retained neighbors.

The heat kernel formulation also requires a source mass. The method notes that directly interpolating the discrete mass matrix is not appropriate because vertex masses are tied to sampling density. A kernel density estimation approach can approximate pointwise mass, but in practice assigning a unit mass to all sources performs well and is much faster. Since the kernel evaluation depends on mass as a linear scalar, the remaining kernel parameters can compensate during optimization.

Geodesic optimization is another important component of the design. Kernel sources must remain on the manifold surface, so standard Euclidean gradient updates would pull them into or away from the surface. The method instead applies a Riemannian gradient descent strategy. A gradient is first projected onto the tangent plane at the current kernel position. That projected gradient defines a velocity vector, and the exponential map is used to move the source along a geodesic on the surface. Momentum, when used, is parallel transported from the old position to the new one. This keeps all source motion intrinsic to the mesh while preserving smooth optimization trajectories.

Density control follows a style similar to Gaussian splatting methods. During optimization, the system tracks a hit count and an energy score for each kernel. The hit count measures how often a kernel is selected as one of the strongest contributors during color formation, while the energy score aggregates its filtered contributions over an accumulation window. Kernels with low hit counts or low energy are removed to reduce overhead. Conversely, kernels responsible for high reconstruction error are densified. The spatial color error is distributed to contributing kernels according to their partition of unity weights. High error kernels are then either cloned or split. Small kernels are duplicated in place, allowing the optimizer to separate them naturally, while larger kernels are split into smaller kernels displaced along their principal geodesic directions. A maximum growth ratio limits how many kernels can be added per step to maintain stability.

The final representation is evaluated continuously, so it can be integrated directly into physically based rendering pipelines. In a differentiable renderer, when a ray intersects the mesh at an arbitrary surface point, the renderer identifies the locally relevant kernels and evaluates the color using the same kernel aggregation rule. Gradients from rendered pixels can backpropagate through the evaluator to all intrinsic kernel parameters.

Training can be performed in two main ways. If a ground truth UV texture is available, the model samples dense points on the mesh surface, evaluates the HKTex color at those locations, and minimizes photometric error against the color obtained from the UV texture. If only multiview images are available, the model is optimized through the differentiable renderer. Camera views are sampled, rays intersecting the mesh are collected, and the rendered color is compared with ground truth pixels. In both settings, optimization updates kernel positions, orientations, anisotropies, extents, sharpness, colors, and the base color through end to end gradients.

Experiment

The experiments evaluate HKTex on a filtered Objaverse subset using rendered multi-view assessments and storage accounting. UV-texture fitting compares the method against neural fields, image-based representations, and vertex-color baselines under matched memory budgets, showing that HKTex provides compact surface-bound representations while preserving structural details and sharp boundaries more effectively than several alternatives. Multi-view inverse rendering further validates its ability to recover appearance from sparse observations, yielding favorable perceptual and structural quality with lower storage, although kernel evaluation adds overhead partially offset by rendering cost.

On a curated Objaverse subset, UV texture fitting shows that intrinsic and explicit baselines vary widely in reconstruction quality and memory demand. High-resolution ground-truth vertex colors give strong texture reproduction but require large storage, while low-resolution UV textures and ImageGS can keep modest budgets but suffer weaker texture fidelity. HKTex is reported to preserve sharper boundaries and more faithful color distributions under comparable or smaller storage. HKTex preserves sharper texture boundaries and more faithful color distributions while staying within similar or smaller storage budgets than the compared baselines. High-resolution vertex colors deliver the strongest texture reproduction among vertex-based options but require substantially more memory. Neural baselines need large storage to approach reconstruction quality, and ImageGS can show artifacts when splat count is limited.

HKTex achieves the strongest reconstruction quality across perceptual, structural, and error metrics while using less storage than the compared neural representations. It also slightly improves upon high-resolution vertex colors in overall appearance quality while requiring similar or lower storage. Its rendering time is higher, likely due to representation evaluation overhead, though ray tracing can dominate total render cost in the multi-view setting. HKTex provides more accurate and perceptually faithful albedo reconstruction than MLP, hash encoded, and high-resolution vertex color baselines. The method reaches better quality with compact storage, suggesting surface constrained representation is effective for sparse multi-view observations.

Experiments on a curated Objaverse subset compare UV texture fitting and albedo reconstruction quality, storage efficiency, and rendering overhead across intrinsic, explicit, and neural baselines. The results show that high-resolution vertex colors provide strong texture reproduction but demand large memory, while low-resolution UV and splat-based methods use less storage but can weaken fidelity or introduce artifacts. HKTex consistently preserves sharper texture boundaries and more faithful color distributions, achieving perceptually faithful reconstruction with compact storage and suggesting that surface-constrained representations are effective under sparse multi-view observations. It requires higher rendering time due to representation overhead, although ray tracing can dominate total cost.