I took COS526: Neural Rendering in Spring 2026. My teammate and I decided to create a model that transfers the style of a 2D image to a 3D asset, creating reusable stylized assets. In this summary, I’ll go over my contributions to the project: multiview-consistent style transfer, and back projection.

Overview of the Pipeline

The general approach involves converting a 3D mesh into a series of 2D views, then using diffusion-based 2D-to-2D style transfer techniques to perform the style transfer, and finally backprojecting the stylized views into PBR textures for the mesh.

The pipeline takes a single style image and an untextured 3D mesh and returns a fully stylized, reusable asset with PBR textures. It runs in three stages: multiview-consistent style transfer stylizes rendered views of the mesh, back projection bakes those views into a UV texture atlas, and a differentiable rendering step refines the normal, roughness, and albedo maps.

Multiview-consistent Style Transfer

Performing style transfer on each view independently results in large inconsistencies across views even with ControlNet and other guardrails. This makes it impossible to backproject a consistent UV texture.

Thus, I chose to use an iterative process that enforces consistency across views, taking advantage of the fact that we have ground-truth transformation matrices for each view (as we rendered it from a mesh).

Rendering Views

I use Kaolin to render 16 views of the mesh — 12 at 30° azimuth increments plus 4 polar views — for full angular coverage. Each view saves an RGB image, a foreground-normalized depth map, and a Canny edge map, with the camera parameters in camera.json.

The camera angles were intentionally over-specified in order to ensure that at every step, there is enough information from the previous angle to properly enforce view consistency. However, this over-specfication lead to the problem of over-constraining the inpainting process (more on this later).

ConsisLoRA Training

The main 2D-to-2D style transfer backbone I used was ConsisLoRA [1], it is not state-of-the-art perse (the Flux models are better at this than SDXL), however given the limited compute we had, this is my best option. I train one content LoRA per view and a single shared style LoRA.

Style LoRA

The style LoRA is two-stage: first a content LoRA is trained on the style image (A [v]) as a structural anchor, then the style LoRA proper (An image in the style of [v]) is trained conditioned on it.

Content LoRAs

Each content LoRA is trained on its own view (prompt A [c], rank 64), anchoring the structure of that specific pose.

Inference Pipeline

Views are stylized in ascending order of |azimuth| + |elevation|, so the frontal pose acts as the anchor and every later view inherits from the ones already done.

First View

The anchor view is generated with SDXL under Depth and Canny ControlNets. Every later view is histogram-matched to it to prevent color drift, and a BiRefNet mask isolates the foreground.

Cumulative View Warping

For each later view I forward-warp all previously stylized views into the current frame to build a color prior: per-pixel depth is recovered from the rendered depth map, unprojected to world space, and reprojected, then composited nearest-view-wins with seam healing and mask erosion to stop color bleeding.

RePaint-inspired Inpainting

SDXL fills the still-unobserved regions with a block-wise RePaint schedule [2], pinning the known pixels at every step and re-noising block boundaries so the generated content blends into the warped prior.

Afterwards, the model is allowed to diffuse freely with no pinning to allow the seams between the pinned and inpainted regions to heal.

Back Projection

UV Rasterization

Colors are baked into a 4096×4096 UV atlas. Each texel gets a world-space position by rasterizing the mesh triangles in UV space and barycentrically interpolating the vertex positions and normals.

Projection and Visibility

Each texel is projected into every view. A view only contributes if the projection is in-bounds, passes a depth test (rejecting occluded texels), and lands inside that view’s BiRefNet foreground mask.

View Selection and Color Sampling

Among the passing views I take the most face-on one — maximizing alignment between the surface normal and the camera — to avoid grazing-angle distortion, sample its color with bilinear interpolation, and fill leftover seam texels by nearest-neighbor propagation.

Results

We performed style transfer of 7 meshes with 8 different style to produce the following:

In general, we found that our current pipeline performs well on well-defined and simple styles such as the sketch (8th row) and the minimal line art (2nd row). It does especially horribly for more realistic styles such as the Rembrandt painting (3rd row) or the Robert Frank photo (5th row) or more subtle styles such as the pixelated style (1st row).

A possible reason for this can be chalked up to model complexity (i.e. the diffusion model is not sufficiently complex for the task). However, my preferred interpretation for why the results for these styles are not good is that UV mapping is insufficient to fully capture the style. For example, a big component of the Rembrandt style is his harsh lighting, something that’s hard to capture in a texture and is more suitable for capture with a shader perhaps. While the pixelated style is better captured with directly editing the mesh.

This opens up avenues of future work in extending the pipeline’s capabilities beyond just PBR textures towards also writing shaders and editing meshes.

Future Work

The fixed 16-view rig wastes compute on simple meshes and under-covers complex ones, so a geometry-aware next-best-view camera placement is a natural next step.

For the cummulative warping, the depth map of the original mesh was used. However, the stylization process introduces small discrepancy in the silhoulette of the mesh. As a result, using the original depth map is inaccurate and can lead to improper warping. A possible future work could learn an optical flow from the original view to the stylize view and replay the flow to transform the depthmap to better reflect the

Finally, the pipeline’s capabilities could be extended to edit the mesh or write accompanying shaders that could allow for the capture of a wider range of styles.

Use of AI in this project

I used Claude Code to help with the implementation of the codebase.