Abstract

We introduce (), a new state-of-the-art, fast pipeline for text-to-3D asset generation. offers 3D asset creation with high prompt fidelity and high-quality 3D shapes and textures in under a minute. It supports physically-based rendering (PBR), necessary for 3D asset relighting in real-world applications. Additionally, supports generative retexturing of previously generated (or artist-created) 3D shapes using additional textual inputs provided by the user. integrates key technical components, and , that we developed for text-to-3D and text-to-texture generation, respectively. By combining their strengths, represents 3D objects simultaneously in three ways: in view space, in volumetric space, and in UV (or texture) space. The integration of these two techniques achieves a win rate of 68% with respect to the single-stage model. We compare to numerous industry baselines, and show that it outperforms them in terms of prompt fidelity and visual quality for complex textual prompts, while being significantly faster.

Figure 1: integrates Meta’s foundation models for text-to-3D ( (Siddiqui et al., 2024)) and text-to-texture ( (Bensadoun et al., 2024)) generation in a unified pipeline, enabling efficient, state-of-the-art creation and editing of diverse, high-quality textured 3D assets with PBR material maps.

Figure 2: Overview of . The pipeline takes a text prompt as an input and performs text-to-3D generation (Stage I, Siddiqui et al. (2024)), followed by texture refinement (Stage II, Bensadoun et al. (2024)). Stage II can also be used for retexturing of generated or artist-created meshes using new textual prompts provided by the user.

1 Introduction

We introduce (), a new state-of-the-art solution for efficient text-to-3D generation. Text-to-3D is the problem of generating 3D content, such as characters, props and scenes, from textual descriptions. Authoring 3D content is one of the most time-consuming and challenging aspects of designing and developing video games, augmented and virtual reality applications, as well as special effects in the movie industry. By providing AI assistants which can double as a 3D artist, we can enable new experiences centred on creating personalized, user-generated 3D content. Generative 3D assistants can also support many other applications, such as virtual product placement in user-generated videos. AI-powered 3D generation is also important for building infinitely large virtual worlds in the Metaverse.

3D generation has unique and difficult challenges not shared by other forms of media generation such as images and videos. Production-ready 3D content has exacting standards in terms of artistic quality, speed of generation, structural and topological quality of the 3D mesh, structure of the UV maps, and texture sharpness and resolution. Compared to other media, a unique challenge is that, while there exist billions of images and videos to learn from, the amount of 3D content viable for training is three to four order of magnitude smaller. Thus, 3D generation must also learn from images and videos which are not 3D and where 3D information must be inferred from partial, 2D observations.

achieves high quality generation of 3D assets in under a minute. It supports Physically-Based Rendering (PBR) (Torrance and Sparrow, 1967), necessary for enabling relighting of generated assets in applications. When assessed by professional 3D artists, significantly improves key metrics for production-quality 3D assets, particularly for complex textual prompts. The faithfulness to the textual prompts is better than other text-to-3D approaches, commercial or not, outperforming techniques that take from three minutes to an hour for generation. The quality of the generated 3D shapes and textures is better or at least on par with these competitors, using a scalable system that is significantly faster and more faithful.

Once the object is generated, its texture can be further edited and customised in 20 sec, with higher quality and at a fraction of the cost compared to alternatives. The same approach can be applied to texturing of artist-created 3D meshes without modifications.

The rest of this technical report describes the pipeline as a whole, discussing how and are integrated, and conducts extensive evaluation studies against the most prominent industry baselines for text-to-3D generation.

Key capabilities.

is a two-stage method that combines two components, one for text-to-3D generation and one for text-to-texture generation, respectively. This integration results in higher-quality 3D generation for immersive content creation. In particular:

• Stage I: 3D asset generation. Given a text prompt provided by the user, Stage I creates an initial 3D asset using our  (Siddiqui et al., 2024) model (AssetGen for short). This step produces a 3D mesh with texture and PBR material maps. The inference time is approximately 30 sec.

• Stage II, use case I: generative 3D texture refinement. Given a 3D asset generated in Stage I and the initial text prompt used for generation, Stage II produces a higher-quality texture and PBR maps for this asset and the prompt. It utilizes our text-to-texture generator  (Bensadoun et al., 2024) (TextureGen for short). The inference time is approximately 20 sec.

• Stage II, use case 2: generative 3D (re)texturing. Given an untextured 3D mesh and a prompt describing its desired appearance, Stage II can also be used to generate a texture for this 3D asset from scratch (the mesh can be previously generated or artist-created). The inference time is approximately 20 sec.

Technical approach.

By building on AssetGen and TextureGen, effectively combines three highly-complementary representations of the 3D object: the view spaces (images of the object), the volumetric space (3D shape and appearance), and the UV space (texture). This process begins in AssetGen by generating several fairly consistent views of the object by utilizing a multi-view and multi-channel version of a text-to-image generator. Then, a reconstruction network in AssetGen extracts a first version of the 3D object in volumetric space. This is followed by mesh extraction, establishing the object’s 3D shape and an initial version of its texture. Finally, a TextureGen’s component regenerates the texture, utilizing a combination of view-space and UV-space generation, boosting the texture quality and resolution while retaining fidelity to the initial prompt.

Each stage of builds on Meta’s series of powerful text-to-image models Emu (Dai et al., 2023b). These are fine-tuned using renders of synthetic 3D data (from an internal dataset) to perform multi-view generation in view space as well as in UV space, resulting in better textures.

Performance.

Integration of the two stages (AssetGen and TextureGen) and their different representations results in a combined model winning 68% of the times in evaluations. In addition to the strength that comes from this new combination, the individual components outperform the state of the art in their receptive functionalities. Specifically, AssetGen advances text-to-3D in several aspects: it supports physically-based rendering, which allows to relight the generated object, it obtains better 3D shapes via an improved representation (based on signed distance fields), and develops a new neural network that can effectively combine and fuse view-based information in a single texture. Likewise, TextureGen outperforms prior texture generator approaches by developing an end-to-end network that also operates in mixed view and UV spaces. Remarkably and differently to many state-of-the-art solutions, both AssetGen and TextureGen are feed-forward generators, and thus fast and efficient after deployment.

2 Method

We start by giving a high-level view of the two components of , namely AssetGen (Stage I) and TextureGen (Stage II), and we refer the reader to the original papers for more details. We start from Stage II as it simplifies setting out the notation.

TextureGen (Bensadoun et al., 2024): core of Stage II.

TextureGen is a text-to-texture generator for a given 3D shape. Namely, given a 3D object $M$ and a textual prompt $y$, it generates a texture $T$ for the object that is consistent with the prompt $y$. The object $M=(V,F,U)$ consists of a 3D mesh $(V,F)$, where $V\in\mathbb{R}^{|V|\times 3}$ is a list of vertices and $F\in\{1,\dots,|V|\}^{|F|\times 3}$ is a list of triangular faces. The object comes with a map assigning each vertex $v_{i}\in V$ to a corresponding UV coordinate $u_{i}\in U\in[0,1]^{|V|\times 2}$. The texture $T$ is a 2D image of size $L\times L$ supported on $[0,1]^{2}$. The texture has either three or five channels, in the first case representing the RGB shaded appearance of the object (with baked light) and in the second case the RGB albedo (base color), roughness and metalness, respectively.

TextureGen comprises several stages. In the first stage, a network $\Phi^{\text{tex}}_{\text{mv}}$ is trained to generate, from the prompt $y$ and the object $M$, several views $I_{1},\dots,I_{K}$ of the object $M$. The generator is joint, in the sense that it samples the distribution $p(I_{1},\dots,I_{K}|y,M)$. In the second stage, the views $I_{1},\dots,I_{K}$ are first re-projected on corresponding texture images $T_{1},\dots,T_{K}$. Then, a second generator network $\Phi^{\text{tex}}_{\text{uv}}$ takes these and the prompt $y$ to output a final texture $T$ sampled from the conditional distribution $p(T|y,T_{1},\dots,T_{K}).$ This step reconciles the view-based textures, which may be slightly inconsistent, and completes the parts of the texture that are not visible in any of the views. Finally, a third optional network $\Phi^{\text{tex}}_{\text{super}}$ takes the texture $T$ and performs super-resolution (up to 4K). Networks $\Phi^{\text{tex}}_{\text{mv}}$, $\Phi^{\text{tex}}_{\text{uv}}$ and $\Phi^{\text{tex}}_{\text{super}}$ are diffusion-based generators, trained on a large collection of 3D assets starting from a pre-trained image generator in Emu family (Dai et al., 2023b).

AssetGen (Siddiqui et al., 2024): core of Stage I.

AssetGen is a text-to-3D object generator: given a textual prompt $y$, it samples both a 3D mesh $M$ and a corresponding texture $T$ from a distribution $p(M,T|y).$ AssetGen also operates stage-wise. First, a network $\Phi^{\text{obj}}_{\text{mv}}$ takes the prompt $y$ and generates a set of views $I_{1},\dots,I_{K}$ of the object. This is similar to TextureGen’s first stage $\Phi^{\text{tex}}_{\text{mv}}$, except that the views are not conditioned on the geometry of the object $M$, which is instead a target for generation. Then, given the views $I_{1},\dots,I_{K}$, a second network $\Phi^{\text{obj}}_{\text{rec}}$ generates a 3D mesh $M$ and initial texture $T$ using a large reconstruction neural network. Differently from network $\Phi^{\text{obj}}_{\text{mv}}$, which models a distribution via diffusion and is thus aleatoric, the network $\Phi^{\text{obj}}_{\text{rec}}$ is deterministic. Images $I_{1},\dots,I_{K}$ contain sufficient information for the model to reconstruct the 3D object without too much ambiguity. For PBR material reconstruction, this is achieved by tasking the image generator to output the shaded appearance of the object as well as its albedo (intrinsic image), which makes it easier to infer materials. Finally, AssetGen refines the texture $T$, by first obtaining auxiliary partial but sharp texture by re-projecting the input views $I_{1},\dots,I_{K}$ into textures $T_{1},\dots,T_{K}$. Then, a network $\Phi^{\text{obj}}_{\text{uv}}$ maps $T,T_{1},\dots,T_{K}$ (defined in UV space) to a fused and enhanced texture $T^{*}$.

: integrated approach.

Finally, we describe the combination of these two methods into a high-quality text-to-3D generator with retexturing capabilities. The idea is to utilize the texture generator in Stage II to significantly improve the quality of the texture obtained from the first-stage 3D object generator. The 3D object generator AssetGen does produce good quality textures, but has two limitations. First, it is not a model specialized for high-quality texture generation, but TextureGen is. Secondly, the texture generator TextureGen is conditioned on an existing 3D shape of the object, which makes it much easier to generate high-quality and highly-consistent multiple-views of the textured object. In other words, network $\Phi^{\text{tex}}_{\text{mv}}$ solves an easier task than network $\Phi^{\text{obj}}_{\text{mv}}$ (due to the additional geometric conditioning) and can thus generate better views, resulting in better high-resolution textures.

In principle, then, we could simply use network $\Phi^{\text{obj}}_{\text{mv}}$ from AssetGen to generate the 3D shape of the object and then network $\Phi^{\text{tex}}_{\text{mv}}$ and $\Phi^{\text{tex}}_{\text{uv}}$ to re-generate a better texture, with semantic consistency guaranteed by utilizing the same prompt $y$ for conditioning the two steps. However, this approach does not work well by itself. The reason is that the texture fusion and enhancement network in TextureGen is trained on the basis of ‘ground truth’ UV maps by 3D artists; in contrast, the assets generated by AssetGen have automatically-extracted UV maps, that differ substantially from artist-created ones.

Fortunately, AssetGen comes with its own texture re-projection and fusion network $\Phi^{\text{obj}}_{\text{uv}}$ which is trained on the basis of automatically-extracted UV maps and can do a better job than network $\Phi^{\text{tex}}_{\text{uv}}$ on this task. Hence, our integrated solution is as follows:

• Given the prompt $y$, run networks $\Phi^{\text{obj}}_{\text{mv}}$ and $\Phi^{\text{obj}}_{\text{rec}}$ and mesh and UV extraction to obtain an initial mesh $M$ and UV map $U$.

• Given the prompt $y$ and the initial mesh $M$, run network $\Phi^{\text{tex}}_{\text{mv}}$ to generate a set of views $I_{1},\dots,I_{K}$ representing a new, better texture in view space. Using the UV map $U$, reproject these images into partial textures $T_{1},\dots,T_{K}$.

• Given the prompt $y$ and the partial textures $T_{1},\dots,T_{K}$, run the network $\Phi^{\text{tex}}_{\text{uv}}$ from TextureGen to obtain a consolidated UV texture $T$.

• Given the partial textures $T_{1},\dots,T_{K}$ and the consolidated texture $T$, run network $\Phi^{\text{obj}}_{\text{uv}}$ from AssetGen to obtain the final texture $T^{*}$. This fixes any residual seams due to the non-human-like UV maps.

3 Experiments

Method	Generation capabilities				Generation time
	Mesh	Texture	PBR materials	Clean topology	Stage I only	Stages I+II
CSM Cube 2.0 (CSM, 2024)	✓	✓	✗	✗	$15^{*}$ min	$1^{*}$ h
Tripo3D (TripoAI, 2024)	✓	✓	✗	✗	$30^{*}$ sec	$3^{*}$ min
Rodin Gen-1 V0.5 (Deemos, 2024)	✓	✓	✓	✓	–	$3^{*,\dagger}$ min
Meshy v3 (Meshy, 2024a)	✓	✓	✓	✗	$1^{*}$ min	$10^{*}$ min
Third-party T23D generator	✓	✓	✓	✗	$10^{*}$ sec	$10^{*}$ min
	✓	✓	✓	✗	30 sec	1 min
10pt$\vphantom{t}{}^{*}$ Averaged approximate estimates, as evaluated from corresponding public APIs.
$\vphantom{t}{}^{\dagger}$ Depends on the complexity of geometry, can range from 2 to 30 min (in 7 % cases failed to converge).

Table 1: Overview of the industry baselines for the text-to-3D task. Comparison of generation capabilities and run times.

We compare against publicly-accessible industry solutions for the task of text-to-3D asset generation. We report extensive user studies to evaluate both the quality (for the baselines that are producing both textures and materials) and text prompt fidelity aspects of 3D generation, and provide qualitative results for both 3D generation and texturing.

3.1 Industry baselines

We compare performance of with leading industry models for text-to-3D generation, which are currently accessible via web demos and public APIs. The summary of their capabilities, that are relevant to text-to-3D generation, and run times is provided in Table 1.

Common Sense Machines (CSM) Cube 2.0 (CSM, 2024). All results for comparisons were generated using the officially provided Cube API, with separate sequential calls for text-to-image and then image-to-3D generation (with the highest quality settings). Website: www.csm.ai.

Tripo3D (TripoAI, 2024). All results are generated using the official Tripo Platform, including both preview and refinement stages. Website: https://www.tripo3d.ai/app.

Rodin Gen-1 (0525) V0.5 (Deemos, 2024). The generations were obtained manually using the official web interface. The pipeline requires running several stages: text-to-image, image-to-shape, texture generation and material generation. To encourage prompt fidelity, we performed generations with the original text prompt at every stage. We also disabled the symmetry flag, as we found it to be hurtful for generating complex compositions. The rest of the settings were set to default. The method failed on 7 % of prompts (27 out of 404) during the meshing stage, likely due to the originally generated geometries being too complex. Website: hyperhuman.deemos.com/rodin.

Meshy v3 (Meshy, 2024a). The results were generated by the official API and with PBR materials, using the corresponding style setting. The rest of the settings were set by default. Website: www.meshy.ai.

Third-party text-to-3D (T23D) generator. We are providing additional quantitative comparisons with another industry-leading text-to-3D generator. The results were generated using the official web interface, including three stages: text-to-image, asset preview and asset refinement. Out of four image options proposed by the interface after the first stage, we always pick the top left one for consistency.

3.2 User studies

Method	All prompts, per stage ($\uparrow$)		Stage II, per prompt category($\uparrow$)
	stage I	stage II	(A) objects	(B) characters	(A)+(B) compositions
CSM Cube 2.0 (CSM, 2024)	–	69.1 %	84.0 %	87.8 %	54.6 %
Tripo3D (TripoAI, 2024)	–	78.2 %	77.6 %	87.9 %	71.6 %
Rodin Gen-1 (0525) V0.5 (Deemos, 2024)	–	59.9 %	66.7 %	70.1 %	48.8 %
Meshy v3 (Meshy, 2024a)	60.6 %	76.0 %	97.2 %	83.2 %	63.5 %
Third-party T23D generator	73.5 %	79.7 %	95.0 %	89.7 %	67.9 %
	79.7 %	81.7 %	96.5 %	84.1 %	73.9 %

Table 2: User studies: prompt fidelity. Stage I corresponds to the first-round text-to-3D generations, and stage II to the results of the final refinement. For simplicity, we consider Rodin Gen-1 to be a single-stage method.

Method	Q0: fidelity		Q1: quality		Q2: texture		Q3: geometry
	Win ()	Loss ()	Win ()	Loss ()	Win ()	Loss ()	Win ()	Loss ()
All annotators
Rodin Gen-1 (0525) V0.5 (Deemos, 2024)	67.6 %	32.4 %	66.2 %	33.8 %	70.9 %	29.1 %	60.3 %	39.7 %
Meshy v3 (Meshy, 2024a)	61.5 %	38.5 %	60.1 %	39.9 %	49.7 %	50.3 %	65.7 %	34.3 %
Third-party T23D generator	57.2 %	42.8 %	60.4 %	39.6 %	58.6 %	41.4 %	60.0 %	40.0 %
Professional 3D artists
Rodin Gen-1 (0525) V0.5 (Deemos, 2024)	68.0 %	32.0 %	59.8 %	40.2 %	69.1 %	30.9 %	56.7 %	43.3 %
Meshy v3 (Meshy, 2024a)	60.0 %	40.0 %	65.3 %	34.7 %	53.7 %	46.3 %	66.3 %	33.7 %
Third-party T23D generator	59.1 %	40.9 %	61.3 %	38.78 %	60.2 %	39.8 %	60.2 %	39.8 %

Table 3: User studies: summary of A/B tests (for models producing textures and materials). The annotators were asked four questions: Q0 – “which 3D asset is the better representation of the prompt?”, Q1 – “which 3D asset has better quality overall?”, Q2 – “which has better texture?”, Q4 – “which has more correct geometry?”. Win and loss are measured for our method (), with respect to each of the strongest baseline methods (stage II, where applicable).

Figure 3: User studies: analysis of prompt fidelity, visual quality, geometry and texture parameters as functions of the scene complexity, as described by the text prompt (aggregated across all annotators). We report win rate for against baselines and highlight the 50% threshold (dashed line) where our method is found to be as good as the baselines.

We conduct a series of user studies on prompt fidelity and visual quality of text-to-3D generations, produced by each of the models. Our pool of annotators consists of two groups: (1) representatives of a general population with no prior expertise in 3D, and (2) professional 3D artists, designers and game developers. We report aggregated results, as well as results obtained by the group with the strongest relevant expertise.

Evaluation benchmark.

For evaluations, we use a set of deduplicated 404 text prompts that were initially introduced with DreamFusion (Poole et al., 2023). For our analysis, we split this set into a number of categories, according to the described content complexity: objects (156), characters (106) and compositions of characters and objects (141). We report each model’s performance on each of the categories separately, as well as the aggregated scores. In all studies, the annotators were shown fly-around $360^{\circ}$ videos of rendered meshes. Text prompt fidelity, overall visual quality, as well as quality of geometries and textures are evaluated for every model either separately, or in randomized A/B tests.

Evaluation results.

User studies results for text prompt fidelity are shown in Table 2. These were obtained independently for each model, by asking the annotators to decide whether or not the prompt correctly describes the generated content. outperforms all considered industry baselines on this metric (in both stages), with the third-party text-to-3D (T23D) generators being the strongest competitor overall.

The A/B test user studies were designed to evaluate text prompt fidelity, overall visual quality, geometry visual quality, and texture details and artefacts for our model compared with baselines producing both textures and PBR materials. We do not perform exhaustive evaluations of our method versus models generating baked textures, due to significant perceptual differences between generations produces by the two classes of models at rendering time and due to practically limited usability of texture-only generations in real-world applications. The results are summarized in Table 3. We first report aggregated scores across all annotators, and then separately from the subset with a strong expertise in 3D. Overall, performs stronger than the competitors according to most metrics, while also being significantly faster.

We observed that annotators with less experience in 3D tend to favour assets with sharper, more vivid, realistic, detailed textures and are not sensitive to presence of even significant texture and geometry artefacts. The professional 3D artists expressed a stronger preference for generations across the whole range of metrics. We observed that their evaluations gave more weight to correctness of geometries and textures.

In Figure 3, we analyze performance rates for visual quality, geometry, texture details and presence of texture artefacts, as functions of the scene complexity as described by the text prompt. The plots show that, while some of the baselines perform on par for simple prompts, starts outperforming them strongly as the prompt complexity increases from objects to characters and their compositions.

3.3 Qualitative results

Figure 4: Visual comparison of text-to-3D generations obtained after ’s Stage I (left) and Stage II (right). In our A/B user studies, the Stage II generations had a win rate of 68 % in texture quality over the first-stage generations.

Figure 5: Qualitative results for text-to-3D generation. We show quality and diversity of text-to-3D generations produced by , across different scene categories (single objects and compositions).

Figure 6: Qualitative comparison of text prompt fidelity with all industry baselines (on challenging prompts).

Figure 7: Qualitative comparison with industry baselines producing textures with PBR materials (on the same set of prompts).

Figure 8: Examples of typical failure modes of different methods.

Figure 9: (Re)texturing results for generated shapes. Examples of meshes produced by Stage I of the pipeline and textured in Stage II with various text prompts, different from the original ones.

Figure 10: (Re)texturing results for generated shapes. Examples of themed scenes produced by Stage II of by augmenting object-specific texturing prompts with the new style information in a coherent manner.

Text-to-3D.

Visual comparisons of Stage I and Stage II generations are given in Fig. 4. The latter tend to have higher visual aesthetics, appear more realistic and have higher-frequency details. Our annotators prefer generations from Stage II in 68 % of the cases. More qualitative examples of text-to-3D generations produced by Stage II are shown in Figure 5 (diverse classes) and Figure 11 (within one object class).

Next, we visually compare performance of our model with industry baselines on the same scenes (Figure 7), additionally on more challenging prompts (Figure 6) and in terms of most common failure cases of both our method and the baselines (Figure 8). Overall, these qualitative observations confirm that, while the alternative methods do well on simple objects, generation of more complex compositions and scenes becomes a bigger challenge for them. There is also a clear trade-off between generating high-frequency details in textures vs exposing visual artefacts. Meshy v3 Meshy (2024a), in particular, has a visually appealing style with highly detailed generations (which are often appreciated in user studies, in particular among non-professionals), but often suffers from Janus effects, inpainting problems and seams in texture maps. Geometry-wise, Rodin Gen1 Deemos (2024) produces quad meshes with correct topologies, but at cost of compromising prompt fidelity and sometimes failing to produce results for complex prompts altogether.

3D asset (re)texturing.

Figure 9 shows qualitative results for the task of asset retexturing: 3D meshes, generated in Stage I, are then passed to Stage II with textual prompts that are different than the original ones. This process allows us to create new assets with the same base shapes, but different appearances. The results show that in addition to implementing semantic edits and performing both global and localized modifications, can successfully imitate different materials and artistic styles. Figure 10 shows how one can retexture whole scenes in a coherent manner, by augmenting object-level prompts used for retexturing with the style information. As discussed in Bensadoun et al. (2024), Stage II can be applied for retexturing of both generated and artist-created 3D assets with no significant changes to the pipeline.

4 Related Work

There is ample literature in both text-to-3D and text-to-texture. We point the readers to (Siddiqui et al., 2024; Bensadoun et al., 2024) for a more extensive discussion and provide here key pointers.

Text-to-3D.

Some methods (Nichol et al., 2022; Jun and Nichol, 2023; Gupta et al., 2023; Yariv et al., 2023; Xu et al., 2024c) train 3D generators on 3D datasets, but the limited availability of such data is an obstacle to generalization. Hence, most recent approaches start from image or video-based generators trained on billions of data samples (Shue et al., 2022; Mercier et al., 2024).

Many approaches (Lin et al., 2022; Qian et al., 2023; Lin et al., 2022; Tang et al., 2023a; Yi et al., 2023; Chen et al., 2023e; Wang et al., 2023a, c; Zhu and Zhuang, 2023; Huang et al., 2023; Qian et al., 2023; Tang et al., 2023b; Yu et al., 2023a; Sun et al., 2023) are based on distillation (Poole et al., 2023). However, distillation is slow (Lorraine et al., 2023; Xie et al., 2024) and may lead to artifacts such as the Janus effect (Shi et al., 2024). Follow-up works have thus built on multi-view-aware image generators (Liu et al., 2023c; Shi et al., 2023; Jiang et al., 2023a; Chen et al., 2023d; Qian et al., 2023; Shi et al., 2024; Weng et al., 2023; Wang and Shi, 2024; Kim et al., 2024; Zhou et al., 2024).

More recent approaches focus on generating several consistent views (Liu et al., 2023b; Long et al., 2023; Liu et al., 2023d; Yang et al., 2023b, a; Chan et al., 2023; Tang et al., 2024c; Höllein et al., 2024; Gao et al., 2024; Melas-Kyriazi et al., 2024; Chen et al., 2024) from which direct 3D reconstruction is possible. However, these methods are susceptible to limitations in the multi-view consistency of the generated mages. Other approaches thus learn few-view robust reconstructors (Li et al., 2024; Hong et al., 2024; Liu et al., 2023a).

Multi-view to 3D.

Many generators thus build on few-view 3D reconstruction. Methods like NeRF (Mildenhall et al., 2020) cast this as analysis by synthesis, optimizing a differentiable rendering loss. These approaches can use a variety of 3D representations, from meshes to 3D gaussians (Gao et al., 2020; Zhang et al., 2021a; Goel et al., 2022; Munkberg et al., 2022; Monnier et al., 2023; Kerbl et al., 2023; Guédon and Lepetit, 2023; Niemeyer et al., 2020; Mildenhall et al., 2020; Müller et al., 2022; Yariv et al., 2020; Oechsle et al., 2021; Yariv et al., 2021; Wang et al., 2021; Darmon et al., 2022; Fu et al., 2022).

When only a small number of views are available, authors train reconstruction models to acquire the necessary priors (Choy et al., 2016; Kanazawa et al., 2018; Mescheder et al., 2019; Liu et al., 2019; Wu et al., 2020; Monnier et al., 2022; Wang et al., 2023b; Hong et al., 2024; Vaswani et al., 2017; Chan et al., 2022; Chen et al., 2022; Xu et al., 2024a; Wei et al., 2024; Zou et al., 2023; Xu et al., 2024b; Tang et al., 2024a; Zhang et al., 2024; Wang et al., 2024; Wei et al., 2024; Tochilkin et al., 2024; Junlin Han, 2024).

PBR modelling.

Several authors have considered reconstruction methods with PBR support too (Boss et al., 2021b, a; Xiuming et al., 2021; Zhang et al., 2021b; Munkberg et al., 2022; Hasselgren et al., 2022; Jiang et al., 2023b; Liang et al., 2023). This is also the case for 3D generators (Chen et al., 2023c; Qiu et al., 2023; Liu et al., 2023f; Xu et al., 2023; Poole et al., 2023).

Texture generation.

Several works have tackled specifically the task of generating textures for 3D objects as well. For instance Mohammad Khalid et al. (2022); Michel et al. (2022) use guidance from CLIP (Radford et al., 2021) and differentiable rendering to match the texture to the textual prompt. Chen et al. (2023b); Metzer et al. (2023); Youwang et al. (2023) use SDS loss optimization Poole et al. (2022) and Siddiqui et al. (2022); Bokhovkin et al. (2023) use a GAN-like approach analogous to (Karras et al., 2019). Other methods use diffusion in UV space (Liu et al., 2024; Cheskidova et al., 2023), but focus on human character texturing. Yu et al. (2023b) uses point-cloud diffusion to generate a texture.

Richardson et al. (2023); Chen et al. (2023a); Tang et al. (2024b); Zeng (2023) combine texture inpainting with depth-conditioned image diffusion, but generate one image at a time, which is slow and prone to some artifacts. Liu et al. (2023e); Cao et al. (2023) improves consistency by alternating diffusion iterations and re-projections to combine them. Deng et al. (2024) generate four textured views jointly, but uses slow SDS optimization to extract the texture. Meshy (Meshy, 2024b) also provide a texture generator module, but its details remain proprietary.

Image generators.

Our generators are based on image generators, which have been studied extensively starting from GANs (Goodfellow et al., 2014). Recent works use transformer architectures (Ramesh et al., 2021; Ding et al., 2021; Gafni et al., 2022; Yu et al., 2022; Chang et al., 2023). Several more operate in pixel space or latent space using diffusion (Ho et al., 2020; Balaji et al., 2022; Saharia et al., 2022; Ramesh et al., 2022; Rombach et al., 2022; Podell et al., 2023). We build on the Emu class of image generators (Dai et al., 2023a).

5 Conclusions

We have introduced , a unified pipeline integrating Meta’s foundation generative models for text-to-3D generation with texture editing and material generation capabilities, AssetGen and TextureGen, respectively. By combining their strengths, achieves very high-quality 3D object synthesis from textual prompts in less than a minute. When assessed by professional 3D artists, the output of is preferred a majority of time compared to industry alternatives, particularly for complex prompts, while being from 3$\times$ to 60$\times$ faster.

While our current integration of AssetGen and TextureGen straightforward, it sets out a very promising research research direction that builds on two thrusts: (1) generation in view space and UV space, and (2) end-to-end iteration over texture and shape generation.

Figure 11: Quality and diversity of generations produced by , for a single object class (“llama”).

6 Acknowledgements

We are grateful for the instrumental support of the multiple collaborators at Meta who helped us in this work: Ali Thabet, Albert Pumarola, Markos Georgopoulos, Jonas Kohler, Uriel Singer, Lior Yariv, Amit Zohar, Yaron Lipman, Itai Gat, Ishan Misra, Mannat Singh, Zijian He, Jialiang Wang, Roshan Sumbaly.

We thank Manohar Paluri and Ahmad Al-Dahle for their support of this project.