Introducing Coder: Text to Code for 3D world generation, gaming and programming.
Creating 3D worlds for gaming, artificial intelligence training, or content is a complex task that requires modeling intricate interactions between various objects, agents, goals, behaviors, and spaces. Typically, large teams of creators and developers work together for weeks or even months to produce the final results.
While our 3D generation capabilities are aimed to democratize and expedite the creation of textured 3D models and radiance fields, Coder is meant to develop entire environments and complex programs. Technical artists, game developers and python programmers typically have to write thousands of lines of code to synthesize scenes and behaviors.
Coder is powered by large language models and program synthesis techniques to translate natural language into interactive 3D worlds and complex python programs, with the ability to search, edit and regenerate code via natural language. Our knowledge graphs and training data are continually growing. Moreover, we make use of search engines, program-synthesis techniques and retrieval-guided code generation to reduce hallucinations produced by large-language models.
Signup for the waitlist and then access it here.
Models
The user types in a series of text prompts in a session window, along with two choices for each prompt: a) context and b) model type. If the context button is turned on then the previous prompt and its response is prepended to the current query. Currently our engine supports several model types as described below. Users can seamlessly combine different model types to solve scenarios. If the context button is turned on then it allows the user to also translate code between languages (e.g. the previous query might be about a python algorithm but the next one about translating this python code to lua for a Roblox task; this allows creative mixing and matching between domains).
blender-python
Use this for programming 3D worlds in blender
python-general
Use this for search based retrieval guided code generation for python or blender-python.
github-python
Use this for retrieval guided python synthesis and regeneration.
roblox
Use this for Roblox game programming.
geometry-nodes-scripting
Use this for text-to-geometry-nodes scripting in blender.
3dobj-library
Use this for text to 3D asset semantic search.
tools-docs
Use this for Blender/Omniverse semantic guide.
Text to 3D Environment Workflow in Blender
Text to 3D Semantic Search
You can use prompt engineering to get 3D models from the internet with varying degrees of properties. The text captions of many 3D models on the web are noisy and sparse. So in addition to existing metadata, we also applied object detection and segmentation algorithms to enhance the text caption. You need to use the 3dobj-library model type for this use case.
Generating Complex Python Programs
Coder can be used to generate complex python programs. Large language models trained to predict code can hallucinate incorrect programs. In order to mitigate this issue, we augment the engine with two types of guidance — retrieval based and search based.
In retrieval based guidance, we crawled code with permissive license from the web (MIT, Apache, GPL etc), optionally compress functions by dynamically creating new functions on the fly and embedding code in vectorized formats. This creates a large long-term memory consisting of code snippets in the most compressed form. Now given a new natural language query, we search through this memory and augment the outputs to the code generation modules for better predictions.
In search-based guidance, we hooked up a live search engine to our backend to get better information retrieval. Use python-general to access this capability.
For instance, here is an example of writing a pytorch nn.Module for 3d ray marching. First the user selected github-python with the following prompt: “A python function to ray march for rendering an image”, followed by another prompt (with context on) “Translate this function into a pytorch nn.Module”.
Creating Roblox Games
Creating games requires complex interactions between 3D assets and goal-directed behaviors. We work with a growing number of game developers who are continuously training Coder to help them write better Roblox games. This model checkpoint (roblox) is a preview release and we will be continuously updating it as we collect more data. While there are other AI code assistance tools like github co-pilot and chatGPT , we are aiming to solve more complex queries, compose knowledge from other model types and provide a single comprehensive user experience to create 3D worlds across domains. Please checkout this blog post for a workflow to create 3D assets and code for Roblox. In the future we aim to support other platforms as well. Please reach out if you would like to partner with us for your platform or needs.