Creating 3D worlds for gaming, artificial intelligence training, or content is a complex task that requires modeling intricate interactions between various objects, agents, goals, behaviors, and spaces. Typically, large teams of creators and developers work together for weeks or even months to produce the final results.

While our 3D generation capabilities are aimed to democratize and expedite the creation of textured 3D models and radiance fields, Coder is meant to develop entire environments and complex programs. Technical artists, game developers and python programmers typically have to write thousands of lines of code to synthesize scenes and behaviors.

Coder is powered by large language models and program synthesis techniques to translate natural language into interactive 3D worlds and complex python programs, with the ability to search, edit and regenerate code via natural language. Our knowledge graphs and training data are continually growing. Moreover, we make use of search engines, program-synthesis techniques and retrieval-guided code generation to reduce hallucinations produced by large-language models. 

Signup for the waitlist and then access it here.

The user types in a series of text prompts in a session window, along with two choices for each prompt: a) context and b) model type. If the context button is turned on then the previous prompt and its response is prepended to the current query. Currently our engine supports several model types as described below. Users can seamlessly combine different model types to solve scenarios. If the context button is turned on then it allows the user to also translate code between languages (e.g. the previous query might be about a python algorithm but the next one about translating this python code to lua for a Roblox task; this allows creative mixing and matching between domains). 

Use this for programming 3D worlds in blender

Use this for search based retrieval guided code generation for python or blender-python.

Use this for retrieval guided python synthesis and regeneration.

Use this for text-to-geometry-nodes scripting in blender.

Use this for text to 3D asset semantic search.

Use this for Blender/Omniverse semantic guide.

Text to 3D Environment Workflow in Blender

You can use prompt engineering to get 3D models from the internet with varying degrees of properties. The text captions of many 3D models on the web are noisy and sparse. So in addition to existing metadata, we also applied object detection and segmentation algorithms to enhance the text caption. You need to use the 3dobj-library model type for this use case.

Coder can be used to generate complex python programs. Large language models trained to predict code can hallucinate incorrect programs. In order to mitigate this issue, we augment the engine with two types of guidance — retrieval based and search based. 

In retrieval based guidance, we crawled code with permissive license from the web (MIT, Apache, GPL etc), optionally compress functions by dynamically creating new functions on the fly and embedding code in vectorized formats. This creates a large long-term memory consisting of code snippets in the most compressed form. Now given a new natural language query, we search through this memory and augment the outputs to the code generation modules for better predictions. 

In search-based guidance, we hooked up a live search engine to our backend to get better information retrieval. Use python-general to access this capability.

For instance, here is an example of writing a pytorch nn.Module for 3d ray marching. First the user selected github-python with the following prompt: “A python function to ray march for rendering an image”, followed by another prompt (with context on) “Translate this function into a pytorch nn.Module”. 

Creating games requires complex interactions between 3D assets and goal-directed behaviors. We work with a growing number of game developers who are continuously training Coder to help them write better Roblox games. This model checkpoint (roblox) is a preview release and we will be continuously updating it as we collect more data. While there are other AI code assistance tools like github co-pilot and chatGPT , we are aiming to solve more complex queries, compose knowledge from other model types and provide a single comprehensive user experience to create 3D worlds across domains. Please checkout this blog post for a workflow to create 3D assets and code for Roblox. In the future we aim to support other platforms as well. Please reach out if you would like to partner with us for your platform or needs.

Introducing Coder: Text to Code for 3D world generation, gaming and programming.

Creating a game requires multiple experts working together for months on end to create realistic 3D worlds with rich goal-directed behaviors. This is a complicated procedure and requires years of experience. Several modern gaming platforms like Roblox focus on user-genererated content and games. However, most of the users on these platforms do not have the skillsets required to build complex and engaging worlds. Our technology democratizes the two critical elements in game design and building — 3D asset creation and coding. 

In this blog post we will do a walkthrough of how to create 3D assets from images, make them Roblox ready and then use CSM’s Coder to write code to make a simple game. You can access and play the final game generated here. 

In the first step, a user uploads videos to our 3D generation platform and gets back 3D assets and radiance fields. Please carefully read this blog and the instructions (capture and  create 3D) on the web app / iOS app before recording and uploading videos. If you don’t follow these instructions carefully, it will most likely not work.

These assets are typically high dimensional and have to be decimated in order to fit the Roblox requirement. The following video shows a workflow to make the assets Roblox-ready.

After converting and importing all 3D assets in Roblox studio, the user can use our AI code engine to help write games. You can access and play the final game generated here. The objects populating this Roblox game were generated from image inputs and can be accessed here.

A longstanding dream in generative AI has been to digitally synthesize dynamic 3D environments – interactive simulations consisting of objects, spaces and agents. Simulations enable us to create synthetic worlds and data for robotics and perception, new content for games and metaverses, and adaptive digital twins of everyday and industrial objects and spaces. Simulators are also the key to imaginative machines endowed with a world model; they allow them to visualize, predict and consider possibilities before taking actions. But today’s simulators are too hard to build and too limited for our needs. Generative AI will change this.

Generative AI requires compressing diverse human experience and culture into a large scale model. Indeed, recent large generative models (such as Dall-E 2, StableDiffusion, Imagen, Make-A-Scene and many others) have shown that this dream may be possible today. A natural evolution in this progression is for AI systems to create a new class of simulator that learns to generate 3D models, long-horizon videos, and efficient actions. These feats are now becoming attainable due to recent advances in neural rendering, diffusion models, and attention architectures.

Common Sense Machines has built a neural simulation engine we call CommonSim-1. Instead of laboriously creating content and spending weeks of effort navigating complex tools, users and robots interact with CommonSim-1 via natural images, actions and text. These interfaces will enable us to rapidly create digital replicas of real world scenarios or make imagining entirely new ones as natural as taking a photo or describing a scene in text. Today, we are previewing these model capabilities. We are excited to see what developers, creators, researchers, and more are able to create! Sign up below for early access to our application, APIs, checkpoints, and code.

CommonSim-1: A simulator that grows and adapts from experience

A new simulation engine requires new interfaces. Instead of needing a specialized toolchain, CommonSim-1 is operated with images, language, and action.  A user (machine or human) shows or describes what they want to simulate and then controls the kinds of outputs they want to measure and observe. We have built mobile and web interfaces that will let anyone access the model and REST APIs for developers and embodied systems that connect directly to CommonSim-1.

 At the heart of CommonSim-1 is a foundation model of the 3D world that is trained on a large-scale, growing dataset of diverse human (and non-human) experiences across a wide range of tasks. We combine publicly available data, our own internal datasets, and task-specific data provided by our partners. 

 With as little as 1 frame and a set of actions (camera or body movements), our foundation world model can generate high resolution videos (512×512). The same architecture can be trained to solve tasks across a variety of embodiments (camera, robot, car). Since this model imagines the future, one can use their imagination 1) as training data for upstream tasks (3D generation, perception), and 2) as part of a system’s  predictive model. 

To create 3D assets we let users upload videos of objects via our mobile or web app. All processing is done on the cloud and results can be exported into standard 3D (obj, usdz, blend, glTF, etc) and Neural Radiance Fields (NeRFs). 

Standard 3D assets can be imported into existing 3D engines for scene compositing and rendering. Our Hybrid Renderer (current support for Blender with more on the way) allows for NeRFs to be composed with traditional 3D assets. The Hybrid Renderer combines the best of both worlds, allowing for highly accurate but fast photorealistic rendering of scenes with physics.  

In addition to rendering pixels, our CommonSim-1 also infers data types such as bounding boxes, masks, and 6-DOF poses. These data are useful for training perception systems without the cost of data annotation. 

Natural language is a powerful means to describe imaginary situations. For instance, if you had a 3D model of a chair and wanted to render it in a completely new environment, it would normally take days or weeks to achieve satisfactory results. Text-to-image models can automate this process. With a mesh or NeRF generated by CommonSim-1, one can type natural-language descriptions into a text prompt and generate unbounded new hybrid scenes.

The above content generation mechanisms are sufficiently precise and flexible to generate large-scale synthetic data and train perception systems. Below are examples where a vision system for detection, segmentation, and 6-DOF pose tracking was trained without any human-annotated data or human-created models. We’ve tested these systems on objects from grocery stores, warehouses, factories, labs, medical theaters, and more. 

This is just a preview of what we’ve been developing and scaling up. Click below if you’d like to join the waitlist for early access. We are also hiring exceptional talent across the board – contact us at hello@csm.ai. 

We previewed some of our 3D generation capabilities in our CommonSim-1 preview blog post. We are now making our images-to-3D generation capabilities widely available through front-end apps, and will also be offering APIs shortly after for even greater accessibility. This should enable creators and developers to rapidly populate 3D worlds for AI training, games and content.

Our iOS app and web app make it easy for anyone to convert images to NeRFs (neural radiance fields) and textured meshes. These assets can be blended and dropped into existing simulation engines. We will be releasing a set of plugins and open-source code to make it easy to integrate the NeRF asset types into existing simulation and content creation workflows. 

As we gather more user feedback and scale up our cloud resources, we will keep inviting new people from the waitlist. A set of gallery assets and the web portal can be viewed here.

Contrary to existing 3D scanning and modeling software, our product is aimed at creating realistic and whole object models. This is essential for any serious gaming, AI training or content workflow beyond visualization. This often requires capturing objects from all sides, breaking several assumptions in classical structure from motion and NeRF pipelines. In order to do this, we support three capture protocols to cover different modeling scenarios:

The detailed instructions can be viewed here (capture and generate). A summary for capturing is described below:

For capturing static objects, users can walk around the object (left video) to cover the half hemisphere as much as possible. If the user captures using the iOS app, there is an option to place a 3D bounding box to indicate the region of interest. In all other cases on iOS or the web app, we deployed an interactive video segmentation model to let users quickly isolate the object of interest from all occluders — this enables an extremely flexible workflow to handle rigid objects that are under occlusion or motion.

The second protocol (middle video) lets the users walk around the object similar to the static protocol but flip it before repeating this process. This enables capturing all sides of the object, including extremely thin objects like a playing card which is very difficult otherwise.

The last protocol (right video) lets the users manipulate objects in their hands. Our video segmenter automatically isolates human hands. We will also shortly be updating the segmenter to automatically isolate the foreground object. For now, the user has to place a few points of interest on the image frame(s) and then the rest is automated.

All asset types including segmentation masks, standardized NeRF formats and textured meshes can be downloaded from the UI by clicking on an object session. Textured meshes can be imported into off-the-shelf game and rendering engines. Currently the NeRF formats are restricted to be loaded in Omniverse or Blender (instructions and github repo coming).

Comparisons to Apple Object Capture and Photogrammetry

We support two different NeRF variants — a fast NeRF and HDNeRF. HDNeRF takes a longer time to converge but often yields higher resolution and smoother geometry for manufactured objects. We combined this approach with a machine learning driven texturing pipeline to generate high resolution meshes with UV unwrapped texture maps. This approach provides the best of both worlds — leveraging NeRFs to model photometric details, precise geometry and a texturing pipeline which is robust to errors due to noise in camera/geometry estimation. We observed that in many cases where HDNeRF works well, it far exceeds the level of geometric realism. 

Generate 3D assets directly from your phone or web browser

Introducing Coder: Text to Code for 3D world generation, gaming and programming.

Creating a Roblox Game with CSM

CommonSim-1: Generating 3D Worlds

Generate 3D assets directly from your phone or web browser