No Environment, No Sensor Simulation: How Frontier AI Models Are Changing 3D Environment Creation
A physics-based sensor model needs a world to sense. A lidar model needs surfaces to reflect its laser beams. A radar model needs geometry and materials that scatter its waves. A camera model needs a scene to see. However accurate the sensor model is, it can only be as realistic as the environment it runs in.
No environment model, no sensor simulation.
That makes the 3D environment one of the most important parts of any virtual testing toolchain, and historically one of the most expensive. Creating 3D environments for sensor simulation has been cumbersome and costly. Roads have to match a valid road network. Hundreds of buildings, poles, signs, and trees have to be modeled and placed. Surfaces have to be textured, and everything has to stay consistent. For sensor simulation the bar is even higher: geometry has to be accurate enough for realistic lidar point clouds and radar reflections, and every surface needs plausible material properties.
All of this has traditionally required significant manual work from specialized 3D artists. As a result, the number and variety of environments available for sensor testing has been limited, and so has test coverage.
Over the past weeks, we explored whether frontier AI models can break this bottleneck. We ran three experiments, each building on the previous one. The results are not perfect, but they show clearly where environment creation for sensor simulation is heading.
Step 1: From OpenStreetMap to a simulation-ready scene in Blender
The first experiment used ChatGPT 6 Astra, Blender, and the open-source Blender Driving Scenario Creator add-on. The goal was to turn real-world map data into a 3D environment that can actually be used for simulation.
The workflow started with two inputs:
An OpenStreetMap import of a real location
The corresponding terrain data
Astra then retraced the road network on the terrain using the Blender Driving Scenario Creator, producing a valid ASAM OpenDRIVE map. That matters, because an environment that only looks right isn't enough. It needs a road description that simulators can work with.
Next, we gave the model photographs of the real location and asked it to develop the scene further by adding:
Buildings
Sidewalks
Street furniture
Environmental details such as vegetation
To check the result, we connected the environment to esmini and ran a traffic simulation directly on the generated map.
Is it perfect? Not yet. Many details still need improvement and human supervision. Even so, AI assistance already speeds up a large share of the tedious environment creation work.
Step 2: Benchmarking frontier models: ChatGPT 6 Astra vs. Claude Opus 5.5
When Claude Opus 5.5 was released, we repeated the experiment with the same inputs and the same goal:
OpenStreetMap data
Terrain data
Blender Driving Scenario Creator for OpenDRIVE creation
Photos of the real location as reference
The result surprised us. Neither model produced a perfect scene, but Claude Opus 5.5 created a more detailed and realistic 3D environment while using significantly fewer tokens. Buildings were generally placed more accurately, environmental details were richer, and the scene needed less manual refinement than the Astra version.
The more important point is not which model "won" this round. It is how quickly these tools are improving. Until recently, building simulation environments from geospatial data, terrain models, and reference photos took a lot of manual work. Today, frontier models can help with road network creation, scene composition, and 3D asset placement well enough to be useful in engineering workflows.
Step 3: Raising the level of detail with Unreal Engine
Sensor simulation benefits from detail: every additional object and surface is something a lidar or radar can interact with. So for the next step, we moved the workflow to Unreal Engine, continued with Claude Opus 5.5, and raised the level of detail considerably.
Most of the assets, textures, and asset placement in the resulting scene were generated with AI. Starting from map data and reference information, the AI helped create a much richer and more realistic environment than the same amount of manual effort would normally produce.
To be clear: no AI was used for the rendering itself. The video uses Unreal Engine's standard rendering pipeline. AI was used only during content creation, for 3D asset generation, texturing, scene composition, and environment development.
The scene still has artifacts, inconsistencies, and many details that a human environment artist would fix. What stands out is not today's quality level but the pace of progress. Not long ago, an environment like this meant hundreds of manually placed assets, a lot of modeling work, and significant artistic effort. AI can now take on many of these tasks and speed up environment creation dramatically.
From 3D environment to sensor simulation
A good-looking scene is only half the job for sensor simulation. Lidar and radar don't see colors and textures. They interact with materials. An environment is only useful for sensor validation if its surfaces behave physically correctly.
So our next step is to enrich these AI-generated environments with ASAM OpenMATERIAL 3D material properties and use them as input for our physics-based lidar and radar models. Then we can simulate not just the geometry of the environment but also realistic sensor interactions with roads, buildings, vegetation, traffic signs, and many other objects.
The long-term workflow is taking shape:
Import map and terrain data
Generate and enrich the environment with AI
Create ASAM OpenDRIVE road networks
Assign ASAM OpenMATERIAL 3D material properties
Run physics-based lidar and radar simulations
Execute traffic scenarios and validation campaigns
Conclusion
Physics-based sensor simulation stands or falls with the environment model. For years, the cost and effort of building these environments has limited how many virtual worlds sensors could be tested in.
Can AI fully automate environment creation today? No. Human supervision is still essential, and many details need manual refinement. But going from geospatial data and reference photos to a physically meaningful, sensor-simulation-ready digital twin is becoming increasingly realistic. Every new generation of models makes this workflow feel less like science fiction and more like engineering reality.
We see the open question this way: the bigger challenge may not be generating visually realistic worlds, but generating physically accurate digital twins that can be trusted for sensor validation. That is the problem we are working on.
