How GM engineers built real-time camera intelligence for safer driving experiences

By Gilboa Shveki, Director of Software Engineering, Camera Services

Today's vehicles depend on cameras to do more than display images of surrounding driving environments. At General Motors, our camera systems enable safety and driver assistance features¹ that support everyday driving experiences - from maneuvering in tight spaces to trailering assistance that reveals hidden obstacles.

By combining multi-camera calibration, computer visioning, 3D geometry, and real-time processing, these systems provide bird's-eye and rear-anchored views that remain stable as vehicles and trailers move, even reconstructing parts of scenes that the cameras cannot directly see.

Delivering these experiences at production scale demands high-resolution sensors, precise alignment, real-time geometric reasoning, and system architectures that remain robust across vehicles, trailers, and real-world conditions.

This post walks through how GM engineers built a real-time camera intelligence stack that transforms raw camera feeds into a stable, vehicle-anchored understanding of the world.

Cameras as measurement, not just visualization

At first glance, generating peripheral views appears straightforward: capture images, display them, help the driver see more clearly. In practice, this is far more complex.

Cameras are mounted at different locations on the vehicle, with wide-angle lenses, and their calibration can drift from one drive to the next. Even when a vehicle uses closed-loop suspension to compensate for changes in load, that adjustment itself can slightly change each camera's position and orientation.

On top of that, mounting orientation tolerances, physical shifts in the sensors, vehicle height changes from tire pressure, and other small perturbations can all introduce error. Tiny orientation errors-less than 0.1°-are enough to create visible seams and misalignments in stitched views. As the vehicle moves, viewpoints change; when a trailer is attached, large portions of the scene, including what lies directly behind the vehicle, become occluded. Yet drivers still expect a coherent, intuitive view that behaves as if the trailer were transparent.

Solving this requires treating cameras not as passive image sources, but as geometric sensors. The system must understand where each camera sits in the world, how image pixels project into 3D space, and how to reconstruct what is hidden, all in real time, under strict latency and compute constraints.

One geometry stack, multiple views

GM's available camera features are built on a unified, vehicle-anchored vision and geometry stack. As an example, available Top-Down View and available Transparent Trailer² are different expressions of the same underlying architecture: shared coordinate systems, continuous alignment, precomputed geometry, and real-time GPU mapping.

Rather than displaying individual camera feeds, Top-Down View synthesizes a perspective as if a virtual camera were positioned above the vehicle. This view is derived from live camera data projected into a shared world coordinate system, so it's not pre-rendered or purely graphical. Each model has different dimensions and locations for the cameras. It's our job to provide a unique configuration for each vehicle and for each type of view.

Multiple wide-angle cameras contribute input, each with different optics and mounting locations. By projecting camera pixels into a common rear-bumper-ground coordinate frame, the system reasons consistently about spatial relationships (curbs, obstacles, lane edges) regardless of which camera observed them.

Accurate online alignment is critical. Small calibration errors-introduced by mounting tolerances, suspension travel, changes in load, or temperature-driven mechanical shifts-quickly show up as visible seams, double edges, or drifting features between views. Continuous refinement of each camera's position and orientation parameters ensures the composite view remains stable across vehicle motion, environmental changes, and long-term use.

Once a camera image is projected into a world coordinate frame, overlapping regions are stitched together and visually harmonized. Color blending smooths transitions, while dynamic field-of-view control supports different driving contexts such as parking or low-speed maneuvers. Graphics overlays - vehicle outlines, steering-dependent guidelines, proximity cues - are rendered in world coordinates, ensuring guidance remains spatially consistent as the vehicle moves.

Transparent Trailer: Reconstructing what the camera can't see

At a high level, the Available Transparent Trailer system fuses two camera streams:

• Rear Vehicle Camera (RVC) mounted on the vehicle
• Rear Trailer Camera (RTC)³ mounted on the trailer

The output is rendered in the RVC's perspective, preserving a consistent driver experience while reconstructing occluded regions using information from the RTC - even as the trailer articulates.

To scale across vehicle platforms and camera layouts, all computations are anchored in a common world coordinate system: rear-bumper ground coordinates.

Each camera provides intrinsic and extrinsic calibration into this shared frame. Rather than building camera-specific logic, the core algorithm remains agnostic to the mounting location. This modularity enables reuse across features and platforms while maintaining consistency across the camera stack.

To build an algorithm block which is agnostic to the camera type, each camera transfers its own "canonical alignment" matrix to the algorithm block.

From pixels to geometry

For the transparent trailer view to stitch well the camera view from trailer to the RVC camera, it needs accurate camera calibration data for both cameras and needs to have the trailer triangulation angle (TAA). To calculate that angle, two methods are used. One method is based on solving differential equations that predict the change in the TAA as a function of the steering parameters and the geometry. The other method uses the car camera to detect the trailer and calculate its TAA using a deep neural network. The TAA is also essential in many other applications like projecting guidelines on the RTC and warning the driver in case he is about to cut a corner while turning with a trailer.

Calculating the TAA and predicting the path of one or more trailers going forward or backward has been a popular research area and there are many scientific papers dealing with this problem.

At the heart of Transparent Trailer is a geometric reconstruction pipeline. Each pixel in the RVC image is treated as a ray projected into 3D space. To efficiently compute how those rays interact with the environment, the system uses a synthetic dome-shaped geometry divided into triangular facets.

Precomputation: Building the dome range Map (CPU)

Before runtime, the system computes a dense range map for the RVC:

• Rays are cast from each RVC pixel into the synthetic dome
• The first intersection point is recorded
• A Bounding Volume Hierarchy (BVH) accelerates ray-triangle intersection tests

This step is static for a given configuration and produces a dense map of 3D intersection points, which is uploaded to GPU memory as a texture. Importantly, BVH is used only during this preprocessing phase - not at runtime - allowing expensive geometry computations to be amortized.

This approach works well for a fixed viewing point. For views from a moving viewing point, a dome that is described by an analytic expression is used and an on-line ray-triangle intersection is not needed.

Real-time rendering of the view

At runtime, the system operates at display frame rate using a hybrid CPU/GPU design. CPU and GPU are always working together.

Dynamic inputs include:

• Synchronized RVC and RTC frames
• Trailer Articulation Angle (TAA), updated roughly every 33 milliseconds
• Trailer profile parameters

From these inputs, the system computes a parametric trailer plane representing the trailer's current position and orientation.

Shader-Based Runtime Processing (GPU)

For each frame, the GPU shader:

• Reconstructs the RVC ray using the precomputed range map
• If the ray intersects the trailer plane:
  • Traces a secondary ray toward the RTC
  • Samples the corresponding RTC pixel
• Otherwise, retains the original RVC pixel

This per-pixel logic runs entirely on the GPU, enabling low latency and smooth visual output even as the trailer moves. The result is a physically consistent image that effectively "sees through" the trailer.

Knowing the cameras' alignment data enables us to add elements into the views in a way that would match the real-world 3D. This is what we do in the Transparent Trailer view, and it can be extended furthermore to augmented reality views. In these views we can plug virtual objects into the view in a realistic way so that the object seems like it is a part of the scene (standing on the road, has the right scale and orientation).

Online alignment

Accurate geometry depends on accurate alignment. The system uses Online Alignment (OLA) to continuously refine RTC extrinsic parameters during normal driving. Rather than relying solely on factory calibration, OLA leverages vehicle motion and visual cues to maintain consistency over time. This capability is essential not only for Transparent Trailer, but also for stable Top-Down and surround-view generation across temperature changes, mechanical tolerances, and long-term use.

To calibrate the cameras, we need both their intrinsic and extrinsic parameters. Intrinsic describes how each camera forms an image: the sensor's position relative to the lens, focal length (magnification), and lens distortion. These distortions are significant because we typically use very wide-field fisheye lenses. Extrinsic describes where each camera is mounted on the vehicle and how it is oriented in 3D space.

To calculate this, we match points in multiple images and feed them into the bundle adjustment algorithm.

Engineering for failure and uncertainty

Building robust camera systems requires actively searching for failure modes. GM engineers use simulation pipelines to generate adversarial geometric conditions, subtly modifying shapes, and pose stress perception and mapping logic. These cases expose rare failure modes before they ever reach customers and have been shown to significantly reduce near-miss events.

The system is also designed to recognize its own limits. An epistemic uncertainty signal distinguishes expected noise from genuine confusion, automatically surfacing high-value edge cases for further analysis and retraining.

Validation spans algorithmic checks, system tests, and end-to-end scenarios, including deterministic behavior across power cycles, cross-validation of BVH-accelerated ray tracing, and continuous integration tests covering trailer connection, articulation changes, and camera reconnections.

A scalable camera intelligence foundation

GM's work on camera intelligence is about building a scalable foundation that can evolve as we innovate. By anchoring all camera views in a shared geometric model and continuously aligning sensors in real time, the same core stack can support surround views, trailering features, and future driver-assistance capabilities across many vehicles and trailer types.

For drivers, the result is straightforward: more reliable views, more consistent guidance, and fewer surprises when visibility is limited. That underlying camera platform is what allows GM to keep enhancing safety and giving customers a greater sense of confidence and peace of mind on the road.

^{1 Safety or driver assistance features are no substitute for the driver's responsibility to operate the vehicle in a safe manner. The driver should remain attentive to traffic, surroundings, and road conditions at all times. Visibility, weather, and road conditions may affect feature performance. Read the vehicle Owner's Manual for more important feature limitations and information.}

^{2 Safety or driver assistance features are no substitute for the driver's responsibility to operate the vehicle in a safe manner. The driver should remain attentive to traffic, surroundings, and road conditions at all times. Visibility, weather, and road conditions may affect feature performance. Read the vehicle Owner's Manual for more important feature limitations and information. Some camera views require available accessory camera and installation. Not compatible with all trailers. See your dealer for details.}

^{3 Requires additional setup/installation and an available accessory camera.}

^References:

^{[1] Z. Leng and M. Minor, "A simple tractor-trailer backing control law for path following," 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, Taipei, Taiwan, 2010, pp. 5538-5542, doi: 10.1109/IROS.2010.5650489.}

^{[2] Z. Zhang. A flexible new technique for camera calibration. IEEE Trans. Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000.}

^{[3] Camera Calibration Explained: Enhancing Accuracy in Computer Vision Applications https://medium.com/perceptron-perspectives/camera-calibration-explained-enhancing-accuracy-in-computer-vision-applications-8ad1494cc5f2}