Autonomous vehicle sensor fusion combines measurements from multiple sensors into estimates of the vehicle's motion and its surroundings. Cameras can help identify what an object is, lidar supplies spatial measurements, and radar contributes motion information. Navigation sensors provide another part of the picture: where the vehicle is and how it is moving.
The useful output is an estimate that planning and control software can interpret, including its time, coordinate frame, and uncertainty. Adding sensors helps only when their information is aligned correctly and the system handles disagreement. A precise-looking result can still be wrong if it combines old observations, mismatched coordinates, or repeated information from the same source.
On This Page
- Two fusion jobs within the vehicle
- How measurements become an estimate
- Choose where information comes together
- Define time and coordinate interfaces
- Recognize failures before trusting the output
- Evaluate performance under the intended workload
- Match the architecture to the application
Two fusion jobs within the vehicle
Localization fusion estimates the vehicle's own state, such as position, orientation, and velocity. Perception fusion estimates the surrounding scene, such as obstacles and their motion. These tasks interact, but their outputs answer different questions: “Where am I?” and “What is around me?”
For localization, inertial measurement units (IMUs), wheel odometry, satellite navigation, and externally estimated poses can contribute different measurements. An IMU contains accelerometers and gyroscopes; it does not directly measure a drift-free global position. The robot_localization state-estimation documentation describes extended and unscented Kalman filters that accept odometry, IMU, pose, and velocity messages.
Perception pipelines may combine image detections with lidar clusters, or fuse learned representations before detecting objects. Autoware's image-projection fusion documentation, for example, describes assigning camera-derived classifications to lidar clusters and enriching point clouds with image information.
A practical architecture therefore has several boundaries: sensor acquisition, time and spatial alignment, estimation or detection, and delivery to the planner. The planner then selects motion, and controllers command the vehicle. Fusion is part of that chain; a successful detection does not establish that the complete vehicle can negotiate the situation.
How measurements become an estimate
A Kalman-family state estimator alternates between prediction and measurement update. Prediction advances the estimated state using a motion model and available inputs. An update compares a new observation with what the model expected, then adjusts the estimate according to modeled uncertainty. The difference is called the innovation or residual.
Covariance represents uncertainty and relationships among errors in the estimated quantities. A detector's classification score answers a different question and should not be substituted for position uncertainty. The model matters too: a ground robot constrained to a plane and a freely moving aircraft should not inherit the same motion assumptions. robot_localization documents these filter choices and a planar operating mode; neither algorithm removes the need to configure its inputs correctly.
PX4 provides a concrete aircraft example. Its navigation-filter documentation describes IMU-based prediction, delayed measurement fusion, and propagation of the result forward to the current time. It also describes statistical checks on innovations. Missing data and data that disagree excessively with the prediction are distinct failure conditions.
Perception has an additional problem: deciding which observations belong together. A lidar box and several radar returns might describe one vehicle, nearby vehicles, or unrelated reflections. Autoware's radar-to-object fusion algorithm selects radar information around lidar-derived boxes and uses selected measurements to estimate object velocity. That makes the association step visible: combining the wrong observations can corrupt an otherwise valid detection.
Radar Doppler measures motion along the sensor's line of sight, rather than independently providing every component of an object's velocity. Autoware documents an optional conversion that uses the detected object's heading. Integrators should therefore ask which parts of a reported velocity were measured and which depend on geometry or a model.
Choose where information comes together
Fusion can occur at different processing stages. The terms early, intermediate, and late are useful shorthand, but implementations draw the boundaries differently. Ask a supplier or development team to identify the actual inputs to the fusion operation.
The comparison below is an engineering synthesis of Autoware's projection and radar modules and the BEVFusion research paper. It describes architecture tradeoffs, not measured rankings.
| Fusion stage | Information combined | Useful property | Integration tradeoff |
|---|---|---|---|
| Early or measurement-level | Points, pixels, or measurements before final detection | Retains detail for subsequent joint interpretation | Large data streams and close alignment requirements reach the shared processing path |
| Intermediate or feature-level | Representations extracted by sensor-specific processing | Combines geometric and semantic cues before final decisions | Feature definitions, model versions, and training assumptions become coupled |
| Late or object-level | Detections, object states, or tracks | Exposes a comparatively compact boundary between processing modules | Information discarded by individual detectors may no longer be available for fusion |
BEVFusion places camera and lidar features in a shared bird's-eye-view representation, then combines them for perception tasks. This is a specific feature-fusion design, not a requirement that every autonomous system use a top-down map. The paper explains why the choice of representation can preserve or discard useful information.
For a modular integration, object-level fusion can offer an understandable handoff between independently developed detectors. For a jointly trained perception stack, intermediate fusion may expose useful cues before either sensor makes a final decision. Those are design starting points. Neither architecture label proves superior accuracy, lower latency, or reliable behavior after a sensor fails.
Define time and coordinate interfaces
Before tuning a model, establish what each input means. A packet's arrival time and the time its measurement was acquired are different. A sensor may also collect a scan over an interval rather than observe the entire scene at one instant.
Autoware's projection-fusion implementation matches inputs using reference timestamps and configured offsets. It waits for associated inputs or a timeout before processing the collected data. The documentation also provides diagnostics for outputs that lack some expected image information. Widening a matching window can admit more data while increasing temporal mismatch; a larger collection is not automatically a better one.
Consider an illustrative vehicle traveling straight at a constant 15 meters per second, with an uncompensated 40-millisecond timing error relative to a stationary scene. Its travel during that interval is:
Distance traveled = speed × time offset = 15 m/s × 0.040 s = 0.60 m.
This is a hypothetical calculation, not a measured fusion error or a recommended tolerance. Turning, moving objects, sensor geometry, and motion compensation change the resulting alignment error. A known processing delay can be compensated differently from an unknown clock offset; record both the measurement time and its age at the consumer.
Spatial alignment needs equal care. Extrinsic calibration describes the relative position and orientation of sensors; intrinsic calibration describes properties within a sensor, such as a camera's projection model. The receiving module must interpret observations in the intended frame using the applicable calibration.
ROS REP 103 illustrates the danger of assuming matching axis names: its body-frame convention uses x forward, y left, and z up, while camera optical frames use z forward, x right, and y down. It also specifies SI-unit conventions. These are ROS conventions, not universal vehicle requirements.
For each interface, document acquisition-time semantics, units, frame identifier, calibration version, uncertainty representation, maximum usable data age, and missing-data behavior. This is a suggested integration contract rather than a universal message definition. Require an example observation carried all the way through the transforms to the consumer; matching field names alone does not prove compatible meaning.
Recognize failures before trusting the output
Sensor disagreement deserves investigation. The robot_localization configuration guide warns about feeding duplicate information into a filter when position, heading, and velocity originate from the same wheel encoders. It also describes conflicting orientation measurements and inappropriate covariance settings causing an estimate to oscillate.
Weather introduces a different problem. Bijelic and colleagues' Seeing Through Fog research investigated unequal degradation across camera, lidar, radar, and other sensing streams. Its experiments show why fusion designed around clean, mutually consistent inputs can struggle when conditions distort them differently. That research does not establish all-weather capability for an arbitrary sensor package.
The diagnostic table combines those findings with the Autoware and PX4 documentation. The checks are editorial engineering recommendations; each symptom can have several causes.
| Observed symptom | Possible cause | Useful next check |
|---|---|---|
| Stationary objects appear displaced between sensors | Wrong transform, calibration, or timestamp interpretation | Separate stationary alignment checks from moving runs; verify the exact transforms and acquisition times |
| Position or heading jumps between plausible values | Conflicting measurements or understated uncertainty | Inspect residuals and reported covariances for each contributing source |
| Confidence improves sharply without better agreement with a reference | Repeated information treated as independent evidence | Trace upstream dependencies, including estimates already fused elsewhere |
| Detection quality falls when one sensor is obscured | The fusion model depends on a stream that has degraded | Compare individual-sensor and fused outputs for the same obscured scenes |
| An output continues after a sensor stops | Prediction or partial-input processing continues | Check data age, contributor status, and the consumer's response to degraded output |
Redundancy also depends on shared infrastructure. As a design review question, ask what remains if sensors lose a common power supply, clock, network path, or compute process. Multiple devices do not create independent failure paths simply by being counted separately.
Define the permitted response to inadequate perception or localization at vehicle level. Continuing at reduced capability, changing the mission, or stopping must follow the actual operating environment and demonstrated vehicle behavior. Fusion software cannot select a universally safe fallback from sensor count alone.
Evaluate performance under the intended workload
Choose measurements that expose the decision the vehicle must make. The following evaluation questions are engineering recommendations derived from the timing, estimation, and adverse-weather issues above:
- Localization: What are position and orientation errors against a suitable reference, and how do they evolve during an outage and recovery?
- Perception: Which relevant objects are missed, which false objects appear, and how stable are object identities and velocities over time?
- Timing: How old is the information when the planner receives it, including capture, buffering, transport, and computation?
- Degradation: What changes when an input is delayed, unavailable, obscured, or plausible but wrong?
- Resources: Does the intended onboard computer sustain the workload within its power, memory, and thermal limits?
Compare alternative configurations on the same scenes, hardware, and operating conditions. Include individual-sensor baselines so the comparison reveals when fusion helps and when another stream makes the result worse. Preserve difficult subsets instead of reporting only an aggregate score. A benchmark improvement in BEVFusion's documented tasks, for example, cannot establish the performance of a different vehicle implementation.
Set application-specific limits before the trial and retain the sensor configuration, calibration, software version, and conditions with the results. Otherwise, teams can compare nominally identical systems that actually use different inputs or interpret output quality differently.
Match the architecture to the application
For a road vehicle, the documented Autoware examples show how camera classifications, lidar geometry, and radar-derived motion can meet in an object representation. Review behavior where observations overlap imperfectly or disagree, as well as the normal detection path.
For a wheeled mobile robot, the robot_localization example makes a different issue prominent: wheel-derived quantities can share the same underlying measurement. A planar model may be useful where its assumptions hold; confirm those assumptions before applying it to slopes or significant three-dimensional motion.
For an unmanned aircraft, PX4 illustrates fusion of navigation measurements with inertial prediction and delayed updates. The relevant handoff is a timely state estimate for control. A road-perception benchmark does not evaluate that function.
Begin an architecture review with the required output and the conditions in which the vehicle must use it. Then follow one measurement through acquisition, alignment, fusion, and consumption, including a deliberate failure case. The decisive question is whether the complete chain supplies usable information and exposes its limits when the inputs deteriorate.
Sources
- robot_localization: State Estimation Nodes. Developer documentation for filter choices, supported inputs, and operating parameters.
- Autoware: Image Projection Based Fusion. Implementation documentation for image/lidar fusion, timestamp matching, timeouts, and diagnostics.
- PX4: Using the Navigation Filter. Developer documentation for inertial prediction, delayed fusion, and observation checks.
- Autoware: Radar Fusion to Detected Object Algorithm. Implementation documentation for radar association and velocity estimation.
- Liu et al.: BEVFusion. ICRA 2023 research paper, arXiv revision 3, describing a shared feature representation and task-specific evaluation.
- ROS REP 103: Standard Units of Measure and Coordinate Conventions. ROS interface conventions for units, axes, and uncertainty representations.
- robot_localization: Configuration Guide. Developer guidance on duplicate inputs, planar assumptions, and covariance treatment.
- Bijelic et al.: Seeing Through Fog Without Seeing Fog. CVPR 2020 experimental research on multimodal perception under adverse weather; conclusions remain tied to its tested configurations.
Last checked: September 7, 2026.
