Humans are visual creatures. We use our eyes to gather visual information and discern directions. If your car could drive itself, what would serve as its "eyes"?

The answer is onboard sensors. They continuously collect environmental information and send them back to the "brain" of the smart car - the computing platform. The surrounding environment is accurately reproduced through perception algorithms. Then the decision-making algorithm will utilize this perception to plan the vehicle's trajectory.

Today we will talk about the common sensors used for intelligent driving systems: camera, millimeter wave radar, and lidar. What are their differences, and how do they compare to each other?

Cameras are the most common automotive sensor. They are installed around the vehicle, capturing images of the surroundings from multiple angles. The camera is also the sensor that has the most resemblance to the human eye. It can capture rich color and detailed information. However, its limitations are also evident. In situations where there is low light or backlighting that affect the "line of sight," the camera, just like human eyes, may struggle to see clearly and lose track of targets.

The key to visual perception technology is software algorithms that analyze high-density information to "recognize" objects and estimate their distances. However, when encountering unfamiliar objects such as irregular obstacles on the road, the system may make wrong decisions due to inaccurate perceptions.

As a result, most camera-based intelligent driving solutions are limited to L2 only. For L3 solutions, the use of complementary sensors becomes necessary.

Millimeter wave radar is an active sensor that uses the millimeter wave band for ranging, detection, tracking, and imaging. It can actively emit electromagnetic waves that penetrate smoke and dust. Mostly unaffected by light and weather, it helps vehicles perceive surrounding objects in real-time, providing mostly accurate distance and speed information.

However, the perception accuracy of millimeter wave radar is not ideal, and it has weak imaging capabilities and low resolution. Since millimeter-wave radar uses reflection, diffused reflection, and scattering on the target object's surface to detect and track the target, the detection accuracy is greatly reduced for "soft targets" with low reflectivities such as pedestrians, animals, and bicycles. Static objects may also be filtered out by the radar as clutter.

The popular "4D" millimeter wave radar is also just another kind of millimeter wave radar. Compared to the traditional 3D millimeter wave radar, 4D millimeter-wave can provide additional height information. Although, it can only output 1,000 points per frame, while a 128-channel lidar can output over 100,000 points per frame.

Lidar is another active sensor. The most common method used is Time of Flight (ToF) ranging. It works by actively emitting laser beams and measuring the time it takes for the beams to travel back and forth from surrounding objects. By emitting millions of laser points per second, lidar can outline its surroundings, such as vehicles and pedestrians in great detail with image-level resolution.

Since lidar can actively emit lasers, it is minimally affected by changes in ambient light. It's able to detect surroundings even in dark environments. Lidar can also directly obtain the size and distance of its surrounding objects, unlike cameras that have to rely on algorithmic estimates. Therefore, lidar performs better at detecting small and irregular obstacles, and at handling complex scenarios. However, the performance of lidar can be affected to some extent in extreme "white-out" weather conditions like heavy rain, snow, and fog.

In conclusion, the three most common sensors used for intelligent driving systems: camera, millimeter wave radar, and lidar each have their own advantages and disadvantages.

The camera is a passive sensor that recognizes color information, but its confidence level is significantly affected by light. Millimeter wave radar has a high confidence level, but due to its low resolution, it cannot effectively classify pedestrians, bicycles, or smaller objects. Lidar is excellent in its ranging capability, confidence level, and image resolution. The biggest disadvantage of lidar is its high price tag, but that is changing.

Lidar costs are rapidly dropping, and more automakers are integrating lidar into their series production vehicles to improve the safety and comfort of their intelligent driving systems. As lidar technology continues to advance, the number of individual components has been greatly reduced, and the automation rate has increased, making lidar affordable to average consumers.

At the perception level, different sensors have their own strengths and limitations. The key lies in optimizing their roles and maximizing their respective advantages. As the capabilities of intelligent driving software continue to evolve, it is expected that more hardware potential will be "unlocked" in the future, providing consumers with more advanced, more comfortable intelligent driving experiences.

*The gif of the millimeter wave point cloud is from Youtube channel - smartmicrogroup