Self-driving cars have always piqued the attention. Although we might not have fully autonomous cars yet, we have cars featuring advanced driver-assistance systems (ADAS) capable of automatic steering, lane changing, parking, and traffic-aware cruise control.

A self-driving car uses numerous sensors for its ADAS, with machine vision being the primary way of detecting, identifying, and calculating the distance of objects and the general surroundings. Without machine vision, self-driving cars with cruise control and autopilot are very unlikely.

What Is Machine Vision?

Machine vision is a technology that allows machines to see and recognize objects in their surrounding environment. It is a subset of computer vision that focuses on the industrial applications of vision-oriented object detection in autonomous machines such as robots and vehicles.

Machine vision today uses AI deep learning algorithms such as convolutional neural networks (CNNs) to create robust and well-generalized models that can accurately identify objects in various conditions. It makes it possible to implement machine vision on various tasks requiring higher reliability in manufacturing, agriculture, robotics, and automotive.

How Does Machine Vision Work in Self-Driving Cars?

Machine vision in self-driving cars can typically be split into three parts: the camera system, processing (edge computing), and AI. This technology stack allows an autonomous vehicle to see, think, and identify signs and obstacles while driving. Let's discuss each part in depth to understand how these technologies tie together to form a self-driving car's machine vision.

Camera System

Machine vision relies on camera systems to gather visual information on the environment. Self-driving cars employ several cameras installed around the car to gather as much visual information as possible.

Two main sensors are used in machine vision cameras: complementary metal-oxide semiconductors (CMOS) and charge-coupled devices (CCD). For self-driving cars, CMOS is often preferred for its fast readout speed, powerful on-ship electronics, and parallel processing capabilities, making it the faster sensor, albeit prone to noise or artifacts. Solutions, such as different lighting modes, digital night vision, and filters, may help the CMOS sensor during less-than-ideal lighting conditions.

For self-driving cars, cameras are installed at certain distances from each other to produce stereoscopic vision. Stereoscopic vision is the ability to combine two or more visual inputs, creating a sense of depth or three-dimensionality in objects and the environment. This, in turn, allows cars to triangulate and calculate the approximate distance between the object and the car.

Since humans have two eyes, we also benefit from stereoscopic vision. You can test it yourself; close one eye and select a small object from your desk. Place your hand on the object's side and ensure the tip of the object is at least two inches away. Try pausing for a few seconds and see how confident you are with your approximation. Now, try opening both eyes and see how your sense of depth is much better.

Edge Computer

While the camera system on the self-driving car gathers data, an onboard computer (edge processor) will process all the inputs in real-time to update the system on the status of the environment immediately. Although typical machine vision tasks can save money using cloud computing, there is simply too much risk for self-driving cars to be connected to the cloud, even if it's for outsourcing the process in its requirements for machine vision.

Using an edge computer for processing input data eliminates latency problems and ensures that data is received, processed, and communicated in real-time. Edge computers for self-driving cars use specialized computers integrating AI graphics processors like NVIDIA's Tensor Core and CUDA Cores.

AI Algorithms

Algorithms have always been a crucial part of machine vision. The algorithm is what allows a computer to identify all the patterns, shapes, and colors provided by the camera system. Using AI over more traditional machine vision algorithms greatly enhances the ability of a self-driving car to reliably identify objects, street signs, road markings, and traffic lights. Many AI algorithms are being used to train self-driving cars. The most popular include:

• YOLO (You Only Look Once): A real-time object detection algorithm that identifies and tracks objects in the car's field of view.
• SIFT (Scale-Icar'sant Feature Transform): Used for feature extraction, it helps the car recognize distinctive landmarks and objects in its surroundings.
• Histogram of Oriented Gradients (HOG): Used for object recognition, it focuses on extracting local patterns and gradients from images.
• TextonBoost: An algorithm that helps with object recognition by analyzing textures in the environment.
• AdaBoost: Employed for data classification, AdaBoost combines multiple weak classifiers to make strong decisions about the objects and obstacles in the vehicle's path.

The Importance of Machine Vision in Self-Driving Cars

Machine vision is the primary way for a self-driving car to sense and understand its surroundings. Without machine vision, it is very likely that self-driving cars will be pushed back to Level 1 on the vehicle autonomy scale and may never reach full autonomy.

With machine vision, self-driving cars are now capable of object classification, lane and signal detection, sign identification, and traffic recognition.

Although many self-driving vehicles now use various sensors, such as LIDAR, RADAR, and SONAR, all rely heavily on machine vision to see the environment, identify objects, and understand the meaning of signs and traffic lights on the road. All these extra sensors are only there to augment machine vision and further the safety of people, animals, and properties.

That said, machine vision can perform independently without the help of other sensors to provide autopilot capabilities. In fact, Tesla's newest self-driving cars dropped RADAR and now solely rely on machine vision for its Autopilot system.

Although this doesn’t mean anything in terms of discrediting the usefulness of other sensor technologies, it does show the importance and power of machine vision in self-driving cars.

The Future of Machine Vision in Autonomous Vehicles

Machine vision is the foundation of self-driving cars. Through machine vision, cars can see and perceive the environment exactly as humans do. While challenges persist, the benefits of machine vision in terms of safety and navigation cannot be understated. As for the future of autonomous vehicles, any further advancements in AI, edge computing, and/or camera technology will surely make self-driving cars more proficient, which will likely move them to a higher level of automation.