Wednesday 3rd March 2021

Why thermal cameras are critical to effective and safe automatic emergency braking

Published on February 18th, 2021

In recent years, the European Union (EU) has introduced a range of mandatory transport safety measures which contributed to an estimated reduction of 50,000 fatal traffic casualties across Europe per year.

These measures include automatic emergency braking (AEB) systems and other advanced driver assistance systems (ADAS). But, says Mike Walters, vice president, product management, components business at FLIR Systems, there is still more to be done to bring this number down to align with the EU’s long-term goal of moving closer to zero fatalities and serious injuries by 2050.

As we move towards a future where automated and autonomous cars become more prevalent on our roads, we need to consistently enhance safety systems to continue building trust and broader acceptance of the technology.

In part, this means ensuring that ADAS systems (like AEB) are safe including the ability to sense, classify, and make split-second Artificial Intelligence-based decisions in all circumstances. The development of this technology has so far provided a starting point to further inform what’s needed for a smarter and safer future.

Current AEB systems explained

Pedestrian detection and AEB systems are critical to improving road safety. In 2019, nearly 2,000 people were killed in road traffic accidents in the UK, 462 of which were pedestrians. While most new vehicles on our roads already feature these systems, which help detect pedestrians in front of a vehicle and trigger the brakes to attempt to prevent or mitigate a potential collision, the reality is that they are not as effective as they need to be.

Current typical AEB or pedestrian detection systems rely on the use of visible-light cameras and, in some cases, radar. The problem is that visible cameras are only as good as the image created in the available light. There are several conditions in which visible cameras have difficulty detecting a pedestrian, including nighttime, shadows, sun glare, inclement weather conditions like snow, dust, smoke, fog and rainfall, and even atmospheric conditions such as pollution.

Introducing thermal

A thermal camera provides complementary data to RGB cameras and radar. Thermal cameras “see” heat, not reflected light, and perform well both in daylight and nighttime conditions. They are able to see through sun and headlight glare, most fog, and far beyond what headlights can illuminate in darkness.

Given their unique capability, when fused together with radar, visible sensors, and a convolutional neural network (CNN) for AEB applications, thermal sensors can improve pedestrian detection in a high number of conditions where current systems struggle.

By adding thermal cameras to existing systems, obstacles of concern (including pedestrians) can be detected earlier and more reliably, resulting in better overall system performance and, therefore, reduced accidents, injuries, and fatalities.

AEB testing protocols

In current AEB testing, commercially available vehicles are only expected to meet standards based on Euro NCAP testing protocols: night time testing for example only includes testing with streetlights which isn’t representative of real world scenarios nor any testing for inclement weather conditions. Advanced testing that includes modified scenarios and potentially challenging real-life cases for current AEB systems is essential if we are to fully determine their effectiveness.

A 2019 study undertaken by the AAA tested several production vehicles with standard AEB systems, highlighting that the vehicles under test (VUT) travelling at 20 mph hit a soft pedestrian target 40% of the time in daytime conditions, rising to 100% at 25 mph during nighttime.

More recent testing shows that a fused thermal AEB system ran through a series of NCAP tests with a VUT stopping 100% of the time in both day and nighttime test cases. The commercially available AEB systems (which did not have a thermal camera) similarly had mixed performance in daytime tests but repeatedly fail at night, striking a soft pedestrian target in all but two out of eighteen test cases.

The thermal-enhanced AEB system improves functionality in the most dangerous situations, including low-light conditions, darkness, and when exposed to blinding conditions such as emerging from a dark tunnel into bright light.

Because the thermal cameras see temperature rather than colour, they can reliably detect pedestrians even in the most challenging conditions, such as a person wearing all black crossing a poorly lit street at night.

No single sensor will provide a solution for all the variables a vehicle will encounter on the road and whilst many ADAS systems are built with radar and/or visible cameras to provide AEB capability additional sensors such as thermal are required to make these systems operate more effectively and in a broader range of conditions.

Incorporating a thermal sensor within a fused AEB system that includes radar, visible cameras offers a greatly improved sensor suite. This would improve braking and impact avoidance performance in common conditions such as darkness, sun glare, fog, and snow, and ultimately move the automotive industry closer to a zero-fatality reality.

There is now an opportunity for automotive manufacturers, suppliers, regulators, and consumers to demand systems that maximise the safety of drivers, pedestrians, and other vulnerable road users.

The author is Mike Walters, vice president, Product Management, Components Business at FLIR Systems.

Comment on this article below or via Twitter: @IoTNow_OR @jcIoTnow