It feels like we’ve been talking about autonomous vehicles for some time, says Gary Ogasawara, CTO of Cloudian. Although multiple big-name car brands such as BMW, Daimler and Ford – along with the likes of Lyft and Uber – have been developing and testing autonomous vehicles for several years, we’re still yet to see them on our roads in significant numbers.
One of the key challenges in autonomous vehicle development is processing and managing the staggering amount of data involved. Autonomous vehicles are essentially rolling computers that are constantly capturing video and sensory data to keep passengers, pedestrians and other drivers safe.
The bulk of this data comes from the vehicles using data to build a picture of their surroundings. Multiple cameras collect image data, such as Light Detection and Ranging (LIDAR) data, which is then processed to form a composite visual representation of what is happening so the car can make driving decisions.
This is then supplemented by radar, telemetry from sensors and GPS data, resulting in the production of huge amounts of information. Analyst house 451 Research suggests that consumer vehicles create between 12 to 15 terabytes per day, while other experts think it could be up to 20TB. And that’s just for one car. For a company running a fleet of autonomous vehicles, the data generated could easily run into the several hundreds of terabytes every day.
So, how can all this data be efficiently and cost-effectively stored and processed?
Bumps in the road
There’s no doubt that the potential of all the data being collected is incredibly exciting. It will allow for vehicles to become safer and more consumer-friendly, which is essential to widespread adoption. However, the sheer volume of data involved creates significant challenges.
Even 5G cannot handle the bandwidth required by a single vehicle, which makes transferring all this data to the cloud for processing grossly impractical. The simple truth is that networking just can’t keep up with the data volume, which is something the industry is currently discovering.
Related to this is the issue of speed, which is where processing and storage meet. From a safety perspective, a large proportion of data has to be processed in real-time so that vehicles can detect hazards and respond accordingly. But autonomous vehicles that rely too heavily on the cloud are likely to experience latency issues that could lead to serious incidents – and the risk rises significantly when multiple vehicles are in the same area.
When we think about the scale involved, it’s just not feasible to push all the data to the cloud. In order for the full potential of autonomous vehicles to be realised, the right underlying technical infrastructure has to be put in place to support the mass of data being created, while removing the risk of latency issues. This is where storing data at the edge – i.e., on the vehicle itself – can make all the difference.
Moving to the edge
The scale involved with autonomous vehicles presents a huge opportunity for edge storage. Indeed, the concept of ‘data gravity’ – that it’s easier and less costly to move compute and applications to where data resides rather than move the data – necessitates that some of the data be stored at the edge, near to where the compute is taking place.
This data can then be filtered to retain what’s useful and needed for immediate action, what’s useful and should be sent to a data centre or the cloud for processing, and what’s not useful and can be discarded.
This filtering process is a fairly simple way of reducing the amount of data that has to be transmitted elsewhere. Anything deemed ‘normal’ can be summarised, while anything anomalous can be collected and processed. It’s a balance that the systems have to get right, as doing so will ensure that the most usable information is stored in the right place. This, in turn, will enable the maximum amount of value to be generated from the data involved.
It will also speed up the development of autonomous vehicles. Having specific datasets stored at the edge enables faster processing. The results of this can then be sent to a data centre or the cloud, while also creating a more powerful infrastructure for manufacturers to leverage.
So far, it’s the storage and management of the vast amount of data that has proven to be one of the major speed bumps hindering autonomous vehicle development. In the years to come, it’s clear that edge storage solutions will play an increasing role in enabling efficient data management and unlocking the full potential of autonomous vehicles.
The author is Gary Ogasawara, CTO of Cloudian.