Robotic cars are great at tracking other cars, and they’re getting better at noticing pedestrians, squirrels, and birds. The main challenge, though, is posed by the lightest, quietest, swerviest vehicles on the road. “Bicycles are probably the most difficult detection problem that autonomous vehicle systems face,” says UC Berkeley research engineer Steven Shladover. Nuno Vasconcelos, a visual computing expert at the University of California, San Diego, says bikes pose a complex detection problem because they are relatively small, fast and heterogenous. “A car is basically a big block of stuff. A bicycle has much less mass and also there can be more variation in appearance — there are more shapes and colors and people hang stuff on them.” Bikes are also being left behind by the machine learning techniques that enable detection systems to train themselves by studying thousands of images in which known objects are labeled. Most of the training, to date, has employed images featuring cars, with far fewer bikes.
Consider the Deep3DBox algorithm presented recently by researchers at George Mason University and stealth-mode robotic taxi developer Zoox, based in Menlo Park, Calif. On an industry-recognized benchmark test, which challenges vision systems with 2D road images, Deep3DBox identifies 89 percent of cars. Sub-70-percent car-spotting scores prevailed just a few years ago.
Deep3DBox further excels at a tougher task: predicting which way vehicles are facing and inferring a 3D box around each object spotted on a 2D image. “Deep learning is typically used for just detecting pixel patterns. We figured out an effective way to use the same techniques to estimate geometrical quantities,” explains Deep3DBox contributor Jana Košecká, a computer scientist at George Mason University in Fairfax, Virginia.
However, when it comes to spotting and orienting bikes and bicyclists, performance drops significantly. Deep3DBox is among the best, yet it spots only 74 percent of bikes in the benchmarking test. And though it can orient over 88 percent of the cars in the test images, it scores just 59 percent for the bikes.
Košecká says commercial systems are delivering better results as developers gather massive proprietary datasets of road images with which to train their systems. And she says most demonstration vehicles augment their visual processing with laser-scanning (ie lidar) imagery and radar sensing, which help recognize bikes and their relative position even if they can’t help determine their orientation.
Further strides, meanwhile, are coming via high-definition maps such as Israel-based Mobileye’s Road Experience Management system. These maps offer computer vision algorithms a head start in identifying bikes, which stand out as anomalies from pre-recorded street views. Ford Motor says “highly detailed 3D maps” are at the core of the 70 self-driving test cars that it plans to have driving on roads this year.
Put all of these elements together, and one can observe some pretty impressive results, such as the bike spotting demonstrated last year by Google’s vehicles. Waymo, Google’s autonomous vehicle spinoff, unveiled proprietary sensor technology with further upgraded bike-recognition capabilities at this month’s Detroit Auto Show.
Vasconcelos doubts that today’s sensing and automation technology is good enough to replace human drivers, but he believes they can already help human drivers avoid accidents. Automated cyclist detection is seeing its first commercial applications in automated emergency braking systems (AEB) for conventional vehicles, which are expanding to respond to pedestrians and cyclists in addition to cars.
Volvo began offering the first cyclist-aware AEB in 2013, crunching camera and radar data to predict potential collisions; it is rolling out similar tech for European buses this year. More automakers are expected to follow suit as European auto safety regulators begin scoring AEB systems for cyclist detection next year.
That said, AEB systems still suffer from a severe limitation that points to the next grand challenge that AV developers are struggling with: predicting where moving objects will go. Squeezing more value from cyclist-AEB systems will be an especially tall order, says Olaf Op den Camp, a senior consultant at the Dutch Organization for Applied Scientific Research (TNO). Op den Camp, who led the design of Europe’s cyclist-AEB benchmarking test, says that it’s because cyclists movements are especially hard to predict.
Košecká agrees: “Bicycles are much less predictable than cars because it’s easier for them to make sudden turns or jump out of nowhere.”
That means it may be a while before cyclists escape the threat of human error, which contributes to 94 percent of traffic fatalities, according to U.S. regulators. “Everybody who bikes is excited about the promise of eliminating that,” says Brian Wiedenmeier, executive director of the San Francisco Bicycle Coalition. But he says it is right to wait for automation technology to mature.
In December, Wiedenmeier warned that self-driving taxis deployed by Uber Technologies were violating California driving rules designed to protect cyclists from cars and trucks crossing designated bike lanes. He applauded when California officials pulled the vehicles’ registrations, citing the ridesharing firm’s refusal to secure state permits for them. (Uber is still testing its self-driving cars in Arizona and Pittsburgh, and it recently got permission to put some back on San Francisco streets strictly as mapping machines, provided that human drivers are at the wheel.)
Wiedenmeier says Uber’s “rush to market” is the wrong way to go. As he puts it: “Like any new technology this needs to be tested very carefully.”
This post was created for Cars That Think, IEEE Spectrum’s blog about the sensors, software, and systems that are making cars smarter, more entertaining, and ultimately, autonomous.