Underscoring their intention to develop vehicles which are capable of full autonomy, Nissan has shown one of their autonomous Leaf prototypes on a public Japanese highway to the press. The auto-pilot system not only kept lane and distance; it also was also able to switch lanes, overtake other cars and merge into traffic at on-ramps. Although the highway segment was short and no details were provided on the quality of the lane markings (or localization algorithms) and merging capabilities, this event certainly shows that Nissan is committed to its vision of fully autonomous driving and aims to be perceived as an innovation leader. The event does not yet show, however, that it prototypes are more advanced than similar prototypes by other auto makers (e.g. Daimler, BMW etc.).
When KPMG released their first white paper on self-driving cars in late 2012, we were surprised at how little thought they had given to the disruptive potential of fleets of driverless vehicles.They have corrected this now and a major headline and probably the guiding question of the report is: “Mobility on Demand: Why own a car?”
This year’s report begins with the realization that the “momentum around self-driving vehicles is astonishing”. The authors acknowledge that the “industry is moving even faster than we predicted”. They look at the variety and number of autonomy-related news events during the preceding months. They conclude that “the technology is evolving at a rapid pace”. While the industry is definitely gaining momentum, the statement may be a bit on the optimistic side with respect to the technology: If we sort through all the announcements and public demonstrations of some cars driving in some state of quite limited autonomy, we can not see that many significant advances have occurred in the past 12 months.
They then look at the history of the first automotive revolution and proceed to report insights from focus groups. This is useful but yields few real surprises (even auto enthusiasts are willing to use self-driving cars; tech brands such as Google are a little more trusted even than premium auto brands with respect to the technology).
The most interesting section deals with a potential decrease in car ownership: “If half of all American families who currently own two or more cars were to give up one of their vehicles, how would that affect the automotive industry”. They point out that the ratio between fleet and retail sales could “change dramatically”. Mobility costs could decrease for the consumer and big data and dynamic pricing will be a key capability for autonomous mobility providers. They also hint at adverse consequences for mass transit systems.
It is great that the major consulting companies are beginning to realize how disruptive this technology will be (and it is nice that they are coming around to a view that we already had published in early 2010 (Autonomous cars – the next revolution looms).
If the momentum continues to increase – which is very likely – next year’s report will be very different. Then they may be at their best and provide a much more detailed look at business models, competitive space, impact, and strategic positioning for automotive companies and other industries. We will be looking forward to the next report!
World politicians are increasingly taking autonomous vehicles seriously. They start to recognize the importance of autonomous technology and the potential to profit from this innovation. French president Francois Hollande just presented a road map to revive French industry by promoting driverless cars and other technologies. Although it does not appear that France is prepared to fund driverless car development directly, they want to help bring about collaborations fostering this innovation. At this point, this may be what is needed most. There are many funds for this kind research available within the EU and venture capitalists are also discovering the high revenue potential in the area of self-driving cars. It is more important to clarify the legal ground rules for testing and operating driverless cars and making sure that these rules allow for fully autonomous operation. Because only fully autonomous operation will unlock the transformative potential of driverless technology and with it the investments needed to bring it about.
France and the other European countries should make clear that the they have the right and the intention to allow the operation of fully autonomous vehicles in their own territory as soon as they are certified to be extremely safe and as long as their operation is technically limited to a clearly specified region within the nation territory. Such a declaration would remove the doubts about the Vienna Convention standing in the way of automotive progress. Removing such legal uncertainties would open the floodgates for investors seeking to gain the pole position in this transformative technology.
France has good potential to accelerate this technology. The nation is open to novel technogies, has a great research base, has leading sensor manufacturers, and – even if the auto makers have invested much effort into autonomous technologies yet – has some well-positioned companies in the space. An example is the company Induct. It has positioned itself on one of the paths towards fully autonomous technology which is most likely to succeed economically: They don’t try to to build the perfect consumer car which can perform autonomously everywhere at every speed; they use autonomous technology to solve the last-mile transportation problem in a limited and well known area and they initially focus on low-velocity operations which greatly reduces risk and therefore makes commercial success more likely. What they need are testing grounds, i.e. communities, airports, campus grounds where the vehicles can operate and be perfected.They have had problems finding testing grounds in Europe and have now found a more welcoming climate in Singapore – which is probably not what the Frence president had in mind…
Nevertheless, France should not go this path alone. It should at least partner with the other European countries to ensure that Europe remains at the forefront of this technology.
In Japan, prime minister Abe toured the streets around the parliament in Tokyo in three driverless cars (by Nissan, Toyota and Honda) in advance of the upcoming Tokyo Motor Show. Abe has made clear that he wants to advance “auto-pilot” technology as part of this economic policy.Japan has a long tradition of cooperation between government and industry to bring new technologies forward and is also a world leader in robotic technology – a key area that will both profit and play a crucial role in advancing driverless technology.
Overall, it is clear that autonomous technologies are starting to appear on the radar screen of world leaders. There are still too many misconceptions about this technology and the way it will come about. The views are still very auto-industry centric, focused on consumer cars, and fallacious ideas such as the ‘auto-pilot’ analogy still cloud their judgment. But behind all of this, world leaders are beginning to realize that a major innovation is in the making.
In another sign that the race for leadership in autonomous car technology is heating up British newspapers (1,2,3) report that UK’s Automotive Council is investing 77 million Euros to deploy a fleet of 100 driverless vehicles in Milton Keynes by 2017. The vehicles will provide taxi services between downtown and the railway station. The vehicles will be fully electric, can carry up to two persons plus baggage and have a maximum speed of 19 km/h. They will be equipped with sensors and software for autonomous navigation. Details have not been finalized but it appears that the project plans to gradually increase the vehicle’s range and autonomy over time. When the first cars will be placed in service in 2015, they will operate on the sidewalk on dedicated lanes. As the project progresses, the vehicles’ range may be extended to include other areas; however, the vehicles will be limited to sidewalks where they will mix with pedestrian traffic or have their own lanes.
The Automotive Council is funded by the British Government and both the Secretary of Business and the Minister of State for Universities strongly support the project. Partners involved in the project are Cambridge University and ARUP, an engineering firm that also oversaw the development of the Heathrow Autonomous PRT Airport Shuttle. British firms already have begun exporting the autonomous PRT technology to other countries and the government hopes that this technology initiative may result in a leading position for the United Kingdom in the upcoming wave of autonomous mobility.
The Milton Keynes project has many advantages: The low speed and limited range allows gaining experience with fleets of mobile taxis while minimizing risks. Running on sidewalks rather than on city streets also reduces potential legal issues. By the time the project reaches its full scale in 2017, it should not be hard to apply many of their learnings to faster moving electric vehicles that can operate on regular urban streets at speeds of up to 50 km/h. The slow speed will also help to secure confidence and trust by the customers. The project will likely have positive effects on pollution by reducing the number trips driven with conventional cars, reduce accidents, increase the adoption of electric vehicles and reduce the costs for local transportation. Thus this project may pave the way for subsequent deployment of autonomous mobility services across the UK and the world.
Whether the Milton Keynes autonomous vehicle fleet will be the first autonomous vehicle fleet world wide which is not limited to separate tracks remains to be seen. There are strong contenders in the United States where Google is likely to introduce similar services (though on city streets) in some locations by 2017, in Singapore where Induct is experimenting with last mile driverless shuttles and probably also Zoox, a new service that will be unwrapped at the upcoming LA Auto Show.
We also expect key automotive manufacturers to announce such initiatives in the next two years. Daimler is particularly well placed for launching autonomous mobility services. In addition, we expect China to make an autonomous mobility services strategy a top priority within the next two years.
The race for the top position in the coming wave of driverless mobility services is still open. But one conclusion should be obvious: The fast path towards fully autonomous vehicles is not based on perfecting driver assistance systems for consumer cars but rather by deploying regionally focused fleets of special-purpose autonomous (and mostly electric) vehicles for urban mobility.
Addition (2013-11-10): A related Automotive Council presentation
As autonomous vehicle technology matures, legislators in several US states, countries and the United Nations are debating changes to the legal framework. Unfortunately one of the core ideas of these legal efforts is untenable and has the potential to cripple the technology’s progress. We show that the idea that drivers should supervise autonomous vehicles is based on false premises and will greatly limit and delay adoption. Given the enormous loss of life in traffic (more than one million persons per year world wide) and the safety potential of the technology, any delay will incur large human costs.
Read the full paper (pdf).
Supervising autonomous cars is neither necessary nor possible
The car industry is innovating rapidly with driver assistance systems. Having started with park-assist, lane-departure warning, etc., the latest systems now include emergency braking and even limited autonomous driving in stop-and-go traffic or on the highway (new Daimler S-Class).
As the systems become more capable, the situations will greatly increase where driving decisions are clearly attributable to a car’s software and not directly to the driver. This raises difficult questions of responsibility and liability in the case of accidents. From a legal perspective, the easiest solution is to keep the driver in the loop by positing a relationship between the driver and the car where the car executes the driver’s orders and the driver makes sure that the car only drives autonomously in situations which it is capable of handling. The driver thus becomes the supervisor who is responsible for the actions of the car’s software to which he delegates the task of driving.
Unfortunately this legal solution can not accommodate advanced driver assistance systems which perform the driving tasks for longer periods in urban, country- and highway traffic. We will call these systems auto-drive systems to distinguish them from the current, simpler driver assistance systems which are typically used for narrow tasks and short times.
The legal model rests on the following two invalid assumptions:
1) An average human driver is capable of supervising an auto drive-system
All ergonomic research clearly shows that the human brain is not good at routine supervision tasks. If a car drives autonomously for many miles without incident, a normal human will no longer pay attention. Period! No legal rule can change this fact. The human brain was not built for supervision tasks. In addition the supervision of a car traveling at high speed or in urban settings is very different from supervising a plane which is on auto-pilot (see below).
If the developers of the auto-drive system build and test their car on the assumption that a human actively monitors the car’s behavior at all times because situations may arise that the car can not handle alone, then accidents will happen because some of the drivers won’t be able to react fast enough when such situations occur.
Even if a human could remain alert during the whole drive, the problem remains how the user can distinguish which situations a car is able to handle and which situations it can not handle. How much knowledge will a driver need to have about the car’s capabilities? Once auto-drive systems evolve beyond the current very limited highway and stop-and-go scenarios, and are capable to drive in rain and urban settings, it will become very difficult for the manufacturer to enumerate and concisely describe the situations the car can or can not handle. It will become impossible for the average driver to memorize and effectively distinguish these situations.
2) Humans need to supervise cars operating in auto-drive mode
We saw in the last section that humans can not be relied upon to correct mistakes of a car while driving. But humans might still be needed to ensure that the car does not attempt to drive autonomously in situations that it can not handle well.
However, the car is equipped with a wide array of sensors and continuously assesses its environment. If it’s autonomous capability has limitations, it must be able to detect such situations automatically. Therefore there is no need to burden the driver with the task of determining whether the car is fit for the current situation.
Instead, the car needs to inform the driver when it encounters such a situation and then requests to transfer control back to the driver.
Therefore any non-trivial driver assistance system must be able to inform the driver when it enters situations it can not handle well. There is no need to require that the casual driver be more knowledgeable than the system about its capabilities.
Auto-pilot: the wrong analogy
The most frequently used analogy for a driver-assistance system is the auto-pilot in a plane. Mentally assigning the status of a pilot to the car’s driver who then watches over the auto-drive system may have appeal. But it overlooks the fundamental differences between both contexts: A car driving autonomously differs very much from a plane on auto-pilot. The nature of the tasks and the required reasoning capabilities differ considerably:
a) Physics of motion: A plane moves in 3-dimensional space trough a gas. Its exact movement is hard to formalize and predict and depends on many factors that can not be measured easily (local air currents, water droplets, ice on the wings). A trained pilot may have an intuitive understanding of the movement that is beyond the capabilities of the software. In contrast, a car moves in 2-dimensional space; its movement is well understood, easy to handle mathematically and predict, even in difficult weather (provided speeds are adequate to the weather).
b) Event horizon. Situations that require split-second reactions are very rare while flying; they occur frequently while driving a car. Thus the hand-off and return of control between human and machine is much more manageable in flight than in a car. There are many situations which an auto-drive system must be able to handle in full autonomy because the time is not there to hand off control to the human.
c) Training. The supervision task is the primary job function of a pilot, requires extensive, continual training and has many regulations to ensure alertness. This does not apply and can not realistically be applied to the average driver.
Therefore the relationship between pilot and auto-pilot can not be used as a model for the relationship between driver and driver-assistance system.
Driver assistance systems can not gradually evolve into auto-drive systems
Much of the discussion on the progress of autonomous vehicle technology assumes that driver assistance systems will gradually evolve to auto-drive systems which are capable of driving on all types of roads in all kinds of driving situations. Initially, auto-drive will be available only for a few limited scenarios such as highway driving in good weather. Thereafter more and more capable auto-drive systems will appear until the systems are good enough to drive everywhere in all situations.
Unfortunately, this evolution is not likely. Cars which drive autonomously can not return control to a driver immediately, when they encounter a difficult situation. They must be capable of handling any situation for a considerable time until the driver switches his attention to the driving task and assesses the situation. These cars can not limit themselves to driving in good weather or light rain only – they must be able to handle sudden heavy rain for as long as the driver needs to return to the driving task which for safety reasons must be more than just a few seconds. At realistic speeds these cars may travel a considerable distance in this time. If the car can safely handle this delay, it must probably be able to travel long distances in heavy rain, too.
The same issue applies to traffic situations: While highways may look like an ideal, well structured and relatively easy environment for driving, many complex situations can arise there at short notice which a car on auto-pilot must recognize and deal with correctly. This includes many low-probability events which nevertheless arise from time to time, such as people walking or riding their bicycle on highways. Driving in urban settings is much more complex and therefore a gradual path of auto-drive evolution is even more unlikely in such settings. Thus there maybe some low-hanging fruit for the developers of auto-drive applications (limited highway-driving); but almost all the rest of the fruit is hanging very far up the tree! Systems that are capable of driving in urban/countryside traffic can not start with limited capabilities. From the first day, they must be able to handle a very wide variety of situations that can occur in such settings.
Regulations that harm
We have already shown that the requirement of supervised driving is neither necessary nor can it be fulfilled for advanced driver assistance systems. But one could argue that the requirement does little harm. This is not the case. Wherever this rule is adopted, innovation will be curtailed. The safer and more convenient features of autonomous vehicles will only be available to the affluent and it will take a long time until most of the cars on the road are equipped with such technology. This means many more lives lost in traffic accidents, much less access to individual mobility for large groups of our population without driver’s license (such the elderly and the disabled), more waste of energy, resources, space for mobility.
Any country that adopts such rules will curtail innovation in car-sharing and new forms of urban inter-modal and electric mobility that become possible when autonomous vehicles mature that can drive without passengers.
It is obvious today that legislation that requires drivers to supervise advanced driver assistance systems will not stand the test of time.
Download as PDF
Changes 2013-09-26: Updated title and part of the text
Oxford’s mobile robotics group has been making rapid progress in the development of driverless cars. As Prof. Paul Newmann explained in a lively lecture last Thursday (as part of the 14th Annual Robotics Systems Conference), it took his group of 20 PhD students just 4 months to build an autonomous car that was able to navigate local streets.
While being equipped with some algorithms for obstacle detection, the car primarily serves as a test bed for advanced navigation algorithms. Similar to Google, the group uses prior knowledge about the roads to be traveled, but their algorithms can work with much simpler and much less expensive sensors. The car does not need 3D LIDAR sensors. It uses a much cheaper 2D Lidar which is affixed to the very front of the vehicle. The rotating laser captures a slice of points with distance information in a single line below the car as well to the right and the left of the car. As the car moves forward and scans line after line a 3D picture gradually emerges. The car determines its position by comparing the data points gathered to its prior knowledge. The sensor can capture about 40 lines per second. This works well for low speeds but would have to be increased for higher velocities.
Prof. Newmann has also come up with a new approach for navigating in snow and rain. Localization can be very difficult when snow changes the environment’s appearance. His solution is only seemingly simple: instead of trying to detect invariant properties of the landscape, he proposes to accept that the environment may have multiple appearances. Thus he adds the different ways that the environment may look to his store of prior knowledge. As the car drives a known area, it identifies that prior view (winter, summer..) which most closely matches the data captured by its sensors and uses it for localization. It will be interesting to see how robust this approach of “experience-based navigation” can be and how many variations of the environment will be needed to allow fully autonomous driving.
The group currently has two driverless car prototypes; one of them is part of a cooperation with Nissan. It will be interesting to see whether Nissan will incorporate some of the groups navigation algorithms into their solution.
As the first major auto maker, Nissan has announced that they will develop fully autonomous vehicles capable of navigating even in urban traffic without supervision. Nissan’s Executive Vice President Andy Palmer claims that – unlike Google’s driverless car prototypes – these vehicles will neither require costly 3D LIDARs nor will they need specially created maps for navigation. Nissan intends to bring the cars to the market by 2020.
Nissan wants to build on the successes of its Leaf Electric Car and further associate its brand image with innovation. In the past year, Nissan has taken major steps to accelerate autonomous vehicle development: They moved their autonomous research group from Japan to Silicon Valley and are building a testing ground for urban autonomous driving which is slated for completion by the end of this year.
While most other car companies are active in the field of autonomous driving and some (such as Volvo) have made general statements about fully autonomous vehicles, the Nissan announcement appears to be the first which is accompanied by action.
From an innovation diffusion perspective it is interesting to see that the commitment to fully autonomous technology does not come from one of the the top three auto manufacturers but from a large contender who sees the technology as a means to gain reputation and market share. Nissan does not seem to be concerned about the medium term business implications – a transformation of the car market from individual ownership to mobility service providers and a significant reduction of the total vehicle demand. They may count on a first-mover advantage; in addition, the combination of electric vehicles (the Leaf) and autonomous capabilities might be ideally suited for fleets of locally operating autonomous taxis.
It remains to be seen whether Nissan will be able to master the complexities of urban traffic without the advanced sensors and prior knowledge which Google is relying on. Nissan certainly has the plate full to catch up with Google. But with this action by Nissan and the ever-clearer intent of Google to challenge the biggest auto manufacturers it is probably just a question of months until the next auto makers will jump on the bandwagon. The next revolution in mobility is picking up speed…
On July 13 a low-cost driverless car developed by Italian computer vision expert Alberto Broggi navigated the roads of Parma, Italy in public traffic. The route included rural and urban roads, two freeway segments, traffic lights and roundabouts.
This is a major accomplishment because almost all current autonomous cars use an extremely expensive LIDAR sensor (which costs between 30000 and 70000 USD). Broggi’s car, in contrast, relies on an array of low-cost sensors including stereo cameras and several low-cost laser sensors. All sensors are hidden from view.
The successful experiment (see full length video) highlights the rapid advances in computer vision. It is probably premature to conclude that autonomous cars will be able to rely mainly on computer vision – the weather situation in Parma was sunny (which has its own problems for computer vision) and cameras have difficulties at night. But the experiment is a strong indication that autonomous car technology is advancing rapidly and that different approaches than Google’s may lead to success.
Congratulations to Broggi and his team. We hope that they will be able to secure sufficient funding to take their work from the prototype stage to a full product soon. Besides the vision algorithms much work is needed to optimize the operational capabilities of the car for dealing with typical and rare traffic situations. This requires hundreds of thousands of kilometers of test driving and much fine-tuning that is beyond the capabilities of research institutes. The upcoming European Research Programm for Research and Innovation might be a good starting point for advancing this technology.
DARPA has just announced the winning teams of their virtual robotics challenge. Each team had to develop software to enable a very advanced humanoid robot to perform disaster-response tasks such as 1) walking to a standard utility vehicle and entering it, 2) driving the vehicle using steering wheel and pedals, 3) walking on straight, uneven and obstacle-laden surfaces, 4) picking up a fire hose, connecting it and opening the valve. The winning teams will be provided with actual physical versions of the simulated robots and move on to the next level of the Darpa Robotics Challenge.
DARPA used the open source simulation environment Gazebo to create a virtual environment where the software of more than 100 competing teams could be tested. They have created a complete virtual world with a street, houses, obstacles etc. and developed a model of the robot which could then be placed into the virtual world and controlled by the team’s software. This approach has many advantages: Different scenarios can be tested, no physical damage can occur when the software does not behave as expected, the teams can use the simulation to optimize their algorithms etc.
A similar approach could be used to test driverless cars before they are released to the public. Official simulation environments would be needed with standard interfaces for the driverless car software. This would have to include standardized models which describe the behaviour of the car’s sensors (and translate the state of the simulation environment into sensor readings).
A standard simulation environment for driverless cars would enable a testing body to verify that a software-based driverless car behaves as expected in many difficult cases. It would allow quantitative claims about the performance of driverless car software and could be used to compare different driverless car systems. Driverless car companies could use such test environments to prove that their software has been properly and carefully developed.
Of course test cases would have to be challenging and representative of the actual scenarios which a driverless car will encounter. This requires significant effort. It is likely over time the number of test scenarios which a driverless car software will have to pass will increase. In the event of driverless car accidents, the testing agency could rapidly analyze the causal chain for the accident and then build additional simulation test cases which the updated driverless car software would have to pass.
As we have argued in our white paper three years ago, it is extremely important to develop a public common simulation platform for driverless cars now!
This is a video released by Bosch in early 2013 about their autonomous driving efforts. They use a similar approach as Google – based on a Velodyne 3D Lidar which means that they have set their sights on urban driving.