Autonomous Vehicle Technology in MiningHow it works and how it’s applied from user assist to full autonomyBy Chris Brown

Rio Tinto has embraced the all-or-nothing approach with Komatsu for its Mine of the Future program in Western Australia. (Photo courtesy of Rio Tinto)

In a lot of ways the agriculture industry led
the way with the widespread application of
autonomous vehicle technology. Autono-mous Solutions Inc. (ASI) started working
with John Deere in U.S. in 2000. Over the
next 10 years, ASI automated every vehicle
in Deere’s fleet for a variety of agriculture
processes. The company then branched
into applications for military R&D and the
automotive industry. In 2006, ASI entered
the mining business working with Phelps
Dodge (now Freeport McMoRan Copper &
Gold), automating a haul truck and a dozer.
The company is now working with Barrick,
Rio Tinto and several other mining compa-nies to develop autonomous technology for
mining applications.

In a relatively short period of time, ASI
has automated 50 different kinds of vehi-cles—not 50 vehicles, 50 different kinds
of vehicles. The company has a good
understanding of what it takes to automate
a vehicle and integrate it into a process.
Most of the work involves surface applica-tions, using GPS-based positioning for nav-igation. The company has some experience
with factory and underground applications,
or what is referred to as GPS-denied posi-tioning. That knowledge is helpful in
addressing deficiencies with GPS.

There are two approaches to introduc-ing autonomous technology in the mine
environment, an all-or-nothing grand vision
approach, where the mine goes from
manned-vehicles to a fully-automated
unmanned fleet in one step, or a moderat-ed user-assisted approach. The user-assist-ed approach has been implemented fairly
successfully in the agriculture industry for
the last several years. Before a mine makes
that decision though, understanding
autonomous vehicles, how they work and
robotics, sheds light on the technology’s
strengths and weaknesses.

Who is looking at robotics? The interest
seems to fall into two camps, which are not
mutually exclusive, but the interest tends
to be dominated by one or the other. Many
major mining companies are considering
autonomous mining to improve efficiency
and productivity. They have a long range
vision and they are expecting to make an
investment to significantly change the min-ing process. These mining companies may
also have “excessive manpower problems,”
or they might operate in remote or unpleas-ant locations or both.

Then, there are also those mine opera-tors with immediate, non-routine safety
problems, such as geotechnical issues. A
highwall or bench failure would be a good
example, where the mine has been told by
regulators or an insurer that they cannot
operate in a certain area and they need an
unmanned vehicle to perform the work.
These companies have safety concerns and
are not necessarily looking to make an
investment. They are not expecting to
improve productivity. If anything, they are
expecting to take a hit on productivity by
using a remote control machine to deal
with a circumstance.

Automation can be used to improve the
mining process. The technology could
allow equipment to work through
breaks/shift changes, to mitigate the
effects of weather or fatigue, and to elimi-nate mistakes by improving the accuracy
and repeatability. Accurately repeating
processes is an important aspect for min-ers. Haul trucks could be spotted correctly
every time (dumping and loading). They
would travel at the right speed all of the
time. They would place materials in the
appropriate place consistently. The
machines would move/rip as much rock as
possible. The system could maximize hole
drilling speed and accuracy.

By being able to improve all of the
pieces in the process, ultimately with the
grand vision of a fully autonomous mine,
engineers begin to redesign the mine and
take a whole new approach.

The all-or-nothing approach is the
approach currently being embraced by the
OEMs. It is a viable option that requires a
significant investment in one single inte-grated solution from one OEM. The poten-tial upside is obvious. This approach will
work best for new mines in remote loca-tions. Aside from the big investment, the
downside would be the reliability risks
associated with a big process change and
working with a single supplier.

Miners could also opt for an incremen-tal approach. They could begin with a user-assisted approach before transitioning to
full autonomy over time. John Deere had an
all-or-nothing vision. They were going to go
from where they were to unmanned farming
in one step. That hasn’t happened in the
last 10 years. Instead, they have taken an
incremental approach where they intro-duced user-assisted systems. Year-by-year, the user-assisted systems took more control
of the functions of the vehicles until ulti-mately farmers have unmanned agriculture.

What is an Autonomous Vehicle?
To truly understand the grand approach as
well as the incremental approach, users
must first get a feel for how the technology
works. Most people have some hazy notion
that it has something to do with GPS. As
the technology on an unmanned vehicle
matures to an autonomous state with see-and-avoid behavior, it progresses through
several stages. It starts with remote control,
which electronically controls all the degrees
of freedom on the vehicle. The operators
have a controller in their hands and they are
looking at the vehicle. The next level is tele-remote operation, the distinction is that the
operators can’t see the vehicle so they are
dependent on a video feed. For the next
level, autonomous (blind), GPS and map-ping software give the machine the ability to
execute a sequence of paths and actions
commanded by the user, but it’s blind. If
there is an obstacle in the way, the vehicle
will plow into it or over it in the case of a
haul truck. This would be the lowest level of
autonomy. Vehicles can perceive their envi-ronment using an array of sensors. The
sophistication varies from see-and-stop
behavior, where a vehicle has a sensor hori-zon that tells it there is something there, but
not enough sensors to tell it how to safely
plan around it, to see-and-avoid behavior.

This Cat 793 haul truck is equipped with autonomous technology for hauling ore.

The first step in automating a vehicle is
to convert it to drive-by-wire by adding actu-ators and hydraulic controls. Once it is drive-by-wire enabled, computers can talk to it.
Most vehicles are not equipped with drive-by-wire controls. The drive-by-wire system
ties into a controller, which acts as the brain
for the vehicle. The controller is getting data
from position sensors, which are primarily
GPS-based, telling the vehicle where it is
located. It also receives data from obstacle
detection (OD) sensors. It knows where it is
and what’s around it. The operator uses that
information to control the vehicle, commu-nicating with the base station command-and-control software over a radio network.
The video system operates in parallel.

The most vehicle-specific aspect of
automating the vehicle is the drive-by–wire
system. When ASI is approached with a
new vehicle, the engineers determine what
devices will be used to control the vehicle.
Everything else—all of the other black
boxes—remain the same from vehicle to
vehicle. To a robotics engineer involved in
autonomous systems, a utility vehicle is no
different than a haul truck even though the
vehicles are worlds apart as far as size and
weight. Typically, the drive-by-wire package
controls the steering, brakes, throttle,
transmission, attachments and ancillary
functions (horn, lights, etc.). It receives
data from OEM systems (speeds, engine
RPM, health, etc.) The goal in designing a
good drive-by-wire system is to minimize
the impact on the base platform and
always allow a human to easily take control
and operate the vehicle.

The division of labor between the com-mand-and-control software and the vehicle
controller is straightforward. On the com-mand-and-control side, the software re-ceives high level goals from the user, e.g.,
go from point A to point B and do some-thing. It’s creating a global path plan to
execute the goal taking into account the
map and the vehicle’s driving capabilities.
It sends the plan either all at once or in
chunks at a time over the radio network to
a vehicle control unit (VCU). On the VCU
locally, the controller takes the goal and
does all of the low level computing to deter-mine how it has to manipulate the throttle,
steering and transmission to execute the
goal. The VCU closes control loops on the
vehicle. It performs onboard safety checks
for off-path error, over-speed, over RPM,
steering limits, redundant sensing, etc.

GPS is not perfect and it requires fil-tering and augmentation, especially in
robust environments such as mining. It
fits hand-in-hand with good map data.
Adding perception and intelligence to the
system mitigates the need for high accu-racy GPS and/or map data. An inertial
measurement device, which calculates
with a six-axis set of accelerometers and
gyros, tells the system how the vehicle is
moving. With vehicle odometry, the system
knows the speed of the vehicle and the
angle of the steering wheel. If it loses
GPS, the system can dead reckon where
the vehicle should be until it gets a new
GPS point. A variety of other sensors per-form tasks of similar complexity.

For GPS-denied situations, infrastruc-ture-based solutions, such as reflectors
and RFID tags, can be positioned through-out the pit. Another technology currently
under development is vision- or laser-based
recognition, where the system recognizes
landmarks. The military is interested in
this technology because of the threat that
GPS could go away. That is the direction
the industry will move toward for
unmanned robotic systems.

Obstacle Detection and Avoidance
ASI uses three types of sensors for obsta-cle detection: lasers, vision and radar. They
all have strengths and weaknesses.

What
stays the same is the obstacle detection
processing software, which takes the data
from whatever suite of OD sensors make
sense for a given application and converts
that into a traversable map for the vehicle,
locations it can and cannot drive. The spe-cific selection of sensors depends on a
variety of factors based on definition. Would the vehicle be looking for other vehi-cles, people or debris on the road? The
vehicle’s speed and the look-ahead dis-tance would also be a consideration for
safety purposes. Environmental factors,
such as rain, snow, dust and fog are also
considerations. The sensors are fairly
expensive, so mining companies would
want to use as few of them as possible on
each vehicle.

On the command-and-control side, the
software should perform multi-vehicle coor-dination. The software has to be able to
take high-level goals from a dispatch sys-tem and coordinate the activities of multi-ple vehicles to achieve those goals. The
best situation avoids deadlocks or stale-mates along the route. Instead, the soft-ware prioritizes the vehicles and prevents
them from ever coming near each other.

A whole new set of safety issues arise
from autonomous mining. Does the mine
allow humans and manned vehicles to
interact with unmanned vehicles? How will-ing is the mine to rely on the OD system or
tagging process to govern safety? One
option is to use GPS-based safety perime-ters and “keep out” zones. That would
probably be impractical for haul trucks that
traverse the entire pit on haul roads shared
with other vehicles. There are also a couple
layers of safety measures built into the sys-tem, while it is working. As previously dis-cussed, long-range path planning for multi-ple vehicles prevents them from encounter-ing each other. If they do encounter anoth-er, then the system moves from a central-ized approach to a decentralized peer-to-peer approach where the vehicles can ulti-mately see each other. Even if they are not
communicating with each other or they are
not updating their position to a central con-troller, they can still see each other.

From a command center, one operator supervises multiple trucks. (Photo courtesy of Cat)

When the system isn’t working properly
for some reason, it needs to fail in a safe
manner. There are a lot of low level fea-tures that assist with the fail-safe system.
An autonomous system must have a soft-ware-independent, stored-energy emer-gency brake system. In the case of com-plete power loss, it will bring the vehicle to
a complete stop. There are several triggers,
such as communications loss, off-path
error, and system failure, which include
redundant sensing on vehicle controls.

If radio communications is lost, then
the big red button in the control room no
longer works. Theoretically, the vehicle can
run without radio communications. It could
get the entire mission from the global con-troller and then off it goes. It does not need
to talk to the global controller again. That’s
probably not the safest way to operate.
Typically, there is some kind of time-out on
the vehicle if it loses communications.

There are many questions surrounding
autonomous mining. At what point will the
technology be proven? There is a big dif-ference between seeing a demo and seeing
a system run 24-7, 365 days a year. How
would a mine recover from failure if it has
no operators? Reliability is obviously a
huge issue. How does the mine know an
OD system is safe enough? What type of
test(s) does it run? There is no industry
standard at this point. Can the system be
integrated into existing operations?

Incremental Options
Working with a user-assisted program, min-ers could move toward the big vision of
autonomous mining incrementally over
time. The 24-7, 365 performance goal
would be tabled for a while, but the mine
could implement the technology at existing
operations and retrofit the technology to
current fleets. Humans are still in the loop
and can be safely integrated into manned
operations. Some of the current user-assist-ed systems for mining include auto-spot
and auto-pilot solutions for haul trucks, col-lision alert-avoidance systems, remote con-trol/tele-remote systems with autonomous
assist and ripping path assist for dozers.

Many machines, such as drills, shovels
and dozers, have GPS and mapping tools
to assist the operator that are also tools
that could be used for autonomous opera-tions. The next move would be to shift up
to a drive-by-wire or steer-by-wire technolo-gy. The users would have the ability to
implement auto-spotting or auto-pilot.
When the truck driver gets close to the
shovel and they are able to communicate
with each other, the autonomous system
takes over and performs the spotting.
Similarly, when the driver enters a long-haul corridor, the auto-pilot takes over and
maintains optimal speed. Data from mine
operators indicates that average speed
tends to slowly decrease over the course of
a shift and they could theoretically gain a
measurable payback for that investment.

One of the problems with automating a
haul truck fleet is the number of vehicles,
which is a big investment. They cannot be
effectively isolated. There are sets of equip-ment that do operate in isolated zones, such
as drills and dozers. The mines could start
by automating multiple unmanned vehicles,
which do not have to interact with manned
vehicles. In certain situations, such as mul-tiple drills working one pattern or multiple
dozers ripping a leach pad, a single opera-tor could supervise two to five vehicles.
Eventually as the group (operators, produc-tion managers and engineers) become more
comfortable with the technology and equip-ment, it can be phased in over time.

Brown has been involved in the develop-ment of robotic systems for military, min-ing, industrial and agricultural applications
for more than 10 years. He is currently vice
president of business development for
Autonomous Solutions, a Utah-based
robotics company focused primarily on
converting off-the-shelf vehicles over to
autonomous operation. This article was
adapted from a presentation that Brown
made at E&MJ's Haulage and Loading con-ference during May 2011.As featured in Womp 2012 Vol 01 - www.womp-int.com