The 2015 DARPA Robotics Challenge Finals - What Was It?

On June 5th and 6th, 2015, the Finals of the DARPA Robots Challenge were held at the Pomona Fairplex in Pomona, California. This was the culmination a three-year competition in which international teams developed hardware and software to complete a series of human-world tasks in a disaster/rescue environment. Organized and funded by the United States’ Defense Advanced Research Projects Agency, the competition was inspired by the aftermath of the 2011 Fukishima Daiichi nuclear powerplant disaster, and was intended to spur development of tools to aid in crisis events which present high-risk to humans. (When existing robots were deployed and utilized to evaluate the extremely dangerous areas of Daiichi site, it became evident that many critically-useful tasks could not be performed by existing robots.)

In previous years, DARPA held the DARPA Grand Challenge and the DARPA Urban Challenge, which pitted competitors against each other and against off- and on-road obstacle courses with motorized vehicles (which were mostly modified existing motor vehicles originally built for humans).

DARPA Robotics Challenge website

We attended the last day of the Robotics Challenge Finals, and what we witnessed was surprising and impressive, if not for obvious reasons.

Admission to the event was free. Trying to manage our time, we spent little time perusing the impressive amount of static exhibits laid out on the tarmac outside the Fairplex’s horse-racing stadium. Once we saw what was actually happening inside the stadium, we chose to prioritize watching the active competition.

I think it was better than I was relatively unprepared about the event. I’d only very briefly skimmed some documentation about the event - enough to know that a parameter of the competition was “intermittent periods of wireless connectivity” between the human controllers and the robots. This was meant to represent real-world conditions in which robots on disaster site might lose communications contact because of structural impedance or because of radio-frequency interference from rescue operations or equipment malfunctions.

I think my wife Joni and I were both surprised at the scale of the presentation, and the polish of the event. It was extremely well produced. Crowds were well-managed. The event’s graphics were well-placed and informative. Extremely elaborate multi-camera video and professional announcing and field reporting work provided a polished video feed to attendees and Internet viewers. Food and facilities (an easy call for the Pomona Fairplex, which hosts many large events annually, including the month-long, 1.5M visitor Los Angeles County Fair) were very good. I remind you that this was an exhibition of the Defense Advanced Research Projects Agency, the part of the United States military complex which a half-century ago created the embryo of what would become the Internet, and also any number of known and unknown defensive and offensive military technology projects. It’s a government project, and that’s not often a recipe for a Good Time.

In the huge covered grandstands at the Fairplex’ horse-racing track, we found seats in the bleachers, and took in the setting: In front of a crowd of a couple of thousand spectators, five stages were constructed. Flanking a center stage for human presentations, four identical three-walled “sets” were constructed (with an open side theatrically facing the audience) as obstacle courses. The robots were presented with eight tasks working in an environment with human ergonomics, and could score a single point for each challenge for a maximum of eight points. The total time required to finish the (optional) number of tasks they chose to complete was recorded. Final places were based upon total points achieved and lowest time required, and the top three finishing place teams were awarded $2 million, $1 million, and $500 thousand respectively. It should be noted that (as with the previous Grand and Urban Challenges) the costs of developing, purchasing and constructing the robots for several of the teams exceeded the the $2M first-place prize (I think I heard a $1M price tag quoted for one of the 3rd-party robots used by several teams), so for many teams, prize money was not a goal in and of itself. As with previous DARPA Challenges, teams were created and sponsored by commercial industry, government agencies, and educational institutions. Expense, creativity and innovation varied accordingly. The twenty-three teams featured competitors from six countries: Japan, Germany, Italy, Republic of Korea, China and the United States.

Multiple tracks defined teams which received some or no funding from DARPA.

Different from previous DARPA Challenges, teams were not required to design their own robots. Several teams chose to acquire existing robots from other enterprises, and modified or designed their own software for autonomous and human control. Among the 3rd-party robots purchased by teams, the most visibly recognizable were the six Atlas robots from Boston Dynamics. Many readers will know Boston Dynamics from YouTube videos of their “Big Dog” quadrupedal walking robot. (Boston Dynamics was purchased by Google in December 2013.) Another popular choice for teams not developing their own robots was the Robotis Thormang.

During the 2013 DRC Trials, robots were allowed to be mechanically tethered to protect them against damage during a fall, and were allowed to use an umbilical cable to provide external power and data connection. For these Finals, robots were disallowed from either of those aids. They would have to carry all the power required to operate for the 60 minute maximum task period onboard, and would have no safety system in the event of a fall (the final Task is a climb of 4 steps, and some of these mostly top-heavy robots weigh in excess of 180 kg/396 lbs, so they can really do themselves some damage). Most, but not all of the robots are bipedal walking robots, primarily because of the assumption that they would be the best solution for navigating in spaces and over obstacles designed for human beings. There are notable exceptions, like Team Aero’s Aero DRC, with four legs which also have wheels, and NASA Jet Propulsion Labs’ aptly-named RoboSimian, which contorts its four long multi-jointed appendages not unlike a tree-dwelling monkey to whatever configuration is appropriate for completing the current task safely.

Highlights from the 2013 DARPA Robotics Challenge Trials, showing safety-tethered and umbilical-connected robots attempting tasks.

Perhaps the most challenging change between the 2013 Trials and the Finals regards remote control and autonomy. For the Trials, robots could be constantly controlled by a physical data connection - wires. In the Finals, not only would all communications between teams and robot be wirelessly communicated, but the communication links would be deliberately degraded intermittently by DARPA organizers to simulate real-world communications problems at rescue sites. To this end, the expectation was that teams would develop software running internally within the robots that allowed them to complete the competition tasks without constant human help. According to DARPA video team interviews, teams typically “drive” robots to the task area and then initiate an autonomous routine to complete the task. Using multiple camera vision and laser 3-dimensional scanners, the robots would have to identify targets: a doorway, a door handle, a valve wheel, handheld power tools, stair steps, a shape on a plywood wall to be cut around - and manipulate the items in their environment to complete the current task. I think that had teams opted NOT to develop autonomous software, they might still be able to attempt every task manually. But because they wouldn’t know when and for how long DARPA organizers would interrupt their communications connection, their hopes of either achieving competitive times or even completing the stage within the maximum 60 minute time were unlikely.

I’m not certain, but I believe that the human team members were only able to use their robots’ sensing mechanisms to make control decisions from their command centers. So even if they chose to manually perform some tasks, they depended upon having designed adequate sensing infrastructure to apprehend the robot’s environment. So if they had only supplied their robot with a single camera view, it might prove difficult to guide a manipulator with no reference as to depth. We saw brief glimpses of 3-dimensional visualizations of objects on team computer monitors which reveals the nature of the robots’ “vision” systems. Most, if not all of the robots use LIDAR laser scanning systems (often visible as spinning devices mounted high on the robot) to provide their robots with a 3-dimensional model of both its environment and itself - robots must “look” at their own manipulator arms and graspers to determine where they are relative to their environment. (Here’s is a sample of this footage in the middle of a 4-hour DARPA video of the event.)

It should be noted that not all robots and teams attempted all tasks. Teams were allowed to choose to bypass some tasks, if the best use of their available resources with respect to the defined tasks was to skip a potential point-earning task in exchange for much lower elapsed total time. This might completely eliminate significant complexity in the design of the robot, and for those teams not expecting to place first, might improve their finishing positions by avoiding robot- or time-killing tasks. (It’s worth noting that the top three finishers were also the only teams to earn the maximum eight points.)

Teams were allowed to declare a “reset” on any given task, which (I think) earned them a 10-minute time penalty but allowed them to attempt the earn that point again.

The Tasks

• Drive Task (1 point) -  the robot drives a utility vehicle over course of about 100 feet, with two traffic barriers creating a chicane about 2/3 of the way down the path. The robot is NOT responsible for getting into the vehicle - only to steer and operate the throttle pedal. Notably, several of the larger robots were “seated” in the passenger seat and operated the controls on the driver’s side - presumably because they would otherwise not clear the steering wheel. The track for this driving task ran the width of the fairground’s horse racing track, so the four competition courses covered a larger area than the simulated task rooms. This proved surprisingly difficult, and there were a lot of human team members pushing the vehicles back from interactions with barriers.

UNLV’s “Metal Rebel” completing the Drive Task successfully. 

(NOTE: You can view any of these videos full-screen by clicking on the button in the lower-right corner of the YouTube video windows.) 

• Egress Task (1 point) - the robot must exit the vehicle and move about six feet away across a goal line. Teams were allowed to quickly modify the vehicles (without tools) to assist their robots in this task. Some extended simple structures out of the sides of the vehicles to serve as “handrails,” or other temporary stabilization points, and one team threw out a box on ropes to serve as a helper step for their walking robot (not all robots walked, some had wheels and tracked belts). This Task was particularly risky, and I heard a comment somewhere that teams didn’t appreciate the jeopardy that this Task presented to their robots early in the 60-minute completion window. Crashing here (and we saw some big crashes stepping out of vehicles) could prevent a team from earning more than the single Drive Task point.

Here’s Worchester Polytechnic Institute’s and Carnegie Mellon University’s joint entry “Warner” rehearsing Egress the week before the event. If this seems slow, know that some robots took 10-15 minutes to complete one task - most of which was standing in one place and “thinking.” 

• Door Task (1 point) - the robot must open the door (with a lever-style handle), push the door open, and walk across the threshold to gain 1 point. Sounds simple, but we witnessed several failures - including big “crashes” - right here. Many ‘bots stood at the doorway for several minutes, apparently assessing the position of the door and handle.

Team NEDO-JSK’s “Jaxon” earns a point by walking through a doorway. There’s nothing like hearing thousands of people cheer because someone walks through a doorway. (I’d accidentally written that as “walks through a door,” which might have been more than a typo at this event.)

Tartan Rescue’s “CHIMP” demonstrates that its unique design allows it to recover from a crisis in the Door Task which would have been a failure for many other designs.

• Valve Task (1 point) - this task was specifically inspired by events in the wake of the Fukishima disaster. A wheel-type control valve handle must be rotated counterclockwise 360 degrees to earn a point. This is trickier than it sounds. When we do a task like this, we’re unconsciously shifting our weight on our feet to compensate for the forces we impart into the wheel.

 

UNLV’s Metal Rebel wins a point at the Valve Task.

• Wall Task (1 point) - in this impressive challenge, robots select from any of four handheld (human) cordless cutting tools on a wall shelf (most robots knocked off at least one other tool during the task - which was not penalized). They then must completely cut a painted shape out of a section of drywall. All robots simply dropped the tool on the floor upon completion of the task, revealing that there was no specification for the final disposition of the tool. Most robots managed this pretty well. A few missed the painted target slightly, cutting slightly through the circumference of the black circle. But all that we saw managed to take a second cut and earn the point.

See NASA JPL’s RoboSimian make the cut to a cheering crowd in the Wall Task.

• Surprise Task (1 point) - the “surprise” of this task is that DARPA didn’t inform teams what the task was until the day before the competition. However, I just found video of a team rehearsing what would be the Surprise Task back in March 2015, so I guess that suggests that it was a surprise choice from a known collection of tasks. The surprise task turned out to be unplugging an industrial electrical cord and plug from one outlet at “chest height,” and plugging into another. If we pulled a plug at this height, we would unconsciously have to shift our weight backward and forward to prevent from tipping as we pulled and pushed on the plug or cord. For some of the bipedal robots we watched, this tipping proved disastrous.

South Korean Team KAIST (who would go on to win the Finals and $2 million) practices the Plug Task months before the Finals.

• Rubble Task (1 point) - to complete this task, the robot must either cross over an uneven arrangement of cinder blocks (these were meticulously identical on each of the four competition stages), or negotiate an alternate obstacle: an assortment of loose “debris” on a smooth floor. They were allowed to simply pick up the debris, but no robot we saw attempted that strategy. Notably, we saw one robot simply push its way through the debris (each piece of debris weighs less than 5 pounds) to cross the yellow line earning it the point for the task, though it did risk catching both ends of some long pieces, which might have prevented it from continuing to roll across the smooth floor.

Team IHMC Robotics “Running Man” attempts unsuccessfully to negotiate the Rubble Task.

• Stairs Task (1 point) - for the last possible point-earning task, robots could climb four steps to a railed platform. There was a handrail on one side of the stairs. Strategies for climbing these stairs varied from the human-like approach of many frighteningly top-heavy bipedal designs to the monkey-like climb of NASA JPL’s RoboSimian. 

Team Tartan Rescue’s “CHIMP” uses the unique tracked belts on its four limbs to complete the final task for the maximum 8 points to win the $500K third-place prize.

“Running Man” shows us all just how NOT to climb the stairs. Despite this fall and the Rubble Task fall on Day 1 of Finals, Team IHMC affected repairs overnight and to compete on Day 2 and finished in 2nd place overall.

A full description of the eight robot tasks is here on DARPA’s overview page for the Robotics Challenge Finals.

Our Takeaway

We were delightfully surprised at the efforts of the Defense Advanced Research Projects Agency to share this project with taxpayers. The event was well run, and the presentation was watchable and captivating. Watching robots slowly attempting each task was thrilling to watch - the sense of risk was not dissimilar to that of an Olympic athlete risking a lifetime of devoted practice and development, only to lose it all in a single fall. Hearing a crowd of people cheering and "Oh!"ing in unison at almost imperceptible accomplishments of these sometimes lifelike machines was unexpected, and added considerably to the experience.

Having watched online videos of the progress of walking robots over the past decade, we were still impressed at the state of the art - no doubt advanced by this competition. It wasn't so much about locomotion, but the apparently autonomous operations. Ironically, because of pauses which sometimes lasted several minutes, it was apparent that we were seeing software routines attempting to interpret the conditions of the environment with which the robots were presented. And the success rate of the competing robots shows how those teams rose to the occasion.

Will tomorrow's robots be better-suited at helping humans to perform perilous tasks in the near future? Most certainly. Moreover, it's easy for us to imagine that in the very near future, we might be seeing machines capable of adapting and reacting to their environment in ways that resemble living animals.

I'm looking forward to it.

More DARPA Robotics Challenge Finals Videos

Time lapse of the winning 44 minute, 28 second, 8-point run of Team KAIST’s “DRC-HUBO,” earning the Daejeon, South Korean team a $2 million prize.

Here is a time lapse video of TEAM IHMC Robotics “Running Man” completing their entire 50 minute, 21 second, 8-point run for a $1 million 2nd-place prize.

Here is DARPA’s YouTube channel, on which they have posted an 8 and half hour video of Day 1’s competition, and almost 9.5 hours of Day 2 (your Tax Dollars at work).

The video directors of the DARPA Robotics Challenge (which was being streamed live to the Internet, as well as to the five huge video displays behind the competition stages) played this hilarious blooper reel from the previous day’s competition even while the last dozen teams were agonizingly tending their robot’s last-ever competition day. It seemed a little cruel to those teams who had suffered these indignities, but I don’t blame the organizers for trying to make the event entertaining - the audience (and we) were in rapt attention during the challenge stages, and oohed, ahhed, gasped, laughed and applauded with more enthusiasm than they might have at many human-based events.

Boston Dynamics demonstrated a couple of their "Spot" quadrupeds - stunning