How Forza Uses Neural Networks To Evolve Its Racing AI

Today on War Stories, Ars Technica is joined by Dan Greenawalt, Creative Director of the Forza franchise, who takes us through the colossal potential (and many challenges) of utilizing deep neural networks to build sophisticated racing AI.

Beginning with the original Drivatar AI in Forza Motorsport for XBOX in 2005, Greenawalt and his colleagues experimented with computer learning, refining their methods from iteration to iteration.

It was once the AI system was unshackled from local hard drives and placed online in Forza Motorsport 5, however, that the floodgates truly opened.

As the AI rapidly evolved, producing unexpected new behaviors, the Forza team quickly learned how challenging it would be to wield this two-edged sword in creating realistic opponents.

[car engine roaring][upbeat music]- A car is very hard to control.Like it is a very complex systemand in a simulation engine,it has loop upon loop upon loop.A tire itself is a spring.Then there's the springsof the suspension.The car has moments of inertia and weightand the way the car moves andall of these things interact.The car gets heat and wind resistance.It's so complex.And the AI has to be able to predict thatI'm Dan Greenwalt, Creative Directoron the Forza of franchise.And this is how we've evolvedto take better advantageof deep neural networks to make AI.[car engine roaring][soft music]I had a pretty atypicalbackground coming into games.I was a martial arts instructor,but it turns out that martialarts does not necessarily paythe bills that great.So I started looking forjobs and a friend of mine.My roommate actually saidthat there were hiringat Microsoft to test games.I loved games.I like working on games.So I started testing and Ireally, really enjoyed the work.I enjoyed the teams.I liked how it was so collaborativeafter working on a bunch ofsports titles NBA inside drive,and then some racingtitles like Midtown Madnessand Motorcross Madnessand Monster Truck Madness,and a whole bunch of madness titles.Eventually the original Xboxwas announced internally.And when some jobs openedup to go full time,I started working full timeand immediately startedworking bizarre creationson Project Gotham Racing.I ended up moving to the UK,lived there for a while andreally enjoyed the work.They ended up giving mecredit as a designer,which was not something I asked for,but I was incredibly honoredand humbled that they did that.They were a great team.I learned so much workingwith Martin Schenleyand the guys from bizarre.[car roaring]So we founded turn 10, about 2001.The original Xbox had beenout for a little while.The group believed thatwe needed somethingto compete with GranTurismo on Sony's box.You know, Gran Turismo was a juggernautand that really cleared the business caseand put the onus ontothis very small team.There were four of us to comeup with a creative pitch.We got together and put thispitch together for Forza.And it was, it was a new take.We were looking at theracing space differently.We're embracing new trends,web 2.0 customization UGC.That was something we thoughtwas really gonna be important.And it worked so well with car culture.You know, I lived in car culture.I love cars.I tuned cars.I raced cars.I loved that stuff.And so did other people on the team.So that idea of customizationand building your carinstead of just buyingwas something we just,we loved Forza obviouslyit's a racing game.It's in the car genre.I think different players are lookingfor different things in Forza.When you think about motor sport,they're looking for simulation racing,at 10 tents competition.It's being under threat.It's driving a car underduress it's mastery,horizon is about exploration.It's about hanging out with your friends.It's about finding newplaces, freedom, fun,jumping the car around,having a great time.The first forts of motorsport, the original,the shift in 2005, itstarted on the original Xbox.So we made a prototypeand that was a trackthat it was stitching togethera whole bunch of different ideas.And we built our physicsengine and quickly realizedthat our physics engineneeded some assists on top.And at that point we werefaced with a fork in the road.We could either layer assistson top that you could turn offor we could change the foundational layerof the physics engine itself.And the engine would become more "Arkady."We immediately went to this ideathat the physics engine was sacredand it has to be cuttingedge at all times.And the layers we add on top are thingsyou can enable or disable that makeit more controllable the controllerthat first prototypethe physics engine wasso hard to control the controllercause you were giving a 100%input left a 100% input right.You could get on the throttleoff the throttle immediately,and anyone who's played racinggames, you watch their hands.That's how most players play.It's incredibly digital,even though they have analog sticks.So we need to put assists on and buffersthat would change your input.They would kind of interpolate and guess,what were you really trying to do?That doesn't happen in a real car,we didn't add that to the physics layer.We put that on top.So sometimes the inspiration is new tech.Like the original Xbox had a hard driveand it had a NIC cardand that allowed us tocreate new experiences.So we were inspired bythings that would use that,that actually birth Drivatarbecause drive Drivatar to are neededto create big data caches.And you really couldn't have donethat without a hard drive.Drivatar is a AI technologythat we worked on in theoriginal Forza Motorsportwith Microsoft research in Cambridgeand at start,it was a machine learningsystem on a hard drive usinga Bayesian Neural Network to record linesand characteristics ofhow somebody drove a car.And that's how it startedsince then Drivatar hasevolved considerablyas new technologies have come in,new ways of doing AIcome in and we've learnedso much from how our players playand all the data we getfrom how the communityin general has interacted with the system.So the Drivatar thatwe work on today bearsvery little resemblanceto the original one.So the original Forza Motorsport,there was this team in Microsoftresearch from Cambridge,and they had a thought that why is itthat Juan Pablo Montoyawho is an F1 driver at the timeand Michael Schumacherwho is another F1 driverat the time, could bothbe so incredibly fast,but they drive so differently.In fact, you could tellwhich driver was driving,which car based on how the car yardand how it drove in the rainand how strictly they went into corners.So they started looking atBayesian Neural Network,learning networks to learn a line basedon how somebody drove.So they would have peoplelike me drive a carin our physics engine andthat would give them a line.And then they would classify that lineand then generalize itto a new application.So they might take a given corner,that I was in and say,okay, we're gonna classifythat as a right angle corner.And you took it at 10 tenthsor eight tenths of the car's capability.You came in with this much yarn.Now we're gonna go to another trackthat you've never driven in another caryou've never driven andwe're going to replay that.But of course you can't replay itbecause it's got new variables.So it has to then use the neural networkto develop a new line.That was a really interesting challenge,but in some ways it wasa mismatch with Forza,which is about building cars,falling in love with cars,which means we have hundreds of carsand they all have upgrades.So that whole like howdifferent is one driverversus another in the same car,that's a level of subtletythat requires the cars to be same.We meet two people in carsthat are radically different,a car may have 400 more horsepoweror a thousand more pounds of weightthat really changes how a car drives.So from the very beginning,we had an interesting piece of technology,but it was at slight mismatchwith the goals of the game.Ever since then,we have encountered new mismatchesthat are for us to eitherdesign around as designersor implement around with technology.[soft music]It's hard to know even where to startwhen it comes to AI challenges.I think you've got to startwith what players expectout of an opponent in a racing game.What they love is seeingtheir friends namesand kind of a direz versionof what they think their players arealmost like a cartoonor a characterization.And we found that when we gavereally subtle representationsof how your friends drivepeople didn't even recognize it.They had to get kind of bombastic.They had to go beyondhow they really drive.And that creates unpredictable AI,which is one of theproblems players don't want.Meanwhile, you're meeting playerswhere their skill level is.If you're a very skilled player,you can lap a car incredibly quickly.So how do we keep the opponents groupedaround you or predictably around you?And what if you have acatastrophic mistake,you don't wanna necessarilyrestart the lap,but if you go off track,it can be quite a whileto get out of the sandand back on the track.Well now the opponents are long gone,so we've got a set of problems here,but underlying those problems area much deeper set of problems.That actually is wheremore of our technology is.So those are more of the design challengesthat our technology then looks to match.Part of those technological problems area car is very hard to control.Like it is a very complexsystem and we're amazing.I mean humans,we will throw a car into a cornerand just intuit that it's gonna be ableto take this corner at 60 miles an hourbecause I've done a couple of laps on it.It probably can go at 80,but we are hitting the limits of 80%of what the car's capableof really quickly.If you're experienced.That's amazing.That's a big challenge for AI.And then on top of that dealingwith us humans on the track.So the opponents, AIopponents have to dealwith this incrediblyunpredictable nature of a humanthat sometimes we'relooking for that simulation.We wanna take it seriously or you come in.And you're just trying to smash things up.Cause some players wanna do that.The opponent, the AI opponents haveto deal with all of that.So the set of problems we have to dealwith in AI is multifactor.It's designed challenges ofhow do we delight the player- [Instructor] Action[car engine roaring]- As technological challengesof how do we get this computer to dealwith the complexity ofthis whole situation?So in the Xbox one the original[chuckles]I still get that confusedon the original Xbox and the Xbox 360,our Drivatar system waslocal to the hard driveand on Forza Motorsport five,we switched that to being on servers.So these were Thunderheadservers for Xbox live.This was a pretty cool changeand brought in a wholenew set of problems.We had Drivatar fairlywell tamed on the local boxbecause it wasn't really being infectedby other people's data.It was really about local DrivatarWe trained and your own Drivatar,which dramatically reduced the amountof variables coming intothe user's experience.When we went to Thunderhead,all of a sudden it was amassive amount of data,which is awesome.It's great for training in neural netlike data is your friendwhen you're training, learning networks.However, I feel like we've had a tigerby the tail ever since then has beenthat your friends arecoming into your gameand their behaviors areshowing up in your game.So technologically, it was really cool.And from a user experience,it's had a couple of sparklersthat are incredible,great user experiences,but it's also had some irritantsthat we've had to look at.How do we shackle and how do we tameand how do we change the user experience?All of a sudden the Drivatar system wasn'tjust using your data.So you might train it with a hundred laps.That's pretty ambitious thatyou actually did a hundred lapsand various cars and various tracks.And that would train up thesystem for how you drive.And then we had trained thedata that was on the diskand was in the game based on, you know,hundreds of thousandsof our own laps right?All of a sudden going to Thunderhead,we had millions of laps within what?A week.It was crazy.And that allowed us to takea system that would classifyAnd that's what neural nets do right?They're classifying you thinkabout the joke in Silicon Valley, right?Not a hot dog.- That's that's it.- That's what AI does, it classifies.And the thing thatclassifies best is not Ait can tell you, it'snot this, it's not that.And the truth is that's magic.I know it sounds not veryambitious to most people,but it's pretty amazing.So all of a sudden we hadmillions of data pointsfor it to classify.So the AI got very fast.It covered weird and spots.It did things we didn't expect.And we had to start like clamping thatwe didn't want it to do this behavior.But the problem is,it's a bit of a black box.You don't know why it'sdoing the behavior.That's what AI is.You feed in data and it feedsout what it wants to do.It doesn't tell you why.it's like a two year old,you just kind of have to take the outputand then you can put a restrictor on itand have it do less of thator modify its output in some fashion.So the data was a huge boom.We started learningabout how players drive.We started getting a laps.We didn't expect things got fastin ways we didn't expectthey got slow in ways we didn't expect.And weird inaccuracyshowed up in our systemthat we really didn't expect.Now I'm gonna have to divesuper deep for a second.So the way that weclassify a track is usingwhat we call a Chi value.It's basically a value from zero to onewhere zero is the inside of the track.And one is the outside of the track.And when we record a line,it's a series of points.Now those points aresaid in that Chi value.So you might saythat if you are towardsthe outside of the track,you were at like 0.9 chi value.And if you're at the inside of a track,maybe you're at a 0.1,or maybe you're lowerthan that like a 0.01.So all the way around every single tracka track is expressed by these chi values,which gives you threelines an outside lineand inside line.And then in the middle line,the line that AI is going to driveor that a player recordedin these chi values.Here's the issue when youwanna classify and generalize,you're trying to say this corner is likethat corner and thatcorner is like this corner.And you're starting to stitchtogether virtual lines basedon a whole bunch of information.Now it's gonna look at the lines aheadand the lines behind andnew inaccuracies coming in,because the corkscrew at Laguna Saika isa very complex cornerand you can't really generalize itto any other corner in the world.And the system may tryand we don't have control over itcause the black box inaccuracy show up.Because if you think about a track,it's easy to conceptualize itas there's an outside and inside.But if you really think about a track,the outside and the inside move,a curbing shows up or doesn't show up,it gets a little bit ofextra concrete over here.And so all of a sudden thescale of that Chi value changesbecause the track's really wide.And when you go from section to section,it changes in width.So the chi value is still accurate,but you're getting little changesthat make it harder forthe system to deal with.What that would end up with is linesthat have an imperceptiblesmall kink as they get replayedbecause they go from one track,which has no curbing toattract that has curbingor whatever, right?There's a whole host of these thingsthat can make a difference.While the line follower sees the kinkand says, "how am I gonna make that?Clearly I have to hit the brakes."And so we got this breaking behaviorthat showed up out of nowhere.So what do you do?You say, well, Hey, don't hit the brakesso sensitively kind ofoverlook some of that stuff,but now you've actuallytold the controller,"Hey, ignore some data."Sometimes you do need to hit the brakes,but this time we'retelling you not to do it.So all of a sudden we'reputting programmingon top of the black box, whichis more like traditional AI,we are actually programmingif this, then that.So we kind of got a hybrid nowwhere we've got a blackbox training some data,and then this thing on top of it,meanwhile, we wanted tokeep some of the behavioryour player did.So we wanted you to make contact.If you were a player that made contactplayers don't alwaysappreciate having contact madewith them.So we then limited thatto only your friends.So strangers Drivatars weren'tallowed to make contact,even if they wanted to.So this is yet again,another limitation putonto this set of data.The sparklers that happenedwere water cooler talk.We'd get people coming around and saying,"Hey, I drove against yourDrivatar and it was doing this.Or it was doing that.It was really fast.It was really slow."And it kind of reminds you of themwhen you play in the game.These were awesome experiences,really delightful for our players,but they came along with a bag of behaviorlike brake checking thatnobody wanted to seeor getting pit maneuvered in a corner.Here's another complexitywatching all of the other carsaround you is very complex.In fact, most playershave very poor perceptionof the cars around them.If they're not on screen in front of them,they don't realize there's a carin the rear quarter panel.Even real drivers struggle with that.Great drivers have almost like radar.They know where everybody is around them.Just like a great soccer player,a great basketball player.Now give that to an AIwhere we have discussions of performanceand we can't all of a suddenhave it go exponentiallythat every single opponent istracking 16 other opponents.So we have to limit that.So for the most part, ourAI is only watching one car.And if it's not you,it might run into youwithout even knowing.In fact it might even hit youand be continuing to hit and not knowthat it's hitting you at all.When we get more performance,we're able to apply that performanceto watching more things.But it's just been one thing after anotherone way you can makethings more efficient isyou can run something offline,maybe on a server, or it couldbe run during load times.So that is when the game is loading,but before you're on track,we could run somethingand cache a table full of information.So for example, there's aBugatti Veyron on track.It's a heavy car.It's incredibly powerful.It's a tricky car for AI to drivebecause its limits are ridiculous.It's so heavy.It is so performantWe could run that trackthrough a lightweightsimulation during load timeand then cache the resultsso that when the AI has to look up,how fast can I take this corner?Oh, through my simulation,I can take this at 60.So I should throw myself at it at 60.I have to break at this point to hit that,that allows us to run more AI on the trackbecause we're now more performant.So every one of these carscould have look up to.Now, if our cars were all the same,then we wouldn't need as much.But the truth is it's not just a Veyron.It could be a Veyron thatyou've thrown upgrades onmaybe racing tires or downforceor some other piece of technology.So we have to run in load timebecause the number ofpermutations across 700 carsand all of those differentupgrades is innumerable.We could not cache that amount,even if we did it on a server.So we have to do that in load time.And we choose to do that and load timeso that we're moreperformant in real time.But now there's other AI on the trackand maybe there's a puddleor maybe there's rain,or maybe there's a human on the track.How am I gonna deal with this?Should I pitch, shouldI not pit am I damaged?I mean, there's a whole lot of thingsthat are part of what we call race craft.Now, each one of these havetheir own set of problems.Places where you can applylearning neural networks,deep neural networks,places where you'rebetter off with cachingand then places whereyou look to simplify.And we've got numerous placesthat we have looked atsimplifying over the years.And honestly when I say,when we get more powerfulboxes we get greedy.That's what we remove.We remove simplificationson things like the AIor the physics engine itself.We will do new research projectsto find new complexitiesand real-world physics andintegrate them into our game.And then in low time,real time or even offline,we will use caching techniquesto reduce some of that complexity.So we can run at 60 frames per second.So in most racing games.There's this conceptcalled rubber banding.It's often highly maligned by players,but I'm gonna try and impartiallyexplain why it exists.And then I can tell you how we use it.So in racing, you don'trestart very often, right?So you start a racelet's say it's five minute long.You run off the track andgetting back on track isa little bit difficult so as a result,you fall way behind the opponent players.We needed a way for you tocatch up to those playersor else you feel like all is lost.And we have a rewindsystem in Forza Motorsportand Forza horizon,which also helps take care of this.So a catastrophic failure,you can actually rewind and redo it,Drivatar and rewind help usbecause we've got amyriad of lines to follow.And we have the ability for you to fixyour catastrophic mistake,but you still end up with wantingto have more pack around you.It's just more entertaining racing.So we've looked at a couple different waysof doing rubber band.We don't change theopponents characteristics,but if they're far ahead of you,we start scaling back their torque.We start scaling down their friction.We start adding weight.We play with aerodynamics.The reason we have to playwith so many parameters,because if you play withany one of them too much,it does crazy stuff, right?You can't double the torque.The tires will just spin.You can't do too much air frictionor car slows to almost a crawlin like 160 miles per hour.So we've tried to be verysparing of where we use it.Because if you come incontact with the car that has,let's say 20, 30% morefriction in its tires,it's not gonna look right.It's not gonna feel rightwhen you collide them.Cause they have more grip.I think the reason it'smaligned by players iswhen it's put on really heavyand you feel like youare gonna win the raceand on the final 500 meters,some car comes whizzing up from behind youand pass you at the last second.That is really irritating.That's something we've avoided,but there's a connectionbetween rubber bandingand the inaccuracies that havecome out of our black box.We have a setting in Motorsport,it's in horizon as well called unbeatableunbeatable AI be is in fact beatableall of our professional eSportsplayers toy with that AI,they are so fast and they are so goodthat they can beat thatAI without problem,we could not make it fast enoughbecause of these kinks in the Chi value,it just would break itself a little bit.It would sort of stop occasionally.The other issue is the line followerSo it sees, it wants to followthis line at a certain speed,but our line follower was digital.So that would mean it would full breakor it would full throttle,creating a controllerthat intuitively knowsexactly how much throttleand how much breakto give on a very complexsystem is pretty hard.So we were doing it that way.And then we would letthe AI use our abs systemand our traction control systemto limit how much power it got.But that's not the fastest way of drivingour pro drivers myself evenI don't drive with traction control.Often it don't drive with abs,which means I get more out of the tires,more out of the car, whichmeans I'm fast with that car.The AI is already driving withone hand tied behind his backby using these assistsbecause it's so binary.So we ended up havingto use rubber banding,not just in the pocketwhen it's around you,but actually to close the gap to drivers,not even pro drivers, but drivers like mesince then, we've done aton of experimentation.This is what's so exciting.We've been experimentingwith a new controller.That's able to give analog andwe've turned off the assists.So the AI now drives itcan do modulator throttle.It can do modulator braking.That's awesome.We also got rid of the Chi value errorby going to a new systemof classifying the track,basically as a result, her test opponentsand our test bed havegotten so crazy fast.It's ridiculous.Our hope is to get rid of rubber bandingfor difficulty entirely,have it completely off.And the only reason there'dbe any rubber bandingever would be outside of the pocketwhen you've had a catastrophic failureand we don't wanna stop the opponents.We never do that.We wanna slow them down a bit.That would be a great use of rubber band.So we have organized a team.That's just been doingexperiments on Drivataras well as on neural networksand how they could be appliedto different places in our game,our build system, ourtools all over the place.And then we see the resultsof how much faster the opponents getwith and without cheating.What about a new controller?What if we cache do things in real timeand they just try new experiments.We see what works.We design something.We think players are gonna react this wayand it turns out they react that way.[car engine roaring][player cheering]You gotta be really carefulwhen you ask players what they want.And even when you ask them what they did.And we found that over the yearsthat players have told usthey use rewind x amount.And then we look at the dataand they actually use it sometimes 30,sometimes 50, sometimes a 100%more than they remembered using it.The feature was incrediblywell used by people,but it wasn't always well loved.It broke up the action.It caused problems withour line recording for AI.So that was one of those features.We had to look at what a players perceiveversus what a players actually do.Same thing with the self reportingfrom players on how rough the AI was.We had to look at what wasreally causing the AIopponent to run into themwas it that the AI didn't see them?Which is a problem.Was it, the AI didn'thave control of the car?Because it was being toobinary with its inputs.Or was it that the AI thoughtI should hit this personbecause that's what I wastrained to do by my Drivatar?Well, we can take care of the third one.We can, you know, get rid of that,but we needed to do radical experimentsto figure out how to getthe car to control betterand how to give it better vision.- [Commentator] Oh yeah, drivers comingfrom everywhere now its a party- We've constantly hadto grapple and wrestlewith what players perceiveand what they expect andwhat we're capable of.I love what we figuredout over the years though.I think we've made amuch more robust system.[sheep bleats]And in I think it wasForza Motorsports six,we added a parameterwhere you could toggle onor off AI aggression, juston your friend opponent.Yes, we found playersthat really loved it.They loved having the rougher raceand they loved having the reminder ofOh yeah, that drives like Johnversus I need a predictable fieldthat never hits me.The biggest thing we're tackling iswe've been improving our controller.don't all the way breakor all the way accelerateand turning off those assists,the Drivatar and the actually AI is ableto control the car ina very nuanced fashionthe player can a more complexsimulation of the car headinginto our caching of that player dataso that when the AI istrying to play the car back,it's trying to control that car.It has a much better intuitionof if I brake the car isgonna slow down at this ratejudging what is a collisionand what's not a collisionand whose fault it is.That's a great thing for deep neural nets.And we've been applying that.The thing that's impressed me, the mostthat we've been playing with isby turning off all theassists, all of the scalers,any sort of cheats or anything like that,but improving the line value.The AI has gotten so fast.So it destroys our unbeatable setting,which was the one that Icould sometimes competewith in the unbeatable settings cheating.And now it's being destroyedby this car controllerand fixed line follower.These are just cool thingsto see happen in technology,and so we're doing that withthe research team in Redmond,and we're just playingaround with this stuffand seeing where it goesand where it's useful.We started looking at neuralnetworks pretty early.it was part of the originalXbox and the hard drive 2005.I mean, there weren't alot of game teams lookingat neural networks as a result.We maybe we got jaded.Maybe we got experienced.I don't know how you want to say it.But when all of a suddeneveryone was talkingabout solve everything with AI,we had a very realistic eye on that.There are some things AI and DNN,Deep Neural Networks areincredibly powerful for,but they're not like peanut butter.They're not good on everything, right?You got to decide where you put them.So a lot of the learninghas been the right amountof experimentation to makethese features come to life.AI doesn't just have tobe applied to opponents.It can be applied to build systems.It can be applied to optimization.It can be applied to awhole lot of different areasof your pipeline and your experience.So I think a lot of the learningwe've had is going really eyes wide openinto deep neural networks.It's a black box.You put in a ton of data,but what you're gonna getout of that is gonna bea bit of a black box.You've got to take what you get.And then you're gonna have to add scalersand things on top.So it's not this like onesize fits all solution.You're kind of doing a deal with the deviland it's powerful.It's powerful,but you got to know what you're doing,but it's just been super interesting.We learned so much about ourplayers and what they love.We learned more aboutracing in the real world.And on top of that,we learned more about what ourtechnology can and can't do.And what's really magic about it.