Dude, Where's My Rocket Launcher?

Back in March I wrote about how the AI conceptualizes item placement. Rather than thinking of position in terms of raw coordinates, bots consider each item (or cluster of nearby items) to be a potential landmark. When statistical tracking is done for areas players are likely to be, for example, they are grouped by which item they are closest to. Statistical tracking is one of the most powerful tools available to AI if you have a sufficiently large and well organized data set. BrainWorks uses this data to analyze which areas of the level a player is most likely to be in, for example. Not only can this information help bots track down enemies; it also helps a hurt bot them avoid enemies. The more data it can track, the better. So by extension, the faster it can translate from raw coordinates to the nearest item, the better. However, this is not a simple request.

If a level has N items in it and a player is located at position (X, Y, Z), what's the fastest way to determine which item is closest to that player?

Well you can just check the distance to every item and use whichever item is closest. If you read last week's column, you'll remember that this search requires linear time. If you double the number of items on the level, it will take twice as long to find the closest. Certainly we can do better than linear time.

There are a number of data structures that can solve this problem. Some of them might require more setup time but give slightly faster search time. The structure I chose is an octree. It's a tree that holds some number of three dimensional locations. If you want to learn more, I recommend the Wikipedia article on quadtrees as well. A quadtree is the two dimensional analog of an octree, so it's a bit easier to visualize.

In the example I gave last week of a filing cabinet holding 100 pieces of paper, 1 cabinet held 10 folders and each folder held 10 papers. This is a total of 111 pieces of data stored (100 papers + 10 folders + 1 cabinet). In computer science terms, each data storage location in a tree is called a "node". The node where searching begins is called the "root node". In the filing cabinet example, the filing cabinet itself is the root node. The cabinet and folder nodes split 10 ways, and the pieces of paper do not contain any sub-nodes. Nodes that do not further branch are called "leaf nodes".

Getting back to the octree example, the BrainWorks AI code needs to store the spacial locations of 50 or even 200 items in a tree. Each node in the octree contains is one of the stored points, and the node branches 8 ways. Visually think of each node as dividing space into 8 octants-- upper north west, upper north east, upper south west, upper south east, and four more counterparts for lower sections. All points that are above, north, and west of the root node are stored underneath the upper north west node, and so on for all other octants.

For example, suppose there are 50 items on a level. One of them is stored in the root-- say it's a rocket launcher. Of the 49 remaining items, 6 of these item locations are above, north, and west of the rocket launcher. The root node will link to a node for one of these six items-- a megahealth. The other five items will be stored undernearth that megahealth.

Again, the octant division property applies to all nodes, not just the root. So the megahealth also divides its space into eight octants, and the five remaining items will be appropriately placed underneath it. Keep in mind that each of these items is guaranteed to be above, north, and west of the rocket launcher. But items that are above, north, and west of the megahealth will be stored in a different octant than those above, north, and east of the health. Given that there are eight possible areas and only five remaining items to store, chances are that these items will all be stored in one of the eight sub-nodes of the megahealth. (If two items happen to both be below, north, and east of the megahealth, then the first will be stored under the megahealth, and the second will be stored under the first.)

Searching a tree is generally proprortional to the depth of the tree-- the greatest distance from the root node to a leaf node. And for a tree of 50 items, the depth is usually 3 or 4. Going up to 100 items might push the depth to 5 for some level layouts. So the octree structure makes answering the question of "which item am I nearest?" much, much faster than checking all 50 or 100 items.

Given that items have fixed locations on the level, the octree needs to be created once when the level starts, but the tree will never modify during the game. (Ignore items that drop from dead players-- those are handled differently and not useful for statistical tracking anyway.) The actual algorithm is pretty simple, a bit slow, and very likely to produce a well balanced tree. Here's how it works.

Given a set of items:

• Average together all their positions to find the center
• Check each item to find which one is closest to the center
• Add that item as the dividing node
• Separate all remaining items into one of the eight octants this center divides
• Repeat the process on each octant that contains at least one item

By "not terribly fast", this algorithm could take up to half a second on a very, very large map. So still inconsequential, and well worth the one-time hit to produce a better tree.

Of course, there's still the question of how you search the tree to find the nearest item, and that I will be explaining next week. Here's some food for thought though: even if you identify which octant a point would reside in, you still need to search up to seven of the eight octants to guarantee you've found the closest item to your input point. Searching this tree is not done like most tree searches. For those of you who like thinking through these kinds of problems, try to figure out how the search algorithm actually works. I'll give the answer next week.

Given a set of item locations arranged in an octree and one player location, describe an algorithm to find the item nearest the player.

Good luck!

Sitting on the Fence

For roughly the past month, I've been interviewing at a few different places looking for a new job. I've felt overqualified at my current place of employment and really wanted some more challenging work. At the end of the process I had two offers to choose from, both of which were really similar and far superior to my current job. I suppose most people might decide based on some simple, simple heuristic like "highest salary", "most attractive coworkers", or "did I flip heads?"

But I'm a compulsive optimizer. Naturally I agonized for two weeks over which offer I should. I was hoping one offer would be clearly better than the other, but both the corporate cultures and salary packages were similar. I analyzed every angle I could think of, from the commute times to little things my interviewers had said. I called friends I knew and got as much inside information about the places. I made risk estimates for the companies based on their respective industries. Talk about overkill! My opinion changed on a daily basis, but after two weeks I finally made my final decision.

This is, of course, the exact opposite of what good AI should do.

In most systems where a computer must analyze some data and make a decision, there's a problem when two choices are very similar. Sometimes the system can get caught in an oscillation state where working on either choice makes the other one better. For example, a Quake 3 bot might decide to pick up armor, but the act of moving towards the armor makes it decide that picking up some health is a better choice. So the bot instead moves towards the health, and on the way decides the armor is better. The bot can end up in a situation of indecision where it picks up neither of the two items, and clearly this is the worst of the three choices.

Item pickup isn't the only situation where this occurs. Target selection and weapon selection can have the same problems. A bot can try to chase down one target but on the way decide it would rather aim at a different one and waste valuable time aiming at neither. Or in theory a bot can switch to one weapon and change its range based on the new choice, but by the time it gets in position it decides another weapon is useful. If you've never seen a BrainWorks bot do this, that's great! I've worked hard to remove these indecisions from the AI, but it's neigh impossible to make AI that works in all situations. There are two basic tactics you can use to solve indecision oscillations, each with their own costs.

Most of the time I give the scoring estimate a bonus of between 10% and 25% for keeping the same decision. In other words, the AI won't change its mind unless the new option is at least 25% better than the old. The higher this factor is, the fewer oscillations will occur. However, the risk is that an option that's genuinely 8% better will be missed, even if it wouldn't cause an indecision loop. Additionally, it's rare but still possible that even with a 25% factor, the bot can get stuck in indecision. This method is best used for situations where the estimated value of one option over another doesn't change that rapidly. This is how general item pickup oscillations are handled, as well as enemy selection.

When a small buffer isn't sufficient, the other option is to prevent the bot from changing its mind until it completes its selection, or at least for a reasonably long duration (such as half a minute). This is guaranteed to solve the problem. But there are two drawbacks. Not only will the bot be more likely to make suboptimal decisions, but it could get caught in a situation where it's very predictable. As such, it's best to apply this tactic only when absolutely necessary and ideally when the action can be completed quickly.

This method is used in item pickup for items the bot is very nearby. If a bot can pickup an item in less than one second, it immediately overrides all other item pickups and grabs it no matter how useful some other item may be. It turns out that even with a bonus factor for keeping the same item selection, bots could still get stuck in indecision loops when they were very close to an item of low value. That's because the estimated points per second is the ratio of two very small numbers, so the estimation has a large margin of error. No bonus factor could stop the bot from switching between the bad nearby item and the good item that was far away. The code forces the bot to spend half second to grab the nearby item, thereby preventing the loop from occurring.

Regarding my new job, I was leaning towards the place that offered slightly less money but seemed to have higher job satisfaction. In the end they decided to beat the salary offered by the other company, so it was a no-brainer to pick the job that has both higher pay and happier workers.

Mission Impossible

One of the basic realities of life is that if something is easy, it's already been done. In the grand scheme of things, all unsolved problems are hard problems. While that might not have been true 300 years ago, the advancements of widespread education and information technology have made it easy to find potentially talented individuals and make sure everyone knows about their work. The result of that for almost every technological problem you can think of, either someone has solved it already or it's very difficult.

So what do you do when you need to do something that hasn't been done before and it turns out the problem is very hard? How do you break an impossible problem into something solvable? While there's no right or wrong way to go about this, there are three basic approaches you can use. Assume that the objective of a perfect solution is either practically or literally impossible, and you just want the best solution that's actually feasible.

#1: Improve on an existing base

The concept here is to focus on the small, incremental progress you can achieve rather than the large, difficult end objective. This is the approach I took when solving the general high level goal selection in BrainWorks. Bots must constantly decide what they are most interested in doing. Should they track down a target or look for more players? In capture the flag, should they defend their flag or get the opponent's? I confess I've added nothing revolutionary to this logic, and the correct choice is something even experienced human players have trouble making. Instead I just added a few pieces of additional logic to help the bots decide. BrainWorks bots will consider tracking down a target it's recently heard but not seen, for example. It's not algorithmically amazing but it's progress and that's what matters.

#2: Break the problem into pieces and only do some of them

I used this approach when designing the item pickup code. Roughly stated, the implemented algorithm is to estimate how many points the bot would get if it were to pick up different items and it selects the item that gives the most points. While the item selection is reasonably good, there's a lot of corners cut for the purpose of reducing complexity. How valuable is a personal teleporter anyway? BrainWorks just values it at 0.6 points and calls it a day, but clearly such an item is far more useful in some situations than others. And the estimations of where enemies are likely to be are based on proximity to items, not on actual geographic locations (like being in a specific room or a hallway). The estimates aren't remotely perfect, but they are still good enough for the purpose they serve: a rough estimate of what item the bot should pick up next. Designing this code wasn't even possible without ignoring all the truly hard parts of item pickup.

#3: Accept defeat and deal with the consequences

This is essentially The Attitude of the Knife. Sometimes things just don't work out and can't be fixed, and all you can do is accept that reality and handle the consequences as best you can. This was the approach I took for the navigation code in BrainWorks. As I've mentioned before, the navigation tables have huge issues on anything other than a flat, two dimensional map. While I'd love to solve this problem and I'm certain I could do it, I just didn't have enough time to tackle it. There were a few clearly bad navigation decisions (such as jumping between ledges) that I wrote code to manually fix, but those were essentially stopgap measures.

While it's not ideal, there's nothing wrong with accepting defeat against hard problems. After all, you aren't the only person in the world who couldn't solve the problem.

The Power of Spite

In almost any situation where there are limited resources, people are presented with the option of spiteful action. These are actions where someone else is penalized for an action where you received no benefit. Most of the time the spiteful play even had a slight cost, such as additional time. In war, there can be spiteful military decisions, such as General Sherman's sowing of salt during his march to the sea during the American civil war. Sowing salt makes the land completely unfarmable for a period of several years, and Sherman's choice to do this crippled a large part of Georgia's agriculture.

Of course, one of the cardinal rules of war is "waste nothing" since resources are tight. While these spiteful actions cost Sherman's troops a little time and resources, they had a tremendously detrimental effect on the Southerners, both demoralizing them and making it harder for the local citizens to survive. Without commenting on the ethics of this decision, I will merely note that Sherman's spiteful actions gave an advantage to his own side, the North.

So in a first person shooter game like Quake 3, players have countless opportunities to make spiteful plays. Every time they pick up an item they don't need, they are potentially preventing another player from getting that item. A good AI player needs to know when it should pick up an item out of spite. And naturally it shouldn't pick up every item, or it would waste far too much time. There has to be a method for deciding when the spiteful play is worthwhile and when it's not.

In BrainWorks, the additional time cost is automatically factored into the item selection code, as extra time spent grabbing items means a loss of points from delaying the bot's main goal. But the bots still need to know exactly how much an item pickup costs the opponents. Once that value is known, the bot estimates its relative score gain as the spiteful point loss divided by the number of opponents. In other words, keeping an item from 1 of 2 opponents is a lot more valuable than keeping it from 1 of 10. The fewer opponents the bot has, the more they prefer spiteful pickups. So while these spite values don't matter much in large group games, they can have a significant impact on a bot's item pickup selection in 1-on-1 matchups.

This still begs the question of exactly how much a spiteful item pickup costs opponents. Players know that it's great to grab armor even when you're at the maximum and grabbing ammo will rarely have an effect. But BrainWorks bots need to derive this fact. The AI does that by predicting how an individual item pickup will increase a players score for each of three typical players. There's a freshly spawned player, who generally wants armor and weapons. There's a hurt player, who benefits the most from health. And there's a decked out player who probably wants powerups. The AI computes the scores for each player with and without each possible item. The item's spite value is defined as the greatest point gain it gives any type of player. The code uses the maximum estimate rather than the average because players that most need an item are the most likely to pick it up.

That means the spite value is really just an estimate of the item's general usefulness. In fact, the bots use these estimations to isolate the most valuable items on the level and time their respawns, and just as a general area that other players are likely to be in.

Making Things Worse

One of the major challenges of game AI is creating AI that's plays at an appropriate skill level for a range a players. Most games need AI in easy, medium, and hard settings so that as players become better, they can face more appropriate challenges. There are three typical methods for solving this problem.

• More enemies appear on higher skill levels
• Higher skill enemies have more health and deal more damage
• Higher skill enemies act more intelligently

The first two are obviously far easier to implement than the last, but more intelligent enemies gives more satisfying results. While additional enemies works fine for a single player first person shooter like Half Life, it's not an option for Quake 3, since the player selects the number of enemies when they pick the level. The second option, adding more health and damage, was exactly what iD software used for the two lowest skill bots in the original AI. While skills 3-5 gained better accuracy, skills 1 and 2 simply dealt less damage and died faster than a skill 3 bot.

Of course it's the last option, coding additional levels of intelligence, that's "real AI". The other methods have their place, generally when deadlines must be met. But the reason good human players beat bad human players at a game isn't that the bad players accidentally had the "deal less damage" option checked. They just get outsmarted. High skill AI needs to outsmart people more often than low skill AI does, and you don't encounter that kind of AI very often in computer gaming.

When designing BrainWorks, I was determined that all skill gradations would be created by additional intelligence, at least where applicable. So, yes, bots will have more accurate aiming as their skill increases. But they also gain abilities. Here are some of the abilities BrainWorks bots gain and their minimum skill levels. (Skill 1 is the lowest, 5 is the highest.)

• Strafe jumping: Skill 4
• Blast shots: Skill 3
• Item Respawn Timing: Skill 3 (1 item), 4 (2 items), and 5 (3 items)
• Simultaneous movement and attacking: Skill 2

That's right, a skill 1 bot stops moving whenever it starts shooting. There are also other abilities that gradually increase in effect as skill increases:

• Aiming speed
• Aiming error correction
• Audio detection of enemies
• Memory of enemies no longer in view

And more that I'm sure I've missed. To make realistic low skill bots, it's best to start with realistic high skill bots and then take away intelligence until the result seems appropriate for the intended human's skill level. AI designed the other way, by taking bad AI and giving it more health and damage, never quite feels realistic.

Turn Left at the Quad Damage

Have you ever gotten directions that included phrases like, "When you see the giant viking statue, go straight and curve left" or "If you pass a third Starbucks, you've gone too far"? People don't always give directions based on street names; sometimes the street names are largely irrelevant. Here in New England, if the street sign is even posted at all, it might not be readable until too late. Directions like "turn left at the Rite-Way Mart" are often easier to follow than "turn left at old route 293".

Believe it or not, bots have the same problem. We might conceptualize a level as rooms, hallways, and open areas, but these places have no names. Humans generally name these areas after a recognizable landmark. In Quake 3, that usually means items. You'd never tell a teammate, "I'm at (540, -973, 108)", which is how the computer records your position. But you would say, "I'm near the rocket launcher".

One requirement of item pickup that I glossed over in a previous post was this: Bots need to know how likely they are to encounter an enemy in a general area. That implies bots need a concept of what an area is. They need to think of enemies being "near the rocket launcher" not "at (540, -973, 108)".

Bots need to think of enemy location in human terms, not computer terms.

The obvious method of approaching this problem is to divide the level into regions, with each region being defined by one item. Technically each region is "the area of space that is closer to this item than any other item". Once you have the level conceptually divided into "near the red armor" and "near the plasma gun", whenever the bot sees an enemy it can say, "Ah, that player is near the red armor, not the plasma gun. I'm more likely to find enemies if I head for the red armor as well." Now whether or not it wants to enemies is a different question, one answered by item pickup. But at least it can track meaningful data about what other players on the level are likely to do.

However, just because it's possible to find out what item a player is nearest doesn't mean it's particularly fast or easy for the computer to determine that. The AI obviously can't spend half a second to calculate data it needs to save 10 times a second. It's okay to spend a bunch of time when the level loads to calculate the best set of regions, but once these calculations are done, bots need to do the location to region calculation very quickly.

I researched a number of different options, but eventually settled on the octree, briefly mentioned in a previous post. Conceptually an octree is a way to layout some number of points in 3-space. In BrainWorks, the item region octree contains the location of all items on the level. Rather than rewrite what has already been written on the subject, I recommend reading the wikipedia article on quadtrees (the two dimensional equivalent of octrees).

For the 99% of readers who don't care how to implement an octree, I'll just say that a well balanced octree will generally let you find which item is closest to any input location very quickly. The run time is O(log₈ N) time, where N is the number of items on the level. That means that finding the nearest item on a level with 64 items will take twice as long as a level with 8 items, and a level with 512 items will take 3 times as long as a level with 8 items. Since even the largest levels rarely have more than 100 items, this calculation is really fast, and that in turn frees up more processor time to think about different AI problems. The bot makes a richer impression to the human playing the game and hopefully the player has more fun too. Sometimes making a game fun to play involves implementing mind-numbingly boring data structures, but the fun just wouldn't be there if they were missing.

Good Enough For Government Work

This might be a surprise if you've read my post on pragmatism, The Attitude of the Knife, but I'm actually an idealist. I don't want things to be good enough; I want them to be perfect, or at least as perfect as pragmatically possible. I use the attitude of the knife to temper my idealism.

Of course, there is tension between idealism and pragmatism when encountering very difficult problems. I don't just mean, "you'll have to be really smart to solve this problem." I mean, "it's mathematically impossible to solve this problem using the computer you have access to." The idealist wants 100% and the pragmatist wants 99% or even just 90%.

The classic example of a mathematically difficult problem is the Traveling Salesman Problem. If you're not familiar with the problem, it works like this: Suppose you have a salesman who wants to visit every city in a certain area exactly once, and he knows the distance between each pair of cities. What's the minimum distance he'll have to travel to visit all the cities and how do you determine it?

Mathematicians have been studying this problem for almost 200 years now. It turns out that determining the perfect solution is really, really hard, but it's possible. Worst case you can just check every possible ordering of cities. That worst case is unbelievably bad, however. If the salesman had 30 cities to visit and a computer could test one trillion options per second (current no single computer exists that is this fast), testing all these options would take much longer than the current age of the universe. Most of the studying mathematicians have done on this problem revolves around ways to get a "pretty good" answer without spending billions of years to do so.

I say I'm an idealist, which means I want things to be perfect. But I'm also ruthlessly pragmatic. These contrasting ideals intersect on the definition of perfect. If it takes 1 trillion years to find the traveling salesman route that takes 38 days but 1 hour to find the route that takes 42 days, then the 38 day solution isn't perfect. It's really 1 trillion years plus 38 days, which is definitively worse than 42 days, 1 hour. A "perfect" solution must take into account the constraints surrounding the problem and not just the problem itself. The mathematically optimal solution is not the perfect solution expressly because it's not practical enough to use.

What does this have to do with Artificial Intelligence, and specifically item pickup? Deciding what items to pickup is very similar to solving the traveling salesman problem, and that's bad news for me as the AI designer! If there are 20 items on a level, a bot wants to pickup between 0 and 20 different items on the way to its final destination. That's similar to a traveling salesman wanting to visit exactly 20 different cities. And since BrainWorks bots can't spend several trillion years to decide what items they're going to pickup, their item pickup code cuts a lot of corners to get a pretty good solution in a reasonable amount of time. This isn't the mathematically optimal solution, but it is a better choice overall.

Here are some of the tricks BrainWorks uses to reduce the computation time:

• Nearby items are grouped into a single cluster. Bots consider picking up the entire cluster at once. This reduces the effective number of things to consider.
• Bots do not consider picking up more than four items at once before going to their final goal.
• Bots only consider picking up the dozen or so items that are relatively near their current position, or not far off the path to their destination.
• When the bot is very near an item (less than a second away), it automatically picks it up rather than doing a full computation.

These changes have a profound effect on the size of the search. Suppose a level with 40 items gets bound into 25 clusters and the bots never consider more than 4 total pickups from the 12 nearest items. Each item pickup decision require testing at most 800 options, rather than more possibilities than there are people on planet Earth.

It's rare that thinking 10 item pickups ahead would really help the bot more than the first two or three choices. And usually the bot doesn't want to travel halfway across the map to pickup some random item. Only the nearby items are worth considering. For the one or two items that really might be that good, the bot specifically notes these items and includes them in the list of possible pickups, no matter how far away they might be. And as it turns out, these results are still "very good". Often the theoretically best solution isn't the best for the problem at hand. If you idealize the results and not the method, being idealistic involves selecting the most pragmatic solution.

At the Heart of it All

As I outlined last week, the item pickup problem in First Person Shooter games is very difficult to solve, and that goes for humans as well. How do the truly advanced players approach this problem? At the heart of it all is one central question:

How will this benefit me?

If you want to know whether to pickup item A or B, you first must know how much item A benefits you, and how much it costs (in additional time to get the item). BrainWorks solves this problem using a customized resource prediction model, handled by ai_resource.c. Be warned, this code is not for the feign of heart. It's gratuitously well commented and documented, but there are so many bits and pieces it's very hard to get your head around everything at once. Think of it like opening the hood of your car, removing the engine block, and taking everything down to the nuts and bolts. Very interesting, but good luck assembling it into a working vehicle.

First, the AI's resource engine must encode the player's current state. This is handled by the resource_state_t structure (defined in ai_main.h). In brief, that information is:

• Health
• Armor
• Weapons
• Ammo per weapon
• Powerups (eg. Haste, Quad Damage)
• Activatable items (eg. Portable Teleporter)

It's easy to predict the results of picking up a specific item. If you want to imagine what happens when the bot grabs a rocket launcher, you'd just change the player state to have a rocket launcher and some rockets for ammo.

Determining how many points a given resource state is worth is significantly more complicated. Why is that? What other information does the bot need to estimate the points it would gain if it had this set of items? In short, it needs information about possible encounters with enemies:

• Chance of encountering an enemy near the bot
• How much damage the bot will deal with each weapon
• How much damage is required to score a kill
• How much damage an attacking enemy will deal to the bot
  

Once the bot has this information, it can estimate the points gained for any duration of time. The encounter rate can (and does) change based on where the bot is, of course. There will be higher traffic near popular items. As a result, the bot needs to remember when it does and doesn't see other players in different regions of the level. This is actually a complicated enough problem that I'll do a full post explaining how it works another day. For now, assume the bot magically knows how likely it is to find an enemy.

The three damage values should be independent of any changes to the resource states. The amount of damage a weapon deals depends on the bot's accuracy-- getting more ammo will let the bot deal that damage longer, but it won't ever make one weapon deal more damage per second than another. As a result, the bot sorts every possible weapon it could use in a priority list at the very start of prediction. The list might read something like, "Use the plasma gun until out of ammo, then the rocket launcher, then the shotgun, then ..." Now that might not be what the bot actually does when it gets in combat, but it uses this as a rough estimate of what might happen.

Naturally the estimated damage each weapon will deal is computed from the bot's historical data of previous weapon use. If in the past the bot had 30% machinegun accuracy and fired 90% of the time the machinegun was loaded, the machinegun's effective damage rate is 27% of it's maximum possible damage rate.

So when the bot needs to predict 20 seconds of time, it might say, "Well I'll get this many points from firing my rocket launcher for 8 seconds, until it's out of ammo. Then the next weapon is a railgun, which I shoot for the remaining 12 seconds and get this many extra points." There would be 2 distinct calculations in that situation. The code uses a similar system for powerups wearing off. In that example, if the bot had a quad damage that wore off after 15 seconds, it would break the time into three segments (0 to 8, 8 to 15, 15 to 20).

Of course this is all well and good if the bot survives, but what happens if the bot predicts its own death? Surely there must be a penalty if the bot needs health but considers options other than getting it. When the bot thinks its health will reach zero, the resource prediction engine stops giving points, and the bot takes a (1 / opponents) point penalty, since getting killed gives one opponent a point.

All that work just to figure out how many points the bot will gain, not even to select the best item for pickup! Next week I'll explain how exactly the item pickup algorithm uses this resource prediction engine to determine the best item to pickup.

A Journey of One Thousand Miles

There is a very old Chinese quotation that goes, "A journey of one thousand miles begins with a single step." More literally it reads, "A journey of one thousand miles begins beneath one's feet." Wherever you go, the first stage in any journey is the place you are in now. This might seem trite, but it's crucial, and all kinds of problems arise if you ignore it.

For example, suppose you take a cross-country trip from Chicago to Los Angeles:


View Larger Map

The basic plan is to take Interstate Highways 80, 76, 70, and 15 to cover most of the 2000 miles. But none of these directions are actually useful unless you can get on I-80 in the first place! If you zoom in on Chicago on the map, you'll see the directions start from outside the Chicago city hall. You'd use a totally different route to get on the highway if you started on the south side of Chicago or one of the suburbs. Understanding your current location is crucial to route planning.

Now last week I promised this post was about item pickup, not navigation. So what does route planning have to do with item pickup? Determining what items to pickup is a kind of of route planning. If a bot wants to go to a different area of the level, there might be three or four different routes it could take to get there, each with different items available on the way. There may be a good reason to take a longer route that has better items. Items define way points between where the bot's current location and where it wants to end up.

The item pickup algorithm determines which items a bot should pick up while it goes somewhere else.

That might seem simple, but as best I can tell, it's a fundamental change in how item pickup AI has been done in every other first person shooter to date. Typically the problem is viewed as finding where the bot wants to go, and choices are picking up items, hunting down enemies, or guarding a specific location. I've approached the item pickup as what you do while you are also hunting down enemies or guarding a location.

There is one crucial difference between item pickup and traditional route planning problems, however. If I'm planning a cross country trip, I'm either trying to minimize travel time or travel distance. Item pickup isn't interested in either of these things. Why do players pick up items? To get more points in the game. Bots need to select the items to pickup between points A and B that will maximize their score, not minimize travel time. That's why it's often correct to take a longer route with better items. The better items will increase the player's score more than showing up at the destination completely unarmed. The item pickup algorithm I've implemented is completely focused on one thing: estimating the score gained from taking different items.

And because the journey of one thousand miles begins with the ground beneath your feet, bots need to know how effective they are at earning points in their current state (health, armor, weapons and ammo values). Then they estimate the value of a particular item by imagining how that item would modify their state and how that would change their rate of acquiring points. The overall item pickup algorithm involves estimating the points the bot gets going from point A to item X to point B, where X is any nearby item. But the actual meat of the algorithm is estimating how many points the bot gets given a specific amount of health, armor, weapons, and ammunition.

Next time we'll dive in and talk about exactly how this is done, and what other structures are necessary to make these computations possible.

Putting It All Together

I've received a number of requests from people who want to see the bots in action, but don't have Quake 3 installed. So I recorded a short demo of the bots in action, spectating from a bot's point of view. All the bots in this game are skill 4 out of 5, so they're intended to give an experienced player a run for their money. While there's a lot to be said about how each individual piece of code works, it's the entire package that matters. Weapon selection is meaningless without good aiming or item pickup, for example. Here's the video:



The biggest thing this demo shows is how realistic their aiming is. The bot's view changes as chaotically and accurately as a real person would, which is to say "generally pretty accurate, but fast enough to be a bit hard to follow". They quickly respond to changes in attention and their aim isn't instantly spot on, but they correct their aiming mistakes relatively rapidly. You can also see some good firing decisions. For example, early on Xaero fights Mynx in a narrow hallway. Xaero correctly chooses the rocket launcher and kills Mynx with two well placed blast hits.

You can also see how the bots do a pretty good job of picking up the crucial items. One of the first things they do when they respawn is grab a weapon, and they make sure to pickup health and armor when needed. It might not seem like a big deal that bots choose to pickup armor when they are near it, but it really is. The original Quake 3 bots were notorious for ignoring perfectly useful items.

This video also highlights areas where the bots could improve. Most notably, the navigation algorithm, which I could not change. Later in the demo, you can see Xaero dodge in place for maybe 10 seconds before Hunter arrives and Xaero kills her. In this situation, Xaero hears Hunter below him and wants to get to her, but he can't figure out how to jump down the ledge. So he dodges in place until she arrives. Once Xaero kills hunter, he goes on his way, picking up the megahealth and some armor.

So BrainWorks represents a lot of progress for First Person Shooter AI, but there's still a lot of room for improvement. As I said before, AI is hard! For those of you interested in more posts describing exactly how BrainWorks, uh, works, fear not! Next post we'll dive into one of the most complicated systems in the code, item pickup, and explain how the BrainWorks bots solve this difficult problem.