Ballistic Programming

Last Wednesday I attended a talk at MIT from Larry Wall, the designer of the Perl programming language. The talk was entitled, "Ballistic Programming", but really it was excuse to talk about Life, the Universe, and Everything. And by everything I mean programming languages, their design, shortcomings and benefits, natural languages, grammar, and how his perspectives on all of these things impact the design of Perl.

Perhaps it's better to say that the talk was about "everything".

At any rate, the study of ballistics is literally about the art of projectiles. It's the source of the the term ballista, and it's based on the Greek verb for "to throw". Pretty much every sport involving a ball is a ballistic sport, as well as a few others, like the javelin toss and gun target practice. Ballistics is the art of throwing an object from one point to another, and the process works like this:

• Select intended destination
• Estimate force required to move object
• Make attempt
• Evaluate effectiveness
• Correct next attempt using these results

Of course, this procedure applies just as well to programming as it does golf or basketball. Larry had an excellent set of diagrams explaining the two major theories of software project design. First, he showed Waterfall, a complete failure of a process in which each stage seamlessly and perfectly flows from one step the next with no backtracking. It's been known for decades that it doesn't work and causes projects to be over budget, but here's are the stages:

• Specification (select destination)
• Design (estimate force)
• Implement (make attempt)
• Test (evaluate)
• Maintenance (correct)

The problem with waterfall is that it has no flexibility anywhere. Sometimes something will come up in implementation that demonstrates an issue with specifications or design, and those portions need to be corrected. But waterfall states that all the specifications have to be done first, then all the design, and so on. Either the whole project gets held up as flaws in the process are found, or the project is completed with flaws, producing something that no one needs and is impossible to maintain.

The other common methodology is known by the laden buzzphrase "Extreme Programming". The two main tenants of extreme programming are that you want lots of feedback loops so problems can be corrected as soon as possible, and that people should program in pairs. Since pair programming isn't relevant to the discussion at hand, I'll just give my two cents and let the issue rest. It's very useful for educating half of the pair and otherwise it's a waste of time, in that it takes more than twice as many worker-hours to do.

Larry Wall joked that in Extreme Programming, you start with the test which shows you how to implement the program. After you've implemented it, you can write your design document, and finally at the end of the project you can write the specifications. In other words, it's the reverse of Waterfall:

• Test
• Implement
• Design
• Specs

Naturally it's easy to run into problems early if you don't know what your design and specs will actually end up as. And just because you are constantly refactoring when you encounter a problem doesn't mean you refactored in the right way. Knowing something needs fixing is different from knowing how to fix it. Sometimes your next attempt isn't even an improvement on the last issue.

That said, it's is a clear improvement over Waterfall. It's easy for Extreme Programming to beat Waterfall in the same way that it's easy to beat a blind man in boxing. Just because you won doesn't mean you're any good.

Waterfall is always bad, but there's "good" Extreme Programming and "bad" Extreme Programming. Whether or not Extreme Programming works depends heavily on how you skip between the different ballistic steps. Sometimes you start in specs and move directly to implementation, then before you even get to test, you detect a problem in design. You fix the design problem, quickly make an implementation fix to reflect it, and then test it. In you can skip from any step to any other step if you are truly following the fast turn-around tenet to its optimum. In other words, Extreme Programming is just another name for:

Cowboy Coding

Losely put, Cowboy Coding is a system in which "every man does as he sees fit." No system can ever tell you the optimal way to construct the program. There will be speccing, design, implementation, and test, but who knows in what quantities and chronology. Good programmers just know from experience and intuition when to move to the next step. Now cowboy coding is used pejoratively by some programming methodology evangelists, especially those in the Agile community (a subset of extreme programming). When bad programmers try to use Cowboy Coding, the result is always a train wreck. But ironically it's the only good part of Extreme Programming, even if its proponents don't realize it. Think about it like this:

Software design metholodigies instruct programmers when to switch from one ballistic step to the next. Sometimes the methodology's suggestion is optimal and sometimes it is not. Good programmers will generally outperform the methodology's suggestions and almost never do worse. Bad programmers don't know why the next step is the correct one, so even if the methodology suggests the right step, their chances of doing the next step well are low.

You can lead a man to reason, but you can't make him think.

No amount of restriction on the software design process or programming language can turn bad programmers into good ones. They tried this with languages like Java and Python, and while they are fine languages, bad programmers still write equally horrific code in them. So you are better off optimizing your design process for the good programmers and letting natural selection weed out the bad ones.

Dodging The Question

There's a class of questions that have no good answers to them. For example:

Have you stopped beating your wife?

The question presupposes something that is hopefully false, and giving either a yes or no answer implies the question is legitimate. The solution is to provide an answer that challenges the premise. In other words:

I never started.

First person shooter games like Quake 3 are really games of questions and answers. Every shot you take at an opponent asks the question, "How will you survive this hit?" And your opponent would prefer to respond, "You aren't going to hit me". Of course, he doesn't have much say for an instant hit bullet weapon like a machine gun or shotgun. Either you hit him or you don't. But these weapons are balanced to do less damage. For projectile weapons like a rocket launcher, it takes the missile some time to reach them, and they might move out of the way by the time it gets there. In return, they deal a lot of damage if the opponent fails at dodging. Whenever you shoot a rocket at an opponent, you are giving them the opportunity to make a mistake by not moving out of the way.

Tactically it seems like a poor choice to give your opponent the opportunity to not to take damage. All other things being equal, you'd rather they didn't have a say in the matter. Why ask a question ("Do you want to take 100 damage?") when you can make a statement ("Here, take 50 damage!")? But missile weapons provide a tactical advantage. They make large sections of the map dangerous, denying the opponent safe access to whatever items and safe cover locations are in that place. As a result, missile weapons give you control over where your opponent might go next.

I believe that no good player should take damage from the grenade launcher or plasma gun. These weapons exist solely for the purpose of controlling where your opponent can run. In particular, the plasma gun stream is so fast, constant, and deadly that it can literally herd the opponent into a position where another weapon can finish them off. If they don't want to be herded there, they are free to take 200 damage per second from the plasma shots. Missile weapons are more about controlling position than actually damaging the opponent.

Of course, none of that matters if the target can't dodge the incoming projectiles in the first place. The original Quake 3 bots were notoriously bad at this, obliviously walking directly into a path of fire. So I made sure to give BrainWorks bots a very effective missile dodging system. Just watching a BrainWorks bot play, you can shoot a straight stream of missiles at them and watch them sidestep out of the way. It's such a simple, obvious thing for a human that it might not even seem that special.

For the bot, it's not that simple. The bot needs to analyze the trajectory of all incoming missiles for possible impact and the bot needs to compute a safe direction to dodge from that. Sometimes there will be no safe areas, but some areas will be safer than others. And the final dodge direction needs to take into account the bot's intended final destination. If the bot only moves to the safest possible location, the attacker will simply herd the bot to a place that's even more dangerous. The bots selection of where to dodge also needs to eventually get them towards their goal location, albiet at a slower, safer pace.

Those are the requirements for the dodging system, and they are all challenging. Next post I'll explain in detail how BrainWorks solves this problem.

Less Is More

Previously I wrote about a Quake 3 modification I made named Art of War, and how it was the inspiration for BrainWorks. But it wasn't the first Quake 3 mod I made. That honor belongs to an unreleased game variant simply titled Less. The concept of Less is simple: All items are less powerful, less health or ammo per pickup. It's the opposite of Excessive Quake. Excessive is fun because who wouldn't want to be uber powerful. Doesn't that mean Less is less fun to play?

I believe that when done correctly, less is more. The actual numbers on an item don't mean anything; only the relative values matter. For example, you could multiply all health, armor, and damage values in Quake by 1000 and the game play wouldn't change. Players would start with 125,000 health and each rocket would do up to 100,000 damage. Strategically nothing has changed. I call this "Pinball Inflation", as pinball games have been adding extra zeros for decades. Some tables now have scores that reach the trillions.

Online RPGs like World of Warcraft have a similar issue, where you can gain levels and deal extra damage, but you fight monsters that have more health. The difficulty of the fights don't change that much. At best you might gain a totally new kind of ability as you gain a level, which gives you another tool at your disposal, and that's the only way the difficulty increases. So for MMOs, the number inflation game is a method of unlocking more content. Players would be daunted if the first time they picked up a class of character, they had 30+ different abilities to choose from. It's better to start them with 3 to 5 and have them gradually learn more.

All that said, it's crucial to realize that gaining levels doesn't actually make your character more powerful, relative to the content you're doing. Sure you might kill goblins in 2 hits instead of 3, but you'll eventually move onto killing stronger goblins. The primary purpose of gaining levels is letting you experience more content in the game while maintaining the same difficulty.

I started the design of Less from a similar standpoint: I wanted Quake 3 to have more content. Reducing the benefit of each item might seem like a strange way to add content, but listen to the logic. A typical game of Quake 3 involves controlling the two to five best items on the map: red and yellow armor, megahealth, and powerups like quad damage and haste. But on the level there are dozens of items that rarely matter, things like boxes of ammo and weapons. You'd think weapons would matter more, but when they respawn every 5 seconds after pickup, pretty much everyone has any weapon they want. And the ammo a weapon provides makes that weapon's associated ammo box irrelevant. There may be 50 items on a level, but only 3 of them matter for a typical game.

The concept of Less was to make all items have a roughly equal play value on average. Then the best players would be those who knew which item was most important in the current situation, and what their opponent needed the most. You could end up with a game where the winner was the person who best controlled a box of rockets and a box of machinegun bullets. And that adds a deeper level of strategy and tactics, thereby adding more gameplay to the game.

You'd be suprised, but it's actually possible to tweak the numbers for each item to make this possible. The game plays like Bizzaro Quake 3, where you start timing the respawn of ammo boxes and small health balls in addition to armor and quad damage. It's definitely the same game on the surface, but the high level strategy of how you play the map is totally different.

While I don't have the source code in front of me, here are some of the changes made, to give you a sense of how dramatically the gameplay shifts.

• Railgun Weapon: Provides 2 slugs (3 seconds) instead of 10 (15 seconds)
• Railgun Slug Box: Provides 5 slugs (7.5 seconds) instead of 10 (15 seconds)
• Lightning Weapon: Provides 24 ammo (1.2 seconds) instead of 100 (5 seconds)
• Lightning Ammo: Provides 40 ammo (2 seconds) instead of 60 (3 seconds)
  
• Quad Damage: Lasts for 8 seconds instead of 30
• Red Armor: 50 armor instead of 100
• Yellow Armor: 25 armor instead of 50
• Orange Health: 25 health instead of 50
• Yellow Health: 15 health instead of 25

When playing Less, you have this constant sense of never having enough of anything, and that in turn creates a sense of fear and tension. The big question is, though, "Do these changes make a better game?" I'm curious what other people think, and next week I'll share my own thoughts on the difference (and connection) between making a game balanced and making a game fun.

The Last Place You Look

When I was growing up, I always found it ironic how lost things were always found in the last place you looked for them. For some reason, whenever I was looking for a lost toy it was never in my bedroom or the play room. I'd find it outside underneath the deck, in the basement, or inside the sofa. It wasn't until I was older that I realized this wasn't coincidence; it was a mathematical truth. Put another way, when you're looking for something and you've found it, you stop looking. Things are guaranteed to be in the last place you look.

That's not the same as saying all places are equally likely to hold the lost object, of course. If I'm looking for the TV remote, it's less probable that the dog took it as a chew toy than that the remote fell between sofa cushions. And it's far less probable that the remote is now part of an exhibit in the Smithsonian Institute.

The general purpose search algorithm that humans use for locating lost objects involves prioritizing possible locations from most to least likely and searching them roughly in that order. The actual order will get rearranged to save travel time. For example, if an object is most likely to be on top of your dresser, second most likely to be on the bathroom counter, and least likely to be under the bed, you'll still probably check your dresser first and under the bed second before heading to the bathroom. But in general the search is from most likely to least likely. That also explains why the place you find an object is always the last place you'd think to look. The optimal searching algorithm stipulates that you search in the least likely places last.

So this brings me back to last week's topic of octrees. I described how to build the structure that BrainWorks uses to determine which item a player is nearest. But I haven't actually explained how BrainWorks uses the octree to do this. Consider this two dimensional tree (a quadtree):


Each item in the tree is one of the colored dots. The player is the black dot with the circles around it. Every rectangle refers to a particular subsection of the tree. In this example, the red dot is the root node of the tree. The lower left section of the tree is divided by the brown dot into four sub-regions, and the brown dot's lower left region is further subdivided because it contains another item (the purple dot). The circles represent the boundary of the potential search space as the algorithm progresses.

When the search algorithm first starts, it checks the distance from the root node to the player. This distance defines a circle around the player. If an item is closer to the player than the root node is, it must be inside that red circle. At this point the algorithm doesn't know that the green or blue dots are inside the circle. It doesn't even know that no items in the lower left quadrant are closer than the red (root) item is. But it can mathematically guarantee that nothing in the upper right quadrant could be closer, so that quadrant is ignored.

Because the player is in the lower left quadrant, the algorithm prioritizes search in that section first. This is the quadrant that contains the greatest area of the red circle. The next most likely place to look is the lower right quadrant, as the player is closer to the lower right quadrant than the upper left. The upper left quadrant is scheduled third. So the areas to check are, in order:

• Lower Left
• Lower Right
• Upper Left

Then the algorithm merely recurses. First it checks the brown item in the lower left. A quick distance check shows the brown item is too far away, but the search continues on the brown region's subquadrants of upper right, then upper left, then lower right. As none of these regions have any items, the search of the brown region terminates quickly. And at this point in time, the red item is still the closest object to the player.

When the algorithm starts checking the lower right quadrant of the red item, it notices the blue item is closer. So from now on, it compares all potential search areas against the blue circle instead of the red one. No further items exist in the lower right quadrant, so the blue item remains the closest item to the player.

Last, the algorithm wants to check the region to the upper left of the red (root) item. However, when compared to the new blue circle, the search algorithm realizes that the region does not intersect the circle. Since the circle is the area of potentially closer items, nothing in the upper left can be closer that the blue item, so the entire upper left region is ignored. And since all regions were prioritized by distance to the player (from closest to farthest), the search process is done. All potential items have been checked, so the blue item is the closest. The algorithm doesn't even need to check how close the green item is to the player because that entire upper left quadrant was skipped.

The octrees in BrainWorks function the same way, only in three dimensions instead of two, and using spheres instead of circles. That answers last week's question of why the algorithm might need to search up to 7 subregions to find the optimal solution. The closest item isn't necessarily in the same section of the octree as the player, but the diametrically opposed region can always be ignored. That leaves 7 of the 8 regions to check. In practice most regions are quickly pruned away when a closer item was found, in the same way the region containing the green item was ignored when the blue item was discovered. Even though the example tree contained seven items, only three of them needed to be checked (Red, Brown, then Blue) before the closest item could be found.

Dude, Where's My Rocket Launcher?

Back in March I wrote about how the AI conceptualizes item placement. Rather than thinking of position in terms of raw coordinates, bots consider each item (or cluster of nearby items) to be a potential landmark. When statistical tracking is done for areas players are likely to be, for example, they are grouped by which item they are closest to. Statistical tracking is one of the most powerful tools available to AI if you have a sufficiently large and well organized data set. BrainWorks uses this data to analyze which areas of the level a player is most likely to be in, for example. Not only can this information help bots track down enemies; it also helps a hurt bot them avoid enemies. The more data it can track, the better. So by extension, the faster it can translate from raw coordinates to the nearest item, the better. However, this is not a simple request.

If a level has N items in it and a player is located at position (X, Y, Z), what's the fastest way to determine which item is closest to that player?

Well you can just check the distance to every item and use whichever item is closest. If you read last week's column, you'll remember that this search requires linear time. If you double the number of items on the level, it will take twice as long to find the closest. Certainly we can do better than linear time.

There are a number of data structures that can solve this problem. Some of them might require more setup time but give slightly faster search time. The structure I chose is an octree. It's a tree that holds some number of three dimensional locations. If you want to learn more, I recommend the Wikipedia article on quadtrees as well. A quadtree is the two dimensional analog of an octree, so it's a bit easier to visualize.

In the example I gave last week of a filing cabinet holding 100 pieces of paper, 1 cabinet held 10 folders and each folder held 10 papers. This is a total of 111 pieces of data stored (100 papers + 10 folders + 1 cabinet). In computer science terms, each data storage location in a tree is called a "node". The node where searching begins is called the "root node". In the filing cabinet example, the filing cabinet itself is the root node. The cabinet and folder nodes split 10 ways, and the pieces of paper do not contain any sub-nodes. Nodes that do not further branch are called "leaf nodes".

Getting back to the octree example, the BrainWorks AI code needs to store the spacial locations of 50 or even 200 items in a tree. Each node in the octree contains is one of the stored points, and the node branches 8 ways. Visually think of each node as dividing space into 8 octants-- upper north west, upper north east, upper south west, upper south east, and four more counterparts for lower sections. All points that are above, north, and west of the root node are stored underneath the upper north west node, and so on for all other octants.

For example, suppose there are 50 items on a level. One of them is stored in the root-- say it's a rocket launcher. Of the 49 remaining items, 6 of these item locations are above, north, and west of the rocket launcher. The root node will link to a node for one of these six items-- a megahealth. The other five items will be stored undernearth that megahealth.

Again, the octant division property applies to all nodes, not just the root. So the megahealth also divides its space into eight octants, and the five remaining items will be appropriately placed underneath it. Keep in mind that each of these items is guaranteed to be above, north, and west of the rocket launcher. But items that are above, north, and west of the megahealth will be stored in a different octant than those above, north, and east of the health. Given that there are eight possible areas and only five remaining items to store, chances are that these items will all be stored in one of the eight sub-nodes of the megahealth. (If two items happen to both be below, north, and east of the megahealth, then the first will be stored under the megahealth, and the second will be stored under the first.)

Searching a tree is generally proprortional to the depth of the tree-- the greatest distance from the root node to a leaf node. And for a tree of 50 items, the depth is usually 3 or 4. Going up to 100 items might push the depth to 5 for some level layouts. So the octree structure makes answering the question of "which item am I nearest?" much, much faster than checking all 50 or 100 items.

Given that items have fixed locations on the level, the octree needs to be created once when the level starts, but the tree will never modify during the game. (Ignore items that drop from dead players-- those are handled differently and not useful for statistical tracking anyway.) The actual algorithm is pretty simple, a bit slow, and very likely to produce a well balanced tree. Here's how it works.

Given a set of items:

• Average together all their positions to find the center
• Check each item to find which one is closest to the center
• Add that item as the dividing node
• Separate all remaining items into one of the eight octants this center divides
• Repeat the process on each octant that contains at least one item

By "not terribly fast", this algorithm could take up to half a second on a very, very large map. So still inconsequential, and well worth the one-time hit to produce a better tree.

Of course, there's still the question of how you search the tree to find the nearest item, and that I will be explaining next week. Here's some food for thought though: even if you identify which octant a point would reside in, you still need to search up to seven of the eight octants to guarantee you've found the closest item to your input point. Searching this tree is not done like most tree searches. For those of you who like thinking through these kinds of problems, try to figure out how the search algorithm actually works. I'll give the answer next week.

Given a set of item locations arranged in an octree and one player location, describe an algorithm to find the item nearest the player.

Good luck!

Needle In A Haystack

For a long time, I kept all my important documents in a single box. That would be stuff like my passport, car title, insurance information, and so on. After my wife and I bought a home, we had enough papers to file that our simple box wasn't sufficient. We bought large filing cabinet and it's served our needs ever since.

It's not that we needed to look up stuff any more frequently than before. But when we went from 100 pieces of paper to 1000, we didn't want it to take 10 times longer to find anything. To find stuff in the box, I riffled through it until I found the paper I needed, sometimes searching the entire thing. More stuff means the search takes longer.

A filing cabinet solves this problem by providing a natural structure for organizing the papers. For example, all the papers on the home can go in one file folder, all the car information in another, and all the identification documents like passports in a third. Then the lookup process becomes simpler, as I can ignore everything that's not part of the folder that keeps whatever I'm looking for.

This is a real savings in time. Imagine you have 100 different pieces of paper. You can either stuff them all in one box or you can organize them into 10 folders, each containing 10 papers. What's the average number of papers you'll have to look at to find any given piece, assuming you always know what folder should contain that piece of paper?

When looking through the box, you could look at anywhere between 1 and 100 pieces with equal probability. So you'll average (1 + 100) / 2 = 50.5 pieces before you find it. When looking through the folders, you have to find the correct folder from 10 options and then the correct paper of the 10 pieces in that folder. It will take between 1 and 10 tries to find the folder, or 5.5 on average. And similarly it takes on average 5.5 tries to find the paper in the folder. That's 11 tries total for the folder compared to 50.5 for the box. The filing system can be searched roughly five times faster!

Or is it really five times faster? What happens if you have 1000 pieces of paper instead of 100? When you have this much information, you could create folders and subfolders to further organize the papers. If you have 10 main folders, each of which contain 10 subfolders, and each of those contain 10 papers, then it will take 5.5 + 5.5 + 5.5 = 16.5 tries to find any one of the 1000 papers. It would take 500.5 tries if they were in a box. So in this case the filing system is about 30 times faster.

The more data there is to store, the more efficient the filing system is compared to the box. That's because the filing system is a whole order of magnitude more efficient. In mathematical terms, searching through the box takes linear time. When you add ten times as much stuff, it takes ten times as long. But searching the filing system takes logarithmic time. Adding ten times as much stuff just means it takes one more search step. Very roughly speaking, searching is proportionate to the number of zeros on the size of the data. Searching 100 things takes twice as long as 10 things, and searching 1000 things takes three times long. Want to search 1,000,000 things? That's six times longer than searching 10 things.

You might be wondering what all of this has to do with artificial intelligence. At the heart of almost all AI is data, and data doesn't mean anything if you can't find it when you need it. That means efficient data storage is the required foundation for artificial intelligence. An algorithm is only as good as the data it has access to, and good AI needs a lot of data. If all the data were just shoved in a box, there's no way the computer could run the AI's thought process fast enough. Indeed in almost all computer science, the way in which data is organized is just as important as the actual data being stored. After all, what good is information if you can't find it? Next week I'll give a concrete example of how BrainWorks puts this concept to use.

Mind Candy

If you haven't worked in the game industry, you might think of a game designer job as a dream job. You get to create your own toys and then play with them! Well it certainly has its moments, but it's important to realize that the job is still a job. There will always be boring parts that need to be done. And more often than not, the job involves creating your own toys and then figuring out why the won't work or aren't fun. In fact, it's a lot like every other programming job except you get to make toys rather than spreadsheets, word processors, or websites. While it's fun to design toys for a living, there's a pretty high cost to it. And I don't mean the "there's boring stuff too" cost that comes with every job. Here is the cost:

The more time you spend designing a game, the harder it is to enjoy playing it.

The problem is that over time, you have to think of the game in terms of its individual core mechanics. But the wonder and enjoyment comes from how those mechanics are organized into something greater. No game has infinite replayability. There always comes a point where you think, "That was fun, but I won't ever want to play this game again." A game might have 10 hours of good fun, or even 100 or 1000. But no game is fun after 10,000 hours. Why is that? Why can't games be fun forever?

Games are mind candy.

They are a thought treat. Personally I approach games as puzzles to solve. The very act of thinking is fun, at least for my mind. The challenge is learning how the make the best move in each novel situation the game presents. As you play a game more, you learn it better. And eventually you simply recognize the situation and can play on autopilot. At that point the game ceases to be fun because it is no longer challenging. It can still be entertaining to execute the actions, and in playing I can encounter a few new challenges, but in general the candy of the game no longer tastes sweet to my mind.

Unfortunately, my job as a game designer isn't done until I've solved the game and determined there are no degenerate solutions. The harder it is to determine a game's optimal solution, the more replayability it has. (Note that replayability just defines how long it takes for the game to cease being fun. It has nothing to do with whether or not the game is fun in the first place, or to what extent.) If it's too easy to solve the game due to an obvious dominant strategy, no one will have fun playing the game once they figure it out and all my design work is for nothing.

While I'm not an author, composer, or movie producer, I suspect this problem extends to all jobs designing entertainment. A really satisfying book is one where all the plot pieces fit together nicely and there are no obvious plot holes. The characters need to feel genuine and you need to understand how their development as people natural extends from their personalities and experiences. No author can convincingly do that without really getting into the minds and lives of these fictional people. To build the depth of a good book, the author must understand that depth before the final editing pass is done and the book is sent to print. They will never have the enjoyment and wonder of watching how the lives of their characters unfold. They sacrifice that potential joy so that other people can experience it too. It's just part of the creative process.

For my part, I find I don't enjoy playing first person shooters as much as I did many years ago. Maybe after a few years break I'll enjoy them again, but that's just the personal price I paid so that other people get more enjoyment from the games they play.

Mission Impossible

One of the basic realities of life is that if something is easy, it's already been done. In the grand scheme of things, all unsolved problems are hard problems. While that might not have been true 300 years ago, the advancements of widespread education and information technology have made it easy to find potentially talented individuals and make sure everyone knows about their work. The result of that for almost every technological problem you can think of, either someone has solved it already or it's very difficult.

So what do you do when you need to do something that hasn't been done before and it turns out the problem is very hard? How do you break an impossible problem into something solvable? While there's no right or wrong way to go about this, there are three basic approaches you can use. Assume that the objective of a perfect solution is either practically or literally impossible, and you just want the best solution that's actually feasible.

#1: Improve on an existing base

The concept here is to focus on the small, incremental progress you can achieve rather than the large, difficult end objective. This is the approach I took when solving the general high level goal selection in BrainWorks. Bots must constantly decide what they are most interested in doing. Should they track down a target or look for more players? In capture the flag, should they defend their flag or get the opponent's? I confess I've added nothing revolutionary to this logic, and the correct choice is something even experienced human players have trouble making. Instead I just added a few pieces of additional logic to help the bots decide. BrainWorks bots will consider tracking down a target it's recently heard but not seen, for example. It's not algorithmically amazing but it's progress and that's what matters.

#2: Break the problem into pieces and only do some of them

I used this approach when designing the item pickup code. Roughly stated, the implemented algorithm is to estimate how many points the bot would get if it were to pick up different items and it selects the item that gives the most points. While the item selection is reasonably good, there's a lot of corners cut for the purpose of reducing complexity. How valuable is a personal teleporter anyway? BrainWorks just values it at 0.6 points and calls it a day, but clearly such an item is far more useful in some situations than others. And the estimations of where enemies are likely to be are based on proximity to items, not on actual geographic locations (like being in a specific room or a hallway). The estimates aren't remotely perfect, but they are still good enough for the purpose they serve: a rough estimate of what item the bot should pick up next. Designing this code wasn't even possible without ignoring all the truly hard parts of item pickup.

#3: Accept defeat and deal with the consequences

This is essentially The Attitude of the Knife. Sometimes things just don't work out and can't be fixed, and all you can do is accept that reality and handle the consequences as best you can. This was the approach I took for the navigation code in BrainWorks. As I've mentioned before, the navigation tables have huge issues on anything other than a flat, two dimensional map. While I'd love to solve this problem and I'm certain I could do it, I just didn't have enough time to tackle it. There were a few clearly bad navigation decisions (such as jumping between ledges) that I wrote code to manually fix, but those were essentially stopgap measures.

While it's not ideal, there's nothing wrong with accepting defeat against hard problems. After all, you aren't the only person in the world who couldn't solve the problem.

Give It A Go

While my primary area of expertise is First Person Shooter AI, I still get a number of questions about other AI applications. One of the hardest game AI problems to date, if not the hardest, is the game of Go. What's so surprising is that Go has simpler rules than Chess, but Go AI is several orders of magnitude more complicated than good chess AI. Now consider that the best chess AI to date has only beaten a grand master, Kasparov, when it was specifically trained to counteract Kasparov's play style. In the first set of games, which Kasparov won, he commented that he did better when he started making what he considered the second best move instead of the best move. Apparently the first AI was so hyper-tuned against Kasparov's play style that it defeat other similarly good plays that it did not think Kasparov would make.

Yes, that's the best the world has done for Chess AI: A program that can defeat one really good player. But Go? The best AI programs just rank as decent amateurs. And listen to how simple the rules are:

• Players take turn placing stones of their color (black or white) on a square board (usually 19x19 but sometimes 9x9).
• Stones of the same color that are adjacent vertically or horizontally are part of the same group and live or die together.
• When a group is completely surrounded (no plays exist that add a stone to the group), that group is captured. Each lost stone is -1 point at the end of the game.
• If a play is made that would cause both the placed stone and an opposing group to be surrounded, the place stone lives and the opposing group is captured.
• You may not make a play that results in a previous board position.
• The game is over when both players consecutively pass, opting not to play a stone. Each player scores 1 point for each empty space that is enclosed only by their color of stones.
  

What makes this problem so hard to solve? Well for starters, the 19x19 board is enormous. There are a lot of possible positions. Moreover, even though the plays are just of single stones, they come together to form different shapes like pixels form a picture. Different shapes have different properties, strengths, and weaknesses. A typical alpha-beta search becomes rapidly unusable when the average branching factor is on the order of 300.

Nevertheless, in the past few years there have been a few significant advancements in the field of Go AI. Rather than trying to solve Go in the same way Chess would be solved, some Monte-Carlo algorithms have shown a lot of promise. In particular, in 2006, "Upper Confidence Bounds applied to Trees" (or UCT) was developed. I'm not an expert on this subject, but in brief, the concept works like this.

While it's very hard to determine which space on a 19x19 board is the best, it's very easy to compute the score of a Go board. Contrast this with Chess where the actual pieces in play are secondary to question of whether or not a King has been captured. So the very rough idea is to try most options (pruning the obviously bad plays) and play a semi-random game for each of them (again, pruning obviously bad plays). Far less time is spent at each stage of the tree. If there are 50 possible moves that aren't terrible, one of them is just selected at random. There's no aggressive search to determine which of those 50 is the best. When the test game is done for a given play, that provides an estimate of how good that particular initial placement is. Other placements are tried and the one with the highest estimated score is selected as the computer's official move.

Of course, the actual algorithm is quite a bit more complicated than that, but that's the general idea behind the innovation. The crucial lesson here for AI designers is this:

Focus on your goal.

In the end, Go is just a score maximization problem. It's very easy to get bogged down in the details of whether a move nets 5 points or 7 points, quickly producing a state where far too many calculations need to be done. Often the best solution involves determining how much processing time you can actually spend, and then working backwards to figure out what methods will give you the most information for the amount of time spent. In Go, calculating the board's score is relatively fast, so latest algorithm that exploited this fact had a major advantage over those that did not.

My Biggest Programming Fear

I have encountered some very strange bugs in my programming career. And most of them have been in BrainWorks, being by far the most complicated piece of software I've ever written. One of the worst involved a build that only crashed when run on someone else's system. On my system it was fine, but it would crash on my friend's system. It wouldn't crash right away, mind you. Sometimes it would take up to two minutes, so even testing to see if the bug was fixed took time. Oh, and his computer was in London while mine was in Los Angeles. Not exactly the easiest problem to solve.

I solved it by compiling in debug flags that would turn on or off entire sections of code and then had him run it. "Okay, load bots but turn off all movement, scanning, and aiming. Does it crash now? What about when Item Pickup is turned off?" Through the course of four builds that gradually narrowed in on particular blocks of code, we eventually found the offending bug.

(If you're curious what caused it, an uninitialized value was improperly being treated as an address, crashing the program whenever it was accessed. Because his machine's memory layout was different from mine, it would occasionally access invalid memory and the operating system would kill the program. For whatever reason, the uninitialized values that showed up on my system always happened to refer to memory BrainWorks was allowed to access, so it never crashed for me.)

It took about eight hours to do, but once you've solved a bug like that, you feel cabable of tackling any bug. Bugs bother me, but they don't frighten me. Know what my biggest programming fear is?

0 total errors

Any time I've spent over two hours working on a particular feature and attempt to rebuild the code base, I expect to see at least one error. I'm a good programmer, but that doesn't mean I'm perfect. I'm a good programmer because I know I'm not perfect. Good programmers assume they will make mistakes-- that's why they add all those safety error checks. If some other piece of code does the wrong thing, their error checks will contain it.

Last week I spent four hours tracking down the issues with bots overvaluing the shotgun. The problem was that the estimation of fire frequency wasn't precise enough. I spent another three hours analyzing the similarities between estimating the chance of hitting with a weapon (hits divided by attacks) and the chance of firing a weapon (attacks divided by potential attacks) and designing functionality that merged the two concepts. Actually writing the code took another 2 hours. Nine hours total including changes that could totally break the AI's ability to even attack, and here I am looking at the results from my very first recompile:

0 total errors

I haven't run it yet, but I'm terrified. Zero errors on the first try? That never happens. That's not even supposed to happen. Odds are the code I wrote has an error somewhere; I just don't know where.

At any rate, I plan to test it this upcoming week and get a new release out this weekend. Hope you enjoy it.

Release Retrospective

While people have been overall pleased and impressed with the BrainWorks release, I've received one common bit of negative feedback. Many people have said the bots use the shotgun too often, especially in situations where it's flat out the wrong weapon to use. Now I don't have the luxury other people have of saying, "that's not a bug, that's a feature!" Or more commonly, "that's working as intended." BrainWorks is intended to feel realistic. If many people say a portion of it doesn't feel realistic, they are right. I have no grounds on which to disagree. There really is a bug in the code because you say there is.

Of course, if you've ever been told quite literally, "this thing doesn't feel right, so go fix it," you know how challenging of a request that is. One downside of fundamentally basing the BrainWorks AI on causality is that debugging is extremely difficult. There's no magic number that tells the bot how often to use the shotgun. Instead, the bot analyzes its situation and incorrectly decides the shotgun is the right weapon to use. To solve this problem, I need to walk through the entire analysis and see how it reaches that conclusion. To make matters worse, the bots don't always choose to use the shotgun. And there are surely situations where the bot correctly chooses the shotgun. It's very hard to isolate the situations where the mistake as made and then narrow down why that mistake occurred.

My solution was to write some debug code that outputted all the data the bot used at each step of weapon selection analysis, starting from accuracy and fire rate data and ending with its final valuation of how quickly a given weapon would score a kill. I had it output this data for all weapons available and just stared at the numbers, checking if each data point was an accurate value of the concept representing it. I found three key contributing issues, two of which are solvable.

#1) The estimation of weapon fire rate was incorrectly bounded

Knowing how frequently the bot will fire a weapon is crucial to determining the actual damage rate of the weapon. If a bot spends 3 seconds aiming at a target and only fires for 1 of those seconds, it will do one third the damage as a bot that spends all 3 seconds aiming-- provided it's always lined up for a good shot. The bot tracks how often it actually fires the weapon, but I gave this value a lower bound of 50%. In other words, the bot assumes it will spend at least 50% of its time firing with the weapon, possibly more.

I added this assumption to deal with another problem. Bots would switch to a weapon they had never fired before, not have a perfectly lined up shot, and think... "I've spent 0 seconds firing and 100 milliseconds aiming, so I spend 0% of my time attacking. This weapon is terrible. I'm going to use something else." And they would never try out the weapon to see that they could in fact make shots with it. One problem with relying on historical data is wide variance in the initial data sets. I ran some tests with most of the weapons and found the fire rates were in the 60% to 80% range, so 50% seemed like a good bound.

The problem was that the shotgun only had a fire rate of between 30% and 50%, meaning it's hard to line up good shots with the weapon. There were situations where the bot would think it fired 50% of the time when in reality it had only attacked 30% of the time, which inflated the estimated value of the shotgun by a solid 60%. (160% of 30 is 50.) So it's no wonder they thought the weapon was good. Lowering the minimum bound to 30% had other problems though. When that happened, sometimes bots would stop using genuinely good weapons like the rocket launcher. That's a sign that a lower bound is not the correct way to solve this problem.

Related to this is the second issue:

#2) The estimation of weapon fire rate doesn't take location into account

If you're wondering why the shotgun is generally a bad weapon, it's because it's so situational. At point blank range, it can do more damage than the railgun in two thirds the time. But the spread on the weapon makes its value decrease rapidly at medium and long range. There are situations where the shotgun is good, but not many of them. In contrast, weapons like the railgun are consistently good in a wider variety of situations.

This means that a bot might shoot more often with the shotgun at point blank range than at long range because the shots are easier to line up. If the bot had a 40% fire rate with the shotgun, that might be a 50% rate when close to the enemy but only 30% when far away. Similarly, it's easy to line up shots with the rocket launcher when the target is below you, since you can just shoot at the floor. If the target is above you, you need a direct hit, and that's very hard.

If the bot knew how the firing rate could change depending on the combat situation, it would have a much better understanding of whether a weapon is a good choice against the current target. Unfortunately, the code only tracks a single firing rate for each weapon. The concept of where the enemy was located doesn't factor into the data, and that's a real issue.

The drawback of tracking fire rate data across each of the bot's 12 combat zones is that it will take much longer to get good estimates on actual fire rates. If there were problems before with bots accumulating fire rate data into one data "bucket", now that there are 12 "buckets", it will take 12 times longer before the data stabilizes. In other words, the issue that the 50% minimum fire rate was trying to address has now gotten 12 times worse.

I'm still thinking about the best way to solve this problem, but I'm leaning towards seeding the data with some reasonable estimates of how often the weapons really should be firing. If there's enough seed data, it should encourage the bot to act reasonably until it has enough of it's own data to make conclusions. That just leaves one more issue:

#3) Bots do not take into account how a good choice now could be bad later

The major drawback a situational weapon like the shotgun has is that when you use it, you give your opponent some control. They have the power to make your weapon worse just by backing up. While this is a concern for all weapons, the more situational a weapon is, the worse it is when your opponent exploits your weapon's weakness. In other words, the shotgun isn't just bad because it's only useful at close range. It's bad because your opponent can choose to be at medium or far range after you spend the time to pull out the weapon.

This is unfortunately a much harder problem to solve, since it has to do with the local level geometry. If your opponent has no escape routes, the shotgun is still excellent in close quarters. I haven't thought about this problem very much, but my intuition says that solving it in BrainWorks is well beyond the scope of the project. You might be able to analyze past reactions opponents had to your weapon choices, but it's not clear how good this data would be, how it would affect the bot's choices, and if this problem is even large enough that it needs such a sophisticated solution.

At any rate, I apologize if this post is a bit more technical than normal, but that's the nature of the work. I'm very interested to hear your ideas for how to tackle these problems. I plan on thinking through possible solutions over this upcoming week and writing the fixes next weekend. If you have thoughts on this, I'd love to hear them.

Turn Left at the Quad Damage

Have you ever gotten directions that included phrases like, "When you see the giant viking statue, go straight and curve left" or "If you pass a third Starbucks, you've gone too far"? People don't always give directions based on street names; sometimes the street names are largely irrelevant. Here in New England, if the street sign is even posted at all, it might not be readable until too late. Directions like "turn left at the Rite-Way Mart" are often easier to follow than "turn left at old route 293".

Believe it or not, bots have the same problem. We might conceptualize a level as rooms, hallways, and open areas, but these places have no names. Humans generally name these areas after a recognizable landmark. In Quake 3, that usually means items. You'd never tell a teammate, "I'm at (540, -973, 108)", which is how the computer records your position. But you would say, "I'm near the rocket launcher".

One requirement of item pickup that I glossed over in a previous post was this: Bots need to know how likely they are to encounter an enemy in a general area. That implies bots need a concept of what an area is. They need to think of enemies being "near the rocket launcher" not "at (540, -973, 108)".

Bots need to think of enemy location in human terms, not computer terms.

The obvious method of approaching this problem is to divide the level into regions, with each region being defined by one item. Technically each region is "the area of space that is closer to this item than any other item". Once you have the level conceptually divided into "near the red armor" and "near the plasma gun", whenever the bot sees an enemy it can say, "Ah, that player is near the red armor, not the plasma gun. I'm more likely to find enemies if I head for the red armor as well." Now whether or not it wants to enemies is a different question, one answered by item pickup. But at least it can track meaningful data about what other players on the level are likely to do.

However, just because it's possible to find out what item a player is nearest doesn't mean it's particularly fast or easy for the computer to determine that. The AI obviously can't spend half a second to calculate data it needs to save 10 times a second. It's okay to spend a bunch of time when the level loads to calculate the best set of regions, but once these calculations are done, bots need to do the location to region calculation very quickly.

I researched a number of different options, but eventually settled on the octree, briefly mentioned in a previous post. Conceptually an octree is a way to layout some number of points in 3-space. In BrainWorks, the item region octree contains the location of all items on the level. Rather than rewrite what has already been written on the subject, I recommend reading the wikipedia article on quadtrees (the two dimensional equivalent of octrees).

For the 99% of readers who don't care how to implement an octree, I'll just say that a well balanced octree will generally let you find which item is closest to any input location very quickly. The run time is O(log₈ N) time, where N is the number of items on the level. That means that finding the nearest item on a level with 64 items will take twice as long as a level with 8 items, and a level with 512 items will take 3 times as long as a level with 8 items. Since even the largest levels rarely have more than 100 items, this calculation is really fast, and that in turn frees up more processor time to think about different AI problems. The bot makes a richer impression to the human playing the game and hopefully the player has more fun too. Sometimes making a game fun to play involves implementing mind-numbingly boring data structures, but the fun just wouldn't be there if they were missing.

A Journey of One Thousand Miles

There is a very old Chinese quotation that goes, "A journey of one thousand miles begins with a single step." More literally it reads, "A journey of one thousand miles begins beneath one's feet." Wherever you go, the first stage in any journey is the place you are in now. This might seem trite, but it's crucial, and all kinds of problems arise if you ignore it.

For example, suppose you take a cross-country trip from Chicago to Los Angeles:


View Larger Map

The basic plan is to take Interstate Highways 80, 76, 70, and 15 to cover most of the 2000 miles. But none of these directions are actually useful unless you can get on I-80 in the first place! If you zoom in on Chicago on the map, you'll see the directions start from outside the Chicago city hall. You'd use a totally different route to get on the highway if you started on the south side of Chicago or one of the suburbs. Understanding your current location is crucial to route planning.

Now last week I promised this post was about item pickup, not navigation. So what does route planning have to do with item pickup? Determining what items to pickup is a kind of of route planning. If a bot wants to go to a different area of the level, there might be three or four different routes it could take to get there, each with different items available on the way. There may be a good reason to take a longer route that has better items. Items define way points between where the bot's current location and where it wants to end up.

The item pickup algorithm determines which items a bot should pick up while it goes somewhere else.

That might seem simple, but as best I can tell, it's a fundamental change in how item pickup AI has been done in every other first person shooter to date. Typically the problem is viewed as finding where the bot wants to go, and choices are picking up items, hunting down enemies, or guarding a specific location. I've approached the item pickup as what you do while you are also hunting down enemies or guarding a location.

There is one crucial difference between item pickup and traditional route planning problems, however. If I'm planning a cross country trip, I'm either trying to minimize travel time or travel distance. Item pickup isn't interested in either of these things. Why do players pick up items? To get more points in the game. Bots need to select the items to pickup between points A and B that will maximize their score, not minimize travel time. That's why it's often correct to take a longer route with better items. The better items will increase the player's score more than showing up at the destination completely unarmed. The item pickup algorithm I've implemented is completely focused on one thing: estimating the score gained from taking different items.

And because the journey of one thousand miles begins with the ground beneath your feet, bots need to know how effective they are at earning points in their current state (health, armor, weapons and ammo values). Then they estimate the value of a particular item by imagining how that item would modify their state and how that would change their rate of acquiring points. The overall item pickup algorithm involves estimating the points the bot gets going from point A to item X to point B, where X is any nearby item. But the actual meat of the algorithm is estimating how many points the bot gets given a specific amount of health, armor, weapons, and ammunition.

Next time we'll dive in and talk about exactly how this is done, and what other structures are necessary to make these computations possible.

Care to Comment?

It seems that everyone has an opinion on programming style standards, and no two people have the same opinion. So rather than tell you what you should do, let me talk about what I did with the BrainWorks code base. In retrospect, I feel like I did three things well and two things poorly. While I'm sure there are well more successes and failures, no one really cares how good or bad your code is as long as maintenance isn't a complete nightmare. I don't code well for your sake-- I code well for my own.

By the way, keep in mind that the Quake 3 code was written in C, so as a language it lacks some fundamental modern concepts such as "constructors" (a way to initialize information), "member functions" (a way to tie information to things the it can do), and a whole host of other things. Check out ai_lib.c for a reimplementation of things like binary search, memory management and hash tables-- things other languages can (and should) take for granted. I tried hard to keep things organized, but the choice of language made it difficult at times.

Success #1: Commenting

If you've looked at any of the BrainWorks code, it will be apparent that I'm a commenting Nazi. While sometimes I go a bit overboard, I cannot tell you how much time I saved by writing good comments. The purpose of commenting is to explain why, in human language, a block of code is doing things. Sometimes this means one line of comment can describe ten lines of code and sometimes you need the reverse. There's no magic ratio since a complicated thought process can be simple to explain to a computer or vice versa. For an extreme example of this, check out the comments for DataPerceiveCorrect() and ViewAxisModify() in ai_view.c. These functions are the very core of BrainWorks' vision algorithms, and both have header comments far longer than the actual code they contain. In both cases, I had to go back to these functions and I'm so glad I wrote down my thoughts. Without explaining why the code was doing things, it would have been much harder to figure out why it was doing the wrong things.

Commenting isn't just for maintenance. I actually write my comments before my code. If I can't explain in English why my code is doing the whatever it's doing, then I don't understand what I should even write.

Success #2: /ai_debug command

When the project doesn't come with a debugger, you can either litter the code with print statements or you can write your own debug interface. While I had to do a lot of the first, the development process was really helped by creating the /ai_debug command. This command lets you modify the behavior of any bot in real time, as well as force it the bot to give data describing what it's doing. For most of the weapon testing, I used this command to give one specific bot a weapon with infinite ammo and then had him output his accuracy with that weapon. I could see how the rates changed when I removed all error from his aim, or when I told his target to stop dodging (or even stop moving). For item pickup, I could make the bots tell me what items they chose to pickup and why they thought it was the best choice. A good test interface made it possible to analyze the vast sets of data a bot uses to make it's choices

Success #3: Overall Architecture

The top level architecture is handled by ai_action.c, and despite some issues with how some things must be written in the C programming language, I believe it's the best designed file in the whole project. It's a birds-eye view of everything a bot needs to do in a given processing frame. If you want to learn more about the BrainWorks code base, this file is the place to start. The comment in BotActions() explains how and when the bot decides to do things. Some bits of code must execute all the time while others must only run every so often. Structuring the bot's outer processing loop in this manner gave me excellent context whenever I needed to add a new feature. It was easy to look at this file and see exactly which other engines the new feature would need to interact with.

Of course, there are a few problems as well.

Failure #1: Lack of const correctness

This is just an inexcusable mistake on my part. I should have used the const keyword more (restricting a value from being accidentally changed) literally hundreds of times. It hasn't been a big part of my programming background, but doing so can prevent bugs from occurring. It also forces me to thing about function interfaces more. When I got to the end of the project and realized it should have been there, I felt like it wasn't worth the time. Everything already worked, but it should have been there from the beginning.

Failure #2: Lack of external data files

There are a few places where data structures are inlined directly in the code, and they should be stored in an external file and processed on startup. In particular, I'm thinking of ai_weapon.c which contains a description of the characteristics of every weapon. While this data shouldn't change, putting it in another format means non-programmers could read and edit it without rebuilding the project. The less you make someone else deal with your code, the better, no matter how well written it is. I suppose I could blame this on the annoyance of text processing in C versus some modern language, but the fault is really mine. If I'm willing to write memory managers in C, text processing shouldn't bother me at all.