Quake II - ERROR: *Index overflow (and Other Horrible Nasty Problems) SOLVED
By Tony "Mad Dog" Ferrara and David "Too old for pseudonyms" Hyde
You can download this document in its entirety plus MapSpy from the MapSpy page.
Most authors who have toiled for long hours on their pet Quake 2 map extravaganza have run into the "Error: *Index overflow" error message and/or related quirks of the Quake 2 engine. This problem has been addressed many times in the past on the Rust discussion board and other less reputable web sites. Unfortunately, the answers to these questions are usually filled with misinformation or just plain wrong answers. To compound the problem, taking the advice given in these answers will often solve the problem, but only temporarily. The two of us have cussed and discussed this and other problems in attempting to defeat them once and for all. We have succeeded only with a lot of help from two industry professionals: Alex Mayberry of Xatrix and Peter Mack of Rogue Entertainment. In particular, Peter Mack has been exceptionally gracious with tolerating our foolishness and sharing his expert knowledge on this subject. Go buy The Reckoning and Ground Zero NOW!!! (If you already have them, buy them a couple of more times. You owe these guys.)
Almost all answers you've seen to this question involve one or more of these statements:
None of these answers will permanently solve the ERROR: *Index overflow problem, though all of them may delay the error until the player reaches a different part of your map. Then again, they may solve nothing at all.
The Index overflow problem is due to images, models, and/or
sounds overflowing the limits of their respective indexes. The
limit for each of these indexes is 256. Therefore, you cannot
have more than 256 unique images, 256 unique models, or 256
unique sounds in a single map. That's the gospel from GodJohn
Carmack. For all of you mod authors, these constants are defined
in q_shared.h as MAX_IMAGES, MAX_MODELS, and MAX_SOUNDS. But what
exactly does "too many images, models, or sounds" mean?
What items does this include?
Images
Brush textures, environment maps, sprites, and model skins are not included in this image limitation. These images are maintained in the "image refresh" array, which may contain up to 1024 items. In our brief experience, it is very unlikely that you will exceed the 256 image limit, which mainly involves images related to the status bar (exclusively images found in the pics directory of PAKx.PAK). In other words, using a lot of textures would just be a burden on system memory, but shouldn't be the cause of a map crashing out. For that reason the remainder of this document will for the most part ignore image issues.
Models
Models include all of the game models (monsters, weapons, gibs, debris, etc.) as well as brush models. Brush models are any entity that contains a brush or brushes (duh), including func_ entities and all trigger brushes (trigger_multiple, trigger_once, trigger_push, etc.). Unique models fill the available models slots, not instances of models, so for example 3 monster_infantry entities on a map would fill only 1 model slot. Brush models are another story however, as each brush entity requires its own model slot. No matter how "identical" you may think 2 func_doors on your map are, they represent 2 unique models and as such require 2 of the 256 available model slots. Key points to remember:
The following models will always be loaded, regardless of the simplicity of your map:
maps/yourmap.bsp | Yes, your map is a model, too |
models/items/armor/effect/tris.md2 | Power shield flash effect |
models/monsters/parasite/segment/tris.md2 models/monsters/parasite/tip/tris.md2 |
Loaded regardless of whether you have parasites in your map. Go figure. |
models/objects/explode/tris.md2 | Standard fireball |
models/objects/flash/tris.md2 | Muzzle flash |
models/objects/gibs/arm/tris.md2 | |
models/objects/gibs/bone/tris.md2 | |
models/objects/gibs/bone2/tris.md2 | Various body/monster parts |
models/objects/gibs/chest/tris.md2 | |
models/objects/gibs/head2/tris.md2 | |
models/objects/gibs/skull/tris.md2 | |
models/objects/gibs/sm_meat/tris.md2 | |
models/objects/grenade2/tris.md2 | Open hand grenade |
models/objects/laser/tris.md2 | |
models/objects/r_explode/tris.md2 | rocket explosion |
models/objects/smoke/tris.md2 | |
models/weapons/v_blast/tris.md2 | Standard issue blaster, equipped with forearm |
models/weapons/v_handgr/tris.md2 | Hand grenade |
models/weapons/v_machn/tris.md2 | Machine gun |
models/weapons/v_shotg2/tris.md2 | Super shotgun |
players/male/tris.md2 | Grunt player model |
22 models minimum |
Notes:
Sounds
The limits on sounds include virtually all sounds you hear during a game. Unique sounds played by target_speakers, monsters, weapons fire, the player, as well as default sounds for pickup items and explosions must total less than 256. CD music is played back rather than loaded, so is not included in this limitation. As with models, each unique sound takes up one index slot, i.e. multiple target_speakers playing the same sound take up no more room in the list than a single target_speaker's sound.
The following sounds are always loaded at startup:
sound/infantry/inflies1.wav | sound/player/step3.wav |
sound/items/damage.wav | sound/player/step4.wav |
sound/items/pkup.wav | sound/player/u_breath1.wav |
sound/items/protect.wav | sound/player/u_breath2.wav |
sound/items/protect4.wav | sound/player/wade1.wav |
sound/items/respawn1.wav | sound/player/wade2.wav |
sound/misc/h2ohit1.wav | sound/player/wade3.wav |
sound/misc/lasfly.wav | sound/player/watr_in.wav |
sound/misc/menu1.wav | sound/player/watr_out.wav |
sound/misc/menu2.wav | sound/player/watr_un.wav |
sound/misc/pc_up.wav | sound/weapons/blastf1a.wav |
sound/misc/talk1.wav | sound/weapons/grenlx1a.wav |
sound/misc/udeath.wav | sound/weapons/lashit.wav |
sound/misc/w_pkup.wav | sound/weapons/noammo.wav |
sound/player/burn1.wav | sound/weapons/railgf1a.wav |
sound/player/burn2.wav | sound/weapons/rocklx1a.wav |
sound/player/fall1.wav | sound/weapons/xpld_wat.wav |
sound/player/fall2.wav | sound/world/electro.wav |
sound/player/fry.wav | sound/world/land.wav |
sound/player/gasp1.wav | sound/world/ric1.wav |
sound/player/gasp2.wav | sound/world/ric2.wav |
sound/player/land1.wav | sound/world/ric3.wav |
sound/player/lava1.wav | sound/world/spark5.wav |
sound/player/lava2.wav | sound/world/spark6.wav |
sound/player/step1.wav | sound/world/spark7.wav |
sound/player/step2.wav | |
player model-specific sounds |
|
sound/player/male/bump1.wav | sound/player/male/jump1.wav |
sound/player/male/death1.wav | sound/player/male/pain100_1.wav |
sound/player/male/death2.wav | sound/player/male/pain100_2.wav |
sound/player/male/death3.wav | sound/player/male/pain25_1.wav |
sound/player/male/death4.wav | sound/player/male/pain25_2.wav |
sound/player/male/drown1.wav | sound/player/male/pain50_1.wav |
sound/player/male/fall1.wav | sound/player/male/pain50_2.wav |
sound/player/male/fall2.wav | sound/player/male/pain75_1.wav |
sound/player/male/gurp1.wav | sound/player/male/pain75_2.wav |
sound/player/male/gurp2.wav | |
65 sounds total (if including italicized sounds, call it 70) |
The sounds in italics may or may not be loaded as the map is played, depending on the environment of your map: is there water to wade through or lava to burn in, etc. This isn't quite consistent with some of the sounds loaded by default for example, while the gasping sounds would never be needed unless a map contains water; they are nonetheless loaded anyway.
The engine loads the male grunt sounds by default, and then replaces these sounds if necessary with sounds for the model the player is using.
Debunking myths
Once an image, model, or sound is loaded into memory, it occupies the same index slot until a new map is loaded. Put another way, images, models, or sounds are never swapped out for other images, models, or sounds. It's obvious, then, that placement of these items and full vs. fast vis are irrelevant as far as solving this error goes. This is a key point, so pay attention: Once the Index overflow error occurs, no amount of hint brushes, areaportals, vis-blocking walls, rearranging or re-sequencing will save you. You must eliminate one or more images, models, or sounds. Adding to the problem however is the fact that this error message does not tell you which index has overflowed. If you wish to avoid blindly deleting sections of your map, you must determine whether you have too many images (not likely), too many models, or too many sounds, then cut back on these items.
Map size is also irrelevant to this problem. Though it's certainly true that a large map is more likely to contain more models and sounds, it is entirely possible to create a very large, detailed map that fits well within the restraints. Total .bsp size is another issue, however, which is addressed later in this document.
There is a common convention floating around that the "Quake II engine cannot handle more than 300 solid entities". This is obviously misleading if not just plain wrong. We have no idea what the Quake II engine can or cannot handle, but clearly the brush entity limitation is far lower than that.
Potential pitfalls
Spawned models can create problems because they are loaded on the fly rather than on startup. This includes their associated sounds and images as well. It is not safe, then, to assume that if your map starts properly that you will not encounter the Index overflow error. Spawned weapons, ammo, items, and monsters, as well as the sounds they produce, will all consume one or more additional index slots if they have not previously been loaded. This situation is likely the reasoning behind the erroneous advice "too many entities in view" and/or "too much ____ occurring simultaneously", since the Index overflow error frequently occurs during an event producing wonderful amounts of gibs, debris, or explosions. (This sort of thing could well produce the "SZ_Getspace: msg overflowed for <player>" error, however.) The Index overflow error instead occurs when the last entity that was spawned or sound that was played had not existed before, and it was the 257th model or sound to be loaded.
It is fairly simple to test all spawned entities in your map by playtesting. Unfortunately, this is not sufficient to insure Index overflows will not occur. If the number of your map's models or sounds are only slightly less than 256 each, it is still a very simple matter to crash the game. Give up? Try the "give" cheats to give yourself weapons/items that do not exist on your map; that should do it. "Give all" and "give weapons" are special cases where the models and their associated sounds are not loaded until the player selects/drops them, which sadly tends to obscure matters because the crash may not occur for some time after the "give" cheat was used.
Another troublesome aspect of this problem is in making your map part of a unit, particularly if multiple authors are involved in creating maps for the unit. Weapons and items carried over from one map to the next count against the 256 model and sound limit for both maps.
The only way of insuring that neither of these situations causes your map to overflow is to limit the number of models and sounds such that the addition of all dropable pickups would not cause an overflow. This includes all item_ entities other than health, plus all ammo_, key_, and weapon_ entities.
Co-op play can also result in straining your map's model budget. Each unique player model used in a co-op game is included in the model list. Interestingly, this limitation does not apply to deathmatch as co-op involves use of the save-game code on the client side, while deathmatch obviously does not.
Related weirdness
Although spawning the 257th model or sound should cause an immediate Index overflow, this is not always the case. If you are one of the fortunate few, you've been surprised by the very rare grunt launcher:
The grunt launcher has been substituted in this case for the much less entertaining grenade launcher, normally given up by our friend the dead gunner. Please don't ask us why, but picking up the grunt launcher results in an Index overflow. (Standing back and admiring him is OK, though.)
Even rarer are the grunt-firing flyers:
Well, OK, we fabricated this picture because we didn't save the original but it's a good approximation of an error we have actually encountered.
Other oddities we've seen include a target_spawner-spawned monster dropping a key (try to make one do that). If your map is exhibiting any similar strange behavior, it is likely that you have exceeded the limits on either models or sounds.
Solutions
Of course the most accurate method of solving this problem is to obtain employment with Rogue Entertainment and ask Peter Mack to stand over your shoulder. This isn't practical advice for most of us, though, so here are a couple of tools you can use:
"Developer 1; logfile 2" is our friend
We stumbled across the "developer" console command when attempting to solve this problem. We had seen it before, but saw no useful purpose for it. For the purposes of solving the Index overflow problem, the developer command displays all of the models, images, and sounds that are loaded while playing through a map. The "logfile 2" command saves all of the text messages displayed in the console to the file qconsole.log, normally in your BASEQ2 directory. NOTE: Many models, images, and sounds are loaded at startup, so to get meaningful information from your logfile you should have "developer" turned on before your map is loaded. Either a) start your map with a command line similar to: "quake2.exe +developer 1 +logfile 2 +map mymap", or b) start Quake2, bring up the console, type "developer 1;logfile 2;map mymap". (No quotes of course.)
The lines you are interested in in the logfile all start with "PackFile" or "FindFile". "PackFile" refers to files found in pakX.pak; "FindFile" refers to files found elsewhere, like in custom \env\ directories for example. The logfile format is somewhat screwy in that many lines are terminated with a line feed only, rather than the normal carriage return-line feed sequence. This might cause your text editor problems. Important: Saved games can complicate things. Quake 2 apparently displays many of the items found in your last saved game in the logfile, even though they are not loaded and do not appear in your map. To use the logfile effectively, you should first delete any saved games. Remember that Quake2 automatically saves games when a map is loaded.
Delete all lines not containing "PackFile" or "FindFile" and sort them according to type. Models have an extension of ".md2", sounds ".wav", and image ".pcx". Delete any duplicates and count each category. Sounds like fun, right? Well, no, you say, this sounds like a whole lot of trouble. You're correct, it is, and that is why we developed:
MapSpy - we're sure you'll insist on paying us for this
The MapSpy program is a Win32 console application that scans through your .map file for all of your entities and brush models. There are several advantages to interpreting the .map file rather than a logfile.
MapSpy has uses other than solving the Index overflow problem, but that is its primary mission. Its principal features:
Startup options - By default, MapSpy produces output for Easy, Normal, and Hard single-player games as well as Deathmatch. Command line options allow you to limit the output to only those game types you are interested in: -easy, -normal, -hard, -sp, and -dm. A sixth command line option, "-cheat", allows you to stress-test your map by insuring that at least one of each item_ entity (other than health) and one of each ammo_, key_, and weapon_ entities are included in the entity list, and that any associated models, sounds, and images are added to the corresponding lists. This option will allow you to see the effect of typing "give all" within the game.
Here's an example of the output produced by MapSpy:
MAPSPY 0.26
from Tony Ferrara and David Hyde
2 guys really, really tired of ERROR: *Index overflow
Results for base1.map -
10/26/98 12:00:00
==========================================================
Entity list, including spawned/dropped entities
Entity | EASY |
NORMAL |
HARD |
DM |
|||
ammo_bullets | 0 |
0 |
0 |
6 |
|||
ammo_cells | 0 |
0 |
0 |
4 |
|||
ammo_grenades | 2 |
2 |
2 |
3 |
|||
ammo_rockets | 0 |
0 |
0 |
1 |
|||
ammo_shells | 6 |
6 |
6 |
8 |
|||
func_areaportal | 1 |
1 |
1 |
1 |
|||
func_button | 2 |
2 |
2 |
1 |
|||
func_door | 4 |
4 |
4 |
2 |
|||
func_explosive | 2 |
2 |
2 |
0 |
|||
--- snip --- |
|||||||
weapon_shotgun | 1 |
1 |
1 |
1 |
|||
weapon_supershotgun | 0 |
0 |
0 |
1 |
|||
worldspawn | 1 |
1 |
1 |
1 |
|||
-------------------------------------------------------- |
|||||||
Total entities | 596 |
596 |
597 |
548 |
==========================================================
Total number of brush models = 34
Models, Sounds, and Images
Used
Totals
EASY |
NORMAL |
HARD |
DM |
|
Models | 78 |
78 |
78 |
88 |
Sounds | 132 |
132 |
132 |
122 |
Images | 39 |
39 |
39 |
47 |
NOTES:
1) These lists contain ONLY those models, images, and
sounds which contribute to the ERROR: Index overflow error.
2) MAPSPY does not check for the existence of sound files.
This includes sounds played by target_speakers, as well
as sounds which SHOULD exist but do not (e.g. trigger1.wav
for trigger_multiple with sounds=3.
3) player/male model and sound/player/male sounds will be
replaced at runtime by the appropriate player model and sounds.
4) Inhibited brush models are included in the 256 model limit.
================== EASY
==================
MODELS
Total number of models = 78
34 brush models
base1.bsp
models/deadbods/dude/tris.md2
models/items/adrenal/tris.md2
models/items/ammo/grenades/medium/tris.md2
models/items/ammo/shells/medium/tris.md2
models/items/armor/effect/tris.md2
--- snip ---
You can download this document plus MapSpy here. For those interested in the internal workings of MapSpy, this zip file also contains a text file enumerating all of the special cases that MapSpy handles, e.g. door sounds vs. the "sounds" property. Comments, suggestions, and criticisms are welcome at either of the e-mail addresses found at the end of this document. In particular, if you add new entities and/or modify existing entity associations in the database, please forward those changes to us. NOTE: Please include "MAPSPY" in the subject line of any correspondence.
Please note that MapSpy will not solve some of the problems mentioned above, particularly co-op play. In fact, it will solve nothing if the mapper declines to read its output. However, it will give you a very good starting point for solving the Index overflow problem in your map. Additionally you can use MapSpy to monitor these indexes as you make new maps, allowing you to add lots of really cool stuff without fear of encountering this error when you're all done.
Techniques for reducing your model and sound counts
So you now understand the problem with overflows but you really, really don't want to eliminate any of those monsters/doors/trains/etc. that you've so carefully worked on. Here are a few ideas for eliminating models or sounds that just might reduce the count enough to save your map.
Like all trigger fields, the trigger_multiple is a model, and takes up one of the 256 models slots. If something is to happen each time a player passes a location, consider using a nearby door to target that event instead. Generally you'll be targeting a func_areaportal with the door anyway. Just give the other entities you wish to trigger the same targetname as the areaportal.
Locked doors are used often in single-player, and such doors require a trigger_multiple with the "triggered" spawnflag set to operate. It is tempting to inhibit the trigger_multiple in deathmatch, and use another one inhibited in single-player games for deathmatch which lacks this restraint. But there is no reason at all to take up 2 model slots with 2 different trigger_multiples. You can either: a) inhibit the door as well in deathmatch games, or b) use a trigger_always, inhibited in all modes but deathmatch, which triggers the trigger_multiple automatically when the map loads.
Buttons and other puzzle-related entities used in single-player are generally inhibited for deathmatch games. The frames or other dress-up brushes associated with them can often be made into func_walls so they can be inhibited as well for a neater appearance in DM. If you're pushing the model limit, you may have to forgo this practice and leave them as ordinary brushes. Put a playerclip brush or two adjoining them so players don't get hung up on their edges.
You may have figured out by now that too many different monster types in the same map can cause problems, particularly with sounds. Remember that our problem is not too many total monsters, but too many unique models or sounds. Eliminating one of the tougher monsters in favor of several weaker types you already have on your map elsewhere anyway will cut down on the total model and sound counts, and may make for a better fight if the monsters are placed well. Eliminating a monster type will particularly help your sound totals, as all monsters have many unique sounds that are not used by any other entity. Following are a few key points that may assist you in your decisions:
Other considerations
As stated earlier in this document, total texture memory and map size do not influence the Index overflow problem in any way. However, thats not to say that you should not also monitor these quantities. Conversations with both Alex Mayberry and Peter Mack revealed that when designing maps for The Reckoning and Ground Zero, their target for total bsp size was 8-9Mb. This is not the file size of your map; rather it is the size reported at the top of the list of models reported by the modellist console command. At the console, type "logfile 2;modellist", then examine the file qconsole.log. Admittedly, Xatrix and Rogue must target lower-end machines (with less RAM) than most of us care about, but if your map is considerably larger than this you should consider breaking it up into multiple maps. Thats not to say that your map will break if you dont heed this advice, but the larger your bsp is, the more likely are game slowdowns due to disk access.
You also need to monitor the total texture memory used by your map. At the console, type "imagelist". The last line, "total texel count", gives the total size of all textures, environment maps, skins, and sprites used by your map. There is no hard and fast rule here, but if you check the original id maps youll find that none of those maps exceed a "texel count" of about 3,000,000. Again, this is a function of available RAM, but the larger your texture memory requirements are, the more likely gameplay in your map will suffer from swapfile access.
Whenever you design a map, you should decide and always keep in mind what your "target machine" is.
Bottom Line: Test your map
Although the MapSpy program should certainly help you in eliminating overflows, the only way to be reasonably certain that your map will not crash is to thoroughly test it. A few pointers:
Acknowledgements
Thanks to Richard Neff, Alex Mayberry, and Peter Mack for their time in reviewing this document. And once again, a huge THANK YOU to Peter Mack for all of his efforts at tutoring the tutors.
22 October, 1998
David Hyde | rascal@magnolia.net |
Tony Ferrara | tonykat127@email.msn.com |