Basics
An army is composed of squares with combatants in it, much like squares with crowds. A square is treated like a creature for all aspects of combat (unless otherwise stated). A combatant square should only contain NPC-classed characters or monsters. A square of combatants has a movement equal to the slowest creature it contains. It follows all the normal crowd terrain rules, imposing the 2-square movement cost. It can also be directed as a crowd can, usually by the general. Once the armies meet, the generals make opposed rolls to attempt to force the line in a desired direction. It is also available on the Equipment and Combat board for easier discussion.
Size
Multiple squares of combatants take on the next largest size category as soon as it is one square larger than its current one, and these need not take the shape of a normal creature. Therefore, a group of combatants becomes Large at 2 squares, Huge at 5 squares, Gargantuan at 10 squares, and Colossal at 17 squares. As a square is 5 feet, it can never be smaller than Medium-sized, and can never be smaller than the size of the creatures occupying it. Area effects must target individual squares. For "buffing" spells and effects (such as bless, or benefits from bardic music), if at least half of the squares of a large group can be affected the entire group can be affected as if it were a single creature. If multiple identical effects are present from different sources, the areas can be combined to determine if the entire group is affected.
A group of combatant squares is treated as the largest size category it fits in, but individual squares can act or be treated separately for certain things. When attacked, the attacker can decide (before the attack is rolled) to attack the group as a whole or as a single square or squares. This determines both the size modifier to Armor Class, and the target Hit Dice. The group may likewise perform actions as a whole or as individual squares. When a square makes an attack of opportunity it is always treated as a single square.
Hit Dice
A square with combatants has Hit Dice equaling the total Hit Dice of the creatures in the square. A single square may contain four times the number of creatures that can normally fit in the square (as if squeezing). If two or more squares with identical combatants are adjacent they are treated as a single entity (see Size, above). If at any time a larger group is split into smaller groups, the Hit Dice are divided as dictated by the space in squares. Combatant squares should always be NPCs, and therefore use an average hit point amount per Hit Die, making the division of hit points between groups easier. A single square can contain as little as one creature (although this creature may be better considered a normal creature instead of a combatant square). A square of combatants can have fewer hit points than total creatures as this is only an abstraction of large groups. A square reduced to 0 hit points is considered dead, and the dead bodies are treated as dense rubble requiring x2 movement to move through, and adding +5 to the DC of Balance checks.
Initiative
Crowds move at Initiative 0. To decide ties (such as among opposing armies), the army controlled by the leader with the highest Charisma modifier goes first. Any uncontrolled squares of combatants (such as if the leader is dead, or the squares are under a fear effect) are moved as a normal crowd. Most uncontrolled armies will hold ground or flee, depending on whether or not they are attacked, as the DM decides.
Armor Class
Armor Class is equal to the AC of a single creature in a group of combatants, modified by the size of the group. As a group cannot be composed of dissimilar creatures there should never be a conflict. Also see Size for rules governing "buffing" effects.
Base Attack/Grapple
A group has a base attack equal to a single creature within it. As a group cannot be composed of dissimilar creatures there should never be a conflict.
Attacks
A group makes attacks at its relevant size if it is composed of multiple squares. A single square acts as though each creature after the first is aiding another to resolve all attacks. If a square's base attack is high enough for multiple attacks it may use these as a full-round action, as normal. It may make attacks of opportunity (see Size) as normal, but always as a single square. Combatant squares are limited to only one attack of opportunity per round regardless of the number of creatures they contain, and may only make one attack of opportunity against each creature. Squares which contain creatures with the Combat Reflexes feat may make a number of attacks of opportunity equal to those of a single creature regardless of the number of creatures they contain. Otherwise, squares of combatants may make any special attack actions they choose (or are directed to).
Special Attacks and Qualities
A square may use any abilities all of the creatures within it possess as if it were a single creature with uses of its abilities equal to the combined uses of the creatures within it. If the ability can be affected through an aid another action it is, as though all creatures after the first were aiding another within a single square (even if the square is part of a larger group, or using the ability as a larger group). Any abilities that do not have uses (such as damage reduction) are applied to the square or group once.
Saves
A square or group of squares make saving throws collectively as if a single creature with saves equal to one creature contained in the group. Any effect that targets a single creature can target a single square, and any area effects (aside from "buffing affects, see Size) are applied to individual squares, although the save is still made collectively. Effects that target multiple creatures must count their targets against each individual creature, although the save is still made collectively.
Abilities
A square or group has ability scores equal to a single creature of the type contained within it. It uses appropriate size modifiers as if a creature of a size equal to that of the group for certain checks (such as Strength checks for trip or bull rush attacks). A square may aid another a number of times equal to each creature after the first for ability checks that allow it, but a large group cannot aid another more times than possible in a single square.
Skills
If a square or group of combatants needs to make a skill check, it follows the rules for aiding another established in Abilities.
Feats
A group has the same feats as a single creature it contains, and may use any of these that a creature its size normally could. Any feat that is dependent on multiple creatures (such as Phalanx Fighting, Complete Warrior) counts each creature possible in the group, as normal. Armies can also be trained with group tactics, allowing combatants to qualify for feats as if they were larger size categories.
Challenge Rating
The CR of a square of combatants is equal to that of a single creature of the square's type, +1 for each doubling within the square (instead of +2). The encounter level of a group of squares is determined as if each square were a creature, as normal.
I think that about covers everything for the long version. I wanted to write it out in full like that first to cover all the angles that could crop up. In short, the long version is more intended if a PC is the general or in some other position of command. Now for the quick-and-dirty explanation.
Short version
A square of combatants is treated as a single creature with Hit Dice equal to the total Hit Dice of all the creatures in it. Use the Size rules above as a guide for this. Any actions the square takes (attacks, attacks of opportunity, etc.) are treated as if all the "extra" creatures in the square were aiding another. Squares can group together to form larger "creatures" (see Size) as needed. Squares make saves collectively, so long as all squares in an area of effect are the same.
These rules are intended for massive battles. When used, it is recommended that each square only attack other squares of combatants unless there is specific reason to do otherwise (such as an order from a commanding character). Likewise, squares of combatants should only take attacks of opportunity when an opponent enters their square. The best application using this method is to have massive bodies of warriors (or similarly-classed NPCs) duking it out (it cuts down on die rolling exponentially), with perhaps some specialized units such as archers or casters acting separately. For fast results (if you want some game mechanic to decide large battles), assume that all squares are taking 10 for everything and compare results. If one side cannot hit another, subtract one point from the damage it can do and add that to the attack roll until an attack is successful (but not more than that).
"Prop" version
Another option is to ignore the possibility of organization and have uncontrolled fighting (such as a riot or chaotic melee). Each square no longer threatens an area, and should be treated like a hazard. Characters entering a square of combat must make a Reflex save (DC 15) or be damaged by stray blows for 1d6 hit points. Single creatures can be randomly encountered as desired (one could make a random encounter table, but I've gone through enough trouble already).
Argument
I think it isn't hard to implement, and should drastically cut down on excessive rolling. This should also allow for definitive results when the DM either needs to know or does not simply want to leave it up to uncertain storytelling. For others, the short version and "prop" rules can be applied simply as a layer on crowd rules, and battles can be decided as the story needs. Some armies will also have "hero units," which would be individual characters or groups of characters that do not follow these rules. These rules are only intended for NPCs (although it's possible they could work for PC class groups). Whether you read the whole thing or not I'd love to hear your comments. I hope the sectioning made it easier to swallow. Enjoy.
Base Attack/Grapple
A group has a base attack equal to a single creature within it. As a group cannot be composed of dissimilar creatures there should never be a conflict.Ok, so I can't have a combo unit? Also - a medium sized unit could only be composed of small or smaller creatures. There could be room here for an extended size progression over colossal for something like 50 trolls.
Swarm/Squad - (from the philologist corner) Calling this swarm rules may be a little not what you want but "Squad" could work. Then you can define squad construction. (Surprise) Had some thoughts along these lines myself. I feel that the most usable component would be guideline for creating the squads, and defining their special attacks based on the constituents. Simple stuff even - five ogres and one goblin sorcerer, four level 2s and a level 5 fighter. That kind of tac-ops.
You say this is intended for massive battles. Numbers? How many creatures are recommended per unit? Where do you think that the line is drawn?
Actually, thanks to the Dungeon Master's Guide II and its mob template, I was able to better form this idea. It now also takes great advantage of Heroes of Battle, particularly the Commander Rank and Morale rules. Here's a link to the new thread.
Attacks
A group makes attacks at its relevant size if it is composed of multiple squares. A single square acts as though each creature after the first is aiding another to resolve all attacks. If a square's base attack is high enough for multiple attacks it may use these as a full-round action, as normal. It may make attacks of opportunity (see Size) as normal, but always as a single square. Combatant squares are limited to only one attack of opportunity per round regardless of the number of creatures they contain, and may only make one attack of opportunity against each creature. Squares which contain creatures with the Combat Reflexes feat may make a number of attacks of opportunity equal to those of a single creature regardless of the number of creatures they contain. Otherwise, squares of combatants may make any special attack actions they choose (or are directed to).
This is the only major gripe I have with the rules: are you seriously suggesting that a platoon of, say, fifty level 1 soldiers can only make one attack per round, albeit at a colossal attack bonus thanks to all the people helping? To me, that just doesn't seem right.
My inclination would be that a group should make a number of attacks based upon A) the base attack bonus of its individual squares, and B) based on how many squares can actually reach and effect the enemy group.
Basically, if a square is adjacent to an enemy square, it makes however many attacks it is entitled to by it's BAB. Squares armed with reach weapons, such as longspears, get to use their reach, and make as many attacks as they are entitled to by their BAB, against any square that is either adjacent to them, or within their reach. A square may only attack the nearest enemy square, regardless of reach, and intervening friendly squares confer a +4 soft cover bonus to AC to the attacked square.
If this would result in a stupid amount of attack rolls per round, I suggest rolling four d20's, dropping the lowest result, and applying the remaining three results equally across all the squares that can attack in that round.
Finally, any squares not actually within attack reach of an enemy square may move at up to their maximum movement speed in order to "lap around" the flanks of the enemy and enter the fray.
Oh, also, I believe that a group (the whole group) should count as being flanked if a significant portion of both of any two opposite sides of the formation (use your discretion here) are in contact with enemy squares, AND the flanking group outnumbers the potential flankees by a ratio of 1.5:1
Other than that point, I like the cut of your jib.