It happens on rare occasion that the characters I GM for end up in some bigger battle. Often this follows lots of preparation and planning of strategies and tactics. Sometimes it just happens that the charaters are pulled in, but almost always this is quite a climactic and importent moment. Being in a battle with scores/hundres/thousands of others is something that happens rarely to most characters and is an event they should not soon forget.

From a GM point of view however, I struggle a bit with the task of keeping these situations as interesting and as suspenseful as I think they should be. I feel this has several reasons:

For the sake of the argument I'm going to assume some sort of medieval battle scenario, but I guess most of it applies to more modern scenarios as well. Also please note that I'm not looking for rules or mechanics on how to simulate mass battles. The battle will turn out the way I want it too (within reason), and not the way some strategy rules tell me. :)

The stream of foes in a mass battle is essentially unlimited. -
The players can defeat some foe, and will be facing the next one almost instantaneously, thus drawing out the single encounters can quickly make the entire battle seem very dull. If we cut corners however, then the magnitude of the battle, the carnage and the deaths will be cheapened and watered down.

How can I strike a balance between endless dice rolling for the long fighting and a cheap feel to mass battles?

The PCs are not alone -
During a mass battle scene the PCs will be constantly surrounded by friends and foes. Positioning and who you have at your side can often be a more deciding factor of whether you live or die, than pure swordsmanship. Thus allies could cheat PCs out of their kills, or save the PCs lives, or retreat and abandon them. A couple of foes too many could overrun and utterly kill the PCs.

How can I keep up player agency, while the characters are thrown back and forth by the tide of battle? (Should I even be trying to do that?)

Some people feel this is not a system-agnostic question, here are some of the systems where the above problem arose to a lesser or higher degree: Rolemaster/MERP (low magic high fantasy), DSA (high magic high fantasy), Star Wars D6 (Sci Fi/Space Opera), Degenesis (post apocalyptic), HEX (pulpy adventure in a hollow earth). But then again, the rules do not really matter much to me here. I can bend or change the rules to make it work if I know how I want to handle a situation. I think my problem lies more with how such mass battle situations can be kept interesting for players.

5 Answers
5

Establish PC's goal in the battle

Never forget that the battle has a purpose but your PC's motivation might be different. Are they fighting to help win the battle to save a city? Are they forced to fight and want to survive together? Are they protecting an NPC friend who will fight no matter what? Do they defend their honor after swearing that the right flank will hold? The battle itself cannot be the goal, there has to be a well-defined challenge for the party.

Assure that players have competence to attain that goal

If they are just fighting to win the battle, they need something to differentiate them from a sea of regular troops. They can be commanders or officers, they can have their own cohort to lead or they could just be veterans instilling hope and courage in others. Otherwise they will have no agency.
Conversely if they wish to attain a different goal, such as defending their friend prince from death, they need to be put near him - or have at least a way to reach him.

Treat the battle like a scene that surrounds the players

It's easy to forget that the battle is just another way of shining light on the characters. Your players are assumed to fight near constantly, unless they can manage to disengage the opponents. Do not treat that as combat encounter, but rather as a puzzle that can be solved through leadership, cunning and violence. Instead of having your barbarian use his rage to fuel his damage output, let him burn it as a mean to attain advantage. So, using that as an example if your players protect an NPC - Alice, The Brave Maiden from death in battle, Brong the Barbarian might spend some of his rage to clear an opening to reach her in time.

Make battle events relevant to the players in their own scope

Which means that as the battle is going they seem to hold their own, but when the enemy sends reinforcements, they might have to fight an elite enemy as if it was a standard one-on-one encounter. If they rally their troops to exploit an opening, describe how lowly soldiers ravage enemy lines with their newfound courage. Also, limit their knowledge - they shouldn't have any idea what happens on the other flank, unless party rogue climbs the remains of a catapult to scout the situation.

Ensure party goals can be met regardless of the outcome of the battle

That's pretty straightforward. Give players an opportunity to carry mortally wounded Alice into the forest. If their flank collapses, let them be warned to cast invisibility to escape. If the fortress is being overwhelmed, let them carry the MacGuffin to safety in the nick of time.

Last but not least

Assume they act as a team like they would normally do

That means protecting each other, keeping formation, using their basic fighting skills to attack and defend. Don't force the players to tediously adhere to the battle grid or roll dice all the time. Just assume they are fighting a battle and throw in a juicy description how Cerion the Fighter shield-bashes a particularly large orc and Daria the Mage blasts him into a group of goblins. If you want to portray battlefield attrition, you can abstract their actions to a single "performance" roll, which would penalize them with wounds or depleting battle supplies for a bad roll, e.g. Daria would use one of her spells or Cerion would lose some hitpoints. You may even let them haggle with you to get out of trouble, offering to use up a spell or lose a piece of equipment instead of being wounded etc.

Mass battles usually abstract mobs as if they were single units. You have to do the opposite - abstract all the fighting chaos into landscape events not different from weather or sunset and have particular bite-sized challenges appear from it.

Refer to fictional battles as reference material

And the classic example of that is Battle of Helm's Deep - let's stick to the film version.
The characters have a goal - to protect people of Rohan in the fortress (and to win a kill count contest). They have competence and a role - they are leaders and champions, even though the battle is commanded by Theoden and elvish force by Haldir, they have an influence over troops around them. They fight all the time, but only interesting bits are shown on screen. They only are aware of their immediate surroundings and what happens at the gates is not shown while they are in the keep. They in the end achieve their goal and even if Eomer wouldn't come, they would still have hope for women to escape from the caves. Finally, they meet specific challenges - to hold the wall first, then to kill the famous Torchbearer Orc which fails, then to hold an opening for the elvish troops to regroup. When a battle takes turn for worse, Gimli and Aragorn heroically defend the bridge allowing soldiers manning the gatehouse to rebuild, and afterwards protect the king while he retreats to the keep. Then they convince him to lead one last sortie, while arranging for the women to flee through the mountain pass. Plenty of small-scale challenges in the midst of total chaos and what a battle to behold!

In my opinion a battle should be treated like a travel session, that is, it's dotted with meaningful encounters that do not make the bulk of the action for the characters (who will be knee deep in fights for hours on end) but will be what they'll utimately remember in the following days.
You mention it: an ally backstabbing them, a party member caught in a trap, an important enemy officer suddenly brought near by the tide of battle. Those are the "encounters" that should take place during your session, only those should be played out. You may want to add some other "battle effects" to your PCs, like reducing their hit points, spells etc as applicable between these encounters. Maybe when they come near the enemy general they'll duck and try to avoid him, being already weary and wounded.

Make The PC's Important

What I've found works best is to highlight the fact that the PC's are extraordinary actors on the battlefield. Set up mini encounters at places that will allow the PC's to turn the tide of the battle. Whether it be taking down the troll that's punching a hole in the allied line, sneaking into the besieged castle to open the gates, or charging in first and striking down the leadership, make it clear the battle is going well - or not well - because of the actions of the PC's at these turning points. You can plunge them into a line fight for a little while, but keep it dramatic - a whole squad crumbles that needs to be replaced by the Fighter with the assistance of the rest of the squad.

Make The PC'S Pick Their Battles

If you want to give the PC's more control over the mini-encounters, give them pairs of opportunities with obstacles between them and the goals. You want to take down the chieftain who just showed his head? You'll need to work with the siege engine crew who are throwing stones at his bodyguards to make it a fair fight. Want to break through the gate? You get to figure out how to get around the boiling oil so the battering ram can be brought into play. Use these in between skill challenges to play up the chaos of battle.

When I have run mass battles for my players, to keep them all of them involved in the fight, one of them, usually the highest level fighter or the one with the best charisma (or equivalent), runs the army and makes the decisions. Since my players are not expert generals, I usually assign an additional player to assist. Remember lines of sight, fog of war and how far they can personally command troops.

The remaining fighters, staying in a group, along with the healers search for enemy commanders or heroes to engage. To keep it interesting, have the enemy refuse to engage, have them back up around their troops to force the PCs to either fight their way through or go around.

When fighting massed, low level, troops, don’t force your PCs to do combat, allow them to kill and take damage on hp/(Difference in fighting ability/AC) (e.g. fighting low level orcs with relatively poor AC and low hp, for every dead orc, take 1 hp in return) basis and force their way through, they are, after all heroes right? When they do engage the Commanders/heroes, have them do a standard battle while all around them the battle rages around them.

The remaining Spell Casters, are your skirmish troops, you don’t want them in the thick of it, but have them on the outside of the battle lines acting as a light tank company, locate and assess the enemy using their spells as they see fit to help either the Army or their fellow PC’s as required. They can also send information to the Army Commander to help obviate the fog of war/lines of sight restrictions.

Medieval battles were fought with men in lines (a tactic that continued through Napoleonic Era and the American Civil War) so skirmish troops (those outside of the main battle lines) have a job of being able to find targets of value.

System Matters Greatly

Although this question was initially labeled "system agnostic", the system you're using greatly changes how mass battles work.

For example, in a game like Primetime Adventures, which completely lacks any combat system at all, a mass battle is like any other conflict and therefore, equally as entertaining. Other games include subsystems for mass battles, which again, doesn't entail, "have a fight, have another fight, have another fight" as an issue.

That said, the rest of this answer is going to assume you're talking about the subset of games that a) focus on small scale/personal battles, b) with some tactical element, c) without any rules for large scale battles.

Pick the Key Points

Endlessly fighting random guys is not that interesting, fun, or a good use of time. Instead, pick key points ("Hold the gate", "Stop that commander!") and make victory conditions based on that. I wrote the Big List of Combat Stakes that covers a ton of potential things beyond "kill more dudes". While fighting may be super useful, things like mobility, trickery, or magic (well, especially magic) can be very useful for achieving goals without it just being a grind of beat downs.

Make some interesting choices available

"Fight more guys" is not a choice, and while it might be tactically interesting in small bits, you can do that just fine without a large battle. No the problem in a large battle is that EVERYTHING is happening at once, everything needs help, and which thing are you going to prioritize?

The gate is about to fall, that ogre champion is making his way to your general, and your best friend is wounded and needs to get pulled to safety... where do you go? What happens if you're given bad orders? What happens if the orders would have been good 10 minutes ago when they were sent but by the time they reached you, the situation has changed? What if the orders were sent by a commander who you're not sure if you can trust?

Incidental Fighting becomes secondary rolls

You can make simple checks or rolls for all the incidental fighting or threats between these things.

"Make an Endurance Check or have this minor penalty as you fight your
way through the melee."

"Make a Perception check to safely find a path between the cover and
wreckage and not take any arrows along the way - you're lose a couple
of hit points"

"You want to get there quickly? It's a Balance check to pick your way
through the mud, bodies and thrashing horses. Otherwise you show up
after 2 rounds pass."

What you'll notice is happening is that you are effectively making rules to cover mass combat because the system you are using doesn't do it particularly well by itself. This is the reason I highly emphasize that system matters - even if you don't use a system that does, at least being familiar with some that do will give you some tricks to import.