It seems like a lot of roleplaying games quite freely allow players to play antisocial characters who can't really be part of the team. I've been part of groups (in D&D) with both a noble defender of the law, and a brutal tyrant whose goal is to gain power at all costs. The characters don't have much incentive to work together, and in fact have plenty of reason to hate each other. That the group doesn't break down into PC-vs-PC combat is only due to the players being more team-oriented than their characters.

It seems to me like you could add a few extra house rules to promote good team-oriented behavior, but I don't know what they would be.

So what kinds of rules should a game have to promote team-oriented behavior?

I'm looking for rules, not gm-techniques. Our group is interested in experimenting a bit with the rules of the game, and we're wondering what kinds of rules would promote a good team. These might be rules about gameplay, or they might be rules about character creation.

"It seems like a lot of roleplaying games have no problems with players playing antisocial characters who can't really be part of the team." Citation needed. While it CAN work, I wouldn't say it's without any problems, and that it requires very dedicated players to get such characters to still be 'part of the team' or at least 'part of the story'.
–
ZibbobzFeb 19 at 14:59

@Zibbobz, perhaps I should rephrase that. A lot of RPGs pose no problems for creating a party that can't work as a team -- they end up with plenty of problems when playing such a party.
–
JoeFeb 19 at 18:02

Aaah, that makes a lot more sense to me. And, I can relate.
–
ZibbobzFeb 19 at 18:08

The best game with a system for party cohesion I've seen is Warhammer Fantasy Roleplay's 3rd Edition. As part of character creation, the players collectively select a 'Party Sheet' that describes the nature of their party.

A number of different party sheets (such as 'Swords for Hire' or 'Servants of Justice') are included in the rule set, each with their own traits and rule mechanics. One of those mechanics is a 'Tension Meter' that tracks how well or how badly the party is cooperating. If tension reaches certain levels, penalties are applied to the party.

Because this is an explicit feature of the rules system there are references to it whenever it is relevant in both the rules and the published scenarios. This makes policing party coherence seem less a matter of arbitrary spite from the GM, and more of an inevitable consequence of character behaviour.

There are a number of other features of the party sheet system. In fact, the whole concept of the party as an entity with stats and rules is particularly inspired. The only other game where I've seen anything similar is Ars Magica's covenants which are the home bases for the characters.

It also leads to the possibility of some kind of parallel to character advancement for the party itself, quite possibly with in-world manifestations such as a guild house or retinues of followers. If you tie the 'advancement' of the party to the degree of cooperation shown by the players you may find that they will start to balance the trade-off of independent or outright hostile actions with the benefits of ongoing party coherence.

One "rule" that always works is to begin character creation with the requirement that the characters be an established group with a particular purpose. It's up to the players to decide what their group and purpose is, but it must be reflected in their character's creation and backstory. Non-exhaustive examples are a merchant team, a squad of soldiers at war, a mercenary company in a noble's employ, a secret society, a cult, a pair of priests and their templars, bandits, the crew of a privateer, and a band of dwarves seeking their lost homeland's treasure.

Most importantly, the PCs need not be restricted to cardboard cutouts of the group's archetype—the cohesion is from the shared purpose, not all being the same kind of character. For example, a merchant team easily has room for one smooth-talking character, a couple warrior types, and an esoteric-knowledge type – they don't all have to be non-combat traders.

Naturally this biases the game towards certain kinds of plots, but this is in many ways an advantage since it gives the GM ways to make adventures personally meaningful to the PCs as a group and individually.

One caveat: the essential point is to avoid having the players create their characters in a vacuum, and you may have players protest that they should be able to make "any" character they want free of limitations. It's more important that the game be fun for everyone than it is to achieve some mythical standard of "pure" player character-creation autonomy, though. Sometimes a game about a raggedy band of mismatched adventurers just doesn't cohere. When it does, it's fun. When it doesn't, working together to create a coherent band of characters is the answer. Besides, the players are making up the group themselves – if a player wants to play a particular kind of character, there is ample opportunity to craft the group to have room for their character idea.

Committee for the Exploration of Mysteries is a GM-less game with major game mechanics that are basically group voting both to decide on the hazards in play but also to rule on whether other PCs' narration is acceptable.

I've had to deal with this issue quite a lot with my gaming groups. The first method I tried was modifying the adventure in a way that forced the players to co-operate. Powerful enemies that needed the entire party to fight, traps that required more than one player to get past.

Alas, I overestimated my player's ability to play antisocial egomaniacs and constantly ended up with most of the party dying each session. It seemed suicide was preferable to co-operation.

The rule I ended up implementing was experience. Players who co-operated during a game got bonus experience for each time they co-operated. Players who did not co-operate lost experience. Any player that got negative experience over 3 sessions, or who lost enough experience do go below level 1 were asked to leave the game.

Brutal, but it would be an effective filter.
–
SevenSidedDieNov 2 '11 at 3:51

2

Yes it is brutal, but the "stick" is really not the way you train the players, it's a way of eliminating the element you don't want. Training the players comes from the reward, because the reward creates a positive belief that this is a better way to play, you get more as a team.
–
VethorNov 21 '11 at 20:47

I approached your question from the player side. What would make me want to put extra effort in team-oriented gameplay? A gameplay (house)rule that rewards teamwork and at the same time make it easier to achieve something together with a companion. The initiative order seems a possible candidate, reflecting how a PC is either inspired or instigated by a companion to better perform.

e.g. 1/day a PC may choose one of the following:

at the beginning of his turn, a PC may choose to invite an ally with lower initiative along his side to take actions together. This ally takes his next round right after you. If the combined goal of their actions succeeds, the ally stays in this initiative order for the remainder of the combat, if not he takes back his original place in the initiative order.

Instead, a PC may choose to join a higher ranked ally, as an immediate action for 1 round to perform actions together. He takes his next round right after the ally. If the combined goal of their actions succeeds, he grants that ally a +x bonus to initiative on his next initiative roll. (I included this to not disadvantage PC's with low initiative as their players wil have less opportunity to invite lower ranked PC's)

Or something likewise. You could also work something out in the same vein as a reward granted by the GM, for successful teamwork. What a combined goal is and what is considered to be successful is a bit vague, I know, also GM's discretion I suppose.

In the Leverage RPG, there are a number of pressures for good team cohesion:

In the setting, all PCs are highly-skilled criminals working together for a good cause. (Just go watch the show, it'll make more sense.) But that means that if anyone is caught, all the rest of the PCs are at risk. So help each other out!

In the rules, there is an optional mechanic for investing experience in Trust - if you Trust another PC, that can be used as a probable complication for you. Unless the other PC also trusts you. In that case it can be leveraged (wordplay!) as a benefit.

Concept Restrictions

As others have mentioned, making the characters from the beginning have to be part of a team is a simple rule. I usually like to define what kind of protagonists fit in this game world - what define "heroic" and general expectations. In some games, stealing and assassination are valid, in other games, the heroes are expected to keep their word and show mercy and try not to kill. If any of that is expected, make that clear at the beginning and make some requirements about character concepts.

Reward Mechanics

Light rules change: "You get experience points as normal as long as you've worked as a team. If you screw over any team member for yourself, you get no experience points."

Major rules change: "You get experience points when you help out a team member. When you risk your life for them, get X amount of points. When you help them in general, get Y amount of points." etc.

It reminds me of a situation on my own session. A guard's tower, filled with orcs, inside a female priest in plate mail battling with the orcs, the rest of the party outside. The necromancer was firing his sling through small window; weak shapeshifter held the door open, while the bowman was helping the priest inside with a valley of arrows.

I agree with DJClayworth above, that rules are not a way to solve the thing. Some of the ideas that spur in my mind:

Face your players with situations, where the only way to survive and/or win is teamwork - the opposing team will steal their treasure or glory if the Defender and Tyrant don't work together;

Put them in a long-term situation where they need each other - people generally hate the Tyrant and the only thing that stops them from whacking him up to bottom is the Defender, who wants to give him a chance. Perhaps this way he will change his ways? Or the Defender is in a prone position, completely reliant on Tyrants power.

Alternatively you may punish those who are not too keen to work with their party members: experience penalties, bad luck, fatal consequences ("See? You didn't help me saving that peasant, now it's his spirit that makes all your food turn yellow!"), bad reputation.

Extension

As per OPs follow up, a couple of suggestions:

All characters need need to come from one family/organization/agency/religion/philosophy

Each character needs to share goal with at least one other

Each character's back story has to be connected with back story of at least one other character; ideally they were friends or good buddies

Characters get additional 25% of experience for encounters or situation where they employ nice teamwork

Employ hazardous (but not lethal) bad luck for characters which don't work with others when they get a chance

I would flip the question on its head. Why not look to sort out ways in which the players can cooperatively create conflict for their characters. The idea that the Noble Defender and Brutal Tyrant should get along is preposterous. Still you don't want them killing each other because well players generally like their characters.

Why look for ways to make them cooperate more against the nature of the characters it is distilling out the potential for entertaining conflict. Rather look for ways to add enjoyable (for the players) conflict of those characters. To this end I would suggest borrowing from the concepts of HeroQuest (roleplaying game not board game) you could allow them to frame the nature of the conflict and what various degrees of success or failure in that conflict will entail and all parties involved agree to the conflict and consequences.

Conflict of some form is the root of what makes roleplaying interesting. Conflict of what the character is and wants to become. Conflict of an enemy of the party. Conflict of greed verse conscious. The more opportunities you have for the party to enjoy the various conflicts the more I think they tend to enjoy the game.

"Why look for ways to make them cooperate more against the nature of the characters it is distilling out the potential for entertaining conflict." I'm not the asker, but I'm betting he doesn't find it nearly as entertaining. There are groups where intra-party conflict can be entertaining to all and there's groups where intra-party conflict is basically just a bunch of people being jerks.
–
Joe BedurndurnDec 26 '12 at 23:49

What I suggest is putting a place a framework that allows with buy-in from the players of the characters in conflict to express that conflict in a reasonable way. So instead of saying no conflict you allow the players to say this is the conflict, it's potential results, and how we'll address it's resolution. If someone is a jerk you certainly don't allow them to force conflict on another player, you say that they will have to work out agreement of scope, resolution, and results of conflict with that other player.
–
FelanDec 31 '13 at 16:50

Simple rules for helping: If your character tries to do something and another character has a relevant skill, then they may help. This gives a small but significant bonus to your roll. (Or maybe they have to test the skill in order to help, or something.) That other player must narrate how their character helps you, otherwise it does not count.

In my experience there is much more cooperation between player characters in games with useful helping rules (e.g. Burning Wheel, Solar system). D&D 3 had rules for aiding another, but a test to give measly +2 provided too little benefit to become significant, IME. Many rulesets do not have any unified system for helping. Consider adding them.

I wouldn't try to do this with house rules - I would try to do it with objectives. Find some common objective that both your Noble Defender of the Law and Brutal Tyrant need to achieve, and ideally which they can only achieve by working together.

Once they have set out on their quest and are surrounded by hostiles then you can rely more on the fact that they need each other to survive. That should give your Noble Defender some interesting moral choices as he balances the BT's bad behaviour against his need to achieve his Noble Goal; maybe your Brutal Tyrant too while he works out exactly how far he can push the Noble Defender.

This is less a system question and more of a group dynamic question. I've never seen one system promote cohesion more because even if it does, as in cooperative play, players can usually ignore the flavor or intent of such system elements.

If its a plot driven game, the gain is less and pressure is much less. If its mechanics and an endless pursuit of plusses, then disunity usually happens more (in my experience). The best way to promote good party cohesion is to have a collaborative atmosphere. Enjoin the party members to some responsibility in game direction, character creation, etc. I also find a "gaming contract" also helps players feel like things are fair and that the focus is on their fun - that always makes people happier and usually play "better".
Here's some examples: