You can challenge anyone to an unarmed brawl. If they accept, don't take your turn; instead, wait until it's their turn, and play rock-paper-scissors against them to determine the winner.

ROCK: Punch. If the punch damage reduces them to 0 HP, they're knocked unconscious. Some weapons like brass knuckles can be used to increase your punch damage, but I'd say the base is d3. In a tie, the one with the highest strength wins.

PAPER: Dodge. You must move, and you can perform one non-combat action (Picking up something, pulling a lever, pushing over a bookshelf, etc. It can still hurt a dude, as long as you don't touch them directly.) In a tie, the one with the highest dexterity wins.

SCISSORS: Grab. Only lasts for the turn. You can throw them as far as your Strength modifier in meters, or disarm them of a weapon or item. When disarmed, the weapon will clatter to the ground a few squares away (They can use a dodge action to pick it back up). In a tie, the one with the highest constitution wins.

Any attack against someone who's in the middle of a brawl will hit the other brawler if it misses. This means it's mostly in your interests to leave this Mano-e-mano. If the enemy does not accept your challenge (Or if they're surprised), then you keep in turn order: You punch, dodge or grab for free, and they take their turn as normal when it rolls around.

Implications:

Yomi - "Reading the mind of your opponent" - makes this into more of a mind-game than a dice roll. You need to judge whether your opponent wants to punch, dodge or grab, then do the opposite. Now, tie breakers are very powerful: They let you win two out of three match-ups. But wait; if the DM knows how strong you are at grabbing, then he'll use a punch - or does he know that you know that he knows?

Grabbing and disarming an opponent influences this yomi, by making the opponent predictable; You know they'll probably want to dodge, so that they can pick up their weapon.

We had a fight near a cliff edge, and one of my players wanted to throw a guy off. However, when the Brawl turn came around, he'd figured out that I knew he wanted to throw a guy off, so he went for a punch. But I knew that he was going to realize that, so I craftily went for a dodge. What I didn't realize is that the bad guy was packed in. The only place he could dodge was straight off the cliff and down to his death.

That tension of suspending your turn until the enemy's turn rolls around is fantastic. While the others take their actions, you're looking across at the DM and pondering your next move.

Edit:

Colossus-Climbing rules

If you grab a large creature, you can climb up them. You can do this once for something giant-sized, twice or more if it's bigger. This gives you an extra dice of damage: d4 after one climb, d6, d8 and so on if you climb higher. If either you or the colossus dodges, you'll fall off.

So you see, you want to attack, but you'll need to grab on if the colossus tries to shake you off with a dodge.