The magic trick, in one specific form, is to make combat in a roleplaying game tense and risky, but without killing PCs off all the time.

Damage in Dogs in the Vineyard
In Dogs in the Vineyard, when you take a blow, you don't roll damage right then. Instead, you build up a pool of damage dice to roll at the end of combat. The better the to-hit roll, the more dice you add to your damage pool, and the better the weapon, the bigger the dice. A solid punch would be 4d6, a flesh wound with a bowie knife would be 3d8, and a gunshot to the body would be 5 or 6 d10.

So at the end of combat you might roll 4d6 3d8 5d10.

Only the two highest dice count for damage, though; you ignore the rest. If the sum of the two highest dice is 2-7, that's no lasting damage; 8-11, minor damage; 12-15, serious damage, 16-19, potentially-deadly damage; 20, instant death.

So here's the magic trick.

As you take damage, you see how potentially serious the damage is. "5d10! Holy crap!" When you're making tactical decisions about whether to stay in the fight or run away, or whether to fight aggressively or defensively, what you have to go on isn't a concrete number of hit points remaining, but an eyeball judgment of how much you've already risked and whether you're willing to risk more.

The feeling in play is viscerally risky.

But then when the fight's over, you roll your damage dice and sum the two highest. The curve of rolling x-many dice and summing the highest two has an interesting and great feature, which is that for reasonable numbers of dice, the second highest result is more likely than the highest. Roll 5d10 and sum the highest two, you're more likely to get a 19 than a 20. Roll 10d10 and sum the highest two, you're more likely to get a 20 than before, but you're still more likely to get a 19 than a 20.

So the most common outcome of getting shot is that you narrowly avoided getting killed outright. The most common outcome of getting stabbed is that you narrowly avoided a life-threatening wound.

During combat, you're thinking about how bad it can possibly be, so the risk is heightened. After combat, it doesn't turn out that bad (usually). You feel like you risked a lot but somehow got lucky and got away with it.

(You can see the magic trick in action right in Kit's comment. 10d10 damage means you're expecting to die, but you only do die from it a quarter of the time. The game's design keeps you hyperaware of risk.)

To be fair, you can get killed from combat even if you don't roll a 20. You might not have healing available, or it might fail. Sure, as the number of d10's increases, the likelihood of rolling a 20 stays relatively under control, but the likelihood of rolling a 16+ starts climbing up high, not to mention the likelihood of rolling a 12+!

I've seen plenty of Dogs die from failed healings... :) I've even seen a Dog die from rolling a 12 after a fist fight.

Gordon, JMendes: Yes, conflicts in Dogs in the Vineyard are still pretty risky. Riskier than fights in AD&D? Quite possibly!

What I'm saying is that Dogs in the Vineyard plays a trick on you that makes combat feel riskier than it is. If somebody (like George) is looking for ways to make fights feel risky, this is a potentially-useful example.

Additionally, DitV's method keeps the risk raising without ending the conflict itself. A rash of bad rolls in D&D, any version really, can end your characters career with the conflict still going on. A rash of bad rolls in DitV only ends in your death if you decide to Take The Blow in some unhealthy places, but not until the conflict is finished and the basic stakes resolved. Of course, the other possibility is that you Give instead of taking that fierce 5d10 of fallout once the gun fires at you. Either way, you say something damned meaningful about your Dog, and the can still be effective either way.

So this derails from Vincent's point somewhat, but if you're looking to "fix" D&D specifically, there is a number of ways to fiddle with the threat factor of combat.
-the first isn't even mechanical: fights aren't about life or death but alternative objectives, the default outcome is losing something (advantage, resources, a specific item, time, your way in the dark, your freedom...) or running away instead of slaughtering or being slaughtered
-the second is also a matter of DM style. Telegraph the threats and odds of failure to the players clearly. So when death is on the line it's their choice whether to risk it, or fall back to safety (at some other cost).
-the third is to twist the death&dying; mechanics a little, which people have been doing forever. Treat hp as the abstract resource that Gary intended them to be, not wounds. So 0 hp can mean a lot of things. Maybe further damage is applied to Constitution, which keeps the tension up but also gives the PCs a big safety buffer. As for save-or-die effects, there are various interpretations, with some, if you fail a save vs. Poison, you're not instantly dead, just mortally poisoned, frothing, spasming...out of the fight but salvable by a timely antidote. Many poisons take hours or days to kill.

I checked the previous thread a couple of times from "my favourites", and only just now realised there was a new one!

Gregor: I take your point, and i appreciate the aesthetic that effectively treats combat as a meaningful conversation by the GM telegraphing his intent to the players; and them making a conscious decision. I have to say that it is however different to the feel that I'm after - a sort of subsystem where the people engaged in combat perceive time differently, identify with their character and are too worried to think logically. Regarding your last point, fully agree that treating hit points as an abstraction was a stroke of genius on the part of the creators of D&D. It took me a long time to realise that, when I was a teenager I thought I was "making it more realistic" by treating it as wounds. But you're right, abstraction is key.

Vincent: for these, and many other reasons, right now i'm standing in front of my computer and after querying about the possibility of a very particular magic trick, much to my amazement you actually just performed it! i don't know what dice the rabbit rolled, but you pulled it straight out of the hat. I've been thinking about this for a very long time (maybe too long) and I think that's the only solution I've personally come across that addresses it 100%. And not only that, but it's so simple that it's transferable to other games with a different theme! I got the temptation to start listing ideas i had in the past, and how they didn't exactly do what I needed them to, while your solution does. But there's no point. I would run and buy that game, but luckily it's already sitting in my library. I have to digest this mechanic and try it out. Thanks!