So one of the assumptions for my ATE game is that manual action shotguns, often loaded with black powder loads, are a common and desirable weapon. Basically, something like a LeFever Nitro Special (Pulp Guns 1 p 25).

I'm trying to figure out the damage for a 12G 2.75" shell loaded with black powder and firing #4 birdshot for an encounter where a bunch of hunters suddenly find themselves in a bad situation.

Based on the damage from the Winchester 1887 in 12G, I don't think that using black powder reduces the slug damage from 12G shotguns, but I'm not sure about that.

Assuming that black powder slug damage is 5d (or 17.5 on average), the multiple projectile rules give the average damage as 1.5 or 1d-2. So these shotguns should be doing 1d-2 pi- (0.5) with RoF 2x150. Does that look correct?

Based on the damage from the Winchester 1887 in 12G, I don't think that using black powder reduces the slug damage from 12G shotguns, but I'm not sure about that.

It's important to realize that the limit on the charge is the point the gun blows up when you fire it, not how much powder you could theoretically stuff in the barrel. The point at which the gun blows up does not depend on what kind of powder you are using, so to first order the maximum damage you can get out of a load you can safely fire out of your gun is independent of the kind of propellant you are using. There are secondary effects where it matters, particularly for very high velocity rounds or very long barrels where the properties of the powder gases start to be important, but they are normally just that: secondary.

A particular cartridge will of course have only a limited amount of volume. Many cartridges were originally designed for black powder, so when full they hold an amount of black powder equal to about the maximum guns built at the time would handle without exploding. The "drams equivalent" written on the side of the shell is how much black powder the charge is equal to. Usually if you filled the shell completely with black powder it would have that number of drams of powder in it, if you fill it with smokeless, you leave it half empty. Notionally you don't have to do this, but if you fill it entirely with smokeless powder to get more damage, it's not safe to fire from a gun designed for the original charge.

So, takeaway, if a shotgun was originally designed for black powder, or to fire a cartridge that was, you probably can't increase the damage by loading smokeless without the barrel bursting. If it was originally designed for smokeless, damage might drop firing black powder if the original statistics were calculated for a specialty shell, but won't if they were computed for a commercial hunting load, which is underfilled so you can fire it from black powder designs without killing yourself.