I think I'd expect an S-shaped curve for fanfic, with a term for the popularity of the original work, and a more exponential-looking curve for original fiction. People who read fanfic tend to read a lot of fanfic, and that gets a certain number of eyes on your work that wouldn't be there if you were publishing original fiction, but it's exceptionally rare for a fanfic to attract readers that aren't either part of the (still relatively small) fanfic community or fans of the original work and usually both.

HPMoR is unusual in that it has managed to attract an audience independent of those considerations, but that audience is, as best I can tell, quite small compared to the numbers a bestselling original fantasy novel can be looking at.

Oh, without question. A bestselling original fantasy novel has many more readers than a popular fanfiction. Agreed.

So now, if I want to do an expected value calculation, I should consider the likelihood of my work becoming a bestselling original fantasy novel vs the likelihood of my work becoming popular fanfiction, and the effort involved in pursuing those paths, and cash those out in terms of expected readers gained per unit of work. Agreed?

Assuming we agree on that: what would you estimate the ratio of those numbers to be for EY, while preserving the various ideological/educational purposes he had for his work?

Yeah, I think we agree on the problem statement. As to the solution, that's an interesting question. Let's do some Fermi analysis.

We first need to know what popular fanfic is actually looking at in terms of readers. ff.net doesn't expose that information, unfortunately, but AO3 does. If a comment on AO3 is roughly equivalent to a review on ff.net, then each review is worth about a thousand hits. Assume that five or ten percent of those hits, for a long work, are unique readers (logged-in users aren't double-counted, but I assume anonymice on dynamic IPs are when their IP changes), and it looks like Methods has seen one to two million readers.

Now, a popular fantasy novel series can be expected to sell somewhere around twenty million copies (there are few single-volume fantasy books that make that list). Assume five books per series and that half of all readers don't get all the way through, and we're looking at somewhere around eight million unique readers.

If EY is risk-neutral, if he mainly cares about maximizing his readership, and if non-bestselling books usually sell relatively few copies, he would have been correct in writing an original series if there was more than a 12% - 20% percent chance of making it into bestseller territory. That sounds high to me, so I'm going to say that the fanfic approach was probably a good one -- although it's worth mentioning that Methods is arguably as much an outlier in fanfic terms as your average bestseller is in publishing, and retroactively extrapolating from its performance might not accurately model the kind of forward prediction that Eliezer would actually have been doing when he was making these choices. Also, if he happens to have any contacts in the publishing industry, that'd skew things somewhat towards original fic -- not all the barriers to entry are based on dumb luck.

although it's worth mentioning that Methods is arguably as much an outlier in fanfic terms as your average bestseller is in publishing, and retroactively extrapolating from its performance might not accurately model the kind of forward prediction that Eliezer would actually have been doing when he was making these choices.

I think it pretty obviously does not accurately represent the choice Eliezer faced at the time. Aside from the inherent advantages of fanfiction (no barriers to entry for him or readers, close interaction with readers so he can debug chapters), there are who knows how many thousands of high-quality fanfics he was competing with? It's worth noting that Eliezer has done a fair bit of original and fanfiction before (http://yudkowsky.net/other/fiction) and AFAIK none of them have been wildly successful even when you consider them being short stories etc.