ShadowCraft Postmortem: If you can see me, you're already dead

Well, it’s that time already. The final tier is done (for some, at least) and there are no plans for more patches to Shadowcraft this expansion. I think it’s time for a Post Mortem... again.

For anyone unaware of the history, Shadowcraft is a theory crafting engine started by Aldriana around Cataclysm’s launch. There were a few graphical interfaces for it, the most popular being Antiarc's UI at shadowcraft.mmo-mumble.com. Both of these open source projects have had many volunteers and handed leadership over at least once. The list of developers on GitHub doesn't really do the scale of these projects justice, in my eyes at least. Indz for example has been a huge help with the engine, particularly with managing procs, and updating the proc system with the patches, and many seem unaware of how much he’s done recently. Shadowcraft as a whole has been supported by much more than 3 or 4 people so far.

With these changes though, with these patch cycles, I think I've steered the current calculations in Shadowcraft in a slightly different direction than originally designed. Shadowcraft at its core is about determining EP weights, but I've slowly pushed towards damage breakdown precision. If you’re familiar with the architecture of the engine, you should be able to see a stark difference between how the Assassination APS is calculated, and how the Combat and Subtlety ones are. I've been trying to slowly migrate from ‘average case’ to varying weights and phases. This hinges on an interesting topic however; it’s the balance of time to compute vs. precision. Given the current trajectory, calculations will take longer with every patch as I try to fit more precise results and formulas in. I’m making a concerted effort to make sure the algorithms are efficient now, but precision carries a higher weight, even if the EP’s don’t change by any noticeable amount.

I've also made a shift towards settings and knobs. Ways to customize the calculations, like you would customize your buffs. It might be too soon to see how this has played out, but I think things like advanced parameters will be good in the long run. It aids users in considering things like (now added) Feint usage, or experiment with encounter specific buffs that might affect energy regen or stats. I think there would have been significantly less confusion about how Combat was handled during ToT if people had easy access to manipulating the GCD.

Another thing is that the game has changed. Despite how simple and minor the ability seems to some, the addition of Shadow Blades has greatly changed how we need to approach APS calculations, even for Assassination. While the game is arguably getting simpler, certain theorycrafting techniques are becoming more antiquated. If you can remember back to when Mists first launched, Combat’s DPS breakdown was horribly flawed, and it didn’t represent actual performance until I restructured it to use phase based calculations. Even since then, the methods for things like Killing Spree had to be changed, improved upon to support things like new trinkets and higher stats to reflect what’s feasible at progression content. I won’t hide that Killing Spree has been a thorn in my side this entire expansion. It’s surprisingly difficult to model.

We also need to start thinking about the future of Shadowcraft. It’s been roughly a year since I stopped raiding, and my account has been completely inactive since the spring. If Shadowcraft is going to be an excellent resource in 6.0, more people need to participate in the engine by reading and committing code. It won't be feasible for Indz and me to do everything forever. With every patch, it will get harder for me to intuitively understand how mechanics work, which will make patching longer, and more prone to errors. Assuming I still exist when 6.0 beta launches, I'll still work on Shadowcraft and other projects, but for various reasons I can't pretend I’ll be of good use forever.

In the last post mortem, there was also a good chunk of discussion about hosting elsewhere. I don't know if we'll be in the same place when 6.0 launches as we were in Mists. I don’t think so, but maybe. It’s worth considering a secondary frontend to open up new opportunities, but with less Shadowcraft developers than ever (it feels like), it's questionable if we'll ever get it.

Feel free to post about ideas for the future of Shadowcraft in addition to reflecting on this expansion. I'd rather not fragment Shadowcraft discussion too much by creating another thread for what will or should be done in 6.0.

-------

I suppose that’s enough rambling though. I’ll go over a list of concerns from the end of Cataclysm.

Speed of Fixing Bugs:I don’t think I’m the person to answer this one. But everyone else should consider it. I’m not happy with how long it took me to get the engine in a reasonable shape before patches hit (most of the time I was doing bug fixing after the patch date).

Web UI Setup Concerns:This didn’t change. I think Shadowcraft should rely on fewer dependencies, but I’m not going to make any efforts to change what’s working. We’re using almost the same exact dependencies overall as the Shadowcraft projects used in Cataclysm.

Changing Fight Settings:I’ve started opening up more of the engine with advanced parameters. I don’t think this will ever be as good as anyone might want, just due to the nature of modeling, but there’s definitely been some progress. Try and compare the Cataclysm settings window with the current one and I think most people would say that it’s better now, even if it’s not used more.

Reforger Wonkiness:This was brought up several times last post mortem. I don’t think it is any better now than it was then. On the other hand, it doesn’t look like it will matter in 6.0.

Waiting for Items to be Implemented:We tried doing beta phases for Shadowcraft before patches hit. I think this helped on multiple fronts and (specifically) Indz did a good job preparing for launch and keeping up with engine fixes and making sure things went smooth. I believe it also reduced the server load slightly on patch day.

Split Shadowcraft into a Simple Web UI and an Advanced Version:This wasn't done, and won’t be done. At least not as many people imagined it. The design philosophy for settings now (see earlier) allows for one build and a simple interface. I don’t want to manage two sets of calculations either.

Known Issues List:We didn't exactly have one. I don't remember us needing one like we might have in Cataclysm. However, Indz has brought messages to the log on the right side of the screen, which many probably saw during 5.4’s launch. I suppose that counts as an improvement.

Drop Location in Tooltip:This was not done. That was one major reason why the wowhead icons were added in Cataclysm.

Custom EP weights and Caps:There are custom EP weights for general secondary stats, but not strike chance stats. Additionally, PvP mode changes the caps to use typical PvP caps, but they are currently not customizable. The lack of strike stat support shouldn’t be an issue in the future though.

Supporting Interruptions and Phases:While Combat calculations are phase based, we don’t support things like fight interruptions or changes in circumstances. This has been talked about before, but it’s actually a pretty big task, and is unlikely to ever be added to Shadowcraft at this rate.

Good Gear Recommendations and Planning:I don't know if this was that big of a problem in Cataclysm, but I don't think we made any efforts to improve it in Mists. Many others should consider this further than I could.

I feel that Shadowcraft's usability has definitely by far improved over the course of the expansion.

In my opinion, the most important update to the engine without a doubt is the dps phase calculations for combat. As complex as it is, it addressed a problem with combat gearing that was progressing over the course of an expansion.

I think you're largely spot on with your bullet points, and I agree with you on all counts (at least all counts where there is room for opinion/perspective).

The changing fight settings and interruptions points are related and in my opinion, I don't think such features need to be implemented. Sims are better at answering questions posed by those types of questions. I don't think it would be healthy for shadowcraft to be a "do everything well" type of tool. It's extremely good at what it does and I think it should maintain its focus as a gearing tool and less as a tool to investigate fight or rotational mechanics. Patchwerk stat valuation should be good enough for a general case. Most encounters spend a significant chunk of time as effectively patchwerk for melee in general, anyway.

A feature I think would be useful though would be agility-based EP evaluation. Agility-based ep evaluation makes it much easier to spot gemming crossover points (such as the point where you switch from agility-based to haste-based gemming for combat. When you see a haste EP of 0.5 is more intuitive than trying to do math in your head to check if 1.427 is half of 2.856. Or even better, a point where haste becomes > 2/3 the value of agility and JCs instead use the haste JC gems (my gear reached this at one point with haste reaching about 0.7x the worth of agility). AP as a stat, does not exist at level 90. It seems counterintuitive to me to gear based on it. Blizzard has hinted that attack power may make a combat in warlords as a stat, but I don't think that detracts from my point since we'll still be gearing secondary stats against agility, I'd suspect. We'll have to see how the stats on gear pan out in warlords.

For the record: I had always hoped to implement more phase-based calculations, but never had time to do it justice. The initial implementation was little more than a direct port of spreadsheet logic, and spreadsheets can't really support phased solutions. So kudos to Pathal (and any others that may have been involved) for finally putting together a working implementation (for at least some cycles).

Regarding time to compute vs precision: is there any way to make this tunable? When one is rapidly iterating on many different sets of gear there is a lot of value in having an "approximate" calculation that will let you know if you're in the right ballpark; but once you're starting to get close - or want to optimize your strategy in detail for a given fight based on what your gear currently is - it could make sense to do significantly longer runs.

This, of course, depends on how different the logical path is between "fast" and "accurate", and also how large the accuracy different actually is. But, conceptually: sometimes people want an answer in one second, and sometimes people want an exact answer; in an ideal world, it'd be nice to support both (although, if you're going to allow long, detailed evaluations that take more than a few seconds to run, you might need to think about server resources a bit - not sure what the traffic volume looks like).

It should be noted that this also applies to reforging to some extent. An exact solution for reforging is incredibly time-consuming, which is why approximate solutions are used instead, which is why their is wonkiness. Having both a quick and dirty solution for rapid iteration, and a "run as long as you need to" solution to get more accurate (and hopefully less wonky) answers would potentially have value.

To shadowboy813:I don't remember there ever being much of an argument for, or against, Agi and AP normalization. I could probably change it to default to Agi to save users some effort, and might be able to add it to adv_params.

To Aldriana:I'm not sure it could easily be tuned. Having "advanced" calculations that take over a second by design sounds like it would require entirely new calculations and methods.

One of the things I'm thinking about is reducing the reliance on convergence. Assassination right now, is the only spec that actually relies on it, and we calculate Combat 4 times, when it should only be done maybe once or twice. Since Combat has 3 to 4 phases, meanwhile the APS table is virtually identical every time. We end up running through the method at least 12 times just to get a final breakdown. I think Shadowcraft can do both, if I can find some way to improve the determine_stats() method (was a part of compute_damage()) Shadowcraft might be able to become more precise and faster at the same time.

I don't intend to strive for a holy grail of precision. There are better ways to determine those values than what Shadowcraft is designed around. But just to better handle synergies that Shadowcraft has shrugged off in the past as insignificant.

I was talking to a friend about a possible way to provide a fast mode and an accurate mode while not overburdening the server. The idea would be a provide a desktop application of some sort that provided a more detailed model however to preserve the interface people are familiar with the desktop application would contain a simple browser and passed computed results to the Web UI for display.

I'm not entirely sure of how to do this, its been something I've been meaning to work on but haven't for lack of time. Looking at documentation it looks feasible and I'll try to put together some sort of proof of concept.

EDIT: This design would also somewhat mitigate the dependency and set issues of the front end because anyone could simply place their own modified module in the desktop application and get all the benefits of the front end without having to set it up.

On the question of more complex fight modeling I think a better goal might be trying to support "dps primitives" representing common dps scenarios, cleave, aoe, burst and then letting people draw their own conclusions based on these primitives. We could then provide some configurable weighting mechanism so people could optimize around whatever mix they want. Supporting these dps primitives would likely be easier to support than trying to support a more arbitrary and freeform modeling scheme which as Shadowboy notes is probably better suited to simulation anyway.

On gear recommendation, a feature like Ask Mr. Robots "boss to coin" might be useful. I also remember most of the issue with gear recommendation were related to item valuation around caps which should become a smaller issue in WoD.

I'm pretty sure that would require rewritting the current UI to either accept additional input via URL, or rewritting the page as you download it. Seems like it would be easier to just make your own UI.

I just released an experimental patch preparing for 6.0 stuff. It's mostly just to illustrate my goal for making the engine more efficient, but there's a few other changes too. I cut out a lot of procs and dead code just to get rid of future dead weight, but removed hit rating stuff as well.

I think the single largest change was doing a remedial check for convergence. That alone removed >15% of the runtime for the combat script. I'm sitting at ~33% more efficient code now overall (script took ~.15s, now takes ~.1s), with potential to get even better. I could probably reduce calculations to a single APS call under specific scenarios. I'd like to break ~40% better code before thinking about improving the precision through better/more phase handling.

I want to start working on simplifying the proc data further than I have. I want to get it where we no longer need to split up proc stats from their behaviour, and just accept input as ('Assurance of Consequence', 580). This paves the way for Challenge Mode support, but I think it should also improve performance.

I'm stuck debating something with myself however. With my own personal idea of how I think Shadowcraft should operate, I think we need to keep hit rating mechanics in the engine, but I'm really itching to just remove everything related to hit chance. I feel like, if there's a chance that hit chance could become a factor in raids again (think possible return of miss chance debuffs - don't stand in Archavon's fart), then Shadowcraft should be able to incorporate that to a degree. But it really can be a burden to code around, with a tiny chance of it ever being a gameplay mechanic in the future. It is essentially dead weight whenever there isn't a miss chance mechanic. Which at the current rate of miss chance mechanics, will be all the fights in Warlords. Hopefully other's have some opinion.

The engine is still somewhat applicable to the current patch, if you're in BiS gear and hit capped the APS methods should still be relevant. The lack of hit rating or support for most Mists procs means you should avoid it unless you know why you want to use this.

But it really can be a burden to code around, with a tiny chance of it ever being a gameplay mechanic in the future. It is essentially dead weight whenever there isn't a miss chance mechanic. Which at the current rate of miss chance mechanics, will be all the fights in Warlords. Hopefully other's have some opinion.

I am not a shadowcraft contributor, so don't give too much weight to my opinion. I think you should remove hit altogether. Shadowcraft doesn't aim to model specific fight mechanics, but is mainly about giving an accurate representation of the easiest thing to model: a Patchwerk fight. I believe it will be substantially easier to maintain and improve if you strip the maximum amount of dead code now, in the between-expansion lull. Since you seem to say the amount of code that would be removed by such a move is non trivial, and that it would significantly simplify developing new code, i'm all for it.

If we get one hit-penalty fight in WoD, we'll adapt Shadowcraft recommendations accordingly, like we always (should) do (and I really don't see how we could get more than one such fight).

I believe parry to be quite more important than hit, since if I understood correctly, we will still be parried when fighting in front, and many boss fights force front placement. So the amount of code simplification needed to make it worthwhile to remove parry mechanics would need to be proportionally greater :-)

I believe parry to be quite more important than hit, since if I understood correctly, we will still be parried when fighting in front, and many boss fights force front placement. So the amount of code simplification needed to make it worthwhile to remove parry mechanics would need to be proportionally greater :-)

Ah, yes, but if you're modeling a Patchwerk-style fight (as you stated previously), then you do not need to worry about parry. By leaving in parry, you're coding for the individual fight instead of for the ideal stats in the Patchwerk model. By delving into Parry (and, fairly, lesser Hit), you're heading towards a simulator instead of a stats calculator.

This is one of those all or nothing scenarios, where I keep both parry and hit, or none at all.

I also don't see a reason why modeling can't gauge encounter mechanic impacts. I'm positive that modeling can calculate whatever you could possibly encounter in a raid. It's really just a question of how smart is the programmer to get it done, and how long does the resulting code takes to compute. I don't think a hit mechanic that reduces your chance to hit by 30% for 5s is really out of range of Shadowcraft. Spreadsheets and modeling have reasonably factored in miss chance for years. It's not going to suddenly become out of range.

This is one of those all or nothing scenarios, where I keep both parry and hit, or none at all.

I also don't see a reason why modeling can't gauge encounter mechanic impacts. I'm positive that modeling can calculate whatever you could possibly encounter in a raid. It's really just a question of how smart is the programmer to get it done, and how long does the resulting code takes to compute. I don't think a hit mechanic that reduces your chance to hit by 30% for 5s is really out of range of Shadowcraft. Spreadsheets and modeling have reasonably factored in miss chance for years. It's not going to suddenly become out of range.

If it's all or nothing then I guess we're back to the beginning: is it worth it to maintain or not given that these two stats/mechanics are relegated to be more fight-specific? While I am a programmer (to an extent), I haven't looked at the source code (because I don't know that language), so I don't know what the implications of keeping it in are. If you think it's worth it to maintain Hit in order to keep in Parry, then keep it in. If not, we the users can simply make adjustments to the output of the calculations for whichever fights will need either mechanic (not that I know what those adjustments might be - I'm not nearly as hardcore as many readers here).

If both are inextricably linked, I vote to drop them both. Just my 2ç (Disclaimer: In all likelihood I won't be progression-raiding in WoD, so I am not going to be one of Shadowcraft's most impacted users.).

What comes to mind is that it would have to be a debuff that was both severe and long-lasting for it to effect EP-values by a noticeable margin. And, I'm being presumptuous now, those are the kind of debuffs you can, for the most part, avoid by proper planning and positioning.

On the other hand I wouldn't put it past blizzard making a fight with some crucial burn phase that has a hit-debuff (or some other shit) just to throw a wrench into stat weights. In which case it'd be neat if SC could handle the potential GCD-capping, energy/cp starvation, and debuff-downtime.

So I wouldn't blame you if you axed it. Especially if it opens up for higher precision in other parts of the modelling.

But it got me thinking that you have to be making some assumptions/simplifications in how SC handles missing right now, since if you had a 'perfect' model, it would cower breaks in combat as an extreme case of miss-percentage. So to what extent (% of miss) is the current model applicable?

---

I think you're mostly right about the known issues thing, with the exception of how the reforge/gem tool works. It gets asked about all the time. But that's the kind of thing that can just as easily be handled by incorporating it in the OP of the relevant thread(s).

As for gear recommendations, I personally don't see the need. There have been ~two pieces per slot each tier, and in WoD there will be no hit/expertise to muddy the waters. You can simply open the drop-down menu and get a list that should be accurate all but the most special cases. The flip side of that is that it'll be less costly to compute gear-upgrades, making it more feasible to implement such a function.

Reasonably close. Lets, for example, assume a shitty scenario where you miss 4 times in a row at 99% chance to hit. That's a .01^4 probability, or 0.00001% chance of happening. It sucks, I'm sure it's happened before, but it's really not statistically relevant. 3 misses in a row at 95% chance to hit is still only 0.0125% chance of happening.

Shadowcraft already knows how to account for subsequent misses using geometric sequences. This affects energy costs and (primitively) GCD cap calculations. But it fails to consider extreme scenarios like the ones already mentioned. They just aren't meaningful.

Something like a 30-50% chance to miss for Combat might need some special attention in the aps method. But other than that, it's been very reasonable so far.