52 Book Reviews: The Big Short

I’ve decided that I want to read and review one book a week for all of 2017. My first book, The Big Short: Inside the Doomsday Machine, was a tantalizing dip into the world of hedge funds and global finance that left me with countless questions even as it entertained me with a cast of real, interesting characters and filled me with no small amount of despair about the global economic environment.

The plot of the novel is what countless speculators and investors—and eventually, the American people—found out after the fact. Moody’s and S&P, two private companies, are responsible for rating the credit-worthiness of institutions from individual companies to national governments. Their methods for rating credit default swaps, while deliberately opaque to the outside world, could be gamed (for example, collateralized debt obligations, which housing mortgages were stuffed into, were rated on the average credit rating of its individual borrowers, so that allowed banks to include both high- and low-rated borrowers so long as the average was maintained). These holes—along with a blind trust in the ratings of these vehicles, and the tendency for every single trading desk to not look too closely at them—allowed the banks to overextend their debt obligations orders of magnitude more than the cash they had on deck.

A group of “underdogs”, hedge fund managers and just plain private citizens with an interest in trading, realized that despite the organization of these vehicles—where defaults in payment could be tiered to protect higher credit-rated tiers—were all, in fact, based on a single economic factor: the ability for all house owners in the US to pay their mortgages. These mortgages were underwritten with insane terms and expectations of borrowers to repay their debts: workers making 17,000 dollars a year purchasing 400,000 dollar houses were not unheard of.

You can probably see where this is going. If many, many borrowers went into default, the entire investment vehicle would become worthless overnight. These underdogs all realized the same thing, and began shorting these vehicles: essentially betting against their success, against the banks who were betting for their ability to finance their debts.

When defaults start occurring, investors start asking for their money back, under the terms of the debt obligation. But there is no money, because the vehicles are worthless, because no one can pay their mortgages. The banks tank. The US government steps in. Some banks are deemed “too big to fail”, and are essentially nationalized. Some banks fail, some get absorbed into larger ones. Every idiot at every trading desk responsible for the untenable situation in the first place walks away with multimillion dollar payouts. The underdogs are rewarded for their stalwart opposition and long bets. The American taxpayers foot the bill.

The author, Michael Lewis, seems to understand that most readers would be out of their depth in terms of the terminology and processes involved in hedge fund management, and, depending on the foundational knowledge required, either describes the jargon as quickly and specifically as he can, or else relies on metaphors to get his point across. I’d have rather he done the former much more, because his metaphors seem either overly simplistic, or still couched in how financial managers think about their work.

For example, the nature of how credit default swaps are arranged is usually depicted as a “tower” whose floors flood in order of their proximity to ground level, which is analogous to the amount of risk each “floor” contributes to the agreement. Halfway through the novel, however, this metaphor is discarded because it’s much easier to think of credit default swaps as having three basic categories of risk. The tower metaphor loses its usefulness then, and it’s unclear why it was introduced in the first place.

The “underdogs” in the story are deep and sympathetic characters, and since the reader probably knows the outcome of the story, have the advantage of being able to root for them the whole time (or not, as it were; Greg Lippmann of Deutsche Bank is portrayed as a canary in the coal mine in this book, but is considered a piece of shit in German-based analyses of international finance who brought an unrealistic American culture to Deutsche Bank and “bet against his own products”, walking away with a tidy sum). Lewis’ domain expertise and colorful prose bring these underdogs to life. I found myself wanting a tenth of their gumption, cynicism, and intelligence.

So, yeah. If you love well-researched retrospectives on global catastrophes, and are willing to walk away with a depressed outlook on the market’s ability to correct for the greed of banks and investors, I’d pick it up.

Project A Part 5: The Entity Editor

Today I started playing with a front-end to modify entities outside of the game proper. Pretty screenshot first:

Doing this properly involved discovering even more tight couplings between components that the Kryo serializer seemed to have problems with. The long and short of it seems to be: if a Component is referenced as a bog-standard field on other components, the serializer gets confused and fails to reify that component when it’s meant to for a given entity.

Each of the individual components has its own way of displaying itself in the table, all of which are rather crude right now. Some of the raw data exists as protobuf messages, making it fairly painless to iterate over them and display all of the keys and values (as seen here). However, Artemis’ components must extend a sentinel class with no methods or fields. This makes it impossible to have a proto message which is also a component. The one place I do this is for items called ‘raws’, which are any items which need to be created from a template (e.g. bottles of water, guns, etc.). And this is kind of a kludge:

publicclassRawsextendsComponent{publicGeneratedMessageV3raws;}

This also means that I can’t have systems subscribe to any of those raw keys, so even if I could make all components protos, there would also need to be an implementation on top of Artemis’ systems to allow them to subscribe to only the proto “aspects” they are interested in.

So what I have to do for now is, for every field on a component, I need a separate line of code with a ton of duplication within the editor UI. So for the field Stats.currentHealth, I have the line

in the list of rows being added to the table view. And this sucks. I guess the most resilient way to handle this would be to use Java reflection on all of the components. It wouldn’t be too bad; just checking to see which fields have primitives or Strings and add them to the UI. So I guess that’s the next step.

After that, I want to be able to modify in-game values using this editor, and then load the save back up. I mean, I’m still conflating saved games with pre-stored game data, but right now they’re using the same mechanism. After a while I can imagine saved games simply being a delta of what the player changed against the pre-stored game data, but that seems a little like overkill.

Project A Part 4: New Year’s Progress

Today I managed to make some progress on my engine for the first time in a while. A bunch of changes I wanted to make occurred to me as I continued playing Fallout 4.

I don’t want a randomly generated world. Part of the major issue with Fallout 4 is the lack of a clear, coherent storyline, and a randomly generated world would only be more arbitrary and banal. I want to create a plot, rich characters, and have the user’s actions change the game world in the way you see in Deus Ex and the original Fallout series. I’m not sure which is a taller order: a well-functioning procedurally generated world a la Dwarf Fortress, or a well-designed, coherent game experience with a rewarding and engaging storyline.

Regardless, if I want to go in this direction, I need to be able to add static content—quests, characters, items, and etc—to my game engine. 90% of it isn’t geared for this sort of retrieval of pre-created content. Earlier versions of the engine dealt with assets like sprites, images, object metadata and the like pretty well. This version strips off all of the image loading and sprite sheet code in favor of a CP437-based user interface and viewport. There was even an iteration where quests were randomly generated; they were just never exposed to the game world.

Up until this point I had never gotten to the point of adding things like quests. I’ve decided to take a close look at how Bethesda’s engines deal with content like that. Based on what I know about the Construction Kit, forms, and the like, I think quests are simply another kind of component. There are flags specific to how quests are built, such as the number of stages. There are also hooks for code to be inserted, such as what should happen when a certain stage in the quest is reached. Since I already have Python bolted into my engine, I don’t see why I can’t write scripts that attach to component objects in this way.

Of course, all of this means that I need some way of adding this content into what will eventually become the resources on disk. Which means:

Saving and Loading

In order to modify a set world, I need to be able to serialize it to disk, and allow the engine to deserialize it on the other end. Artemis, the entity component system implementation I’m using, has a library that combines its knowledge about how to store and load entities in the world with the Kryo serialization framework.

There was a lot of trial and error in watching the serializer walk down the object graph and attempt to serialize everything it came across. Once I realized it was painless to write custom serializers, everything went a bit more smoothly, though some of the shotgun surgery designed to get heavy or transient objects out of the way of the serializing framework will probably pay off in the long run. Investigating what objects stored references to what in their fields exposed a lot of places where logic should have been migrated to one of the component systems.

In addition, there were a ton of assumptions about the order in which important objects were created. At one point or another, the high level objects which need to be interleaved are:

The Universe, which contains some metadata about the game world as a whole, as well as a grid-based reference system for all of the cells in the universe. It’s supported by a lightweight UniverseParameters configuration governing the seed and other important universal values. It is also responsible for huge swaths of the procedural generation algorithm.

The World is the abstraction specific to Artemis; the manager through which entities, components, and systems are manipulated.

The UiStateController is meant to be where all of the back and forth between the UI and the game world take place. This is part of the aftermath of decoupling UI code from the game’s run loop, which was itself part of the aftermath of enabling a way to alternate between real-time and turn-based simulation.

The StateManager is my thin wrapper around the World, and is responsible for ensuring that all systems interested in getting events from entities are subscribed.

In the original implementation there was a clear and obvious order to how these things were instantiated. Now that the universe can be pre-loaded, and the world is configured independently from the rest of the game state, the fragile connections between these objects becomes a lot more pronounced. That will probably require some refactoring down the line.

My Own Construction Kit

Finally, and one of the biggest consequences of re-thinking the game’s content, the need for a custom editor becomes apparent. Outside of the game engine itself, I need a way to deserialize a saved world and attach components to objects. In this way, the editor can save changes to the objects in the game world, and then the modified world can be loaded in. This mechanism is exactly the same as the save/load mechanism.

Of course, I’m going into this blind, with no real conception of the shortcomings of Bethesda’s approach. Perhaps this makes my game more moldable, perhaps less so.

There’s still a huge distance to cover in making this work properly. How do I split up game content into separate files? How can I have one new file overwrite components in old files? I’m not sure any of this is impossible, but it’s going to be a lot of work to come up with this scheme at the same time I’m trying to design the editor to put it into action. We’ll see what happens next.

Project A Part 3: From July to October

I had meant to keep a seriously thorough engineering log of my work on Project A, in order to write some nice, spaced out blog posts about it. Instead, I worked my ass off on it for three months and didn’t keep a scrap of notes. So now I’m looking through my commit log to see if I can make some coherent sense of the work I’ve done.

Here’s a great screenshot showing off some of the user-visible changes.

Things that don’t have a full write-up yet: line-of-sight falloff of lighting, individual stats and needs meters, a world map which includes roads, and an inventory which handles stacked items.

World Generation

World generation was improved, massively, between this post and the last. I finally managed to implement Voronoi world generation in a way that was comprehensible to me. I’ve been writing worldgen related code for a while, and a proper Voronoi implementation always eluded me. I am whole-heartedly standing on the shoulders of
Martin O’Leary, Amit Patel, and James Humphreys. It’s not completely up to snuff—it doesn’t handle biome changes, elevation, or running water—but it’s a fantastic start, and I can’t wait to add more detail to it.

Time Scaling

I’ve managed to implement proper time scaling, so that crossing a tile in the overworld takes two hours and has all the requisite effects on hunger, thirst, and other status effects.

Once you descend into an overworld tile it’s real time, and NPCs move about and time goes forward in real time. And then once you initiate combat, you’re in a turn-based system like Fallout 2 or Xcom, which makes sure that all things in everybody’s turn happens in a causally-correct order, ignoring time.

This all has to happen in the context of a game loop which may take a different number of milliseconds to update the game logic in each frame, so it will act the same whether it’s going 30FPS or 60FPS.

Before this, the entire game was turn based, so the game loop was completely unattached to the stepping forward of game logic. Now they can be dependent on each other or not depending on where the player is in the world and whether they’re in combat. If you cross a tile in the overworld, it takes “two hours” and your stats are affected as if all that time passed. But if you go into a cell and wander around for two hours (real time, as the clock is ticking) the same effects to your stats will occur. But if you’re in combat, time freezes and only ticks forward based on turns and action points.

And, so that players can see the trail of bullets, all animations (like explosions and bullets moving towards their target) cause all turns to stop until the effects of everything that happened last turn are handled.

The real-time outside of combat is great because it enables stealth. You’ll be able to see NPCs moving about, and their cone of vision, and have areas of light and dark. And if you take someone out and no one notices, combat isn’t initiated, so the flow of realtime isn’t affected.

The practical upshot of all this is that now action points can be implemented, and actions corresponding to a certain AP cost, as you can see below.

In-Game Console and Scripting

I’ve agonized forever about how to get in-game scripting in a Java game. One day it hit me like a flash: embed a Python interpreter. I’m a huge Python fan, and being able to off-load game behavior, as well as fuck with the game while developing it, was a huge step forward for me. Game scripting comes with its dangers—hiding errors until runtime, for one—but I think the benefits will soon outweigh the disadvantages.

In Conclusion

Do yourselves a favor and keep regular engineering logs. It will make personal retrospectives and public changelogs so much easier.