Ninji's Thingtechnical stuff by Ninji
http://wuffs.org/
Mon, 30 Jan 2017 20:39:03 +0000Mon, 30 Jan 2017 20:39:03 +0000Jekyll v3.1.6A Rant About Proxying API Requests on iOS (and others)<p>Wherein I try to find a way to redirect API requests from Twitter clients through an external proxy, for request modification shenanigans. It’s easy everywhere… except for non-jailbroken iOS. Because Apple.</p>
<hr />
<p>I’m a heavy Twitter user, and I’m not happy with any of the clients that exist right now - I’ve wanted to develop my own for a long time, but due to my lack of time and motivation, it’s never really gone anywhere.</p>
<p>My current mobile client is a modified version of Fenix for Android, with a few tweaks that make it work better for my uses (a larger tweet cache, a streaming toggle, and loading images through a compression proxy when on mobile data).</p>
<p>This sort of works, but it’s not optimal. These changes only work on that app and I have to manually port them to new versions. I’ve lived with it up until now, but I’m planning on switching to iOS, where installing modified apps is considerably more difficult (especially without a jailbreak).</p>
<p>So, I hit upon an idea: if I create a Twitter API proxy, I can implement many of the features I rely on (or want to have) without having to modify clients at all, and make other useful tweaks - for instance, if I keep a persistent userstream connection open and then simulate the server-side API in my proxy, then clients can connect/disconnect all they want without ever triggering Twitter’s rate limiting. And I can force images to be compressed simply by replacing the image URLs in API responses.</p>
<p>This should work on any platform, with any app - even desktop apps.</p>
<hr />
<p>Except for one thing. How <em>DO</em> you get your client to connect to the proxy?</p>
<p><em>Twidere on Android</em> lets you provide a different API URL. <em>Twitter for iPhone</em> has an “API root” option which does the same (although it may have been removed in a newer app version; I’m running the last iOS 6 compatible build). Most other clients are locked to the default <code class="highlighter-rouge">api.twitter.com</code>.</p>
<p>This means you’ve got to hijack connections to it somehow. SSL is required, so this will require a self-signed certificate for <code class="highlighter-rouge">api.twitter.com</code> and <code class="highlighter-rouge">userstream.twitter.com</code>, but this shouldn’t be too much of a problem as even iOS lets you install root CAs.</p>
<p>On Android, this isn’t a massive deal. If you root, you can modify your hosts file (with the caveat that you’ll break <em>Twitter for Android</em>, as it <em>forces</em> certificate pinning). If you don’t want to do this, you can always install a patched client.</p>
<hr />
<p>The situation on iOS is different. If you jailbreak, you can modify your hosts file - that’s pretty straightforward, but you can’t depend on a jailbreak being available for newer iOS devices. And if you don’t, what CAN you do?</p>
<ul>
<li><strong>DNS server:</strong> Doable, but you can’t assign custom DNS servers for mobile data, only for individual wifi profiles.</li>
<li><strong>HTTP proxy:</strong> Doable, but you can’t set a proxy username/password for mobile data, and you have to set up settings individually for each wifi network.</li>
<li><strong>VPN:</strong> Okay, this one might work. But isn’t VPNing all your traffic just to hijack Twitter connections a little overkill?!</li>
</ul>
<p>It’s a complete mess.</p>
<p>iOS allows you to specify a global http proxy, to be used on mobile data AND all wifi networks, which seemed like the perfect solution at first. But this option can only be enabled on a “supervised device”, which is the mode used for businessy enterprise deployments or some junk like that. And you need a Mac running Yosemite (or newer) to run <em>Apple Configurator</em>, the tool that lets you supervise a device.</p>
<hr />
<p>So what other options do I have? My iPhone 3GS doesn’t have data access, so I’m unable to test out anything for real until I actually get a new phone.</p>
<p>I might still be able to use a http proxy - I found out online that you can modify configuration files in an iTunes backup, which would potentially allow me to edit <code class="highlighter-rouge">SystemConfiguration/preferences.plist</code> and insert settings that the UI won’t let me add.</p>
<p>I think you can force a http proxy on mobile data through the APN settings, but there doesn’t seem to be a way to set a username/password here. I won’t run an open HTTP proxy, so that idea’s out.</p>
<p>Barring these, a VPN is probably my only choice.</p>
<hr />
<p>It’s nice that iOS has been slowly becoming more open, but silly restrictions like this are amazingly frustrating. What harm is there in allowing unsupervised devices to assign a global http proxy, for instance? (Even if it has to be configured using a profile and not through the device UI.)</p>
<p>It probably won’t stop me from getting an iPhone, because I’m sick of Android and there aren’t really any viable alternatives. Ah, well, I guess it’ll be a change… and hopefully it won’t take too long for an iOS 9 / iPhone 6S+ jailbreak to be available.</p>
Sat, 19 Sep 2015 21:00:00 +0100http://wuffs.org/2015/09/19/proxying-api-requests-on-ios.html
http://wuffs.org/2015/09/19/proxying-api-requests-on-ios.htmlHow I Began Modding NSMBWii<p>I’m planning to write a series of posts about low-level modifications to <em>New Super Mario Bros. Wii</em>, the 2009 Nintendo game. This is a sort of ‘prequel’ to that series, where I write some fluff about how I got to this point and the history/background of NSMBW mods.</p>
<p>Later posts will be more technical, with more of a focus on low-level reverse-engineering stuff and on how I achieved the things I did in Newer SMBW.</p>
<hr />
<p>I’ve always enjoyed tinkering with software and seeing what makes it tick, and when I discovered the ‘ROM hacking’ scene in 2005, I was immediately hooked. At the time, I’d taken an interest in discovering and playing classic Nintendo games (through emulation) - especially the Mario series - and I thought it was ridiculously cool that you could modify games and add things like custom levels and graphics.</p>
<p>Of course, with my limited skills, I couldn’t do much other than use existing tools for editing specific games - I don’t think I’d ever touched a hex editor at that point, and even if I had, I wouldn’t have known how to do anything with a ROM…! On older consoles, all the game code and data - graphics, levels, music, sound effects, etc - were all shoved into the same file.</p>
<p>One thing that makes modern games much easier to delve into - at least, for somebody with the level of skill I had at that point - is that they use a standard file structure which you can easily extract. This was first used by Nintendo on the DS and GameCube. It doesn’t mean you can immediately obtain any data you want, as most of it will be in a non-standard format (e.g. the ‘nsbmd’ model files used by many first-party Nintendo games), but it’s still better than no structure at all, and games often share formats with each other for resources like textures, models and UI layouts.</p>
<h2 id="modding-nsmb-on-the-ds">Modding NSMB on the DS</h2>
<p>When <em>New Super Mario Bros.</em> was released for the Nintendo DS in May 2006, I immediately wanted a level editor for it, but I didn’t quite have the skills to do that at the time, so I waited to see if somebody else would… to no avail. Eventually I decided to try and build my own, and in July 2007 I started doing just that.</p>
<p>I was still very much a novice, but I slowly taught myself how to figure out simple file formats and build half-usable software. From then on, I rewrote my NSMB editor a couple of times - adding more features and learning more each time - and eventually handed it off to somebody else as an open-source C# project.</p>
<p>Nintendo released the inevitable sequel to that game in November 2009, as <em>New Super Mario Bros. Wii</em>, and of course, I simply <em>had</em> to dig into it! It was leaked to pirate sites a week before the release (thanks to a store in Australia that broke the street date), and I couldn’t resist getting my grubby hacker paws all over it.</p>
<p>I quickly figured out that the level format was barely changed from the DS iteration, and it only took me a couple of days to put together a prototype level editor and post a teaser screenshot on the GBATemp message board.</p>
<h2 id="reggie-and-newer-smbw">Reggie! and Newer SMBW</h2>
<p>I asked my home IRC channel for a name for this new editor, and for some reason or other, we ended up calling it “Reggie!”. I think the reason might have been Bob’s Game and his <b><i><font color="red">REGGIE!</font></i></b> cry, but I can’t remember for sure… :p I built up a small team of like-minded people - who helped out massively with documenting the many object settings in the game, and testing, among other tasks - and over the next few months, we developed Reggie into a rather substantial project. <i>(We eventually released it in March 2010 after about four months of development.)</i></p>
<p>At the same time that this was going on, after the announcement of Reggie!, some users on GBATemp were proposing and discussing a project they called ‘Newer Super Mario Bros. Wii’. To us (the Reggie! dev team), this idea seemed outright <strong>absurd</strong> - we’d only just figured out how to edit level files, and weren’t sure how much more we could achieve with the game. Yet the Newer thread on GBATemp was planning a full revamp of the game, and discussing all sorts of ideas for things like new worlds, characters and boss battles.</p>
<p>One night in December 2009, I was bored and decided I’d jump into their chat room and see what they said about me being there. They accused me of being an impostor, but once I proved I was myself, I ended up sticking around for a while. Following some initial hesitation, the Reggie! and Newer teams ended up joining forces. They received access to pre-release builds of our editor, and we received valuable feedback and extra testing. A fine exchange, right?</p>
<p>We still didn’t really believe Newer would end up going anywhere - it was an extremely ambitious project - but we figured we’d finish our level editor anyway. A project like that would definitely require significant amounts of patches to the game code - what people often call ‘ASM hacking’ - and doing this to a Wii game would be infeasible (especially for somebody like me)… or so I thought. I had a small bit of assembly knowledge thanks to prior tinkering with SMB1 (6502 on the NES), and NSMB (ARM on the DS), but no PowerPC, and figuring out how a large game like this worked seemed downright impossible.</p>
<h2 id="but-what-if-it-did-happen">But what if it DID happen??</h2>
<p>Even so, one of the other members of the Reggie! team (megazig) had some experience with reversing games and Wii code, and he looked into NSMBW and found a few interesting bits of info which we ended up using in Reggie, like structures and values extracted from the code responsible for reading level files. This sparked my own interest in disassembling the game. I got some advice from megazig, a combined .elf file I could analyse from comex (in the next post I’ll explain why this was necessary), and then I started digging through the game code.</p>
<p><em>“What the hell have I got myself into?”</em>, I thought. I looked through pages and pages of identical-seeming PowerPC assembly and had no idea where to begin. There were quite a few interesting strings in the game executable, like the names of files and individual resources, but none of these seemed like a good starting point, especially when I had no specific goal in mind.</p>
<p>Another problem was that when I began, I understood assembly well enough to identify what individual instructions were doing, but I couldn’t understand what a given block of code was supposed to do - before I could really make sense of it, I had to translate it to pseudo-C and read <em>that</em>.</p>
<p>I was about to give up and go back to Reggie! when I struck gold. I was scrolling through the disassembly when I came upon a list of strings that seemed suspiciously like the names of game objects. Things like EN_STAR_COIN, EN_KURIBO, OBJ_MORTON and PEACH_CASTLE_BLOCK. I looked at the code that referenced this, and saw that it was reading an entry from the name list and storing it into a structure. So I thought, what if this code is related to object creation?</p>
<p>I painstakingly translated that function to C and it appeared to confirm my suspicion, and as I went on exploring from there, I started learning more and more about the game (and reverse-engineering in general). With nothing other than very limited guidance from megazig, I had to figure out how things like class constructors/destructors, virtual functions and new/delete were handled, and how they correlated with my existing C++ knowledge.</p>
<p>And as I learned more about the game’s internals, and started developing code patches (which worked both on the Dolphin emulator and on real consoles), I realised that Newer actually <em>could</em> happen - perhaps in a less ambitious form than originally envisaged, but it was possible. We were gradually gaining more technical abilities and we were also creating better custom content. Over the next three years, we worked on adding new things and on using our increasing knowledge to make the game better… and in June 2013, we finally released the game.</p>
<h2 id="summary">Summary</h2>
<p>The final version of Newer includes over 33,000 lines of C++ code written by Tempus and I (not counting the header files which bind to Nintendo’s APIs), over 8,000 lines of assembly, and compiles to around 450kb of <em>new code alone</em>. This includes the following:</p>
<ul>
<li>a complete re-write of the game’s world map system which supports multi-layered maps built out of 2D graphics, with animated elements, and a very limited set of 3D objects (all designed in a tool we created)</li>
<li>new user interface elements (a custom HUD on the world map, a ‘Star Coins’ collection progress display, and an item shop, among others)</li>
<li>new actors (objects, blocks, enemies) which you can interact with in-game</li>
<li>a full set of custom bosses</li>
<li>a new powerup (the Hammer Suit)</li>
<li>tweaks to existing features and elements, to add features that weren’t present in the original game, or fix bugs</li>
</ul>
<p>… and that doesn’t take into account all the work we spent on reverse-engineering the game engine, building auxiliary tools (mainly the map designer, but I also have a set of private tools for limited tasks like model and UI editing) and on things we scrapped (like our initial plans to use 3D world maps, which failed due to the lack of available modellers).</p>
<p>So that’s Newer and Reggie! in a rather large nutshell. Now that I’ve covered the backstory behind these projects, and how I got to this point, my next posts will focus on the technical side of Newer - all sorts of fun things like the Wii’s software architecture, how we compile code, and how we get it to run inside the game.</p>
Sat, 27 Jun 2015 01:42:50 +0100http://wuffs.org/2015/06/27/nsmbw-intro.html
http://wuffs.org/2015/06/27/nsmbw-intro.htmlnsmbwIntroduction<p>This post marks my second attempt to start a blog! The last one barely got off the ground, but I feel like I’ve got more to write about now, so perhaps I’ll actually keep it going?</p>
<p>I’m not sure if I’ll keep on writing, but it’s worth a try. I usually end up monologuing to people about things like programming and reverse-engineering.. so I figured, why not try to divert this into a blog instead?</p>
Fri, 26 Jun 2015 19:56:50 +0100http://wuffs.org/2015/06/26/introduction.html
http://wuffs.org/2015/06/26/introduction.htmlmeta