amoshttps://fasterthanli.me/
Recent content on amosHugo -- gohugo.ioen-usSun, 26 Apr 2020 16:10:00 +0000Thread-local storagehttps://fasterthanli.me/blog/2020/thread-local-storage/
Sun, 26 Apr 2020 16:10:00 +0000https://fasterthanli.me/blog/2020/thread-local-storage/Welcome back and thanks for joining us for the reads notes&hellip; the thirteenth installment of our series on ELF files, what they are, what they can do, what does the dynamic linker do to them, and how can we do it ourselves.
I've been pretty successfully avoiding talking about TLS so far (no, not that one) but I guess we've reached a point where it cannot be delayed any further, so.A no_std Rust binaryhttps://fasterthanli.me/blog/2020/a-no-std-rust-binary/
Mon, 13 Apr 2020 23:40:00 +0000https://fasterthanli.me/blog/2020/a-no-std-rust-binary/In Part 11, we spent some time clarifying mechanisms we had previously glossed over: how variables and functions from other ELF objects were accessed at runtime.
We saw that doing so &ldquo;proper&rdquo; required the cooperation of the compiler, the assembler, the linker, and the dynamic loader. We also learned that the mechanism for functions was actually quite complicated! And sorta clever!
And finally, we ignored all the cleverness and &ldquo;made things work&rdquo; with a three-line change, adding support for both GlobDat and JumpSlot relocations.More ELF relocationshttps://fasterthanli.me/blog/2020/more-elf-relocations/
Mon, 06 Apr 2020 20:30:00 +0000https://fasterthanli.me/blog/2020/more-elf-relocations/In our last installment of &ldquo;Making our own executable packer&rdquo;, we did some code cleanups. We got rid of a bunch of unsafe code, and found a way to represent memory-mapped data structures safely.
But that article was merely a break in our otherwise colorful saga of &ldquo;trying to get as many executables to run with our own dynamic loader&rdquo;. The last thing we got running was the ifunc-nolibc program.Safer memory-mapped structureshttps://fasterthanli.me/blog/2020/safer-memory-mapped-structures/
Tue, 03 Mar 2020 19:40:00 +0000https://fasterthanli.me/blog/2020/safer-memory-mapped-structures/Welcome back to the &ldquo;Making our own executable packer&rdquo; series, where digressions are our bread and butter.
Last time, we implemented indirect functions in a no-libc C program. Of course, we got lost on the way and accidentally implemented a couple of useful elk-powered GDB functions - with only the minimal required amount of Python code.
The article got pretty long, and we could use a nice distraction. And I have just the thing!I want off Mr. Golang's Wild Ridehttps://fasterthanli.me/blog/2020/i-want-off-mr-golangs-wild-ride/
Fri, 28 Feb 2020 10:30:00 +0000https://fasterthanli.me/blog/2020/i-want-off-mr-golangs-wild-ride/My honeymoon with the Go language is extremely over.
This article is going to have a different tone from what I've been posting the past year - it's a proper rant. And I always feel bad writing those, because, inevitably, it discusses things a lot of people have been working very hard on.
In spite of that, here we are.
Having invested thousands of hours into the language, and implemented several critical (to my employer) pieces of infrastructure with it, I wish I hadn't.GDB scripting and Indirect functionshttps://fasterthanli.me/blog/2020/gdb-scripting-and-indirect-functions/
Thu, 27 Feb 2020 21:00:00 +0000https://fasterthanli.me/blog/2020/gdb-scripting-and-indirect-functions/In the last article, we cleaned up our dynamic linker a little. We even implemented the Dynamic relocation.
But it's still pretty far away from running real-world applications.
Let's try running a simple C application with it:
// in `samples/puts.c` #include &lt;stdio.h&gt; int main() { puts(&quot;Hello from C&quot;); return 0; } $ cd samples/ $ gcc puts.c -o puts $ ../target/debug/elk ./puts Loading &quot;/home/amos/ftl/elk/samples/puts&quot; Loading &quot;/usr/lib/libc-2.30.so&quot; Fatal error: Could not read symbols from ELF object: Parsing error: String(&quot;Unknown SymType 10 (0xa)&quot;): input: 1a 00 10 00 a0 bf 0b 00 00 00 00 00 c1 00 00 00 00 00 00 00 Even if we did add that variant to SymType, I'm sure there'd be other relocation types, and other subtle things we're not doing quite right yet.Working with strings in Rusthttps://fasterthanli.me/blog/2020/working-with-strings-in-rust/
Wed, 19 Feb 2020 14:30:00 +0000https://fasterthanli.me/blog/2020/working-with-strings-in-rust/There's a question that always comes up when people pick up the Rust programming language: why are there two string types? Why is there String, and &amp;str?
My Declarative Memory Management article answers the question partially, but there is a lot more to say about it, so let's run a few experiments and see if we can conjure up a thorough defense of Rust's approach over, say, C's.
Clear, simple, and wrong Let's start with a simple C program that prints its arguments.Dynamic linker speed and correctnesshttps://fasterthanli.me/blog/2020/dynamic-linker-speed-and-correctness/
Fri, 14 Feb 2020 12:20:00 +0000https://fasterthanli.me/blog/2020/dynamic-linker-speed-and-correctness/In the last article, we managed to load a program (hello-dl) that uses a single dynamic library (libmsg.so) containing a single exported symbol, msg.
Our program, hello-dl.asm, looked like this:
global _start extern msg section .text _start: mov rdi, 1 ; stdout fd mov rsi, msg mov rdx, 38 ; 37 chars + newline mov rax, 1 ; write syscall syscall xor rdi, rdi ; return code 0 mov rax, 60 ; exit syscall syscall And our library, msg.Dynamic symbol resolutionhttps://fasterthanli.me/blog/2020/dynamic-symbol-resolution/
Sun, 02 Feb 2020 12:20:00 +0000https://fasterthanli.me/blog/2020/dynamic-symbol-resolution/Let's pick up where we left off: we had just taught elk to load not only an executable, but also its dependencies, and then their dependencies as well.
We discovered that ld-linux walked the dependency graph breadth-first, and so we did that too. Of course, it's a little bit overkill since we only have one dependency, but, nevertheless, elk happily loads our executable and its one dependency:
$ ./target/debug/elk ./samples/hello-dl Loading &quot;/home/amos/ftl/elk/samples/hello-dl&quot; Found RPATH entry &quot;/home/amos/ftl/elk/samples&quot; Loading &quot;/home/amos/ftl/elk/samples/libmsg.A half-hour to learn Rusthttps://fasterthanli.me/blog/2020/a-half-hour-to-learn-rust/
Mon, 27 Jan 2020 14:30:00 +0000https://fasterthanli.me/blog/2020/a-half-hour-to-learn-rust/In order to increase fluency in a programming language, one has to read a lot of it. But how can you read a lot of it if you don't know what it means?
In this article, instead of focusing on one or two concepts, I'll try to go through as many Rust snippets as I can, and explain what the keywords and symbols they contain mean.
Ready? Go!
let introduces a variable binding:Loading multiple ELF objectshttps://fasterthanli.me/blog/2020/loading-multiple-elf-objects/
Sun, 26 Jan 2020 22:30:00 +0000https://fasterthanli.me/blog/2020/loading-multiple-elf-objects/Up until now, we've been loading a single ELF file, and there wasn't much structure to how we did it: everyhing just kinda happened in main, in no particular order.
But now that shared libraries are in the picture, we have to load multiple ELF files, with search paths, and keep them around so we can resolve symbols, and apply relocations across different objects.
After a long period of trawling through references, painstakingly turning C struct definitions into nom parsers and hunting down valid enum values&hellip; it's time for some graphs.The simplest shared libraryhttps://fasterthanli.me/blog/2020/the-simplest-shared-library/
Wed, 22 Jan 2020 10:30:00 +0000https://fasterthanli.me/blog/2020/the-simplest-shared-library/In our last article, we managed to load and execute a PIE (position-independent executable) compiled from the following code:
; in `elk/samples/hello.asm` global _start section .text _start: mov rdi, 1 ; stdout fd mov rsi, msg mov rdx, 9 ; 8 chars + newline mov rax, 1 ; write syscall syscall xor rdi, rdi ; return code 0 mov rax, 60 ; exit syscall syscall msg: db &quot;hi there&quot;, 10 The big improvement in that article was that we started caring about relocations.ELF relocationshttps://fasterthanli.me/blog/2020/elf-relocations/
Sun, 19 Jan 2020 18:30:00 +0000https://fasterthanli.me/blog/2020/elf-relocations/The last article, Position-independent code, was a mess. But who could blame us? We looked at the world, and found it to be a chaotic and seemingly nonsensical place. So, in order to blend in, we had to let go of a little bit of sanity.
The time has come to reclaim it.
Short of faulty memory sticks, memory locations don't magically turn from 0x0 into valid addresses. Someone is doing the turning, and we're going to find out who, if it takes the rest of the series.Position-independent codehttps://fasterthanli.me/blog/2020/position-independent-code/
Mon, 13 Jan 2020 21:00:00 +0000https://fasterthanli.me/blog/2020/position-independent-code/In the last article, we found where code was hiding in our samples/hello executable, by disassembling the whole file and then looking for syscalls.
Later on, we learned how to inspect which memory ranges are mapped for a given PID (process identifier). We saw that memory areas weren't all equal: they can be readable, writable, and/or executable.
Finally, we learned about program headers and how they specified which parts of the executable file should be mapped to which memory areas.Running an executable without exechttps://fasterthanli.me/blog/2020/running-an-executable-without-exec/
Sun, 12 Jan 2020 19:00:01 +0000https://fasterthanli.me/blog/2020/running-an-executable-without-exec/In part 1, we've looked at three executables:
sample, an assembly program that prints &ldquo;hi there&rdquo; using the write system call. entry_point, a C program that prints the address of main using printf The /bin/true executable, probably also a C program (because it's part of GNU coreutils), and which just exits with code 0. We noticed that entry_point printed different addresses when run with GDB, but always the same address when run directly.What's in a Linux executable?https://fasterthanli.me/blog/2020/whats-in-a-linux-executable/
Sun, 12 Jan 2020 19:00:00 +0000https://fasterthanli.me/blog/2020/whats-in-a-linux-executable/Executables have been fascinating to me ever since I discovered, as a kid, that they were just files. If you renamed a .exe to something else, you could open it in notepad! And if you renamed something else to a .exe, you'd get a neat error dialog.
Clearly, something was different about these files. Seen from notepad, they were mostly gibberish, but there had to be order in that chaos. 12-year-old me knew that, although he didn't quite know how or where to dig to make sense of it all.Crafting ICMP-bearing IPv4 packets with the help of bitvechttps://fasterthanli.me/blog/2019/making-our-own-ping-14/
Thu, 12 Dec 2019 07:22:00 +0000https://fasterthanli.me/blog/2019/making-our-own-ping-14/So. Serializing IPv4 packets. Easy? Well, not exactly.
IPv4 was annoying to parse, because we had 3-bit integers, and 13-bit integers, and who knows what else. Serializing it is going to be exactly the same.
Right now, we don't have a way to serialize that.
Let's take the version and ihl fields, both of which are supposed to take 4 bits, together making a byte. We could serialize them like this:Crafting ARP packets to find a remote host's MAC addresshttps://fasterthanli.me/blog/2019/making-our-own-ping-13/
Thu, 12 Dec 2019 07:21:00 +0000https://fasterthanli.me/blog/2019/making-our-own-ping-13/Alright. ALRIGHT. I know, we're all excited, but let's think about what we're doing again.
So we've managed to look at real network traffic and parse it completely. We've also taken some ICMP packets, parsed them, and then serialized them right back and we got the exact same result.
So I know what you're thinking - let's just move our way down the stack again - stuff that ICMP packet in an IP packet, then in an Ethernet frame, and then serialize the whole thing.Parsing and serializing ICMP packets with cookie-factory.https://fasterthanli.me/blog/2019/making-our-own-ping-12/
Thu, 12 Dec 2019 07:20:00 +0000https://fasterthanli.me/blog/2019/making-our-own-ping-12/In the last part, we've finally parsed some IPv4 packets. We even found a way to filter only IPv4 packets that contain ICMP packets.
There's one thing we haven't done though, and that's verify their checksum. Folks could be sending us invalid IPv4 packets and we'd be parsing them like a fool!
This series is getting quite long, so let's jump right into it.
Let's read a bit of RFC 791:The many rewrites of the itch.io desktop apphttps://fasterthanli.me/blog/2019/itch-app-rewrites/
Fri, 22 Nov 2019 16:00:00 +0000https://fasterthanli.me/blog/2019/itch-app-rewrites/I started working on the itch.io desktop app over 4 years ago.
It has arguably been my main project ever since, along with companion projects like butler, capsule and many smaller libraries.
I'm fuzzy on the initial history, but I remember the codebase went through a lot of changes. As early as 2014, the whole codebase was ported from vanilla JavaScript to TypeScript. In 2016, I released a timeline of all the changes.Parsing IPv4 packets, including numbers smaller than byteshttps://fasterthanli.me/blog/2019/making-our-own-ping-11/
Thu, 21 Nov 2019 19:21:00 +0000https://fasterthanli.me/blog/2019/making-our-own-ping-11/Hello and welcome to Part 11 of this series, wherein we finally use some of the code I prototyped way back when I was planning this series.
Where are we standing? Let's review the progress we've made in the first 10 parts: first, we've started thinking about what it takes for computers to communicate. Then, we've followed a rough outline of the various standards and protocols that have emerged since the 1970s.Improving error handling - panics vs. proper errorshttps://fasterthanli.me/blog/2019/making-our-own-ping-10/
Thu, 21 Nov 2019 19:20:00 +0000https://fasterthanli.me/blog/2019/making-our-own-ping-10/Before we move on to parsing more of our raw packets, I want to take some time to improve our error handling strategy.
Currently, the ersatz codebase contains a mix of Result&lt;T, E&gt;, and some methods that panic, like unwrap() and expect().
We also have a custom Error enum that lets us return rawsock errors, IO errors, or Win32 errors:
pub enum Error { Rawsock(rawsock::Error), IO(std::io::Error), Win32(u32), } First of all, I want to address something: When is it okay to panic?Rust 2020: Fundinghttps://fasterthanli.me/blog/2019/rust-2020-funding/
Wed, 06 Nov 2019 17:30:02 +0000https://fasterthanli.me/blog/2019/rust-2020-funding/Blog posts that praise Rust are many but funding is generally in short supply.
If even a small percentage of the money Rust saves companies was put back into the ecosystem it would help secure the future of the platform tremendously.
Multiple sources of funding It is unreasonable going forward to expect the same handful of companies to provide all the funding.
Although Rust was born (or so I'm told) at Mozilla, it seems pretty clear that it has outgrown it significantly, and many big companies now benefit from Rust.Consuming Ethernet frames with the nom cratehttps://fasterthanli.me/blog/2019/making-our-own-ping-9/
Tue, 05 Nov 2019 19:30:02 +0000https://fasterthanli.me/blog/2019/making-our-own-ping-9/Now that we've found the best way to find the &ldquo;default network interface&rdquo;&hellip; what can we do with that interface?
Well, listen for network traffic of course!
use rawsock::open_best_library; use std::time::Instant; fn main() -&gt; Result&lt;(), Error&gt; { let lib = open_best_library()?; let iface_name = format!(r#&quot;\Device\NPF_{}&quot;#, netinfo::default_nic_guid()?); let iface = lib.open_interface(&amp;iface_name)?; println!(&quot;Listening for packets...&quot;); // doing some low-cost logging over here let start = Instant::now(); iface.loop_infinite_dyn(&amp;mut |packet| { println!( &quot;{:?} | received {} bytes&quot;, start.Binding C APIs with variable-length structs and UTF-16https://fasterthanli.me/blog/2019/making-our-own-ping-8/
Tue, 05 Nov 2019 19:30:01 +0000https://fasterthanli.me/blog/2019/making-our-own-ping-8/Okay, I lied.
I'm deciding - right this instant - that using wmic is cheating too. Oh, it was fair game when we were learning about Windows, but we're past that now.
We know there's IPv4 routing tables, and we know network interfaces have indices (yes, they do change when you disable/enable one, so ill-timed configuration changes may make our program blow up).
We also know how to call Win32 APIs directly, and we know that WMI really is more geared toward system administrators than it is toward developers.Finding the default network interface through WMIhttps://fasterthanli.me/blog/2019/making-our-own-ping-7/
Tue, 05 Nov 2019 19:30:00 +0000https://fasterthanli.me/blog/2019/making-our-own-ping-7/Let's set aside our sup project for a while.
Don't get me wrong - it's a perfectly fine project, and, were we simply rewriting &ldquo;ping&rdquo; for Windows in Rust, we could (almost) stop there.
We're currently using the operating system's facility to speak ICMP, which is great for a bunch of reasons: we can be sure that whatever flaws there are in the implentation, all &ldquo;native&rdquo; Windows programs suffer from it as well.Program in C (Parody song)https://fasterthanli.me/blog/2019/program-in-c/
Thu, 31 Oct 2019 07:30:00 +0000https://fasterthanli.me/blog/2019/program-in-c/ Once upon a time, @Cinememer wrote some alternative lyrics to &ldquo;Under The Sea&rdquo;. I couldn't resist singing them!
Download high-quality audio (FLAC) Download audio (OGG) Download audio (MP3) Download lyrics (TXT) Instrumental (Neverland Orchestra) The builder pattern, and a macro that keeps FFI code DRYhttps://fasterthanli.me/blog/2019/making-our-own-ping-6/
Sat, 12 Oct 2019 07:31:00 +0000https://fasterthanli.me/blog/2019/making-our-own-ping-6/Our ping API is simple, but it's also very limited:
pub fn ping(dest: ipv4::Addr) -&gt; Result&lt;(), String&gt; // called as: ping(ipv4::Addr([8, 8, 8, 8])).unwrap(); It doesn't allow specifying the TTL (time to live) of packets, it doesn't allow specifying the timeout, it doesn't let one specify the data to send along, and it doesn't give us any kind of information on the reply.
Let's change that now.
We could take all of these as arguments:A simple ping library, parsing strings into IPv4 addresshttps://fasterthanli.me/blog/2019/making-our-own-ping-5/
Sat, 12 Oct 2019 07:30:00 +0000https://fasterthanli.me/blog/2019/making-our-own-ping-5/We've just spent a lot of time abstracting over LoadLibrary, but we still have all the gory details of the Win32 ICMP API straight in our main.rs file! That won't do.
This time will be much quicker, since we already learned about carefully designing an API, hiding the low-level bits and so on.
Let's add an icmp module to our program. Actually, we've been dealing with an IPAddr all this time, it also sounds like it could use its own package:Designing and implementing a safer API on top of LoadLibraryhttps://fasterthanli.me/blog/2019/making-our-own-ping-4/
Fri, 04 Oct 2019 07:30:00 +0000https://fasterthanli.me/blog/2019/making-our-own-ping-4/It's refactor time! Our complete program is now about a hundred lines, counting blank lines (see the end of part 3 for a complete listing).
While this is pretty good for a zero-dependency project (save for pretty-hex), we can do better.
First off, concerns are mixed up. In the same file, we:
Expose LoadLibraryA / GetProcAddress Expose the Win32 ICMP API Use both of the above to ping 8.8.8.8 Second, a lot of low-level details are exposed, that really shouldn't be.FFI-safe types in Rust, newtypes and MaybeUninithttps://fasterthanli.me/blog/2019/making-our-own-ping-3/
Mon, 30 Sep 2019 19:44:00 +0000https://fasterthanli.me/blog/2019/making-our-own-ping-3/It's time to make sup, our own take on ping, use the Win32 APIs to send an ICMP echo. Earlier we discovered that Windows's ping.exe used IcmpSendEcho2Ex. But for our purposes, the simpler IcmpSendEcho will do just fine.
As we mentioned earlier, it's provided by IPHLPAPI.dll, and its C declaration is:
IPHLPAPI_DLL_LINKAGE DWORD IcmpSendEcho( HANDLE IcmpHandle, IPAddr DestinationAddress, LPVOID RequestData, WORD RequestSize, PIP_OPTION_INFORMATION RequestOptions, LPVOID ReplyBuffer, DWORD ReplySize, DWORD Timeout ); Compared to MessageBoxA, there's a lot more types going on!Windows dynamic libraries, calling conventions, and transmutehttps://fasterthanli.me/blog/2019/making-our-own-ping-2/
Sat, 28 Sep 2019 18:32:00 +0000https://fasterthanli.me/blog/2019/making-our-own-ping-2/So, how does ping.exe actually send a ping? It seems unrealistic that ping.exe itself implements all the protocols involved in sending a ping. So it must be calling some sort of library. Also, since it ends up talking to the outside world via a NIC (network interface controller), the kernel is probably involved at some point.
In reading files the hard way - part 2, we learned about dynamic libraries (like libc), and the Linux kernel, and how syscalls allowed us to ask the Linux kernel to do our bidding.A short (and mostly wrong) history of computer networkinghttps://fasterthanli.me/blog/2019/making-our-own-ping/
Fri, 27 Sep 2019 18:46:15 +0000https://fasterthanli.me/blog/2019/making-our-own-ping/When I launched my Patreon, I vowed to explain how computers work. But in 2019, computers rarely work in isolation. So let's take the time to write a few articles about how computers talk to each other.
The history of network protocols and standards is long and complicated. Starting with a comprehensive review would prove quite tedious, especially if such a review was done in isolation from modern use.
So, instead, as we did with files, we'll start by accomplishing concrete tasks and work our way back to understand which protocols are involved and how they fit together.Declarative memory managementhttps://fasterthanli.me/blog/2019/declarative-memory-management/
Thu, 19 Sep 2019 18:10:02 +0000https://fasterthanli.me/blog/2019/declarative-memory-management/It feels like an eternity since I've started using Rust, and yet I remember vividly what it felt like to bang my head against the borrow checker for the first few times.
I'm definitely not alone in that, and there's been quite a few articles on the subject! But I want to take some time to present the borrow checker from the perspective of its benefits, rather than as an opponent to fend with.Reading files the hard way - Part 3 (ftrace, disk layouts, ext4)https://fasterthanli.me/blog/2019/reading-files-the-hard-way-3/
Sat, 31 Aug 2019 00:00:02 +0000https://fasterthanli.me/blog/2019/reading-files-the-hard-way-3/So far, we've seen many ways to read a file from different programming languages, we've learned about syscalls, how to make those from assembly, then we've learned about memory mapping, virtual address spaces, and generally some of the mechanisms in which userland and the kernel interact.
But in our exploration, we've always considered the kernel more or less like a &ldquo;black box&rdquo;. It's time to change that.
In the belly of the beast In part 2, we've used gdb to step through our userland program, readfile.Reading files the hard way - Part 2 (x86 asm, linux kernel)https://fasterthanli.me/blog/2019/reading-files-the-hard-way-2/
Sat, 31 Aug 2019 00:00:01 +0000https://fasterthanli.me/blog/2019/reading-files-the-hard-way-2/Looking at that latest mental model, it's.. a bit suspicious that every program ends up calling the same set of functions. It's almost like something different happens when calling those.
Are those even regular functions? Can we step through them with a debugger?
If we run our stdio-powered C program in gdb, and break on read, we can confirm that we indeed end up calling a read function:
Cool bear's hot tip GDB is an open-source debugger that runs on Linux, macOS (sometimes), and Windows (with some limitations).Reading files the hard way - Part 1 (node.js, C, rust, strace)https://fasterthanli.me/blog/2019/reading-files-the-hard-way/
Sat, 31 Aug 2019 00:00:00 +0000https://fasterthanli.me/blog/2019/reading-files-the-hard-way/Everybody knows how to use files. You just open up File Explorer, the Finder, or a File Manager, and bam - it's chock-full of files. There's folders and files as far as the eye can see. It's a genuine filapalooza. I have never once heard someone complain there were not enough files on their computer.
But what is a file, really? And what does reading a file entail, exactly?
That's what we're about to find out.Huffman 101https://fasterthanli.me/blog/2019/huffman-101/
Fri, 09 Aug 2019 00:01:00 +0000https://fasterthanli.me/blog/2019/huffman-101/Let's play a game: your objective is to guess a word, but you can only ask yes or no questions. You should also aim to ask as few questions as possible.
You might have played a variant of this game before, guessing famous actors or musicians. You'd usually ask questions like &ldquo;Are they alive?&quot;, or &ldquo;Have they won an Oscar&rdquo;? And that would allow you to narrow down the possibilities, until you finally resort to a list of direct guesses (&ldquo;Is it Amy Adams?And now for a bit of an announcementhttps://fasterthanli.me/blog/2019/patreon/
Fri, 09 Aug 2019 00:00:00 +0000https://fasterthanli.me/blog/2019/patreon/Hey all, thanks for checking in!
After much soul searching, I have arrived to the following conclusion:
Teaching folks about stuff is my jam. I've been writing multiple articles that sort of read like course material, if there was no dress code, maybe?
In 2013, I organized a 1st year Computer Science student project. Instead of making them implement &ldquo;control tower software&rdquo; for a fictional airline, I decided to go for something real - the BitTorrent protocol.Rust modules vs fileshttps://fasterthanli.me/blog/2019/rust-modules-vs-files/
Mon, 01 Jul 2019 00:00:00 +0000https://fasterthanli.me/blog/2019/rust-modules-vs-files/A while back, I asked on Twitter what people found confusing in Rust, and one of the top topics was &ldquo;how the module system maps to files&rdquo;.
I remember struggling with that a lot when I first started Rust, so I'll try to explain it in a way that makes sense to me.
Important note All that follows is written for Rust 2018 edition. I have no interest in learning (or teaching) the ins and outs of the previous version, especially because it was a lot more confusing to me.Celebrating Mario Makerhttps://fasterthanli.me/blog/2019/celebrating-mario-maker/
Tue, 21 May 2019 00:00:00 +0000https://fasterthanli.me/blog/2019/celebrating-mario-maker/I've been watching a lot of Super Mario Maker videos this past month. Probably a hundred hours! This game is like a world onto itself, and it was fascinating to learn its design language and patterns.
With Super Mario Maker 2 coming out soon, I thought I'd show off some of the cool stuff I've seen, to celebrate Mario Maker.
We'll start with some basic elements of Mario Maker (with screenshots), and then move on to a lot of video clips showing cool stuff.Rust generics vs Java genericshttps://fasterthanli.me/blog/2019/rust-vs-java-generics/
Thu, 09 May 2019 00:00:00 +0000https://fasterthanli.me/blog/2019/rust-vs-java-generics/In my previous article, I said I needed to stop thinking of Rust generics as Java generics, because in Rust, generic types are erased.
Someone gently pointed out that they are also erased in Java, the difference was elsewhere. And so, let's learn the difference together.
Java generics I learned Java first (a long, long time ago), and their approach to generics made sense to me at the time.
Here's how Java sees generics: Every instance of a class is an object, and every object inherits from Object.Recursive iterators in Rusthttps://fasterthanli.me/blog/2019/recursive-iterators-rust/
Wed, 08 May 2019 00:00:00 +0000https://fasterthanli.me/blog/2019/recursive-iterators-rust/I've been looking for this blog post everywhere, but it doesn't exist, so I guess it's my turn to write about Some Fun with Rust.
The task at hand Let's say you have a recursive, acyclic data structure, like so:
struct Node { values: Vec&lt;i32&gt;, children: Vec&lt;Node&gt;, } This allows you to represent a tree-like structure:
[1, 2, 3] /\ / \ / \ / \ / \ [4, 5] [6, 7] Now let's say you want to iterate over the values of the root node and all its children, recursively, so that you get the sequence [1, 2, 3, 4, 5, 6, 7].Ludum Dare 44 post-mortemhttps://fasterthanli.me/blog/2019/ld44-postmortem/
Tue, 30 Apr 2019 00:00:00 +0000https://fasterthanli.me/blog/2019/ld44-postmortem/So, I tried doing Ludum Dare 44, and I gave up on it, and I feel like absolute crap so I'm going to attempt to write these feelings away in a celebratory post-mortem &ldquo;I failed!&rdquo; post.
In typical me fashion, this post is going to be a jolly mix of a lot of things - hopefully explaining what I was going for is going to make me feel better about the end result.2018 Retrospectivehttps://fasterthanli.me/blog/2018/2018-retrospective/
Fri, 21 Dec 2018 19:00:00 +0000https://fasterthanli.me/blog/2018/2018-retrospective/The year is drawing to a close, and I'm going off on a much-needed holiday next week. This seems like a good time to look back at the past twelve months!
I can't believe that shipped 2018 was the year of foundational work. As far as &ldquo;work work&rdquo; is concerned, I spent the first 9 months finishing up my largest project ever, the itch v25 rewrite.
Source: itch.io desktop app screenshotTesting itch v25https://fasterthanli.me/blog/2018/itch-v25-pt3/
Thu, 20 Sep 2018 16:30:00 +0000https://fasterthanli.me/blog/2018/itch-v25-pt3/We've just taken a look at the development for v25 of the itch app, so it's time to take a look at some testing infrastructure.
Unit tests When I started working on the app, I was happy to get anything running at all. But it became clear pretty quickly that without proper testing, regressions would keep happening.
But unit testing that beast is pretty hard - for three major reasons:itch v25 development environmenthttps://fasterthanli.me/blog/2018/itch-v25-pt2/
Fri, 14 Sep 2018 17:01:00 +0000https://fasterthanli.me/blog/2018/itch-v25-pt2/One of my goals with the v25 release was to make it much easier to develop, test, and distribute the app.
Base tools I typically develop the itch app under Windows 10 or a Debian-based Linux.
The set-up is mostly the same for both OSes. You've got your standard unix tools, tmus for panes, git, and that's about it. I prefer zsh, but bash (with a few aliases stolen from prezto) will do when I'm using msys2.A bit of perspective on itch v25https://fasterthanli.me/blog/2018/itch-v25-pt1/
Thu, 13 Sep 2018 19:01:00 +0000https://fasterthanli.me/blog/2018/itch-v25-pt1/It's a bit daunting to write a postmortem for the v25 release of the itch.io desktop app.
In part because, it being mostly a reliability/performance update, it's not press-friendly. But also, because there's so much to share!
Before I jump into specific subjects, I want to throw a bunch of numbers at you so you can get a better sense of the scope of the project.
The platform itch.io is home to about 125'000 games at the time of this writing.itch v25 and healthhttps://fasterthanli.me/blog/2018/itch-v25-pt0/
Thu, 13 Sep 2018 19:00:00 +0000https://fasterthanli.me/blog/2018/itch-v25-pt0/Back in August of 2017, I started writing an article about my work on the itch.io app. I quickly realized one article would not cut it.
So I wrote a second. And a third. And a fourth. In August of 2017, I was pretty sure I was nearly done with the work. It looked &ldquo;pretty good if you asked me&rdquo;, and there was &ldquo;probably just a lot of testing&rdquo; to be done before if it was &ldquo;good to ship&rdquo;.Remembering the departedhttps://fasterthanli.me/blog/2018/remembering-the-departed/
Thu, 19 Jul 2018 00:00:00 +0000https://fasterthanli.me/blog/2018/remembering-the-departed/In April of 2012, I wrote a few paragraphs about suicide.
I still get regular e-mails about it, from folks who are currently struggling. They often ask if I've written anything else about the subject. They often thank me for writing it, or say it helped them in some way.
It's been over 6 years since I wrote this article, and a lot has happened since. Maybe it's time I write about this again.Abouthttps://fasterthanli.me/about/
Tue, 22 Aug 2017 00:00:00 +0000https://fasterthanli.me/about/Hello and welcome! I'm Amos.
I was born in 1990 and grew up playing the piano, building dumb robots out of LEGO and servos, and trying to create games with The Games Factory, C/OpenGL and later various Java libraries.
My first big gamedev project: Stratagem, an XML-driven strategy game engine. I studied Computer Science at EPFL, in Switzerland. While I was there, I designed and helped run a 1st year CS project where students implemented the BitTorrent protocol.Efficient game updateshttps://fasterthanli.me/blog/2017/efficient-game-updates/
Mon, 16 Jan 2017 00:00:00 +0000https://fasterthanli.me/blog/2017/efficient-game-updates/itch.io has large &amp; often-updated games. How do we distribute these efficiently?Things that can go wrong when downloadinghttps://fasterthanli.me/blog/2017/downloads-gone-wrong/
Mon, 02 Jan 2017 00:00:00 +0000https://fasterthanli.me/blog/2017/downloads-gone-wrong/This is why we can't have nice thingsitch.io app timeline 2016https://fasterthanli.me/blog/2016/10-months-of-itch/
Sat, 01 Oct 2016 00:00:00 +0000https://fasterthanli.me/blog/2016/10-months-of-itch/That's amore*andfallhttps://fasterthanli.me/blog/2015/andfall/
Tue, 25 Aug 2015 00:00:00 +0000https://fasterthanli.me/blog/2015/andfall/Forgive me father, for I have sinnedooc generics and flawed designshttps://fasterthanli.me/blog/2015/ooc-generics-and-flawed-designs/
Mon, 19 Jan 2015 00:00:00 +0000https://fasterthanli.me/blog/2015/ooc-generics-and-flawed-designs/What can I say&hellip;S-exps in your browserhttps://fasterthanli.me/blog/2014/sexps-in-your-browser/
Mon, 03 Nov 2014 00:00:00 +0000https://fasterthanli.me/blog/2014/sexps-in-your-browser/<a href="http://xkcd.com/224/">http://xkcd.com/224/</a> was right all along.Happy stay with us dayhttps://fasterthanli.me/blog/2014/happy-stay-with-us-day/
Wed, 10 Sep 2014 00:00:00 +0000https://fasterthanli.me/blog/2014/happy-stay-with-us-day/It's world depressing day 2014! Time to write another article.Cross-compilation noteshttps://fasterthanli.me/blog/2014/cross-compilation-notes/
Thu, 07 Aug 2014 00:00:00 +0000https://fasterthanli.me/blog/2014/cross-compilation-notes/Autotools, CMake, raw make&hellip; how do you build for another platform?On the subject of timehttps://fasterthanli.me/blog/2014/on-the-subject-of-time/
Tue, 11 Feb 2014 00:00:00 +0000https://fasterthanli.me/blog/2014/on-the-subject-of-time/It's a fine Wednesday of February, and I'm sitting in my living room at four in the morning, typing these words. Just a few minutes ago, I poured myself a half glass of wine and smoked a cigarette, celebrating the end of &ldquo;my first week&rdquo;, as a matter of saying. I'm done with paid work for the week, and I get a few days to enjoy doing what I'm really interested in these days: creating games.Three gamedev surpriseshttps://fasterthanli.me/blog/2014/three-gamedev-surprises/
Sat, 08 Feb 2014 00:00:00 +0000https://fasterthanli.me/blog/2014/three-gamedev-surprises/Despite their peaceful appearance, game developers actually lead thrilling lives! Here are three things I learned (or re-learned) about yesterday that I'd like to share with you, in the form of assumptions that revealed false.
VSync is relatively straightforward. Right? As an obsessive-compulsive, bipolar, perfectionist game dev, getting your game to run smoothly on all kinds of operating systems, graphics cards, and drivers combination is something of a holy grail. Many look for it, but let's be honest here, it never really turns out as expected.Fast font packing for fun and profithttps://fasterthanli.me/blog/2014/fast-font-packing-for-fun-and-profit/
Fri, 07 Feb 2014 00:00:00 +0000https://fasterthanli.me/blog/2014/fast-font-packing-for-fun-and-profit/Being creative is hard work, let's go optimizing instead! My graphics engine dye was pretty naive about displaying text, and it was wasteful. Let's see how I made it all better with this one weird tip.
Disclaimer: Even after a few years I'm still very much an OpenGL newbie. Please don't hit me with crowbars.
Once upon a time, OpenGL was easy to use - and also falling out of relevancy as far as high-performance 3D graphics were concerned.Lestac: The Making Ofhttps://fasterthanli.me/blog/2013/lestac-the-making-of/
Wed, 18 Dec 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/lestac-the-making-of/Ludum Dare 28 is over! Let's review five things you might not know about Lestac.The quest for ooc.vimhttps://fasterthanli.me/blog/2013/the-quest-for-ooc-vim/
Fri, 06 Dec 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/the-quest-for-ooc-vim/&lsquo;We become what we behold. We shape our tools and then our tools shape us.&rsquo; &ndash; Marshall McLuhanrock 0.9.8 is outhttps://fasterthanli.me/blog/2013/rock-0-9-8-released/
Mon, 18 Nov 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/rock-0-9-8-released/The latest version of rock is fresh out of the llama's lair. Let's review what this new iteration contains.And then there were fewer bugshttps://fasterthanli.me/blog/2013/and-then-there-were-fewer-bugs/
Tue, 15 Oct 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/and-then-there-were-fewer-bugs/A series of nasty bugs have been keeping me busy recently. Let's see why those happened and how I fixed them.rock 0.9.7 + new websitehttps://fasterthanli.me/blog/2013/rock-0-9-7-released/
Wed, 09 Oct 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/rock-0-9-7-released/A new release of rock is available, along with a fully revamped website.Game Design: The Binding of Isaachttps://fasterthanli.me/blog/2013/game-design-the-binding-of-isaac/
Sat, 22 Jun 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/game-design-the-binding-of-isaac/Ed's blasphemous roguelike is still going strong. While waiting for Rebirth, why not take a look at its design?The Choice Ep. 1: Debriefinghttps://fasterthanli.me/blog/2013/the-choice-episode-1-debrief/
Fri, 31 May 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/the-choice-episode-1-debrief/Episode 1 is just the beginning. Do not delude yourself. This evaluation is far from over.Damian Sommer on The Yawhghttps://fasterthanli.me/blog/2013/damian-sommer-on-the-yawhg/
Wed, 29 May 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/damian-sommer-on-the-yawhg/Four questions to the friendly canadian dev about his latest interactive adventure.The iterative nature of arthttps://fasterthanli.me/blog/2013/the-iterative-nature-of-art/
Mon, 27 May 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/the-iterative-nature-of-art/Words of wisdom from Dominique &lsquo;Dom2D&rsquo; Ferland, independent game designer.oocdoc, Part 4 — sourcepathhttps://fasterthanli.me/blog/2013/oocdoc-part-4-sourcepath/
Sat, 25 May 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/oocdoc-part-4-sourcepath/Continuing our ooc documentation generator, we are preparing the terrain for serious work with multiple ooc modules!sam 0.2.0 releasedhttps://fasterthanli.me/blog/2013/sam-0-2-0-released/
Sat, 25 May 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/sam-0-2-0-released/sam keeps your ooc libraries up-to-date. See what's new in this release!The best way to learnhttps://fasterthanli.me/blog/2013/the-best-way-to-learn/
Sat, 25 May 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/the-best-way-to-learn/Words of wisdom from Noel Berry, independent game developer.oocdoc, Part 3 — parsinghttps://fasterthanli.me/blog/2013/parsing/
Fri, 24 May 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/parsing/After reviewing a few alternatives, I'm building my own documentation generator. I'll show you how!The shortest ooc quinehttps://fasterthanli.me/blog/2013/the-shortest-ooc-quine/
Fri, 24 May 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/the-shortest-ooc-quine/No, seriously. You won't find shorter than that. And if you do, I want to know what hardware you're using.An ooc quinehttps://fasterthanli.me/blog/2013/quine/
Thu, 23 May 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/quine/Quine, noun: a computer program which takes no input and produces a copy of its own source code as its only output.oocdoc, Part 2 — brummihttps://fasterthanli.me/blog/2013/brummi/
Thu, 23 May 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/brummi/Today we take a look at a very minimalist tool for generating ooc docs by Friedrich Weber: brummi!oocdoc, Part 1 — NaturalDocshttps://fasterthanli.me/blog/2013/naturaldocs/
Wed, 22 May 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/naturaldocs/Starting a series on everyone's favorite software problem: generating good documentation! Today, we take a look at NaturalDocs.Experiments in happinesshttps://fasterthanli.me/blog/2013/happiness/
Sun, 19 May 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/happiness/The past year has been rough, but I'm finally getting out of it. Here's what I learned, and why this blog is really colorful now!Isaac rubs his back on non-existent doorshttps://fasterthanli.me/blog/2013/collision/
Wed, 20 Mar 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/collision/Haven't blogged in a while. Life's fine, project are a-plenty, but I just wanted to make a more lasting post about one particular issue that struck me as funny when programming Paper Isaac.
Bugs, bugs, bugs What's infuriating when letting others play an early prototype is that you hear constantly the same things. Some bugs are non-trivial to fix, some you're just not motivated to fix now&hellip; sometimes you just have your head elsewhere, gotta focus, or are elbow-deep in some other piece of code and the damn walls can wait.Things I struggle withhttps://fasterthanli.me/blog/2013/struggle/
Wed, 27 Feb 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/struggle/Putting thoughts in bits I think about lots of things but when it comes down to writing them, drawing them, implementing them, it's not that easy. Even with years of practice in each of these trades, it's still an uphill battle.
Which is why I am not going to read that article after I wrote it and will go straight to publication.
Not assuming nobody cares While I have always been surrounded by caring and loving people, I still automatically assume that whenever I put out something (a blog post, a game, a library, anything), nobody cares.Next power of twohttps://fasterthanli.me/blog/2013/npot/
Fri, 22 Feb 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/npot/While looking to write a pure ooc version of ftgl, I was reading the source of ftgl-gl3 and I stumbled upon this piece of code:
static inline GLuint NextPowerOf2(GLuint in) { in -= 1; in |= in &gt;&gt; 16; in |= in &gt;&gt; 8; in |= in &gt;&gt; 4; in |= in &gt;&gt; 2; in |= in &gt;&gt; 1; return in + 1; } This is needed because dealing with power-of-two textures (32x32, 64x64, 128x128, etc.rock 0.9.6 is on the loose!https://fasterthanli.me/blog/2013/codename-loki/
Wed, 20 Feb 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/codename-loki/Just 8 days after the last release, rock 0.9.6 is out.
To update, run git pull &amp;&amp; make rescue as usual. To install from scratch, clone the repo, cd into it, and run make rescue from there - it'll download the latest bootstrap, compile itself from C, then recompile itself from ooc.
Running rock -V should give you something like this:
rock 0.9.6 codename loki, built on Wed Feb 20 15:09:08 2013Android development with rock 0.9.5https://fasterthanli.me/blog/2013/ooc-android/
Wed, 13 Feb 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/ooc-android/rock 0.9.5 is out! It's the meanest, slimmest, baddest rock release yet.
To update, run git pull &amp;&amp; make rescue as usual. To install from scratch, clone the repo, cd into it, and run make rescue from there - it'll download the latest bootstrap, compile itself from C, then recompile itself from ooc.
Running rock -V should print this happy little version line:
rock 0.9.5 codename panda, built on Wed Feb 13 00:39:52 2013NeverJam: the game jam jam gamehttps://fasterthanli.me/blog/2013/neverjam/
Wed, 30 Jan 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/neverjam/Our January project was ambitious: a 2D puzzle game, a-la lemmings with a twist, with big and numerous levels. And of course, all using our homegrown tools, from the compiler to the level editor to the UI system and game framework.
However, January ended too soon, and, sleepless nights notwithstanding, I had to resolve to publish something completely different. It was a good occasion to get to know Twine.sam, homebrew-mingw, etc.https://fasterthanli.me/blog/2013/braindump/
Thu, 24 Jan 2013 00:00:00 +0000https://fasterthanli.me/blog/2013/braindump/I want to write blog posts, but right now I have too much to do.
So instead, here are bullet points:
I wrote an ooc tool named sam, which helps you keep your git repos up-to-date, and helps to remind you what to push when switching workstations. It's pretty neat, and portable.
A while ago, I started working on homebrew for Windows, or rather, for MinGW+MSYS. Provided you have msysgit and Ruby in your PATH, it'll let you brew install most packages.Having fun with oochttps://fasterthanli.me/blog/2012/ooc-tips/
Mon, 31 Dec 2012 00:00:00 +0000https://fasterthanli.me/blog/2012/ooc-tips/Unfortunately, the ooc language could have better documentation. In the meantime, I'd like to blog about about some features that might not be very well-known.
Nested functions Here's a program that prints 1, 3, 5, 7, 9:
import structs/ArrayList main: func { list := ArrayList&lt;Int&gt; new() addIfOdd := func (i: Int) { if (i % 2 != 0) list add(i) } for (i in 0..10) { addIfOdd(i) } list map(|i| i toString()) join(&quot;, &quot;) println() } If you're doing something repeatedly, and it won't be too readable with a closure, you can simply declare a nested function!The perils of ooc argumentshttps://fasterthanli.me/blog/2012/ooc-arguments/
Mon, 31 Dec 2012 00:00:00 +0000https://fasterthanli.me/blog/2012/ooc-arguments/The ooc language is known to be friendly to C libraries, and we have a slew of them covered on GitHub, but one common hurdle is how to correctly declare extern functions.
Argument types For an ooc function prototype, there are many types of arguments. You can go with regular variable declarations, like so:
something: func (a: Int, b: Int, c: String) But in this case, a and b have the same type, so you can also use multi-declarations to shorten it a bit:Cross-platform game distributionhttps://fasterthanli.me/blog/2012/game-distrib/
Mon, 24 Dec 2012 00:00:00 +0000https://fasterthanli.me/blog/2012/game-distrib/ooc makes it easy to compile your application on all major platforms (Windows, OSX, Linux) - the compiler itself runs there, and the SDK supports all these platforms with basic functionality: data structures, file handling, time handling, networking, etc.
But between getting your application running on your dev environment with all the libraries installed, and getting it into a neat package for your users to run without having to install any dependencies by hand, there's a bag of tricks.Ludum Dare #25 Post-mortemhttps://fasterthanli.me/blog/2012/ld25-postmortem/
Thu, 20 Dec 2012 00:00:00 +0000https://fasterthanli.me/blog/2012/ld25-postmortem/Last week-end, I participated to Ludum Dare for the fourth time in a row!
Downloads: Linux (64) | OS/X | Windows
Story So here is our entry: Legithief. The backstory is simple, yet cunning: you are an ordinary thief practicing ordinary acts of thievery in the houses of ordinary people to make a living. But one day.. you are quietly robbing yet another home, when you are suddenly smashed in the head with a bat.AOT vs JIT: Why don't we do both?https://fasterthanli.me/blog/2012/scissors/
Tue, 04 Dec 2012 00:00:00 +0000https://fasterthanli.me/blog/2012/scissors/I wanted to take some time to write about a piece of software I've been working on lately, just so you know how I've been spending the last few weeks.
Rationale A few years ago, I designed a programming language: ooc. Even though I've done my fair share of Java, C, Ruby, JavaScript, and even some Perl, Scala, Python, PHP, etc., I still find myself going back to ooc because it gives me access to C libs, relatively high-level constructs, and it forces me to write code that's not too smart.In the mind of a suicidal personhttps://fasterthanli.me/blog/2012/suicide/
Fri, 06 Apr 2012 00:00:00 +0000https://fasterthanli.me/blog/2012/suicide/Switzerland In a recent popular vote, the Swiss people decided against passing from four mandatory weeks of holidays per year to six. Everywhere in the world, the white cross on a red background is a symbol of hard work, reliability, efficiency, and professionalism.
Switzerland has been successfully flipping off its neighbors, by sticking with its own currency and only contracting bilateral agreements when it wants to, on its own terms. A luxury that not many countries on this side of the ocean can afford.