Arscoin

"Make a cryptocurrency," they said. "All the cool kids are doing it," they said. And so we did. And for most of launch day, it fell on its ass and flailed around like a dying giraffe.

Cyrus Farivar's piece on Arscoin tells you how the currency was created, and Andrew Cunningham's piece on mining tells you how to get them. But while Cyrus and Andrew and other staffers were busy in early January ferreting out the means to create Arscoin, tech wizard Lee Aylward and I had our own task: we had to figure out what kind of infrastructure we needed to put behind our funny money. Getting it working proved to be frustrating and occasionally hilarious—and, as is so typical with the Internet, once launch day hit all the planning and preparation went flying out the window.

Now that we're on our third day of public Arscoin mining, things seem to be mostly stable. But what exactly goes into the backend of a scrypt-based mining operation? How do pools look from the administrative side? What kind of hamster-powered wreck of a server did we choose to host this on? Why didn't we plan better, and what kind of magic did we throw at this mess to make it work right? And that stuff we did to fix the pool—why didn't we just do that in the first place?

I can't tell you anything about how the other guys admin their pools, but I can tell you how we did it: frantically, one explosion at a time.

Genesis

The BrodkinBot, hard at work.

When Cyrus initially proposed the currency, the biggest problem we had was in deciding what to name it. At first, it seemed obvious that we'd go with some kind of play on Jon Brodkin's name, due to a long-running in-joke in the staff IRC channel (we even have an IRC bot in the channel that will spit out ludicrous variations on "Jon Brodkin" on command).

However, "Jon BrodCoin" wasn't meant to be. We quickly realized that a coin based on a silly in-joke wouldn't have nearly enough name cachet (sorry, Jek Borkins), and we needed to tie in the Ars brand. So, Jon BrodCoin became Arscoin.

Once we had our currency starter kit in hand, we all began running the QT-based client and solo mining Arscoins. It became obvious that pool mining would be the way to go once we opened this up to the public, because it was going to be the only way to actually let everyone join in the fun and make some Arscoins. Without a pool, the crazies with 500 GPUs would dominate the solo mining and snap up all the coins.

Cyrus did a bit of digging into his endless virtual rolodex and came up with contact info of some smart Bitcoin & Litecoin pool admins to offer us assistance, including Ahmed Bodiwala. Bodiwala suggested that in order to set up pool mining, we should download his fork of the stratum-mining protocol, along with the PHP-based MPOS application to actually administer the pool.

And so, we did.

Bliss

Lee Aylward requisitioned us an Amazon EC2 Micro instance to test with, and we set about installing the stratum and MPOS prerequisites. Aylward worked on setting up DNS entries and acquiring SSL/TLS certificates for "coins.arstechnica.com," while I beat the installation into shape. The Ubuntu-imaged micro instance got Nginx and PHP-FPM to serve out MPOS, as well as MySQL to hold data for stratum, and MPOS and memcached to make stuff go fast. We fed it an updated version of Python and stuffed it full of all the necessary libraries to make everything work right.

Underneath the covers, the server also runs the same "arscoind" protocol daemon (adapted from litecoind and bitcoind) that we'd previously been running on our desktops and laptops while we were solo mining. Aylward set up a second instance of the protocol daemon on a spare Ars Web server in order to host the Arscoin Hat Store, and he wrote a bit of code to auto-generate online wallets for Ars users.

We went "live" with the Arscoin pool on the evening of January 21—that's when stratum connections started working. All the Ars staffers who were solo mining began transferring their growing hoards of Arscoins out of their locally hosted wallets and deleting their local copies of the blockchain; after a few days, everyone had shifted to pool-based mining, and the blockchain existed only on the EC2 micro pool server and the store Web server.

Running ten instances at once, you can get maybe 1.5M hashes per second total.

For weeks, things ran perfectly. We joyfully threw as much CPU and GPU power as we could at the pool, getting nearly a hundred workers across dozens of computers and video cards running. We spun up further dozens of Amazon GPU instances to hammer the pool (and also to see what kind of hash rates we could get out of them—turns out, because they're Nvidia-based, running ten instances at once you can get maybe 1.5M hashes per second total, and it cost about $7 per hour to run all ten). Ken Fisher quickly dominated all of us with his multiple AMD-based GPUs.

While we banged on the hardware, Cyrus and Andrew and others contributed to getting the feature articles running. Lee Aylward threw together a store and added hats for readers to buy; we'd tested the hats out without revealing their purpose, and they drew immediate comments, so we figured interest would be high. Finally, we had to await the blessing of Condé Nast's legal team, who wanted to make sure we weren't going to cause an international tax law crisis with our fancy new cryptocurrency.

The evening before we were scheduled to go live, Aylward took our trusty Micro instance—which had borne up without complaint under the testing load—and upgraded it to a Medium EC2 instance. This added plenty of RAM and some additional CPU, and we felt ready to face the inevitable Arscoin rush.

Collapse

At 0700 CT on March 5, our trio of Arscoin stories went live. For the first hour or so, things held together fairly well. We watched as the CPU load on coins.arstechnica.com began to spike; the load average quickly climbed up and settled in between 5-6.

This is what it looks like when servers cry.

At first, the chief offender was mysqld, the MySQL daemon. Amazon EC2 instances aren't the greatest database performers, and so Aylward quickly spun us up an EC2 RDS instance and migrated the MySQL database over to it. We watched in consternation as the comments on the three Arscoin stories began to fill with more and more complaints about coins.arstechnica.com being unavailable, but we were pretty confident that giving MySQL more breathing room would fix things.

After about ten minutes, Aylward had the migration done; we bounced the relevant services and watched in horror as things got so much worse.

As so often happens with layered application stacks, removing one bottleneck exposed another. Within seconds, the Nginx Web server app began to throw errors, saying that it was unable to communicate with PHP-FPM ("Resource temporarily unavailable," it said, taunting us with the word "temporary"—like, maybe if we were patient enough things would fix themselves, right?). Folks attempting to access coins.arstechnica.com were now seeing 502 "Bad Gateway" errors, as Nginx repeatedly failed to get a good response from the upstream PHP handler.

Rather than extensively troubleshoot, Aylward took a stab in the dark and quickly changed Nginx and PHP-FPM from unix socket-based communication to TCP port communication, thinking that the number of requests might be bumping into the upper limit of one of the Linux kernel's file or IO parameters. The change sort of worked; the PHP-FPM errors stopped, but now instead of HTTP 502 errors, visitors saw HTTP 504 "Gateway Timeout" errors. PHP-FPM had gone totally uncommunicative. Attempting to restart it via its upstart job once yielded success, but the process it said it was spawning didn't actually exist; shortly after that, upstart joined PHP-FPM in non-responsive bye-bye land.

Aylward and I both made the sign of the cross and rebooted the server.

Lee Hutchinson / Lee is the Senior Reviews Editor at Ars and is responsible for the product news and reviews section. He also knows stuff about enterprise storage, security, and manned space flight. Lee is based in Houston, TX.