Behind the effort to archive Nintendo’s disappearing social network

Scraping and saving millions of posts before Nintendo deletes them all on Nov. 7.

On November 7, another piece of Nintendo's short online history will go away forever. That's when Miiverse, the artwork and message sharing service embedded into many Wii U games, will be shutting down because, as Nintendo puts it, "among other reasons, many users are shifting to social networking services." While users can request an archive of their own Miiverse posts and drawings, the millions of memes, bad jokes, incisive game criticism, and more currently hosted on Miiverse will be removed from Nintendo's public servers in a few months (some features of certain Wii U games will also be impacted, as detailed in this FAQ).

That massive purge of player-created data wasn't acceptable to professional Web developer Tim Miller. Shortly after Nintendo announced the Miiverse shutdown last week, Miller started up an effort to permanently archive the network's public contents, complete with a GitHub page and the support of the Web preservationists at Archive Team.

"Any time a social network goes down, we lose a ton of data," Miller told Ars in an IRC chat. "Part of history, our culture, is lost. In Miiverse, especially in the art section, you can see people really investing a lot of their time and energy in it. And being able to save that for others to see and experience is extremely important.

"Just look at the people still making Splatoon art, right now, even though it's going away in a few months," he continued. "It's clear some people out there care about it, and preserving this data would be [a] great thing for all. It's important to remember that, when you give your data to companies like this, they can just as easily throw it away. And with that, so goes years of history."

While Miller says he understands why Nintendo is shutting down Miiverse in the face of other, broader social networks, he's not happy with the way the company is handling all the user-created data it has accumulated in the last few years. "Right now, Nintendo is going 'Remember to click this button, and we'll give you a zip at some point, now it's your problem,'" he told Ars. "To me, that's a giant 'fuck you.' At least it's better than other companies that just close up shop and don't announce it, but they could do better."

The scrape race

Enlarge/ Wonderful dot-art like this is part of what the Miiverse preservation effort is trying to save.

The Miiverse archiving effort is still in its early days. Because Miiverse has no public API, the team is still figuring out how to parse Miiverse URLs to get at all the network's public data through HTML scraping. Once that's done, Miller and other volunteers will set up VM scripts that can crowdsource the downloading and processing effort through Archive Team's Warrior tool.

So far, the team has identified more than 2 million distinct Miiverse users by following the trail of friends lists. While some of those users hide part of their profiles behind privacy settings, the vast majority of the posts themselves are public, Miller says.

It'll be a bit of a race to scrape the millions of published Miiverse posts and drawings from all those users before the November 7 shutdown, but Miller is confident that it's possible with Archive Team's resources. After that, Miller says he wants to organize it all into a database and maybe build a public website so people can search through the Miiverse archive at their leisure.

Further Reading

Around the industry, though, publishers without much care for preservation are dooming to extinction many games that exist solely as digital downloads. Earlier this year Nintendo shut down the DSiWare shop on original DS hardware (though most of those games continue to be available through the 3DS line). Last year, Sony shut down PlayStation Mobile, cutting off access to plenty of great Vita titles from smaller indie publishers.

Saving all of those titles (or even a worthwhile subset) will be a gargantuan job for current and future gaming historians. For now, though, it's nice to see a coordinated effort to save the public history of Nintendo's first social network.

Kyle Orland
Kyle is the Senior Gaming Editor at Ars Technica, specializing in video game hardware and software. He has journalism and computer science degrees from University of Maryland. He is based in the Washington, DC area. Emailkyle.orland@arstechnica.com//Twitter@KyleOrl

24 Reader Comments

Nintendo's aggressiveness to closure and failure in the online space is why I'm extremely cautious about buying from their digital storefronts despite loving services like Steam. I think I've only purchased Pokémon Blue from the 3DS store, because it's an experience I don't care about. I've had it before, it was just fun to have. Anything else though, I still get the Game Card version, because I know Nintendo is eventually going to shutter online services with little notice or care for customers' purchases. I have a LivingDex in Pokémon Sun (meaning I've not just completed the Pokédex, I have at least one of every numbered species at all times) and if I had the digital version of Sun and something happened to my 3DS... Bye bye everything, and any other games. The Game Card version is not affected by what happens to my 3DS outside of theft.

The software was hand-drawing/animation sharing tool, but it allowed comments. People ended up using the comments to have many long discussions. It was very popular. One cool thing about it was that all the comments were in the person's handwriting! (no keyboard, virtual or otherwise).

Nintendo didn't know how to monetize it, so they used the excuse of coming out with the 3DX to shut the whole thing down to the grief of millions of kids. They later came out with a 3D version of Flipnote, but no one trusted them again after that.

Note, per my previous comment on Flipnote Hatena, do a search for Flipnote Hatena in Youtube to see some of the animations kids created with the program that came free with every DSi. This doesn't show the social aspect of it, which was quite extensive (both my kids were involved in it--their first social media experience).

Can I just say a huge thank you to all the digital historians out there who are keeping this information available? As a side note, maybe ARS can do a feature on how somebody would go about performing this work. I would definitely be interested!

On the one hand, the idea that an online service and all of its content is going away forever, not just archived and de-activated but deleted is sad and a little disturbing. This is one reason why always-online gaming and required "official" servers (no ability to create your own dedicated server) really sucks. Some games, even single player games, will become unplayable ever again if their publisher decides to kill the servers unless someone hacks it or the publisher decides to patch the official and always online parts of their DRM out of the game (fat chance of that...).

On the other hand, I am sure that it is very expensive for a company to maintain servers for an online service that few people are using. It is unreasonable to expect that every single online service that ever has been or ever will be created is going to continue to exist for the rest of eternity.

Someone has to shoulder the cost for hosting the content if we as a society deem the content worth preserving. I think that if organizations like Archive.org want to host the data, then more power to them and I wish Nintendo would work with them to make it easier, instead of having to rely on this hack-together bubblegum and duct tape solution of HTML scraping.

Can I just say a huge thank you to all the digital historians out there who are keeping this information available? As a side note, maybe ARS can do a feature on how somebody would go about performing this work. I would definitely be interested!

I second that: I'd be very interested in a technical article about how to archive/preserve information like this.

I remember when they shutdown their online forums YEARS ago! I remember the rankists, people who tried to rank up by creating and replying to posts. I tried to do this too, but not even a year into it they shutdown their forums. lol

While I do think that preserving Miiverse is a good thing, I do wonder if it exists in a legal gray area. Has that ever been explored? Are there terms or service or whatever that would make this sort of thing technically illegal?

Nintendo's aggressiveness to closure and failure in the online space is why I'm extremely cautious about buying from their digital storefronts despite loving services like Steam. I think I've only purchased Pokémon Blue from the 3DS store, because it's an experience I don't care about. I've had it before, it was just fun to have. Anything else though, I still get the Game Card version, because I know Nintendo is eventually going to shutter online services with little notice or care for customers' purchases. I have a LivingDex in Pokémon Sun (meaning I've not just completed the Pokédex, I have at least one of every numbered species at all times) and if I had the digital version of Sun and something happened to my 3DS... Bye bye everything, and any other games. The Game Card version is not affected by what happens to my 3DS outside of theft.

Unfortunately, on the Switch you can get game cards for your games but you'd lose the save file with your Switch's death. All save games (regardless of download vs physical) are saved to the system, so they're basically the same as playing games off of a (faster) optical disc. And Nintendo hasn't yet provided a way to back up and/or sync our save games.

This is something I've been pondering quite a bit lately, in a more general sense.

So much of what is created these days is purely digital ... What kind of artifacts are we going to leave for the archaeologists?

Most written text from antiquity has disappeared as well. Parchment and papyrus degrade, so needed to be regularly copied to ensure the texts' survival. What survived from antiquity is what a small group decided was worthy of preservation.

We have Vitruvius' De Architectura in its entirety because Charlemagne's court deemed it worthy of preservation. A significant subset (maybe a third) of the works of Galen survive, because these were considered vital medical texts. Meanwhile, the works of Sappho, revered by the ancients, are almost entirely lost, because medieval monks likely decided they were unworthy of preservation.

Just like in ancient times, we need to work to preserve our accumulated knowledge. If something isn't copied, it will likely eventually be lost. That is why organizations such as the Internet Archive are so important.

I'm getting increasingly bemused by Apple's approach here. Presumably the release date for iOS 11 will be announced at the iPhone event next week and it will be out before the end of the month. Yet I've just checked the App Store and they're still selling games like Baldur's Gate I & II, Icewind Dale, XCOM and the Cave shmups which are 32 bit apps and will cease working without even a warning about impending obsolescence.

Either they've got cast iron assurances that apps which haven't been updated in 2 or 3 years will be patched in the next few weeks (which might be possible for the Infinity Engine ports as Overhaul Games did release the port of Planescape as a 64 bit app - I'm very sceptical about XCOM and the Cave games) or they just don't care that it's a class action suit waiting to happen.

Why do people even want every forum post they ever made preserved for all eternity? If I want something saved, I save it myself in multiple places rather than leaving just a single copy on some random forum somewhere.

All this grandiose talk about preserving our accumulated knowledge and culture makes me laugh. There's barely any knowledge worth preserving long term on most forums. Most of the actual useful information on internet topics is only relevant for a few years or even less. As for the art and culture arguments, if you just save every hideous MS Paint meme without discrimination the volume is simply too much for anyone to get any value out of it. Anything really worthwhile should be saved in other places than on a forum.

I don't personally care much about the Miiverse or it's contents myself but I salute Tim Miller's, Archive Team's et. al. efforts and the idea itself.So much so that I'm now running a AT Warrior as I have the internet connection, a "server" I run 24/7, will to do it and so on.