Catching the Blockchain Trainhttp://decentralized.blog/2017-09-14T16:44:00+02:00Ten terrible attempts to make IPFS human-friendly2017-09-14T16:44:00+02:002017-09-14T16:44:00+02:00Mark Porstag:decentralized.blog,2017-09-14:/ten-terrible-attempts-to-make-ipfs-human-friendly.html<p><strong>This article is part 6 of the Blockchain train journal, start reading here: <a href="catching-the-blockchain-train.html">Catching the Blockchain Train</a>.</strong></p>
<h2>IPFS human-friendly naming; is it possible?</h2>
<p>Did you see the traffic-magnet-title I came up with? It is a bit over the top; we're just looking at how the addressing of content can be …</p><p><strong>This article is part 6 of the Blockchain train journal, start reading here: <a href="catching-the-blockchain-train.html">Catching the Blockchain Train</a>.</strong></p>
<h2>IPFS human-friendly naming; is it possible?</h2>
<p>Did you see the traffic-magnet-title I came up with? It is a bit over the top; we're just looking at how the addressing of content can be made a bit easier than <code>ipfs cat /ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV</code>.</p>
<p>As mentioned in the previous post, adding some mutable "label" to point to (immutable) IPFS content is sometimes desired.</p>
<p>An example is the content of this blog: the original address (hash) for the home page <code>/ipfs/QmcPx9ZQboyHw8T7Afe4DbWFcJYocef5Pe4H3u7eK1osnQ/</code> is different from the current (ah no, already expired!) home page <code>/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV/</code>.</p>
<p>Mutable links are essential, but not enough for us humans. They are hard to read, let alone remember.</p>
<p>What we'll do in this episode is to explore different ways to:</p>
<ul>
<li>resolve this problem of adding mutable links, and</li>
<li>to make the links more human-friendly, while</li>
<li>trying to stay decentralized where possible.</li>
</ul>
<p>We'll see the following methods in action:</p>
<ol>
<li>The IPFS gateway</li>
<li>IPNS</li>
<li>DNS TXT IPNS Records</li>
<li>Browser extensions</li>
<li>Namecoin </li>
<li>Blockstack ID</li>
<li>Ethereum Name Service</li>
<li>Proquint Pronounceable Identifiers</li>
<li>Name shorteners</li>
<li>Filecoin</li>
</ol>
<p><strong>TL;DR</strong>
To make IPFS more human-friendly, in the context of a decentralized IPFS powered website, there is currently no other way than sacrificing a part of decentralization using HTTP-to-IPFS gateways and old school DNS.
Naming systems in the blockchain sphere (Blockstack, Namecoin, EthNames) are possible candidates to improve the UX while keeping it 100% decentralized, but at this point of time none of these work out of the box. Filecoin might be the way to go, but that is still far away.</p>
<h2>The IPFS gateway</h2>
<p>The most obvious step to make content available on the old skool internet with a regular browser is to make use of the IPFS gateway. These gateways speak HTTP and access content through their IPFS node.</p>
<p>Any gateway will do, but the gateway where content is pinned is of course faster. Examples of accessing the previous (before this post) home page of this blog:</p>
<ul>
<li><a href="http://decentralized.blog:8080/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV/">http://decentralized.blog:8080/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV/</a></li>
<li><a href="https://gateway.ipfs.io/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV/">https://gateway.ipfs.io/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV/</a></li>
<li><a href="http://localhost:8080/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV/">http://localhost:8080/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV/</a> (if you have a local IPFS node running)</li>
</ul>
<p>So while that works, it has a couple of problems:</p>
<ul>
<li>when the content of the home page changes these URL's still point to old content</li>
<li>the hash is not very human-friendly (a.k.a. ugly)</li>
<li>it introduced an element of centralization (sort of, there is no dependence on a particular single gateway)</li>
</ul>
<h2>IPNS</h2>
<p>To solve the first of these three problems we can, of course, use IPNS. Any time the content changes we publish the latest content hash like:</p>
<div class="highlight"><pre><span></span>ipfs name publish QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV
Published to QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN: /ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV
</pre></div>
<p>Now we can access the home page through any gateway but with the mutable IPNS link: <a href="http://decentralized.blog:8080/ipns/QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN/">http://decentralized.blog:8080/ipns/QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN/</a></p>
<p>Much better, but still ugly...</p>
<p>Note: IPNS is still a bit shaky and forgets published names after about 12 hours. You might want to run a cron job to republish every 8 hours or so.</p>
<h2>DNS TXT IPNS Records</h2>
<p>Now, let's get rid of the hash. The white paper mentions the use of a DNS TXT record. This is what Wikipedia has to say about it:</p>
<blockquote>
<p>A TXT record (short for text record) is a type of resource record in the Domain Name System (DNS) used to provide the ability to associate some arbitrary and unformatted text with a host or other name, such as human readable information about a server, network, data center, and other accounting information.</p>
</blockquote>
<p>So the owner of a domain can associate any data to this domain by adding a TXT record. It is like a key-value store where only the owner of the domain can write, and the world can read.</p>
<p>This is used by IPNS to look up the <code>/ipns/&lt;peerID&gt;</code> path if you insert a valid domain as the path.</p>
<p>This is how IPNS normally resolves:</p>
<div class="highlight"><pre><span></span># Publish
ipfs name publish QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV
Published to QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN: /ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV
# Normally IPFS resolves the peerID hash
$ ipfs name resolve /ipns/QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN
/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV
</pre></div>
<p>To be able to resolve a domain we need to add a <a href="https://github.com/ipfs/go-dnslink">dnslink</a> to the TXT record for that domain.</p>
<p>After doing so for the domain name of this blog (decentralized.blog) pointing to the <code>/ipfs</code> link</p>
<div class="highlight"><pre><span></span>$ dig txt decentralized.blog
.
.
<span class="p">;;</span> ANSWER SECTION:
decentralized.blog. <span class="m">900</span> IN TXT <span class="s2">&quot;dnslink=/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV&quot;</span>
</pre></div>
<p>we can resolve by domain name</p>
<div class="highlight"><pre><span></span>$ ipfs name resolve decentralized.blog
/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV
</pre></div>
<p>which allows us to use a much nicer address:</p>
<p><a href="https://ipfs.io/ipns/decentralized.blog/">/ipns/decentralized.blog</a></p>
<p>This works, but every time I change the content, I have to change the DNS TXT record and wait for propagation. The cool thing is that I can make the DNS TXT record point to a <code>/ipns</code> link, and simply do a <code>ipfs name publish</code> when content has changed:</p>
<div class="highlight"><pre><span></span>$ dig TXT decentralized.blog
.
.
<span class="p">;;</span> ANSWER SECTION:
decentralized.blog. <span class="m">900</span> IN TXT <span class="s2">&quot;dnslink=/ipns/QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN&quot;</span>
<span class="c1"># points to /ipns path now</span>
</pre></div>
<p>Note, I should have done this in the first place, but I was led on the wrong track by this example in the white paper:</p>
<blockquote>
<div class="highlight"><pre><span></span># this DNS TXT record
ipfs.benet.ai. TXT &quot;ipfs=XLF2ipQ4jD3U ...&quot;
# behaves as symlink
ln -s /ipns/XLF2ipQ4jD3U /ipns/fs.benet.ai
</pre></div>
</blockquote>
<p>That is probably a typo, although the ipfs.io domain also points to an immutable <code>/ipfs</code> path for some reason:</p>
<div class="highlight"><pre><span></span>$ dig TXT ipfs.io
.
.
<span class="p">;;</span> ANSWER SECTION:
ipfs.io. <span class="m">60</span> IN TXT <span class="s2">&quot;dnslink=/ipfs/QmPCawMTd7csXKf7QVr2B1QRDZxdPeWxtE4EpkDRYtJWty&quot;</span>
</pre></div>
<p>More about how IPFS resolves paths <a href="https://discuss.ipfs.io/t/how-does-resolution-and-routing-work-with-ipfs/365/3">here</a>.</p>
<p>So now we have a human-friendly URL but at the cost of adding more centralization (the DNS).</p>
<h2>Browser extensions</h2>
<p>I was hoping there was a browser extension that allowed me to browse a web of <code>/ipfs</code> and <code>/ipns</code> paths, but the extensions that do exist are not made for that purpose.</p>
<p>Because it is still interesting, I'll mention it anyway. The most up to date extension is <a href="https://github.com/ipfs/ipfs-companion">ipfs-companion</a>. There are browser extensions available for <a href="https://addons.mozilla.org/en-US/firefox/addon/ipfs-companion/">Firefox</a> and <a href="https://chrome.google.com/webstore/detail/ipfs-companion/nibjojkomfdiaoajekhjakgkdhaomnch/">Chrome</a>.</p>
<p>One of the cool things it does is helping to distribute data by redirecting requests to the public gateway to your local node.</p>
<p>Another exciting project is the <a href="https://github.com/beakerbrowser/beaker">Beaker browser</a> that used to support IPFS, but not anymore. It is a browser for distributed content based on <a href="https://datproject.org/">Dat</a>.</p>
<h2>Namecoin</h2>
<h3>What is Namecoin?</h3>
<p><a href="https://namecoin.org/">Namecoin</a> is a very interesting protocol based on blockchain technology. I won't go into it here in much depth, because I want to start exploring the mother of all blockchains first (i.e., Bitcoin, but you knew that).</p>
<p>I just realized, this is episode 6 of <a href="catching-the-blockchain-train.html">Catching the Blockchain Train</a>, and we haven't seen a single blockchain yet! And worst, we are not going into that here either... Please bear with me, and we will get there eventually.</p>
<p>So, Namecoin. This is what they say about themselves on the website:</p>
<blockquote>
<p>Namecoin is an experimental open-source technology which improves decentralization, security, censorship resistance, privacy, and speed of certain components of the Internet infrastructure such as DNS and identities.</p>
<p>(For the technically minded, Namecoin is a key/value pair registration and transfer system based on the Bitcoin technology.)</p>
<p>Bitcoin frees money – Namecoin frees DNS, identities, and other technologies.</p>
</blockquote>
<p>The part about DNS might be able to help us here. Namecoin facilitates the <code>.bit</code> TLD, which has its own <a href="https://bit.namecoin.org/">portal</a>.</p>
<h3>Getting a .bit domain</h3>
<p>To get started we need to get hold of a .bit name. I rented one a while ago at an exchange, but I now <a href="https://bit.namecoin.org/your_website.html">see</a> that there is a better way: using a Namecoin client. If you go the exchange way, realize that you don't have the private keys to control the domain (change value in the KV pair), so you don't own it.</p>
<p>To get your own .bit domain name, follow the steps at <a href="https://wiki.namecoin.org/index.php?title=Register_and_Configure_.bit_Domains">Register and Configure .bit Domains</a>.</p>
<p>You'll see that you need a bit of namecoin (NMC) to get your .bit domain, but mining is not really an option (it is done alongside Bitcoin mining and hard), but you can also buy some. Here is a list of exchanges where you can buy NMC, <a href="https://coinmarketcap.com/currencies/namecoin/#markets">Namecoin Markets</a>, but I'd recommend <a href="https://shapeshift.io/#/coins">ShapeShift</a> if you already own some bitcoin or altcoin. You need 0.02 NMC which is about <a href="https://coinmarketcap.com/currencies/namecoin/">nothing</a>.
You can use this <a href="https://namecha.in">namecoin explorer</a> to see if your desired name is still available.</p>
<h3>How a .bit lookup works</h3>
<p>.bit domains have a unix-path equivalent like this: <code>d/somedotbitname</code>. These paths are the key of a record stored in the Namecoin's blockchain. So to see what the value is you have to have access to a Namecoin node, and ask for the value of the record with key <code>d/somedotbitname</code>.</p>
<p>That value can be anything, but in practice, it is either a reference to nameservers (standard DNS) or directly to an IP address, or to a <a href="https://en.wikipedia.org/wiki/Bitmessage">Bitmessage</a> address. Others add their email and say the .bit address is for sale.</p>
<p>We need access to a namecoin node to query it, but there are online REST API's (introducing a central point of failure) we can use, like <a href="https://namecoin.webbtc.com/api/name">webbtc</a>:</p>
<div class="highlight"><pre><span></span># Bitmessage address value
$ curl http://namecoin.webbtc.com/name/d/mysite.json
{
&quot;name&quot;: &quot;d/mysite&quot;,
&quot;value&quot;: &quot;BM-2cUyrUNq91XqdPSKvcHSytuED9nnTazf5r&quot;,
&quot;txid&quot;: &quot;78e212c47b41b3ae80413a7064c7bee044f8f2bc27362e9c37d5474b68bfd9e3&quot;,
&quot;address&quot;: &quot;NFE3ED8C3BGRXuzjBRD3YGJV6jrmbS5h3Q&quot;,
&quot;expires_in&quot;: 23292
}
# IP address value
$ curl http://namecoin.webbtc.com/name/d/dappersoftware.json
{
&quot;name&quot;: &quot;d/dappersoftware&quot;,
&quot;value&quot;: &quot;{\&quot;ip\&quot;:\&quot;207.111.216.146\&quot;,\&quot;map\&quot;:{\&quot;*\&quot;:{\&quot;ip\&quot;:\&quot;207.111.216.146\&quot;}}}&quot;,
&quot;txid&quot;: &quot;da72c6b0cf85c94d7b2d4f2f8534d1cdafa47ca94fd00489bf205f00c3fc1cb2&quot;,
&quot;address&quot;: &quot;N2hrunEcP3PNxdKvihMKCoEnvLUcCcBhKw&quot;,
&quot;expires_in&quot;: 4344
}
</pre></div>
<h3>How browsing .bit websites works</h3>
<p>Obviously the normal DNS system doesn't know what to do with a <code>.bit</code> domain, so in a browser that won't resolve to anything. The Namecoin team provides a layer on top op the namecoin node to be able to browse <code>.bit</code> domains: <a href="https://github.com/namecoin/ncdns">ncdns</a>. I'll leave it at that since we are diverging from our goal...</p>
<h3>How do we resolve a .bit domain to an IPFS hash?</h3>
<p>What are we trying to do again? So much exciting stuff is going on that we almost got lost in the shapeshiftbitmessagewebbtcncdnschain. Focus!</p>
<p>The goal of this exploration is to be able to pass around a human-friendly address like <code>dweb.bit</code> to bring up an IPFS-powered website. That would be possible if the IPNS system knew how to read the value of <code>dweb.bit</code> in the Namecoin blockchain and interpret it and resolve it to an IPFS address.</p>
<p>It's just like we saw with the DNS TXT record trick, but now the IPFS node would need to have access to the Namecoin blockchain (or a centralized public API like webbtc we saw before).</p>
<p>So: <code>/ipns/dweb.bit</code> needs to resolve to <code>/ipfs/&lt;some multihash&gt;</code>. The value field of a .bit record can be anything, so that is easy to solve. The other part would be that the IPNS resolver implementation needs to recognize .bit domains and access the Namecoin network somehow. This was not implemented as can be easily shown:</p>
<div class="highlight"><pre><span></span>$ ipfs name resolve dweb.bit
Error: Could not resolve name.
</pre></div>
<p>The IPFS-Namecoin integration is discussed <a href="https://github.com/ipfs/notes/issues/41">here</a>. Personally, I would start by providing a central .bit lookup service on ipfs.io, and extend that later with a Namecoin resolver implementation in the ipfs code. It doesn't look like it has priority now in the IPFS project though.</p>
<p>In any case, I'm ready: <a href="https://namecha.in/name/d/dweb">dweb.bit</a> points to <code>{ "ipns": "/ipns/QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN" }</code>.</p>
<h2>Blockstack ID</h2>
<p>Another contestant for the naming layer that is mentioned in Juan Benet's presentations is Blockstack.</p>
<h3>What is Blockstack?</h3>
<p>This is what Wikipedia has to say about it:</p>
<blockquote>
<p>Blockstack is the first implementation of a decentralized DNS system on top of the Bitcoin blockchain. It combines DNS functionality with public key infrastructure and is primarily meant to be used by new blockchain applications. </p>
</blockquote>
<p>That sounds very similar to what Namecoin does, but it uses the Bitcoin blockchain instead of its own fork. In the past, <a href="http://blog.onename.com/namecoin-to-bitcoin/">it used</a> the Namecoin blockchain, but the idea is that it should be <a href="https://www.youtube.com/watch?v=Itp6eRv1wRQ">blockchain agnostic</a>.</p>
<p>On the homepage of Blockstack a bigger goal is stated:</p>
<blockquote>
<p>A New Internet for Decentralized Apps</p>
<p>Blockstack is a new decentralized internet where users own their data and apps run locally. A browser portal is all that’s needed to get started.</p>
</blockquote>
<p>But for now, let's limit our focus to the decentralized DNS system.</p>
<h3>How to register a Blockstack ID</h3>
<p>The naming system is called Blockstack ID, and the TLD is <code>.id</code>. You can get your own .id at <a href="https://onename.com">onename</a> or use the terminal as described here: <a href="https://blockstack.org/docs">Blockstack CLI</a>.</p>
<p>Let's try that:</p>
<div class="highlight"><pre><span></span># First install blockstack with pip (Python2 only, booh!)
# Create a wallet; not mentioned in the documentation, but required to do first
$ blockstack setup
# Check price for the desired name, the shorter the name, the more expensive
$ blockstack price dweb.id
{
&quot;name_price&quot;: {
&quot;btc&quot;: 0.016,
&quot;satoshis&quot;: 1600000
},
&quot;preorder_tx_fee&quot;: {
&quot;btc&quot;: 0.00219904,
&quot;satoshis&quot;: 219904
},
&quot;register_tx_fee&quot;: {
&quot;btc&quot;: 0.0020583,
&quot;satoshis&quot;: 205830
},
&quot;total_estimated_cost&quot;: {
&quot;btc&quot;: 0.02213352,
&quot;satoshis&quot;: 2213352
},
&quot;total_tx_fees&quot;: 613352,
&quot;update_tx_fee&quot;: {
&quot;btc&quot;: 0.00187618,
&quot;satoshis&quot;: 187618
}
}
# Get a bitcoin address to pay the fee to
$ blockstack deposit
{
&quot;address&quot;: &quot;3BmAHjCLgELuRdg3jM1MMgm5twXPWWxr7s&quot;,
&quot;message&quot;: &quot;Send bitcoins to the address specified.&quot;
}
# Start the API server; also not documented, but the error on register is helpful
$ blockstack api start
{
&quot;status&quot;: true
}
# Now register a name
$ blockstack register dweb.id
Calculating total registration costs for dweb.id...
Registering dweb.id will cost about 0.02213352 BTC.
Use `blockstack price dweb.id` for a cost breakdown
The entire process takes 48 confirmations, or about 5 hours.
You need to have Internet access during this time period, so
this program can send the right transactions at the right
times.
Continue? (y/N): y
{
&quot;message&quot;: &quot;Name queued for registration. The process takes several hours. You can check the status with `blockstack info`.&quot;,
&quot;success&quot;: true,
&quot;transaction_hash&quot;: &quot;c4fc237f4b13d9925f8180fbe9e131cd0a1ee54d9b8c7455c4b2e2464fbe1315&quot;
}
# Check the status
$ blockstack info
{
&quot;cli_version&quot;: &quot;0.14.4.2&quot;,
&quot;consensus_hash&quot;: &quot;47f015a3959a80e7ce9c5fb425833860&quot;,
&quot;last_block_processed&quot;: 482064,
&quot;last_block_seen&quot;: 482070,
&quot;queues&quot;: {
&quot;preorder&quot;: [
{
&quot;confirmations&quot;: 0,
&quot;name&quot;: &quot;dweb.id&quot;,
&quot;tx_hash&quot;: &quot;c4fc237f4b13d9925f8180fbe9e131cd0a1ee54d9b8c7455c4b2e2464fbe1315&quot;
}
]
},
&quot;server_alive&quot;: true,
&quot;server_host&quot;: &quot;node.blockstack.org&quot;,
&quot;server_port&quot;: 6264,
&quot;server_version&quot;: &quot;0.14.4.0&quot;
}
</pre></div>
<p>This is a Bitcoin transaction which can be inspected here <a href="https://live.blockcypher.com/btc/tx/c4fc237f4b13d9925f8180fbe9e131cd0a1ee54d9b8c7455c4b2e2464fbe1315/">c4fc237f4b13d9925f8180fbe9e131cd0a1ee54d9b8c7455c4b2e2464fbe1315</a>.</p>
<p>About 5 hours later the registration was completed:</p>
<div class="highlight"><pre><span></span># Check if we own the dweb.id name now
$ blockstack names
{
&quot;addresses&quot;: [
{
&quot;address&quot;: &quot;3PR8Js7pLt5Bagk5MZHffXoR43EpQrJHxT&quot;,
&quot;names_owned&quot;: [
&quot;pors.id&quot;,
&quot;dweb.id&quot;
]
}
],
&quot;names_owned&quot;: [
&quot;error&quot;
]
}
# Hmm, sort of I guess :)
</pre></div>
<p>All in all, it seems more like an identity thing, but the <code>.id</code> domains can also be used as a DNS record (BNS in this case). Blockstack also introduced namespaces, the bit after the dot (now id), which is obviously expensive.</p>
<h3>How a .id lookup works</h3>
<p>There is a command for lookups in the client:</p>
<div class="highlight"><pre><span></span>$ blockstack lookup timblee.id
<span class="o">{</span>
<span class="s2">&quot;profile&quot;</span>: <span class="o">{</span>
<span class="s2">&quot;@type&quot;</span>: <span class="s2">&quot;Person&quot;</span>,
<span class="s2">&quot;account&quot;</span>: <span class="o">[</span>
<span class="o">{</span>
<span class="s2">&quot;@type&quot;</span>: <span class="s2">&quot;Account&quot;</span>,
<span class="s2">&quot;identifier&quot;</span>: <span class="s2">&quot;timbl&quot;</span>,
<span class="s2">&quot;proofType&quot;</span>: <span class="s2">&quot;http&quot;</span>,
<span class="s2">&quot;proofUrl&quot;</span>: <span class="s2">&quot;https://gist.github.com/timbl/04e8ac7c81cd2dee2f51a5e8c672188d&quot;</span>,
<span class="s2">&quot;service&quot;</span>: <span class="s2">&quot;github&quot;</span>
<span class="o">}</span>,
<span class="o">{</span>
<span class="s2">&quot;@type&quot;</span>: <span class="s2">&quot;Account&quot;</span>,
<span class="s2">&quot;identifier&quot;</span>: <span class="s2">&quot;timberners_lee&quot;</span>,
<span class="s2">&quot;proofType&quot;</span>: <span class="s2">&quot;http&quot;</span>,
<span class="s2">&quot;proofUrl&quot;</span>: <span class="s2">&quot;https://twitter.com/timberners_lee/status/740677355950080001&quot;</span>,
<span class="s2">&quot;service&quot;</span>: <span class="s2">&quot;twitter&quot;</span>
<span class="o">}</span>
<span class="o">]</span>,
<span class="s2">&quot;image&quot;</span>: <span class="o">[</span>
<span class="o">{</span>
<span class="s2">&quot;@type&quot;</span>: <span class="s2">&quot;ImageObject&quot;</span>,
<span class="s2">&quot;contentUrl&quot;</span>: <span class="s2">&quot;https://s3.amazonaws.com/97p/lUU.jpeg&quot;</span>,
<span class="s2">&quot;name&quot;</span>: <span class="s2">&quot;cover&quot;</span>
<span class="o">}</span>
<span class="o">]</span>
<span class="o">}</span>,
<span class="s2">&quot;zonefile&quot;</span>: <span class="s2">&quot;</span><span class="nv">$ORIGIN</span><span class="s2"> timblee.id\n</span><span class="nv">$TTL</span><span class="s2"> 3600\n_http._tcp URI 10 1 \&quot;https://blockstack.s3.amazonaws.com/timblee.id\&quot;\n&quot;</span>
<span class="o">}</span>
</pre></div>
<p>The zonefile bit is the interesting part in our use case; we can store routing information in there.</p>
<p>Also, there is an API (it seems a bit shaky, but it works for some accounts): <a href="https://core.blockstack.org/v2/users/werner">https://core.blockstack.org/v2/users/werner</a>.
For <a href="https://core.blockstack.org/v2/users/pors">pors.id</a> the API doesn't work, but the <a href="https://explorer.blockstack.org/name/pors.id">blockstack explorer</a> does return data for this account.</p>
<p>So how can we use all this?</p>
<h3>How do we resolve a .id domain to an IPFS hash?</h3>
<p>As said, the zone file entry seems to be the best spot to store routing information. This can be done with the CLI client with the <code>blockstack update</code> command. But to what value do we need to set it? That all depends on what the Blockstack and IPFS teams have agreed on.</p>
<p>A bit of Googling and digging in the Blockstack and IPFS forums didn't give me any answers and my hope was set on this duo-presentation <a href="https://www.coindesk.com/events/construct-2017/videos/">Muneeb Ali &amp; Juan Benet: Blockstack IPFS "CTO Briefing" at CONSTRUCT 2017</a>, but not a word about integration there (at least not the addressing bit).</p>
<p>There is some work about the integration between Blockstack and IPFS as a <a href="https://github.com/blockstack/blockstack-core/issues/430">data storage layer</a>. But that's not what we are looking for here.</p>
<p>So just as with Namecoin, there is no way to use Blockstack as the name layer for IPFS.</p>
<p>But again, I want to be ready when they are so I updated my zonefile:</p>
<div class="highlight"><pre><span></span>$ <span class="nb">echo</span> <span class="s1">&#39;{&quot;ipns&quot;:&quot;ipns/QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN&quot;}&#39;</span> &gt; new_zone_file.txt
$ blockstack update dweb.id new_zone_file.txt
<span class="o">{</span>
<span class="s2">&quot;message&quot;</span>: <span class="s2">&quot;Name queued for update. The process takes ~1 hour. You can check the status with `blockstack info`.&quot;</span>,
<span class="s2">&quot;success&quot;</span>: true,
<span class="s2">&quot;transaction_hash&quot;</span>: <span class="s2">&quot;026c739761ec6ed60119b4d80da2ccf81f274d558f53f377d6862a6020d767b8&quot;</span>,
<span class="s2">&quot;zonefile_hash&quot;</span>: <span class="s2">&quot;4db474dd1dda7502c6152d248b7a24302f6e104a&quot;</span>
<span class="o">}</span>
</pre></div>
<p>See the resulting Blockstackified zone file <a href="https://explorer.blockstack.org/name/dweb.id">here</a>.</p>
<p>I will revisit Blockstack in depth in a future post, as soon as we got a better understanding of blockchain technology in general.</p>
<h2>Ethereum Name Service</h2>
<p>We start to see a trend here: very cool blockchain powered name systems, but no support in IPFS. Let's see how ENS fares.</p>
<h3>What is ENS?</h3>
<p>From the documentation:</p>
<blockquote>
<p>ENS is the Ethereum Name Service, a distributed, open, and extensible naming system based on the Ethereum blockchain.</p>
<p>ENS can be used to resolve a wide variety of resources. The initial standard for ENS defines resolution for Ethereum addresses, but the system is extensible by design, allowing more resource types to be resolved in future without the core components of ENS requiring upgrades.</p>
</blockquote>
<p>We haven't looked at <a href="https://www.ethereum.org/">Ethereum</a> yet, but that will, of course, be the case in an upcoming post (multiple posts most likely). It is next to Bitcoin the most important blockchain around.</p>
<p>In short: Ethereum is building a decentralized virtual machine where you can run applications. From the Ethereum website:</p>
<blockquote>
<p>Ethereum is a decentralized platform that runs smart contracts: applications that run exactly as programmed without any possibility of downtime, censorship, fraud or third party interference.</p>
<p>These apps run on a custom built blockchain, an enormously powerful shared global infrastructure that can move value around and represent the ownership of property. This enables developers to create markets, store registries of debts or promises, move funds in accordance with instructions given long in the past (like a will or a futures contract) and many other things that have not been invented yet, all without a middle man or counterparty risk.</p>
</blockquote>
<p>As said, more on that later, for now we just look at the naming system.</p>
<p>To understand the ENS basics, I recommend to:</p>
<ul>
<li>read <a href="http://docs.ens.domains/en/latest/introduction.html">ENS documentation introduction</a> and</li>
<li>watch <a href="https://www.youtube.com/watch?v=pLDDbCZXvTE">ENS Ethereum Domain Name System</a>, slides <a href="https://arachnid.github.io/devcon2/#/title">here</a>.</li>
</ul>
<p>The currently available TLD is <code>.eth</code>. Let's try to get one...</p>
<h3>How to register a .eth name</h3>
<p>The registration for a .eth name is set up as an auction. The details are not really relevant, but you can read more about it here: <a href="https://registrar.ens.domains/">registrar.ens.domains</a>.</p>
<p>This is also the officail app, but it didn't work for me for some reason. This one works much nicer: <a href="https://www.myetherwallet.com/#ens">myetherwallet</a>. It operates in combination with an Ethereum browser or this browser extension: <a href="https://metamask.io/">metamask</a>.</p>
<p>You need some Ether (the Ethereum currency) and a bit patience, and you will be the owner of a <code>.eth</code> address.</p>
<p>Progress can be followed here <a href="https://etherscan.io/enslookup?q=yourname.eth">etherscan.io</a>.</p>
<h3>How a .eth lookup works</h3>
<p>There is a <a href="https://github.com/ethereum/ens/blob/master/ensutils.js">javascript library</a> that allows you to hook into the ENS system. This code needs to run on the Ethereum virtual machine. Currently, there seems no other way to interact with it.</p>
<h3>How do we resolve a .eth domain to an IPFS hash?</h3>
<p>Well, we don't. Although IPFS is mentioned once in the ENS documentation (in the intro) there is no code that makes integration possible. </p>
<h2>Intermission: now what?</h2>
<p>As we have seen so far, there is no (easy) way to use blockchain-based names to address IPFS content. We want to be able to pass around <code>.eth</code>, <code>.id</code>, and <code>.bit</code> addresses and know it will resolve in a trusted way to the correct piece of content. I.e., the content that the owner of such an address wants it to point to.</p>
<p>One way forward that will certainly happen for Ethereum and Blockstack is that IPFS will be used as one of the supported file systems. In that case, the user lives in the Ethereum or Blockstack world and will be able to use <code>.eth</code> or <code>.id</code> addresses.</p>
<p>The alternative, where IPFS itself resolves these addresses is harder because IPFS nodes need trusted access to nodes of the other network. For example, this <a href="https://github.com/llopv/go-ipfs/commit/c206f41072f405b09c05b7a0ed929218e1c15c55">proof of concept</a> relies on having access to a Namecoin node on the same machine where the IPFS node runs. This is obviously not feasible (who wants to run a node for every naming system that comes out next to its IPFS node?).</p>
<p>A proposal to solve this that I like is <a href="https://github.com/ipfs/go-ipfs/issues/3942">pluggable ipns resolvers</a>. I have added my own 2 cents to it, let's see where it leads. But till there is a solution, we still have a couple of contestants, starting with...</p>
<h2>Proquint Pronounceable Identifiers</h2>
<p>Another round another chance!</p>
<p>From the white paper:</p>
<blockquote>
<p>There have always been schemes to encode binary into pronounceable words. IPNS supports Proquint. Thus:</p>
<div class="highlight"><pre><span></span># this proquint phrase
/ipns/dahih-dolij-sozuk-vosah-luvar-fuluh
# will resolve to corresponding
/ipns/KhAwNprxYVxKqpDZ
</pre></div>
</blockquote>
<p>I'm not sure if this is an improvement, especially because the definition of "pronounceable" is stretched a bit here.</p>
<p>As an aside, this remembers me of my first open source contribution <code>Sort-of-pronounceable password generator</code> back in 2000. There was no github yet, so the source code is gone, but someone put it in a <a href="https://pastebin.com/HaDBpB8Z">pastebin</a> 11 years later (I found it when I Googled for my own name, ahem). To make sure this <a href="http://localhost:8080/ipfs/QmZBN1F7r6FFPndxRxvYCvLS3cHWzdo74crmzEykjj6wgB">historic piece of code</a> never gets lost I added it to IPFS.</p>
<p>Back to Proquint! This is actually implemented in IPFS. So let's give it a try:</p>
<div class="highlight"><pre><span></span>$ ipfs name resolve -r /ipns/dahih-dolij-sozuk-vosah-luvar-fuluh
EU�o�C.��M�
<span class="c1"># yuk</span>
</pre></div>
<p>So yeah that works, but <code>/ipns/KhAwNprxYVxKqpDZ</code> is not something valid.</p>
<p>You can replay it with these online tools:</p>
<ul>
<li><a href="http://l-ra.github.io/proquint/js/proquint.html">Proquint online tool</a></li>
<li><a href="http://lenschulwitz.com/base58">Base58 encoder/decoder online</a></li>
</ul>
<p>To encode a valid IPNS peerId is a bit more work to figure out. I'm not interested in that now and also lazy, so I'll leave that to anyone who likes to <a href="https://github.com/Bren2010/proquint/blob/master/proquint.go">dive into that</a> :)</p>
<p>Another solution mentioned in the white paper is...</p>
<h2>Name shorteners</h2>
<p>From the white paper:</p>
<blockquote>
<p>Services are bound to spring up that will provide name shortening as a service, offering up their namespaces to users. This is similar to what we see today with DNS and Web URLs:</p>
<div class="highlight"><pre><span></span># User can get a link from
/ipns/shorten.er/foobar
# To her own namespace
/ipns/XLF2ipQ4jD3UdeX5xp1KBgeHRhemUtaA8Vm
</pre></div>
</blockquote>
<p>So this is based on the ipns feature it can resolve domains via a DNS TXT record.</p>
<p>Resolving goes like this:</p>
<ul>
<li>A request for <code>/ipns/shorten.er/foobar</code> is resolved to <code>/ipns/&lt;hash-of-shortener-service-provider&gt;/foobar</code>.</li>
<li>This file <code>/ipns/&lt;hash-of-shortener-service-provider&gt;/foobar</code> <em>points</em> to the peerID address of <code>foobar</code>, like: <code>/ipns/XLF2ipQ4jD3UdeX5xp1KBgeHRhemUtaA8Vm</code></li>
</ul>
<p>Now, this pointing is not as easy as it sounds. This was probably the intended use of what is described in the white paper as Peer Links:</p>
<blockquote>
<p>As encouraged by SFS, users can link other users' Objects directly into their own Objects (namespace, home, etc). This has the benefit of also creating a web of trust (and supports the old Certificate Authority model):</p>
<div class="highlight"><pre><span></span># Alice links to bob Bob
ipfs link /&lt;alice-pk-hash&gt;/friends/bob /&lt;bob-pk-hash&gt;
# Eve links to Alice
ipfs link /&lt;eve-pk-hash/friends/alice /&lt;alice-pk-hash&gt;
# Eve also has access to Bob
/&lt;eve-pk-hash/friends/alice/friends/bob
# access Verisign certified domains
/&lt;verisign-pk-hash&gt;/foo.com
</pre></div>
</blockquote>
<p>Unfortunately, the <code>ipfs link</code> command is not implemented yet which probably explains why no-one implemented an IPFS name shortener.</p>
<p>There is a way to do something similar to peer links with <code>ipfs files</code>, but it is not suitable to build a name resolver. See here for my search for a solution: <a href="https://discuss.ipfs.io/t/are-peer-links-supported/1035">Are Peer Links supported?</a>.</p>
<p>Again no success here, but there might be hope for the future...</p>
<h2>Filecoin</h2>
<p>As mentioned in a previous post, Filecoin will be the cryptocurrency to help incentivize the file storage marketplace. This means that there will be a blockchain available in the IPFS network. The primary function of this blockchain is to pay and get paid for storage, but since it's there, it can also be used for other functionality. Like identity or a name system. </p>
<p>In other words, just like Namecoin, Blockstack and ENS provide blockchain enabled identity/naming, Filecoin can do the same. We'll have to see how this all plays out in the next months (years?) but this seems like an attractive option since the IPFS nodes and the Filecoin nodes will most likely be one and the same or at least tightly integrated.</p>
<p>The <a href="https://filecoin.io/filecoin.pdf">Filecoin white paper</a> doesn't mention anything about naming, but it is still very early to say anything definite about it.</p>
<p>This is an fascinating introduction <a href="https://www.youtube.com/watch?v=e02czCnCuCM">Filecoin | White Paper Breakdown and Token Sale Analysis</a> although he gets into a hate speech for a couple of minutes about the ICO (which I think is not completely undeserved).</p>
<p>OK, more than enough about friendly naming I'd say! Time to move on to the next step in our journey to decentralizing this blog (more soon in the next episode...).</p>Understanding the IPFS White Paper part 22017-08-21T17:44:00+02:002017-08-21T17:44:00+02:00Mark Porstag:decentralized.blog,2017-08-21:/understanding-the-ipfs-white-paper-part-2.html<p><strong>This article is part 5 of the Blockchain train journal, start reading here: <a href="catching-the-blockchain-train.html">Catching the Blockchain Train</a>.</strong></p>
<h2>The IPFS White Paper: IPFS Design</h2>
<p>The IPFS stack is visualized as follows:</p>
<p><img src="img/ipfs-stack.jpg" width="100%" /></p>
<p>or with more detail:</p>
<p><img src="img/thin-waist.jpg" width="100%" /></p>
<p><em>I borrowed both images from presentations by Juan Benet (the BDFL of IPFS).</em></p>
<p>The IPFS design …</p><p><strong>This article is part 5 of the Blockchain train journal, start reading here: <a href="catching-the-blockchain-train.html">Catching the Blockchain Train</a>.</strong></p>
<h2>The IPFS White Paper: IPFS Design</h2>
<p>The IPFS stack is visualized as follows:</p>
<p><img src="img/ipfs-stack.jpg" width="100%" /></p>
<p>or with more detail:</p>
<p><img src="img/thin-waist.jpg" width="100%" /></p>
<p><em>I borrowed both images from presentations by Juan Benet (the BDFL of IPFS).</em></p>
<p>The IPFS design in the white paper goes more or less through these layers, bottom-up:</p>
<blockquote>
<p>The IPFS Protocol is divided into a stack of sub-protocols responsible for different functionality:</p>
<ol>
<li>Identities - manage node identity generation and verification.</li>
<li>Network - manages connections to other peers, uses various underlying network protocols. Configurable.</li>
<li>Routing - maintains information to locate specific peers and objects. Responds to both local and remote queries. Defaults to a DHT, but is swappable.</li>
<li>Exchange - a novel block exchange protocol (BitSwap) that governs efficient block distribution. Modelled as a market, weakly incentivizes data replication. Trade Strategies swappable.</li>
<li>Objects - a Merkle DAG of content-addressed immutable objects with links. Used to represent arbitrary data structures, e.g. file hierarchies and communication systems.</li>
<li>Files - versioned file system hierarchy inspired by Git.</li>
<li>Naming - A self-certifying mutable name system.</li>
</ol>
</blockquote>
<p>Here's my alternative naming of these sub-protocols:</p>
<ol>
<li>Identities: name those nodes</li>
<li>Network: talk to other clients</li>
<li>Routing: announce and find stuff</li>
<li>Exchange: give and take</li>
<li>Objects: organize the data</li>
<li>Files: uh?</li>
<li>Naming: adding mutability</li>
</ol>
<p>Let's go through them and see if we can increase our understanding of IPFS a bit!</p>
<h2>Identities: name those nodes</h2>
<p>IPFS is a P2P network of clients; there is no central server. These clients are the nodes of the network and need a way to be identified by the other nodes. If you just number the nodes 1,2,3,... anyone can add a node with an existing ID and claim to be that node. To prevent that some cryptography is needed. IPFS does it like this:</p>
<ul>
<li>generate a <a href="https://en.wikipedia.org/wiki/Public_key_infrastructure">PKI</a> key pair (public + private key)</li>
<li>hash the public key</li>
<li>the resulting hash is the NodeId</li>
</ul>
<p>All this is done during the <code>init</code> phase of a node: <code>ipfs init</code> &gt; the resulting keys are stored in <code>~/.ipfs/config</code> and returns the NodeId.</p>
<p>When two nodes start communicating the following happens:</p>
<ul>
<li>exchange public keys</li>
<li>check if: <code>hash(other.PublicKey) == other.NodeId</code></li>
<li>if so, we have identified the other node and can e.g. request for data objects</li>
<li>if not, we disconnect from the "fake" node</li>
</ul>
<p>The actual hashing algorithm is not specified in the white paper, read the note about that here:</p>
<blockquote>
<p>Rather than locking the system to a particular set of function choices, IPFS favors self-describing values. Hash digest values are stored in multihash format, which includes a short header specifying the hash function used, and the digest length in bytes. </p>
<p>Example:</p>
<p>\<function code>\<digest length>\<digest bytes></p>
<p>This allows the system to (a) choose the best function for the use case (e.g. stronger security vs faster performance), and (b) evolve as function choices change. Self-describing values allow using different parameter choices compatibly.</p>
</blockquote>
<p>These multihashes are part of a whole family of self-describing hashes, and it is brilliant, check it out: <a href="https://github.com/multiformats/multiformats">multiformats</a>.</p>
<h2>Network: talk to other clients</h2>
<p>The summary is this: IPFS works on top of any network (see the image above).</p>
<p>Interesting here is the network addressing to connect to a peer. IPFS uses <a href="https://github.com/multiformats/multiaddr">multiaddr</a> formatting for that. You can see it in action when starting a node:</p>
<blockquote>
<p>Swarm listening on /ip4/127.0.0.1/tcp/4001</p>
<p>Swarm listening on /ip4/172.17.0.1/tcp/4001</p>
<p>Swarm listening on /ip4/185.24.123.123/tcp/4001</p>
<p>Swarm listening on /ip6/2a02:1234:9:0:21a:4aff:fed4:da32/tcp/4001</p>
<p>Swarm listening on /ip6/::1/tcp/4001</p>
<p>API server listening on /ip4/127.0.0.1/tcp/5001</p>
<p>Gateway (read-only) server listening on /ip4/0.0.0.0/tcp/8080</p>
</blockquote>
<h2>Routing: announce and find stuff</h2>
<p>The routing layer is based on a DHT, as discussed in the <a href="understanding-the-ipfs-white-paper-part-1.html">previous episode</a>, and its purpose is to:</p>
<ul>
<li>announce that this node has some data (a <code>block</code> as discussed in the next chapter), or</li>
<li>find which nodes have some specific data (by referring to the multihash of a block), and</li>
<li>if the data is small enough (=&lt; 1KB) the DHT stores the data as its value.</li>
</ul>
<p>The command line interface and API don't expose the complete routing interface as specified in the white paper. What does work:</p>
<div class="highlight"><pre><span></span># tell the DHT we have this specific content:
$ ipfs dht provide QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG
# ask for peers who have the content:
$ ipfs dht findprovs QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG
QmYebHWdWStasXWZQiXuFacckKC33HTbicXPkdSi5Yfpz6
QmczCvhy6unZEVC5ukR3BC3BPxYie4jBeRApTUKq97ZnEo
QmPM3WzZ3q1SR3pzXuGPHD7e6z3eWEWCnAvrNw7Wegyc8o
QmPKSqtkDmYsSxQbqNkZG1AmVnZSFyv5WF7fFE2YebNBFG
QmPMJ3HLLLBtW53R2YixhVHmVb3pkR2bHni3qbqq23vBSv
QmPNHJJphV1TB6Z99L6r9Y7bKaCBUQ67X17hicnEmsgWDJ
QmPNhiqGg81o2Perk2i7VNvvVuuLLUMKDxMNwVauP8r5Yv
QmPQJRgP3Vxi52Ho7HfnYdiCRJTRM1TXwgEnyjcwcLuKfb
QmNNxr1ZoyPbwNe2CvYz1CVyvSNWsE8WNwDWQ9t9BDjnj5
QmNT744VjtRFpDYB25EVLx7ha1zAVDKsd3qFjxfQLjPEXq
QmNWwGRWTYeut6qvKDhJBuEJZnbqMPMfuF81MPvHvPBX89
QmNZM5NmzZNPkvH2kPXDYNAB1cAeBNfxLyM9B1crgt3VeJ
QmNZRDzSJybdf4rmt972SH4U9TF6sEK8q2NSEJpEt7SkTp
QmNZdBUV9QXytVcPjcYM8i9AG22G2qwjZmh4ZwpJs9KvXi
QmNbSJ9okrwMphfjudiXVeE7QWkJiEe4JHHiKT8L4Pv7z5
QmNdqMkVqLTsJWj7Ja3oKwLNWcAYUkRjSZPg22B7rvKFMr
QmNfyHTzAetJGBFTRkXXHe5om13Qj4LLjd9SDwJ87T6vCK
QmNmrRTP5sJMUkobujpVXzzjpLACBTzf9weND6prUjdstW
QmNkGG9EZrq699KnjbENARLUg3HwRBC7nkojnmYY8joBXL
QmP6CHbxjvu5dxdJLGNmDZATdu3TizkRZ6cD9TUQsn4oxY
# Get all multiaddr&#39;s for a peer
$ ipfs dht findpeer QmYebHWdWStasXWZQiXuFacckKC33HTbicXPkdSi5Yfpz6
/ip4/192.168.1.14/tcp/4001
/ip6/::1/tcp/4001
/ip4/127.0.0.1/tcp/4001
/ip4/1.2.3.4/tcp/37665
</pre></div>
<p><code>ipfs put</code> and <code>ipfs get</code> only work for ipns records in the API. Maybe storing small data on the DHT itself was not implemented (yet)?</p>
<h2>Exchange: give and take</h2>
<p>Data is broken up into <code>blocks</code>, and the exchange layer is responsible for distributing these blocks. It looks like BitTorrent, but it's different, so the protocol warrants its own name: BitSwap.</p>
<p>The main difference is that wherein BitTorrent blocks are traded with peers looking for blocks of the same file (torrent swarm), in BitSwap blocks are traded cross-file. So one big swarm for all IPFS data.</p>
<p>BitSwap is modeled as a marketplace that incentivizes data replication. The way this is implemented is called the BitSwap Strategy, and the white paper describes a feasible strategy and also states that the strategy can be replaced by another strategy. One such a bartering system can be based on a virtual currency, which is where <a href="https://filecoin.io/">FileCoin</a> comes in. </p>
<p>Of course, each node can decide on its own strategy, so the generally used strategy must be resilient against abuse. When most nodes are set up to have some fair way of bartering it will work something like this:</p>
<ul>
<li>when peers connect, they exchange which blocks they have (<code>have_list</code>) and which blocks they are looking for (<code>want_list</code>)</li>
<li>to decide if a node will actually share data, it will apply its <code>BitSwap Strategy</code></li>
<li>this strategy is based on previous data exchanges between these two peers</li>
<li>when peers exchange blocks they keep track of the amount of data they share (builds credit) and the amount of data they receive (builds debt)</li>
<li>this accounting between two peers is kept track of in the <code>BitSwap Ledger</code></li>
<li>if a peer has credit (shared more than received), our node will send the requested block</li>
<li>if a peer has debt, our node will share or not share, depending on a deterministic function where the chance of sharing becomes smaller when the debt is bigger</li>
<li>a data exchange always starts with the exchange of the ledger, if it is not identical our node disconnects</li>
</ul>
<p>So this is set up kind of cool I think: game theory in action! The white paper further describes some edge cases like what to do if I have no blocks to barter with? The answer is simply to collect blocks that your peers are looking for, so you have something to trade. </p>
<p>Now let's have a look how we can poke around in the innards of the BitSwap protocol.</p>
<p>The command-line interface has a section <code>blocks</code> and a section <code>bitswap</code>; those sound relevant :)</p>
<p>To see bitswap in action, I'm going to request a large file <code>Qmdsrpg2oXZTWGjat98VgpFQb5u1Vdw5Gun2rgQ2Xhxa2t﻿</code> which is a video (download it to see what video!):</p>
<div class="highlight"><pre><span></span># ask for the file
$ ipfs get Qmdsrpg2oXZTWGjat98VgpFQb5u1Vdw5Gun2rgQ2Xhxa2t﻿
# in a seperate terminal, after requesting the file, I inspect the &quot;bitswap wantlist&quot;
$ ipfs bitswap wantlist
QmYEqofNsPNQEa7yNx93KgDycmrzbFkr5oc3NMKXMxx5ff
QmUmDEBm9a8MYyqRdb3YQnoqPmqAo4cEWdKQErirFJdSWD
QmY5VJPbsRZzFCTMrFBx2qtZiyyeLhsjBysyfC1fx2gE9S
QmdbzYgyhqUNCNL8xU2HTSKwao1ck2Gmi5U1ygjQuJd92b
QmbZDe5Dcv9mJr8fiqp5aJL2cbyu64tgzwCS2Vy4P3krCL
QmRjzMzVeYRE5b6tDF3sTXMV1sTffno92uL3WwuFavBrWQ
QmPavzEJQw8atvErXQis6C6GF7DRFbb95doAaFkHe9M38u
QmY9fs1Pkr3nV7RkbGdfGh3q8HuKtMMCCUp22AAbwPYnrS
QmUtxZkuJuyydd124Z2cfx6jXMAMpcXZRF96QMAsXc2y6c
QmbYDTJkmLqMm6ojdL6pLP7C8mMVfVPnUxn3yp8HzXDcXf
QmbW9MZ7cwn8svpixosAuC7GQmUXDTZRuxJ8dJp6HyJzCS
QmdCLGWsYQFhi9y3BmkhUreX2S799iWGyJqvnbK9dzB55c
Qmc7EvnBPf2mPCUCfvjcsaQGLEakBbUN9iycnyrLF3b2or
Qmd1mNnDQPf1BAjFqDHjiLe4g4ZFPAheQCniYkbQPosjDE
QmPip8XzQhJFd487WWw7D8aBuGLwXtohciPtUDSnxpvMFR
QmZn5NAPEDtptMb3ybaMEdcVaoxWHs7rKQ4H5UBcyHiqTZ
.
.
.
# find a node where we have debt
$ ipfs dht findprovs Qmdsrpg2oXZTWGjat98VgpFQb5u1Vdw5Gun2rgQ2Xhxa2t
QmSoLMeWqB7YGVLJN3pNLQpmmEk35v6wYtsMGLzSr5QBU3
QmSoLnSGccFuZQJzRadHn95W2CrSFmZuTdDWP8HXaHca9z
QmUh2KnjAvgEbJFSd5JZws4CNvt6LbC4C1sRpBgCbZQiqD
Qmc9pBLfKSwWboKHMvmKx1P7Z738CojuUXkPA1dsPrvSw2
QmZFhGyS2W833nKKkbqZAU2uSvBbWUytDJkKBHimwRmhd6
QmZMxNdpMkewiVZLMRxaNxUeZpDUb34pWjZ1kZvsd16Zic
Qmbut9Ywz9YEDrz8ySBSgWyJk41Uvm2QJPhwDJzJyGFsD6
# try one to see if we have downloaded from that node
$ ipfs bitswap ledger QmSoLMeWqB7YGVLJN3pNLQpmmEk35v6wYtsMGLzSr5QBU3
Ledger for &lt;peer.ID SoLMeW&gt;
Debt ratio: 0.000000
Exchanges: 11
Bytes sent: 0
Bytes received: 2883738
</pre></div>
<p>Thank you <code>QmSoLMeWqB7YGVLJN3pNLQpmmEk35v6wYtsMGLzSr5QBU3</code>; what a generous peer you are!</p>
<p>Now, have a look at the <code>block</code> commands:</p>
<div class="highlight"><pre><span></span># Let&#39;s pick a block from the wantlist above
$ ipfs block stat QmYEqofNsPNQEa7yNx93KgDycmrzbFkr5oc3NMKXMxx5ff
Key: QmYEqofNsPNQEa7yNx93KgDycmrzbFkr5oc3NMKXMxx5ff
Size: 262158
$ ipfs block get QmYEqofNsPNQEa7yNx93KgDycmrzbFkr5oc3NMKXMxx5ff &gt; slice_of_a_movie
# results in a binary file of 262 KB
</pre></div>
<p>We'll have another look at how blocks fit in in the next chapter.</p>
<p>The three layers of the stack we described so far (network, routing, exchange) are implemented in <a href="https://libp2p.io/">libp2p</a>.</p>
<p>Let's climb up the stack to the core of IPFS...</p>
<h2>Objects: organize the data</h2>
<p>Now it gets fascinating. You could summarize IPFS as: Distributed, authenticated, hash-linked data structures. These hash-linked data structures are where the Merkle DAG comes in (remember our previous episode?). </p>
<p>To create any data structure, IPFS offers a flexible and powerful solution:</p>
<ul>
<li>organize the data in a graph, where we call the nodes of the graph <code>objects</code></li>
<li>these objects can contain data (any sort of data, transparent to IPFS) and/or links to other objects</li>
<li>these links - <code>Merkle Links</code> - are simply the cryptographic hash of the target object</li>
</ul>
<p>This way of organizing data has a couple of useful properties (quoting from the white paper):</p>
<blockquote>
<ol>
<li>Content Addressing: all content is uniquely identified by its multihash checksum, including links.</li>
<li>Tamper resistance: all content is verified with its checksum. If data is tampered with or corrupted, IPFS detects it.</li>
<li>Deduplication: all objects that hold the exact same content are equal, and only stored once. This is particularly useful with index objects, such as git trees and commits, or common portions of data.</li>
</ol>
</blockquote>
<p>To get a feel for IPFS objects, check out this <a href="https://ipfs.io/ipfs/QmNZiPk974vDsPmQii3YbrMKfi12KTSNM7XMiYyiea4VYZ/example#/ipfs/QmP8WUPq2braGQ8iZjJ6w9di6mzgoTWyRLayrMRjjDoyGr/graphmd/README.md">objects visualization</a> example.</p>
<p>Another nifty feature is the use of unix-style paths, where a Merkle DAG has the structure:</p>
<p><code>/ipfs/&lt;hash-of-object&gt;/&lt;named-path-to-object</code></p>
<p>We'll see an example below.</p>
<p>This is really all there is to it. Lets see it in action by replaying some examples from the <a href="https://gateway.ipfs.io/ipfs/QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG/quick-start">quick-start</a>:</p>
<div class="highlight"><pre><span></span>$ mkdir foo
$ mkdir foo/bar
$ <span class="nb">echo</span> <span class="s2">&quot;baz&quot;</span> &gt; foo/baz
$ <span class="nb">echo</span> <span class="s2">&quot;baz&quot;</span> &gt; foo/bar/baz
$ tree foo/
foo/
├── bar
│ └── baz
└── baz
$ ipfs add -r foo
added QmWLdkp93sNxGRjnFHPaYg8tCQ35NBY3XPn6KiETd3Z4WR foo/bar/baz
added QmWLdkp93sNxGRjnFHPaYg8tCQ35NBY3XPn6KiETd3Z4WR foo/baz
added QmeBpzHngbHes9hoPjfDCmpNHGztkmZFRX4Yp9ftKcXZDN foo/bar
added QmdcYvbv8FSBfbq1VVSfbjLokVaBYRLKHShpnXu3crd3Gm foo
<span class="c1"># the last hash is the root-node, we can access objects through their path starting at the root, like:</span>
$ ipfs cat /ipfs/QmdcYvbv8FSBfbq1VVSfbjLokVaBYRLKHShpnXu3crd3Gm/bar/baz
baz
<span class="c1"># To inspect an object identified by a hash, we do</span>
$ ipfs object get /ipfs/QmdcYvbv8FSBfbq1VVSfbjLokVaBYRLKHShpnXu3crd3Gm
<span class="o">{</span>
<span class="s2">&quot;Links&quot;</span>:<span class="o">[</span>
<span class="o">{</span>
<span class="s2">&quot;Name&quot;</span>:<span class="s2">&quot;bar&quot;</span>,
<span class="s2">&quot;Hash&quot;</span>:<span class="s2">&quot;QmeBpzHngbHes9hoPjfDCmpNHGztkmZFRX4Yp9ftKcXZDN&quot;</span>,
<span class="s2">&quot;Size&quot;</span>:61
<span class="o">}</span>,
<span class="o">{</span>
<span class="s2">&quot;Name&quot;</span>:<span class="s2">&quot;baz&quot;</span>,
<span class="s2">&quot;Hash&quot;</span>:<span class="s2">&quot;QmWLdkp93sNxGRjnFHPaYg8tCQ35NBY3XPn6KiETd3Z4WR&quot;</span>,
<span class="s2">&quot;Size&quot;</span>:12
<span class="o">}</span>
<span class="o">]</span>,
<span class="s2">&quot;Data&quot;</span>:<span class="s2">&quot;\u0008\u0001&quot;</span>
<span class="o">}</span>
<span class="c1"># The above object has no data (except the mysterious \u0008\u0001) and two links</span>
<span class="c1"># If you&#39;re just interested in the links, use &quot;refs&quot;:</span>
$ ipfs refs QmdcYvbv8FSBfbq1VVSfbjLokVaBYRLKHShpnXu3crd3Gm
QmeBpzHngbHes9hoPjfDCmpNHGztkmZFRX4Yp9ftKcXZDN
QmWLdkp93sNxGRjnFHPaYg8tCQ35NBY3XPn6KiETd3Z4WR
<span class="c1"># Now a leaf object without links</span>
$ ipfs object get /ipfs/QmdcYvbv8FSBfbq1VVSfbjLokVaBYRLKHShpnXu3crd3Gm/bar/baz
<span class="o">{</span>
<span class="s2">&quot;Links&quot;</span>:<span class="o">[</span>
<span class="o">]</span>,
<span class="s2">&quot;Data&quot;</span>:<span class="s2">&quot;\u0008\u0002\u0012\u0004baz\n\u0018\u0004&quot;</span>
<span class="o">}</span>
<span class="c1"># The string &#39;baz&#39; is somewhere in there :)</span>
</pre></div>
<p>The Unicode characters that show up in the data field are the result of serialization of the data. IPFS uses <a href="https://github.com/google/protobuf">protobuf</a> for that I think. Correct me if I'm wrong :)</p>
<p>At the time I'm writing this there is an experimental alternative for the <code>ipfs object</code> commands: <code>ipfs dag</code>:</p>
<div class="highlight"><pre><span></span>$ ipfs dag get QmdcYvbv8FSBfbq1VVSfbjLokVaBYRLKHShpnXu3crd3Gm
<span class="o">{</span>
<span class="s2">&quot;data&quot;</span>:<span class="s2">&quot;CAE=&quot;</span>,
<span class="s2">&quot;links&quot;</span>:<span class="o">[</span>
<span class="o">{</span>
<span class="s2">&quot;Cid&quot;</span>:<span class="o">{</span>
<span class="s2">&quot;/&quot;</span>:<span class="s2">&quot;QmeBpzHngbHes9hoPjfDCmpNHGztkmZFRX4Yp9ftKcXZDN&quot;</span>
<span class="o">}</span>,
<span class="s2">&quot;Name&quot;</span>:<span class="s2">&quot;bar&quot;</span>,
<span class="s2">&quot;Size&quot;</span>:61
<span class="o">}</span>,
<span class="o">{</span>
<span class="s2">&quot;Cid&quot;</span>:<span class="o">{</span>
<span class="s2">&quot;/&quot;</span>:<span class="s2">&quot;QmWLdkp93sNxGRjnFHPaYg8tCQ35NBY3XPn6KiETd3Z4WR&quot;</span>
<span class="o">}</span>,
<span class="s2">&quot;Name&quot;</span>:<span class="s2">&quot;baz&quot;</span>,
<span class="s2">&quot;Size&quot;</span>:12
<span class="o">}</span>
<span class="o">]</span>
<span class="o">}</span>
$ ipfs dag get /ipfs/QmdcYvbv8FSBfbq1VVSfbjLokVaBYRLKHShpnXu3crd3Gm/bar/baz
<span class="o">{</span>
<span class="s2">&quot;data&quot;</span>:<span class="s2">&quot;CAISBGJhegoYBA==&quot;</span>,
<span class="s2">&quot;links&quot;</span>:<span class="o">[</span>
<span class="o">]</span>
<span class="o">}</span>
</pre></div>
<p>We see a couple of differences there, but let's not get into that. Both outputs follow the IPFS object format from the white paper. One interesting bit is the "Cid" that shows up; this refers to the newer <a href="https://github.com/ipld/cid">Content IDentifier</a>.</p>
<p>Another feature that is mentioned is the possibility to <code>pin</code> objects, which results in storage of these objects in the file system of the local node. The current go implementation of ipfs stores it in a <a href="https://github.com/google/leveldb">leveldb</a> database under the <code>~/.ipfs/datastore</code> directory. We have seen pinning in action in a previous post.</p>
<p>The last part of this chapter mentions the availability of object level encryption. This is not implemented yet: <code>status wip</code> (Work in Progress; I had to look it up as well). The project page is here: <a href="https://github.com/ipfs/specs/tree/master/keystore">ipfs keystore proposal</a>.</p>
<p>The <code>ipfs dag</code> command hints to something new...</p>
<h2>Intermission: IPLD</h2>
<p>If you studied the images at the start of this post carefully, you are probably wondering, what is IPLD and how does it fit in? According to the white paper, it doesn't fit in, as it isn't mentioned at all!</p>
<p>My guess is that IPLD is not mentioned because it was introduced later, but it more or less maps to the Objects chapter in the paper. IPLD is broader, more general, than what the white paper specifies. Hey Juan, update the white paper will ya! :-)</p>
<p>If you don't want to wait for the updated white paper, have a look here: the <a href="https://ipld.io/">IPLD website</a> (Inter Planetary Linked Data), the <a href="https://github.com/ipld/specs/tree/master/ipld">IPLD specs</a> and the <a href="https://github.com/ipld/ipld#implementations">IPLD implementations</a>. </p>
<p>And this video is an excellent introduction: <a href="https://www.youtube.com/watch?v=Bqs_LzBjQyk">Juan Benet: Enter the Merkle Forest</a>.</p>
<p>But if you don't feel like reading/watching more: IPLD is more or less the same as what is described in the "Objects" and "Files" chapters here.</p>
<p>Moving on to the next chapter in the white paper...</p>
<h2>Files: uh?</h2>
<p>On top of the Merkle DAG objects IPFS defines a Git-like file system with versioning, with the following elements:</p>
<ul>
<li><code>blob</code>: there is just data in blobs and it represents the concept of a file in IPFS. No links in blobs</li>
<li><code>list</code>: lists are also a representation of an IPFS file, but consisting of multiple blobs and/or lists</li>
<li><code>tree</code>: a collection of blobs, lists and/or trees: acts as a directory</li>
<li><code>commit</code>: a snapshot of the history in a tree (just like a git commit).</li>
</ul>
<p>Now I hear you thinking: aren't these blobs, lists, and trees the same things as what we saw in the Mergle DAG? We had objects there with data, with or without links, and nice Unix-like file paths.</p>
<p>I heard you thinking that because I thought the same thing when I arrived at this chapter. After searching around a bit I started to get the feeling that this layer was discarded and IPLD stops at the "objects" layer, and everything on top of that is open to whatever implementation. If an expert is reading this and thinks I have it all wrong: please let me know, and I'll correct it with the new insight.</p>
<p>Now, what about the <code>commit</code> file type? The title of the white paper is "IPFS - Content Addressed, Versioned, P2P File System", but the versioning hasn't been implemented yet <a href="https://github.com/ipfs/notes/issues/23">it seems</a>.</p>
<p>There is some brainstorming going on about versioning <a href="https://gist.github.com/flyingzumwalt/a6821e843366d606aeb1ba53525b8669">here</a> and <a href="https://github.com/oduwsdl/ipwb/issues/61">here</a>.</p>
<p>That leaves one more layer to go...</p>
<h2>Naming: adding mutability</h2>
<p>Since links in IPFS are content addressable (a cryptographic hash over the content represents the block or object of content), data is immutable by definition. It can only be replaced by another version of the content, and it, therefore, gets a new "address".</p>
<p>The solution is to create "labels" or "pointers" (just like git branches and tags) to immutable content. These labels can be used to represent the latest version of an object (or graph of objects).</p>
<p>In IPFS this pointer can be created using the Self-Certified Filesystems I described in the previous post. It is named IPNS and works like this:</p>
<ul>
<li>The root address of a node is <code>/ipns/&lt;NodeId&gt;</code></li>
<li>The content it points to can be changed by publishing an IPFS object to this address</li>
<li>By publishing, the owner of the node (the person who knows the secret key that was generated with <code>ipfs init</code>) cryptographically signs this "pointer".</li>
<li>This enables other users to verify the authenticity of the object published by the owner.</li>
<li>Just like IPFS paths, IPNS paths also start with a hash, followed by a Unix-like path.</li>
<li>IPNS records are announced and resolved via the DHT.</li>
</ul>
<p>I already showed the actual execution of the <code>ipfs publish</code> command in the post <a href="getting-to-know-ipfs.html">Getting to know IPFS</a>.</p>
<p>This chapter in the white paper also describes some methods to make addresses more human-friendly, but I'll leave that in store for the next episode which will be hands-on again. We gotta get rid of these hashes in the addresses and make it all work nicely in our good old browsers: <a href="ten-terrible-attempts-to-make-ipfs-human-friendly.html">Ten terrible attempts to make IPFS human-friendly</a></p>
<p>Let me know what you think of this post by tweeting to me <a href="https://twitter.com/pors">@pors</a>!</p>Understanding the IPFS White Paper part 12017-08-10T11:11:00+02:002017-08-10T11:11:00+02:00Mark Porstag:decentralized.blog,2017-08-10:/understanding-the-ipfs-white-paper-part-1.html<p><strong>This article is part 4 of the Blockchain train journal, start reading here: <a href="catching-the-blockchain-train.html">Catching the Blockchain Train</a>.</strong></p>
<h2>Oh no, a White Paper!</h2>
<p>The crypto currency/blockchain world loves its white papers, and IPFS is no exception. It started with the famous <a href="https://bitcoin.org/bitcoin.pdf">Bitcoin: A Peer-to-Peer Electronic Cash System</a> white paper by …</p><p><strong>This article is part 4 of the Blockchain train journal, start reading here: <a href="catching-the-blockchain-train.html">Catching the Blockchain Train</a>.</strong></p>
<h2>Oh no, a White Paper!</h2>
<p>The crypto currency/blockchain world loves its white papers, and IPFS is no exception. It started with the famous <a href="https://bitcoin.org/bitcoin.pdf">Bitcoin: A Peer-to-Peer Electronic Cash System</a> white paper by Satoshi Nakamoto (which in turn references to another white paper you don't want to know about). We will have a look at the Bitcoin white paper in a future post when we dive into crypto currencies.</p>
<p>Two things you need to know about white papers:</p>
<ol>
<li>They are distributed in PDF format (because it is harder to change than HTML!)</li>
<li>white papers are hard, scary and contain at least one mathematical formula.</li>
</ol>
<p>The IPFS white paper is here: <a href="https://ipfs.io/ipfs/QmR7GSQM93Cx5eAg6a6yRzNde1FQv7uL6X1o4k7zrJa3LX/ipfs.draft3.pdf">IPFS - Content Addressed, Versioned, P2P File System
(DRAFT 3)</a>. And, as it should, hosted on IPFS.</p>
<p>To understand how IPFS works it's a good idea to walk through this white paper step by step. I'll limit this post to chapter 2 of the white paper, having a look at the underlying technologies:</p>
<ul>
<li>Distributed Hash Tables</li>
<li>Block Exchanges - BitTorrent</li>
<li>Version Control Systems - Git</li>
<li>Self-Certified Filesystems - SFS</li>
</ul>
<p>Mapped on a ISO like stack:</p>
<p><img alt="IPFS is based on these technologies" src="img/ipfs-based-on.png"></p>
<p>OK, here we go!</p>
<h2>Distributed Hash Tables (DHT)</h2>
<h3>What is a DHT?</h3>
<p>A DHT is like a Python dict or Perl hash (if you have the key, you can retrieve the value), but the data is distributed over multiple nodes. The Wikipedia article <a href="https://en.wikipedia.org/wiki/Distributed_hash_table">Distributed hash table</a> gives a good introduction.</p>
<p>In the case of IPFS, the key is a hash over the content. So ask an IPFS node for the content with hash QmcPx9ZQboyHw8T7Afe4DbWFcJYocef5Pe4H3u7eK1osnQ and the IPFS node will lookup in the DHT which nodes have the content. </p>
<p>How the specific value is found efficiently (fast, with a little as possible network requests) and how to manage the DHT so that changes (nodes that enter/leave the network, or new entries to the table) are absorbed easily is different for the various DHT implementations that exist.</p>
<p>One such an implementation is called Pastry, and I like these two videos where the <a href="https://www.youtube.com/watch?v=WqQRQz_XYg4">routing</a> (how to find values) and the <a href="https://www.youtube.com/watch?v=p8iugvHeGcg">dynamics</a> (how to handle node additions/removals) are explained really well.</p>
<h3>Kademlia and friends</h3>
<p>Back to the white paper. Three DHT implementations are mentioned explicitly, starting with Kademlia. Kademlia has a <a href="https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf">white paper</a> of its own but let's not go there...</p>
<p>Kademlia is the DHT protocol that is used in almost all popular P2P systems and again the Wikipedia article on <a href="https://en.wikipedia.org/wiki/Kademlia">Kedemlia</a> is a great introduction.</p>
<p>In short, Kedemlia uses the ID of the nodes to get step by step closer to the node with the desired hash (from the Wikipedia article): </p>
<blockquote>
<p>When searching for some value, the algorithm needs to know the associated key and explores the network in several steps. Each step will find nodes that are closer to the key until the contacted node returns the value or no more closer nodes are found. This is very efficient: Like many other DHTs, Kademlia contacts only O(log(n)) nodes during the search out of a total of n nodes in the system.</p>
</blockquote>
<p>The details can start to get quite complicated and I don't think it won't add to our goal of understanding IPFS, so we move on. If you like this sort of thing: this is a <a href="https://www.youtube.com/watch?v=kXyVqk3EbwE">nice introduction</a> and if you want to dig deeper the link to the Kademlia white paper is ^^^.</p>
<p>IPFS mentions two other DHT implementations that are added to standard Kedemlia:</p>
<ul>
<li>Coral DSHT: this improves the lookup performance and decreases resource use.</li>
<li>S/Kademlia: makes Kademlia more resistant against malicious attacks.</li>
</ul>
<h3>The IPFS DHT in practice</h3>
<p>The DHT is used in IPFS for routing, in other words:</p>
<ol>
<li>to announce added data to the network </li>
<li>and help locate data that is requested by any node.</li>
</ol>
<p>The white paper states:</p>
<blockquote>
<p>Small values (equal to or less than 1KB) are stored directly on the DHT. For values larger, the DHT stores references, which are the NodeIds of peers who can serve the block.</p>
</blockquote>
<p>So let's have a look if we can access the DHT directly and add and retrieve small data blocks. </p>
<div class="highlight"><pre><span></span>$ ipfs daemon <span class="c1"># make sure this runs (and install/setup ipfs first)</span>
$ <span class="nb">echo</span> <span class="s1">&#39;my tiny text&#39;</span> <span class="p">|</span> ipfs add <span class="c1"># add content to the node, smaller than 1KB</span>
QmfQKcXMLGvCxx6opNDwb1ptD1LJER6MPHdsMHCB1CXpFF
$ ipfs cat QmfQKcXMLGvCxx6opNDwb1ptD1LJER6MPHdsMHCB1CXpFF <span class="c1"># just check</span>
my tiny text
$ ipfs dht get /ipfs/QmfQKcXMLGvCxx6opNDwb1ptD1LJER6MPHdsMHCB1CXpFF
<span class="c1"># returns not found</span>
</pre></div>
<p>So now we have that text there we would like to access it from the DHT, but apparently, the only <code>ipfs dht get</code> request that is supported is for keys starting with <code>/ipns/</code>.</p>
<p>OK, and so we create an IPNS and see if we can query that directly:</p>
<div class="highlight"><pre><span></span>$ ipfs name publish QmfQKcXMLGvCxx6opNDwb1ptD1LJER6MPHdsMHCB1CXpFF <span class="c1"># point a IPNS address to our content</span>
Published to QmYebHWdWStasXWZQiXuFacckKC33HTbicXPkdSi5Yfpz6: /ipfs/QmfQKcXMLGvCxx6opNDwb1ptD1LJER6MPHdsMHCB1CXpFF
$ ipfs resolve QmYebHWdWStasXWZQiXuFacckKC33HTbicXPkdSi5Yfpz6 <span class="c1"># check it</span>
<span class="c1"># never returns, hmm IPNS doesn&#39;t seem to be ready for production</span>
$ ipfs dht get /ipns/QmYebHWdWStasXWZQiXuFacckKC33HTbicXPkdSi5Yfpz6 <span class="c1"># same thing but directly</span>
<span class="c1"># does return binary data starting with 4/ipfs/QmfQKcXMLGvCxx6opNDwb1ptD1LJER6MPHdsMHCB1CXpFF</span>
</pre></div>
<p>It sort of does work, but it is limited to IPNS for now it seems.</p>
<p>What else can we do with the DHT?</p>
<p>Use the DHT to find out which nodes can provide some data:</p>
<div class="highlight"><pre><span></span>$ ipfs dht findprovs QmfQKcXMLGvCxx6opNDwb1ptD1LJER6MPHdsMHCB1CXpFF
QmYebHWdWStasXWZQiXuFacckKC33HTbicXPkdSi5Yfpz6
QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN
<span class="c1"># ^^^ there are both my nodes, so that works</span>
<span class="c1"># Now ask the DHT for the address of the first peer that was returned</span>
$ ipfs dht findpeer QmYebHWdWStasXWZQiXuFacckKC33HTbicXPkdSi5Yfpz6
/ip4/176.92.234.78/tcp/4001
/ip4/85.74.239.218/tcp/38689
/ip4/127.0.0.1/tcp/4001
/ip4/192.168.1.30/tcp/4001
/ip6/::1/tcp/4001
/ip4/192.168.1.3/tcp/4001
/ip4/176.92.234.78/tcp/40443
<span class="c1"># Oooh nice! We will visit this address notation in the next episode.</span>
</pre></div>
<p>All in all, we saw a bit of the DHT in action, but not as intuitive and functional as I hoped for. </p>
<p>Time to move on to the next exciting technology applied by IPFS:</p>
<h2>BitTorrent</h2>
<h3>How does BitTorrent work?</h3>
<p>We all know about BitTorrent, but some of us (yeah, me too) need to dig deeper to really get it. This presentation is a great introduction: <a href="https://www.youtube.com/watch?t=961&amp;v=kxHRATfvnlw">Feross Aboukhadijeh: WebTorrent - JSConf.Asia 2014</a>. The speaker talks about how he implemented a BitTorrent client that can run in the browser, thanks to WebRTC (which is a cool technology that might come in handy later on for our project). It also explains the working of the DHT again to drive it home.</p>
<p>More to read or watch:</p>
<ul>
<li><a href="http://web.cs.ucla.edu/classes/cs217/05BitTorrent.pdf">Peer-to-peer networking with BitTorrent</a> (PDF). If you read only one article, make it this one.</li>
<li><a href="https://www.youtube.com/watch?v=nmgzOQf6NbQ">Peer to Peer Content Delivery - BitTorrent</a> (video) + <a href="https://courses.cs.washington.edu/courses/cse461/17wi/section-slides/9-p2p.pdf">the slides</a> (PDF).</li>
<li><a href="http://bittorrent.org/beps/bep_0052.html">The BitTorrent Protocol Specification v2</a> (the spec, not hard to read).</li>
</ul>
<h3>BitTorrent and IPFS</h3>
<p>The exchange of data (blocks) in IPFS is inspired by BitTorrent but is not 100% BitTorrent. The white paper mentions two BitTorrent features that IPFS uses:</p>
<ul>
<li>tit-for-tat strategy (if you don't share, you won't receive either)</li>
<li>get rare pieces first (improves performance and more, see the first PDF above)</li>
</ul>
<p>A notable difference is that where in BitTorrent each file has a separate swarm of peers (forming a P2P network with each other) where IPFS is one big swarm of peers for all data. The IPFS BitTorrent variety is called BitSwap and I'll go over that in the next episode.</p>
<p>Let's try if we can see something of the swarm in action.</p>
<div class="highlight"><pre><span></span># Make sure you have the daemon running
$ ipfs swarm peers
/ip4/104.131.131.82/tcp/4001/ipfs/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ
/ip4/104.236.151.122/tcp/4001/ipfs/QmSoLju6m7xTh3DuokvT3886QRYqxAzb1kShaanJgW36yx
/ip4/128.199.219.111/tcp/4001/ipfs/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu
/ip4/178.62.61.185/tcp/4001/ipfs/QmSoLMeWqB7YGVLJN3pNLQpmmEk35v6wYtsMGLzSr5QBU3
</pre></div>
<p>When you just start the daemon you get connected to a couple of "seed peers". After a while the number of peers grows rapidly.</p>
<p>Another command to query the swarm:</p>
<div class="highlight"><pre><span></span>$ ipfs swarm addrs
QmNRuQrwtGgeVQgndTny4A4Rukg7GR7vzDJrVJxUBfqevk <span class="o">(</span><span class="m">4</span><span class="o">)</span>
/ip4/127.0.0.1/tcp/4001
/ip4/172.31.41.39/tcp/4001
/ip4/35.167.26.28/tcp/4001
/ip6/::1/tcp/4001
QmNSJohkvvBVmHeN7KUZnm4X84GA6Znbv6ZmvsTAjbw3AB <span class="o">(</span><span class="m">5</span><span class="o">)</span>
/ip4/10.0.1.8/tcp/4001
/ip4/127.0.0.1/tcp/4001
/ip4/174.44.163.74/tcp/16012
/ip6/::1/tcp/4001
/ip6/fd15:462e:f8fd:695e:9161:27dd:7f78:d242/tcp/4001
QmNTJyhCYcbv5GdnqtdEwTfJCgY6pV4PJiNTtAmuoxnQak <span class="o">(</span><span class="m">3</span><span class="o">)</span>
/ip4/127.0.0.1/tcp/4001
/ip4/94.176.232.68/tcp/4001
/ip6/::1/tcp/4001
QmNTZy7TfXvsHczwwV3EYbxRZN5gthgicG9GiroD7C4ZrP <span class="o">(</span><span class="m">4</span><span class="o">)</span>
/ip4/127.0.0.1/tcp/4001
/ip4/172.20.255.127/tcp/4001
/ip4/54.229.227.53/tcp/4001
/ip6/::1/tcp/4001
.
.
.
</pre></div>
<p>These are the addresses in the swarm the node knows of, the hash on top is the peerId.</p>
<p>With this information, you can connect to a peer like</p>
<div class="highlight"><pre><span></span>$ ipfs swarm connect /ip4/114.91.202.180/tcp/34746/ipfs/QmfTgdg6GkqJtUrWAYo69GjcLrjQq9LjTjgW3KZ1ux1X6U
connect QmfTgdg6GkqJtUrWAYo69GjcLrjQq9LjTjgW3KZ1ux1X6U success
</pre></div>
<p>More about BitSwap later...</p>
<h2>Let's take a break</h2>
<p>Pfew, that was a lot of theory and dry material, and we are not even halfway! So to let it all sink in and to free up the brain for the rest go for a walk in the park and listen to this podcast: <a href="https://tim.blog/2017/06/04/nick-szabo/">The Quiet Master of Cryptocurrency — Nick Szabo</a>. It goes all over the place, but I found it very interesting. It zooms out from bit level to things like "what is money?".</p>
<p>Highly recommended listening, and when you're done...</p>
<h2>Version Control Systems - Git</h2>
<p>The white paper's entire section on Version Control Systems is reproduced here:</p>
<blockquote>
<p>Version Control Systems provide facilities to model files changing over time and distribute different versions efficiently. </p>
<p>The popular version control system Git provides a powerful Merkle DAG object model that captures changes to a filesystem tree in a distributed-friendly way.</p>
<ol>
<li>Immutable objects represent Files (blob), Directories (tree), and Changes (commit).</li>
<li>Objects are content-addressed, by the cryptographic hash of their contents.</li>
<li>Links to other objects are embedded, forming a Merkle DAG. This provides many useful integrity and workflow properties.</li>
<li>Most versioning metadata (branches, tags, etc.) are simply pointer references, and thus inexpensive to create and update.</li>
<li>Version changes only update references or add objects.</li>
<li>Distributing version changes to other users is simply transferring objects and updating remote references.</li>
</ol>
</blockquote>
<p>Let's step through these lines one by one to understand it:</p>
<blockquote>
<p>Ad 1. Immutable objects represent Files (blob), Directories (tree), and Changes (commit).</p>
</blockquote>
<p>Git only adds data. So blobs, trees, and commits are immutable. What the last version of either of these is, is determined by special references (see point 4 and 5).</p>
<blockquote>
<p>Ad 2. Objects are content-addressed, by the cryptographic hash of their contents.</p>
</blockquote>
<p>In git, the file or directory name is not used when they are referenced. Git hashes the content (or listing or commit) with SHA1 and uses these hashes in its database.
This article makes it very insightful how that works: <a href="https://medium.freecodecamp.org/git-internals-for-curious-developers-a1e44e7ecafe">Git under the hood</a> (Start reading at "Now let’s look at how Git does all this")</p>
<blockquote>
<p>Ad 3. Links to other objects are embedded, forming a Merkle DAG. This provides many useful integrity and workflow properties.</p>
</blockquote>
<p>Oh my, a Merkle DAG! No worries, it is actually not as scary as it sounds. </p>
<p>A <a href="https://en.wikipedia.org/wiki/Merkle_tree">Merkle tree</a> is a binary tree where the parent contains the hash of the concatenation of the hashes of the two children. This explains the integrity property: any change in a data block results in a change of the root node. With just a little bit of meta-data (uncles and parents, which can be untrusted) and a trusted root node, we can verify the validity of the block.</p>
<p>Now, a Merkle DAG is not the same as a Merkle Tree. The difference is explained here: <a href="https://bmcorser.github.io/2014/12/22/merkle-dag.html">Merkle DAG</a> and <a href="https://github.com/jbenet/random-ideas/issues/20">JRFC 20 - Merkle DAG</a></p>
<p>In short: the Merkle DAG is more general as it is not a binary tree but a graph, and any node can contain data, not just the leaf nodes as in the Merkle Tree.</p>
<p>It's still all a bit vague, but I'll address it again when we have a look at the IPFS Merkle DAG.</p>
<blockquote>
<p>Ad 4. and 5 Most versioning metadata (branches, tags, etc.) are simply pointer references, and thus inexpensive to create and update. Version changes only update references or add objects.</p>
</blockquote>
<p>This is visualized here: <a href="http://eagain.net/articles/git-for-computer-scientists/">Git for Computer Scientists</a> where the branch, the HEAD, and tags are simply references to commits.</p>
<blockquote>
<p>Ad 6. Distributing version changes to other users is simply transferring objects and updating remote references.</p>
</blockquote>
<p>What it says here, no need to explain I think :)</p>
<h2>Self-Certified Filesystems - SFS</h2>
<p>This is used to implement the IPNS name system for IPFS. It allows us to generate an address for a remote filesystem, where the user can verify the validity of the address.</p>
<p>From the white paper:</p>
<blockquote>
<p>SFS introduced a technique for building Self-Certified Filesystems: addressing remote filesystems using the following scheme</p>
<div class="highlight"><pre><span></span>/sfs/&lt;Location&gt;:&lt;HostID&gt;
</pre></div>
<p>where Location is the server network address, and:</p>
<div class="highlight"><pre><span></span>HostID = hash(public_key || Location)
</pre></div>
<p>Thus the name of an SFS file system certifies its server.</p>
</blockquote>
<p>I think that speaks for itself, we will see it in action in the next episode: <a href="understanding-the-ipfs-white-paper-part-2.html">Understanding the IPFS White Paper part 2</a></p>
<p>Let me know what you think of this post by tweeting to me <a href="https://twitter.com/pors">@pors</a>!</p>Getting to know IPFS2017-08-03T19:11:00+02:002017-08-03T19:11:00+02:00Mark Porstag:decentralized.blog,2017-08-03:/getting-to-know-ipfs.html<p>This article is part 3 of the Blockchain train journal, start reading here: <a href="catching-the-blockchain-train.html">Catching the Blockchain Train</a>.</p>
<h2>Let's read a bit (reading is good for you!)</h2>
<p>I already listed this article in the previous episode, but I mention it again because it is a great place to start to understand …</p><p>This article is part 3 of the Blockchain train journal, start reading here: <a href="catching-the-blockchain-train.html">Catching the Blockchain Train</a>.</p>
<h2>Let's read a bit (reading is good for you!)</h2>
<p>I already listed this article in the previous episode, but I mention it again because it is a great place to start to understand IPFS and its context: <a href="https://ipfs.io/ipfs/QmNhFJjGcMPqpuYfxL62VVB9528NXqDNMFXiqN5bgFYiZ1/its-time-for-the-permanent-web.html">HTTP is obsolete. It's time for the distributed, permanent web</a>.</p>
<p>A couple of things we can pick up from it (a bit simplified):</p>
<ul>
<li>IPFS consists of a network of peer-to-peer nodes (aka computers that talk to each other directly)</li>
<li>These nodes can store content (any type of file)</li>
<li>Content is represented by a hash and is immutable (if the content changes, so does the hash)</li>
<li>A node can request content from other nodes by using this hash. This is pretty cool: there is a permanent relation between the hash and the content itself. Unlike the current web where the content behind a URL can change.</li>
<li>Nodes can decide to store a copy of any content</li>
<li>The more nodes store a piece of content the harder it is to get rid of it (the permanent web).</li>
<li>The network also gets faster that way, similar to bittorrent getting faster when the number of seeders goes up (IPFS is partly based on the bittorrent protocol, but one of the differences is that it <a href="https://www.reddit.com/r/ipfs/comments/48ab4z/why_do_people_say_ipfs_is_permanent_torrents/d0z08of/">prevents duplicate pieces of identical content</a>)</li>
</ul>
<p>A bit more detail:</p>
<ul>
<li>Content (files) is broken up into blocks, and each block gets a hash</li>
<li>Duplicate blocks (identical content but different hash) are removed from the network</li>
<li>IPFS has a directory concept (another hash) that points to the hashes of the content inside</li>
<li>Git like version control is used for blocks</li>
<li>To solve the problem of finding the last version of some content is solved by IPNS</li>
<li>IPNS allows anyone to create a unique public link to any content that is out there. </li>
<li>So an IPFS hash points to some immutable content, regardless the version; the IPNS address points to some file or directory determined by the creator of that address</li>
<li>This IPNS address is implemented as a pubkeyhash which is similar to the public address in a Bitcoin wallet.</li>
<li>IPNS hashes are long and ugly. Also, browsers don't speak IPFS. There are a couple of solutions to bridge the gap to the current time that we will look into later.</li>
</ul>
<p>To read and discover more about IPFS, this page on github is a great starting point <a href="https://github.com/ipfs/ipfs">ipfs/ipfs</a>. Make sure to check out the <a href="https://github.com/ipfs/ipfs#quick-summary">quick summary</a>.</p>
<p>OK, it is all still a bit abstract I guess, so time to play.</p>
<h2>Terminal time (move those fingers buddy!)</h2>
<p>Now we understand the basics lets start playing a bit.</p>
<p>To get up and running and to be able to play along, first <a href="https://ipfs.io/docs/install">install</a> (I recommend the ipfs-update approach) and follow the <a href="https://ipfs.io/docs/getting-started/">getting started</a>.</p>
<p>Now it is time to set the first step into making this blog decentralized. BTW what a great domain this site has! Well, as long as you are reading the centralized version :)</p>
<p>I uploaded the statically generated website to my server and it looks like this:</p>
<div class="highlight"><pre><span></span>├── archives.html
├── author
│ └── mark-pors.html
├── authors.html
├── catching-the-blockchain-train.html
├── categories.html
├── category
│ └── blockchain-train-journal.html
├── feeds
│ ├── all.atom.xml
│ └── blockchain-train-journal.atom.xml
├── getting-to-know-ipfs.html
├── index.html
├── picking-a-decentralized-storage-system.html
├── tags.html
└── theme
├── css
... // ...
└── youtube.png
</pre></div>
<p>To shoot a file into the interplanetary file system, we simply do:</p>
<div class="highlight"><pre><span></span>$ ipfs init <span class="c1"># this creates your node&#39;s peer ID</span>
$ ipfs daemon <span class="c1"># start your local node</span>
$ ipfs add catching-the-blockchain-train.html
added QmXyZcrThrfWQSTKzPiNT4Nd2RcqcVQ3tr7rmFqHYZ3fq4 catching-the-blockchain-train.html
</pre></div>
<p>Now my node can serve this file and tell the network there is a piece of content (my first blog post!) available, just in case someone requests this hash: <code>QmXyZcrThrfWQSTKzPiNT4Nd2RcqcVQ3tr7rmFqHYZ3fq4</code>.</p>
<p>So, in your terminal:</p>
<div class="highlight"><pre><span></span>ipfs cat QmXyZcrThrfWQSTKzPiNT4Nd2RcqcVQ3tr7rmFqHYZ3fq4
</pre></div>
<p>should return the same as (for as long as I host it over HTTP of course):</p>
<div class="highlight"><pre><span></span>curl http://decentralized.blog/catching-the-blockchain-train.html
</pre></div>
<p>Notice that when you request the file for the second time through ipfs, it gets it very fast. That is because your node now has a copy of the content locally. Pretty cool huh?</p>
<p>The ipfs node comes with an HTTP gateway, so we can also access the file in a browser: <a href="http://decentralized.blog:8080/ipfs/QmXyZcrThrfWQSTKzPiNT4Nd2RcqcVQ3tr7rmFqHYZ3fq4">http://decentralized.blog:8080/ipfs/QmXyZcrThrfWQSTKzPiNT4Nd2RcqcVQ3tr7rmFqHYZ3fq4</a></p>
<p>This is what happened here (from the getting started document):</p>
<blockquote>
<p>The gateway served a file from your computer. The gateway queried the DHT, found your machine, requested the file, your machine sent it to the gateway, and the gateway sent it to your browser.</p>
</blockquote>
<p>Note: this obviously introduces a central server into the picture, but at least anyone can run such a gateway. So it is still sort-of-decentralized. We have to look for something better later on.</p>
<h2>Decentralize it! At last some progress</h2>
<p>To push the whole website onto the interplanetary file system can't be more simple, just add the <code>-r</code> flag and point to the directory containing all static files.</p>
<div class="highlight"><pre><span></span>$ ipfs add -r ../../www/dcb
added QmWu4hsywXoSrw5JRhjUadyMnDuxuvcgExwVD62cWeLVjb dcb/archives.html
added QmUCJ6z2EfikXEDUYbxfjdkrX2zw62XCAM93HHC61Qrmwg dcb/author/mark-pors.html
added QmV13vUiKU64oHoxbp5MN7bnehraW46de82PLvndbisn64 dcb/authors.html
added QmXyZcrThrfWQSTKzPiNT4Nd2RcqcVQ3tr7rmFqHYZ3fq4 dcb/catching-the-blockchain-train.html
added QmRiWHfYLQEJoKz2dDq2sXnW2keiKZG1nHbnaN55TTXN8b dcb/categories.html
added QmSCx2eV3BjHJWoXMBc7rKakPv3fAcewyroFCRmovcsYSy dcb/category/blockchain-train-journal.html
added QmVxVLT22KLGasBLxzSv6yXofx3GVXmWEd3HqSjPp1X6Zz dcb/feeds/all.atom.xml
added QmdKJY5VVd8Qm5xsLt8Vj5XwvTbe5mCQhK6DnZ9LkqTHwz dcb/feeds/blockchain-train-journal.atom.xml
added QmYpyQ3MXYZum3q2E9pRxUWnJXK74cLvFbuMDYukK9ivK1 dcb/index.html
added QmXkfv9ZXb9evxPXTMJC8fvgeZymYpTQo3CVFv7mDoSRTN dcb/picking-a-decentralized-storage-system.html
added QmSyuSjmzhwkoj6qH7s4MmKeB98Ub1DZ1Hm5anBBnR9yVu dcb/tags.html
added QmSfLETTMQc1b9cBJENH1S4NK5qniXSf79KSP8h5CbU3U8 dcb/theme/css/main.css
... // ...
added QmU8CmEYWV5aUuqoToq1xiyqftdY8MHQ8MdViA2ex23uyV dcb/theme/images/icons/youtube.png
added QmYBVeJkBRdJYj8jr3YTGWE537h26YqcfjvbX4ttjZ65X2 dcb/author
added QmPwQPX86RQi6MrZQigh2tWgztcbf46EjhRg1hzMzELvHN dcb/category
added QmfRxCPp417G3NTqEpBmwtGwtUKFzgQjBRvxNBq88HhC9F dcb/feeds
added QmYUnPcDPcnKA3WFembvNYG48m1oUWa7QLNWZxrkLv1vcE dcb/theme/css
added QmfBq8BDXxhu9cLAG4qhxqY6K5xDQCBWyoYqX5fTmQbfwW dcb/theme/images/icons
added Qme4Dt6vQ1eAVcEs7dY55cAMhTsRJxTakANhZjX9EKvGfF dcb/theme/images
added QmciTMAUiEqMpar2u9n1dJNJStmrP6fjMESyH6ZS5eo56H dcb/theme
added QmcPx9ZQboyHw8T7Afe4DbWFcJYocef5Pe4H3u7eK1osnQ dcb
</pre></div>
<p>Note: my first try failed when I tried to add a symlink to the network, that doesn't work.</p>
<p>See that the hash for <code>dcb/catching-the-blockchain-train.html</code> is the same as we previously got? That's the de-duplication in action.</p>
<p>Now when we take the bottom hash from this list and feed it to the gateway, we see the blog: <a href="http://decentralized.blog:8080/ipfs/QmcPx9ZQboyHw8T7Afe4DbWFcJYocef5Pe4H3u7eK1osnQ/">http://decentralized.blog:8080/ipfs/QmcPx9ZQboyHw8T7Afe4DbWFcJYocef5Pe4H3u7eK1osnQ/</a>. All relative links work just fine; it's beautiful!</p>
<p>In case my gateway is down, just try the one provided by IPFS: <a href="https://gateway.ipfs.io/ipfs/QmcPx9ZQboyHw8T7Afe4DbWFcJYocef5Pe4H3u7eK1osnQ/">https://gateway.ipfs.io/ipfs/QmcPx9ZQboyHw8T7Afe4DbWFcJYocef5Pe4H3u7eK1osnQ/</a> or your local gateway: <a href="http://127.0.0.1:8080/ipfs/QmcPx9ZQboyHw8T7Afe4DbWFcJYocef5Pe4H3u7eK1osnQ/">http://127.0.0.1:8080/ipfs/QmcPx9ZQboyHw8T7Afe4DbWFcJYocef5Pe4H3u7eK1osnQ/</a>. Now is that cool or what?</p>
<p>There is still a small issue: it is the previous version of the blog because I'm still writing this article and when I upload it when I'm done the hash changes, but then the new hash won't be in the article...</p>
<p>There is a solution for that:</p>
<h2>A permanent address for this blog with IPNS</h2>
<p>The usage of IPNS is explained here: <a href="https://ipfs.io/ipfs/QmNZiPk974vDsPmQii3YbrMKfi12KTSNM7XMiYyiea4VYZ/example#/ipfs/QmP8WUPq2braGQ8iZjJ6w9di6mzgoTWyRLayrMRjjDoyGr/ipns/readme.md">The Inter-Planetary Naming System</a>.</p>
<p>So in our case, on the node that runs on my server, I do:</p>
<div class="highlight"><pre><span></span>$ ipfs name publish QmcPx9ZQboyHw8T7Afe4DbWFcJYocef5Pe4H3u7eK1osnQ
Published to QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN: /ipfs/QmcPx9ZQboyHw8T7Afe4DbWFcJYocef5Pe4H3u7eK1osnQ
</pre></div>
<p>This results in an address for the blog that won't change: <code>QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN</code>.</p>
<p>This address points (when I executed the command just right now) to the directory containing the static files (index.html is served when it is in that directory BTW).</p>
<p>When you read this, the address <code>QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN</code> is still the home page address, but it will contain new content (I will do another <code>publish</code> when I'm done writing this post).</p>
<p>So our static blog is now decentralized and available through <a href="http://decentralized.blog:8080/ipns/QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN/">http://decentralized.blog:8080/ipns/QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN/</a>. Note the <code>ipns</code> part instead of <code>ipfs</code> this time.</p>
<p>IPFS is pretty cool, and it makes our lives much easier. Before we move on to improving our blog I'd like to look a little bit under the hood the understand the workings of IPFS (and IPNS) a bit more... </p>
<p>Read on here: <a href="understanding-the-ipfs-white-paper-part-1.html">Understanding the IPFS White Paper part 1</a>.</p>Picking a Decentralized Storage System2017-08-02T19:11:00+02:002017-08-02T19:11:00+02:00Mark Porstag:decentralized.blog,2017-08-02:/picking-a-decentralized-storage-system.html<p>This article is part 2 of the Blockchain train journal, start reading here: <a href="catching-the-blockchain-train.html">Catching the Blockchain Train</a>.</p>
<h2>Some reading material to get started</h2>
<p>A couple of articles that I found useful when learning about decentralized storage:</p>
<ul>
<li><a href="https://media.consensys.net/decentralized-storage-the-backbone-of-the-third-web-d4bc54e79700">Decentralized Storage: The Backbone of the Third Web</a></li>
<li><a href="https://ipfs.io/ipfs/QmNhFJjGcMPqpuYfxL62VVB9528NXqDNMFXiqN5bgFYiZ1/its-time-for-the-permanent-web.html">HTTP is obsolete. It's time for …</a></li></ul><p>This article is part 2 of the Blockchain train journal, start reading here: <a href="catching-the-blockchain-train.html">Catching the Blockchain Train</a>.</p>
<h2>Some reading material to get started</h2>
<p>A couple of articles that I found useful when learning about decentralized storage:</p>
<ul>
<li><a href="https://media.consensys.net/decentralized-storage-the-backbone-of-the-third-web-d4bc54e79700">Decentralized Storage: The Backbone of the Third Web</a></li>
<li><a href="https://ipfs.io/ipfs/QmNhFJjGcMPqpuYfxL62VVB9528NXqDNMFXiqN5bgFYiZ1/its-time-for-the-permanent-web.html">HTTP is obsolete. It's time for the distributed, permanent web</a></li>
<li><a href="https://ethereum.stackexchange.com/a/2422/16115">What is the difference between Swarm and IPFS?</a></li>
</ul>
<h2>Comparing decentralized storage systems</h2>
<p>Our decentralized blog pages are in need of some sort of decentralized hosting. To build that from scratch seems like a lot of work and will probably result in a blockchain train crash.</p>
<p>So let's see what is already out there we could use for our project.</p>
<p>I searched a bit and came up with:</p>
<ul>
<li><a href="https://storj.io/">Storj</a></li>
<li><a href="https://sia.tech/">SIA</a></li>
<li><a href="https://maidsafe.net/">MaidSafe</a></li>
<li><a href="https://ipfs.io/">IPFS</a></li>
<li><a href="https://zeronet.io/">ZeroNet</a></li>
<li><a href="https://www.ethereum.org/">Ethereum Swarm</a></li>
<li><a href="https://www.bigchaindb.com/">BigchainDB</a></li>
<li>did I forget a serious contender? Please <a href="https://twitter.com/pors">let me know</a>.</li>
</ul>
<p>I’m not yet qualified to compare these technologies (still trying to catch the train!), but I’ll add some observations for each:</p>
<h3>Storj</h3>
<p>From the home page:</p>
<blockquote>
<p>Blockchain-based, end-to-end encrypted, distributed object storage, where only you have access to your data.</p>
</blockquote>
<p>That doesn't sound like something that is handy for a blog, where we want content to be public.</p>
<p>Storj looks like a great product though, it is a bit like <a href="https://www.resilio.com/">Resilio Sync</a>, but with a blockchain added to create a marketplace for buying or renting out disk space on the network.</p>
<h3>SIA</h3>
<p>SIA is another Dropbox killer, more or less the same as Storj.</p>
<h3>MaidSafe</h3>
<p>Where Storj and SIA pitch their service as a faster and cheaper way to store your data, Maidsafe promotes their SAFE network by scaring you. Cloud providers, government, and hackers can not be trusted is the message in their video. Apart from the funny name, it is very similar to the two services above.</p>
<h3>IPFS</h3>
<p>Now, this is a different beast. The first three services are <code>incentive platforms</code>, which means they have their own cryptocurrency to support their marketplace. IPFS also has such a thing, <a href="https://filecoin.io/">FileCoin</a>, but it is neatly split from IPFS which is more like a decentralized storage protocol.</p>
<p>IPFS stands for <code>interplanetary file system</code>, but they hide the original name from their homepage. Too funny for a serious business!</p>
<p>On the home page it says:</p>
<blockquote>
<p>IPFS is the Distributed Web</p>
<p>A peer-to-peer hypermedia protocol to make the web faster, safer, and more open.</p>
</blockquote>
<p>IPFS seems like an great option for our project.</p>
<h3>ZeroNet</h3>
<p>ZeroNet doesn't have a cheesy video on their home page, so I had to actually <em>read</em>. However, their awesome presentation is 100x better than the four videos from the others combined: <a href="https://docs.google.com/presentation/d/1_2qK1IuOKJ51pgBvllZ9Yu7Au2l551t3XBgyTSvilew/pub?start=false&amp;loop=false&amp;delayms=3000&amp;slide=id.g9a1cce9ee_0_4">ZeroNet preso</a>.</p>
<p>They also seem to operate a lot less like a serious business compared to the others, showing more of the idealistic open source spirit.</p>
<p>On the home page:</p>
<blockquote>
<p>Open, free and uncensorable websites, using Bitcoin cryptography and BitTorrent network</p>
</blockquote>
<p>Oh, oh, that is what <em>we</em> are trying to do here! A competitor! Nah, let's just keep an eye on the code, maybe we can learn a thing or two.</p>
<h3>Ethereum Swarm</h3>
<p>On our trip to become blockchain developers, we have no other choice than to dive into Ethereum. We will address that in a future episode. But their decentralized file storage system, named Swarm, must be considered here.</p>
<p>It took me a while to find the source code for this project, but <a href="https://github.com/ethereum/go-ethereum/tree/master/swarm">here it is</a>!</p>
<p>From the <a href="http://swarm-guide.readthedocs.io/en/latest/introduction.html#introduction">documentation</a>:</p>
<blockquote>
<p>Swarm is a distributed storage platform and content distribution service, a native base layer service of the ethereum web 3 stack. The primary objective of Swarm is to provide a sufficiently decentralized and redundant store of Ethereum’s public record, in particular to store and distribute dapp code and data as well as block chain data. </p>
</blockquote>
<p>It seems that Swarm is an incentive platform enforced with Ethereum smart contracts, so not the best choice for our purpose. We gotta stay hands-on here. We want to add a coin later ourselves!</p>
<h3>BigchainDB</h3>
<p>From their site:</p>
<blockquote>
<p>BigchainDB is complementary to decentralized storage, processing and communication building blocks. It can be used side by side with higher-level decentralized computing platforms and applications, and protocols for identity, financial assets, intellectual property and sidechains. BigchainDB fills a gap in the decentralized stack.</p>
</blockquote>
<p>This is a brilliant solution, almost too good to be true. It is not just decentralized storage, but a full blown database.</p>
<p>It is built on top of MongoDB and decentralized, so data is immutable, and there is no central authority.</p>
<p>For our current project it might be a bit overkill, but let's keep an eye on it.</p>
<h2>The winning decentralized storage system</h2>
<p>OK, I’ll be honest, I already picked one before I did this quick research what is around :-)</p>
<p><strong><em>The winner is IPFS! Yay!</em></strong></p>
<p>The runner-up we might have a look at at a later stage: BigchainDB!</p>
<p>Even in retrospect, IPFS seems like a good choice: all competitors have their "flaws" in the context of this project. By applying "selection by deduction", only IPFS remains.</p>
<p>After this very scientific approach in picking IPFS, let's hope it is indeed useful for us. </p>
<p>Let's move on to part 3: <a href="getting-to-know-ipfs.html">Getting to know IPFS</a></p>Catching the Blockchain Train2017-08-01T13:34:00+02:002017-08-01T13:34:00+02:00Mark Porstag:decentralized.blog,2017-08-01:/catching-the-blockchain-train.html<h2>What is this and for who?</h2>
<p>This is a blog for techies who (almost) missed the blockchain train. Hmm, I also need to write a song "the blockchain train," it just sounds so good!</p>
<p>I'm a bit late to the blockchain/Bitcoin party, and when you made it past the …</p><h2>What is this and for who?</h2>
<p>This is a blog for techies who (almost) missed the blockchain train. Hmm, I also need to write a song "the blockchain train," it just sounds so good!</p>
<p>I'm a bit late to the blockchain/Bitcoin party, and when you made it past the previous line I assume you are too!</p>
<p>No worries, there is still plenty of time to add blockchain technology to your skillset. I am confident that what we see happening now is just the very beginning of a decentralized Internet. So even though we feel we are late, we can still catch the blockchain train. Especially because you and I are pretty smart! Right?</p>
<p>My approach to educating myself is two-fold:</p>
<ul>
<li>
<p>First by reading as much as I can about the blockchain, Bitcoin, smart contracts, Ethereum, dapps, ipfs, and all of that. </p>
</li>
<li>
<p>In parallel, I want to apply what I learn by turning this blog into a decentralized app (dapp) as much as possible. I will work in "naive mode" and start building and using technologies I found on my path, and backtrack (you know, refactor) when I see it was the wrong approach.</p>
</li>
</ul>
<p>The code will, of course, be open source and available here: <a href="https://github.com/blockchaintrain/blog">blockchaintrain</a>.</p>
<p>BTW this is not a tutorial; it's just a journal of my activities.</p>
<h2>What I'm reading</h2>
<p>I'm reading three books in parallel right now (next to the online article here and there):</p>
<ol>
<li><a href="https://www.goodreads.com/book/show/35432045-mastering-bitcoin">Mastering Bitcoin: Programming the Open Blockchain</a>: a thorough technical introduction into the world of Bitcoin. I got stuck halfway when I read it a couple of years ago. This time I mean to finish it.</li>
<li><a href="https://www.goodreads.com/book/show/34890064-blockchain-for-dummies-for-dummies">Blockchain for dummies</a>: yeah, I know...</li>
<li><a href="https://www.goodreads.com/book/show/26457167-decentralized-applications">Decentralized Applications: Harnessing Bitcoin's Blockchain Technology</a>: not very well received, but I like it so far.</li>
</ol>
<p>I'll keep you updated on other books or articles I read.</p>
<h2>Decentralized blog design part 1</h2>
<p>The first version of this blog (you're reading it now) is a statically generated blog with <a href="https://github.com/getpelican/pelican">Pelican</a>*, and I uploaded the files to my server with SSH.</p>
<p>So my progress towards a fully decentralized blog is 0/100!</p>
<p>Excellent! That means there is still a lot to discover.</p>
<p>In the next article, I will describe how I made a first step: load the content of the blog from a decentralized file system.</p>
<p>*: if you don't like Python, you can use Hugo (Go) or Jekyll (Ruby) or
<a href="https://www.netlify.com/blog/2017/05/25/top-ten-static-site-generators-of-2017/">any one of these</a>. Let me know if you follow along with a different site generator, and I'll add it to this blog.</p>
<h2>What can you expect in the next episodes?</h2>
<ul>
<li>Using a decentralized file system</li>
<li>From static pages to dynamic content (so from files to a dapp)</li>
<li>Supporting identity</li>
<li>Introduction of colored coins or sidechains or maybe even a mini-ICO</li>
<li>Whatever seems like a useful feature to add...</li>
</ul>
<h2>Wanna join me?</h2>
<p>If you want to play along and contribute, please leave a note in the comments! Uhm, as soon as I have figured out how to support comments in a decentralized manner. Till then, tweet to me <a href="https://twitter.com/pors">@pors</a>!</p>
<p>Continue reading here, part 2 of the Blockchain train journal: <a href="picking-a-decentralized-storage-system.html">Picking a Decentralized Storage System</a>.</p>