be and your thank you so the money and because a freelance Python development from the UK columns and this is an introductory talk really about facing higher the media distributed hash tables answer basic ideas on sure you all know the right thing my feet 4 on and online expectation of that you know that with the prices of so what we'll do is explore so the core concepts of space and time and I'll tell you the story of how I used a society creates a fundamental personal project was to build a distributed hash table which are called the area use 3 reference in his keynote this morning you have 1 is rather Austrian history so this isn't an exhaustive of exposition facing there's an awful lot have to leave out it simplifies a lot as well my main is on you with just enough information so you can continue exploring the module in the concept found therein afterward and it's also a personal pedagogical exercise as if I can explain facing in a simple total weight it's it demonstrates my clarity of thought about facing kind an understanding of what I think of as you can hear I'm talking very quickly because I have brought a lot of information that they give to you so yes perhaps like American

01:22

Ukraine will find this talk so the 1st thing that needs to be answered is what spacing Heidi well the patent documents are very clear about this they they state that this module provides infrastructure for writing single-threaded concurrent code using carry genes would like to I access over sockets and the result is running NetWare clients and servers and other related primitive article that I want to go up and down and you have from memory but no I understand all the more to it it from the documentation it doesn't give me a 2nd or a feel of how might you use small

01:56

arms and such documentation can make model of their little bit intimidating and turns of the realm of esoteric lead to behave like treaties so he talks Korea fixed with some of and the out

02:09

talks in any way what those facing higher do well we can do a lot better than you can keep the answer simple to what they think I do as treaty says it lets you write code that concurrently handles asynchronous network based on the answer must be clear about what I mean by those terms concurrency is some things in here it happens not 1 on asynchronous so literally means not synchronize there's no way to tell when something may happen in the network of course as you know immediately communicating with with another device you survive the Internet and find it so it is actually 1 of the input output when program communicates with the

02:51

outside world so the problem clearly stated that messages arrive and depart from that would be predictable time and Eysenck I'll let you do with such interactions simultaneously so given the simple purpose I want place think idea into a practical context on so let's talk about distributed hash table so again that start with the obvious question what is distributed hash table while I'm assuming you know what's a hash table is because it's always the same thing as victim parts and we are not Europe path that role and I think it's reasonable to expect you to all understand what it is it's a simple peak value data on this is because it is distributed because the whole is built from several independent yet related part so it's a little bit like this abstract encyclopedia that the whole encyclopedia is made of performance that independent yet related to each of archaic distributed hash table is a structure that consists of many independent nodes that of the collaborate with each other over the network but this is also the centralized that's a new node is more important than than than the other so there are no client-server relationship engaged right solution it and that

04:12

would have no and so what is the hash table a DFT is its key value based on why would I want to implement 1 of these well it's a really interesting programming problem is fascinating properties as I mentioned there's no single point of failure of the GHG algorithm by look condemn on the efficiency of the huge number of nodes has good handling of fluid network membership note the joint work on a solid foundation for more complex services such as the ones that the whole the rest of this morning on this test in the real world of BitTorrent users to get any of the 3 methods as well and and obviously if you're if you like older it's an example of decentralized samples which a fascinating on more just technical and political levels what have so guess what distributed hash table Mozart have to conquer handle lots of asynchronous network based on my own which means that they think so we have a context of the how do they think I 0 how do I use a sink to make all this work so let's introduce some core concepts this 1 being given the common quite simply this encoded just keeps looping on each iteration of the loop basically just to think 1st thing is that it holds for all I events that occurred during the time it took to complete the previous iteration of the and the other thing that it does is that it it runs any callbacks that that need to be running during this current iteration of the of the loop also carries out various housekeeping the difficult facts that have yet to be executed that's not something reading and so it's important again this is sort of an articulatory took to define appalling and callbacks are so appalling is discovering the status of something external to the program on in a single I this is network related higher than that of callback is some code has to be executed with on event has occurred that's been detected by polling and this metaphor obviously the kids appalling are we there yet are we there yet are we there yet and and the mother is saying election when we get there and this creates a sort of call back in she is promising do something when some condition is met by that gets the intervention and again a graph

06:32

models so it's important to note that polling takes place once during each iteration of the loop i of and I discovered by Paul violence discovered by the calling determine with cold packs to execute join in the current iteration of all pending callbacks executed on after that the other of the success of the existing 1 and only can't continue while this is happening it is brought so the next iteration cannot start until all the sequence to be executed callbacks then

07:13

in some sense so you will be asking that there's something wrong here is

07:17

that sound very unfortunately concurrency is also a problem and there actually more than 1 way to do it so it's worth looking at was taken sometimes examine why

07:29

works in the way it it does so in this case additional fitting model several problems and have task is the record of the the records by change retrieve data in different ways task the rights of changes that task rights exchanges what's the problem here while task a sobering other radical containing of these changes so we have lots of data which is something that we want to avoid so why don't wait until about 1 task is finished before continuing so 1st you may then be followed by c and so on and this is Eastwood standard sort of terministic but what happens if they need to wait for something such as a call to another machine on that 1 while the program has to wait until the eighties network all complete out and in that situation that we can't get along with others that this you still hanging around the age so given the situation this slide programmers brothers blocks and this is an exchangeable but with writing code in history and the parliament to network-based events which is precisely the sort program that basing it on this has been designed to help with and so you probably ask yourself why not just get on with task B and C while we wait for the results in the network over and above the view thinking and then actually you've you've described quite succinctly

08:54

what they think just so what is the most important side of the tool that you're paying attention so singularities of interest and this means that never faced by is non-blocking answer with well the program does not wait for a reply from the locals for continuing program and well if I call back be and when the result of that report is that in the meantime the program continues to hold In responsible network related events and callbacks execute drawing iteration of the of the of the week after the lecture that I have is detected

09:35

confused while you should be because actually this is how we as humans think about the conference so in the real world we make plans all the time the washing machine finishes is an expected events and how close tries to call for when these expected event happened how committees stop efforts to the washing

09:58

basket and also we multitasking the same way that same kind does was to between the things we need to do while we wait for other things happen we know we will have time just please the orange juice Weil-Petersson the execute while we make breakfast we don't just stick something bridge in the market where put on for 5 minutes and then watch what I don't see you probably as with other things this is exactly the sort of thing that I'm talking we move on to this in 1 of those that anyone think eventually it's it's that this is how we get the world around us so much so that the what was on the micro it out so long my last month of the microwave goes paying and I is the expected events and

10:44

take project sons and so a single avoid potentially confusing and complicated competency will retain the benefits of string the country code and the other is the fundamental advantage of using but we plan ahead expected events by defining qualified to be called when such events you UK and in the meantime and we sequentially handle the callbacks relates with that made happened in the fact that questions before we actually start off power and synchronous 100 tasks creating how do such tasks waiting for non-blocking network-based and how callbacks to foreign to handle

11:22

the eventual result of to answer these questions we need to understand reading futures and top so I'm going to attempt to explain curvature in another 3 slides encouraging in sense is object represents an error in resenting an activity that eventually complete In order to decorate function that returns an important thing to know about the protein so that they may be defend the yield from syntax and when interpreting suspended this allows the event to get on with the other thing that readings results generated so the lazily generate results according to coaching doesn't actually start execution the yield from other objects and that when the yielded from object as a result the current team continues from the yield from statement that suspended and this is called re-entry and at the end of the chain of view from this object that returned results already except rather yielding from some of the current

12:31

so that this node that they so yeah that's just crazy that OK so this is a decorative routine method that handles an incoming HTTP requests as part of my upstream something is giving from the current routine created by the functional see something with the response that generates a list of current loopholes by the building from the readings created by the payload book read called which is in the reads the raw data posted as part of the question you have to stop trying to carry out of 101 data derived the curve was again while waiting for the current routine created by the self-growth process data method of itself which 1 of the things which is called the data of among the task encapsulated by this protein is complete returns a response the books dreamcoat routine guess the return results resumes execution from where each unit from this it's so we

13:38

cannot asynchronous activity happens but how do we handle the results are handled result of the parents were radical

13:45

works well for this you need to understand fusion task which are trying explain in just 2 slides on so future if you don't twist it this will be the future of represents a result that may not yet be available yet callback functions are added to a sort to do is to be executed when is and the task is simply future that wraps parenting and the resulting object is realized when the when the carotene conflicts of the 6 future has resolved

14:23

which sounds overeager to so here's some example of exist so the 1st thing I do is I created back should handle also talk again anything does is called it looks the result of an example is a created task associated with a certain routine that will that will cause and then I actually that task the of the the callback divided by the and in the task

14:53

object is automatically resolve the co routine that's associated with a cold and at the end I expect go remember generators of lasting up and running the event itself OK so here's another perspective on this little bit mind-bending somebody marriage and like point and that we used to work with the 1st part of the part of the mean function and understand the value of the features and tasks are going to be like first-class function call we can also pass them into functions and return them as value which is modeled knows what the result is that we can start and callbacks so let's do a

15:36

quick recon before we come to the sort of the end of the talk of the wave of 5 minutes and so and evaluating this will become the pulse of Ireland managers event handling callback parity of object represents activity that it should be complete source decorated functions and the

15:55

subject of future represents a response that may not be able to get and the city callbacks executed when it results in a set of features of the mysterious got transferred of and tasks are boilerplate future losses that rock proteins and these are realised that every single it's so a give out the but very quickly look at how they they think it works in practice with DHT put 1st gets do that I need to very quickly explain

16:26

how DHT you so as as I mentioned before the is made of and the classic way to visualize this is where the context and you'll see

16:36

what each node has a unique ID that is within the set of all possible values for certain hashing function you want what hashing functions of fuel in all holders things that more than I in my case it might be used to shot fired from the idea value indicates the nodes position in the Gulf and in this way we can tell whether node is located in the abstract network work so we can tell who is close to and how far away the nodes of features so there is some sort of

17:07

notion of distance DAG and this is a key value pair key is turned into a hash again using the same fashion of a shot of the text and the value is stored and nodes used by these are close so the hash of

17:24

the key the so this is similar to understanding where to look things up in a more people in encyclopedia going back to the beginning of articles are words all keys in our case and associated given definitions of values of that is stored in volumes cover some alphabetical range within the global

17:43

encyclopedias so of all the loans the age where

17:48

several along the z can handle oops no matter what each node maintains something called the routine table that tracks the steps of this and its interaction with with its peers results in exchange of information about the nodes that has been talked to and that's how the rooting table is populated initially kept date because nodes keep bumping into each of them answers the tablespoons splits the cost-effectiveness of into buckets look it's contain same number of pairs that will look at the cover a smaller range close to the local and they are therefore the local no knows more close nodes as you you can see me or at about half of 1 of and the the nodes by knowing the

18:35

different book it's in that the score at each node i has some very simple rules they behave quote is a very simple rules it doesn't really matter what those rules on just put a few of them to illustrate what I mean so but the important thing about the state was immediately put value chain to be able to get energy to get this set and each 1 of those fundamental actions requires the look of the case of so that you know which nodes you want interact with along these interactions are asynchronous and lookups or also parallel concurrent because you can offset the appears at the same time the about information you need so how does the look work because of of the 6 degrees of separation again in which always features that they talk

19:19

about what it so that I want a value with the key to the hash is in a position at 6 o'clock as the targets of the 1st thing I do is I asked the nodes in my kitchen table that are closest to the uninvaded

19:40

with notes from there table of of of

19:42

the here and this keeps going recursively until I find nodes that and I actually had that there are actually close to the target until I can't find any nodes that are at the top the

19:55

list so the set requires a look and how is this handled in the role of facing higher well look what is future because it's something whose result because you know so we finished looking up we need to interrogate on that there of the

20:13

state of the look up the progress of finding diagnosed cloaked to talk is held within the look of instance and the lookup results because it's it's future when the result it's not on the result is you need to be of value or not found exception in the case of the text operations which can to be in 20 closest is known nodes in the reached in of the books a will be then is the 29 stole this election so whereas network out as a sing higher handled this network approach on how do nodes in the DHT handle down 1 aspect of the so called concepts 567 streams transport protocols streams are high-level abstractions that allow you to send and receive traffic down the network of from the network using we don't want transports are provided by a single I handle low-level network activity such as TCP and UDP and they have a low level of violent and offering and them the tools the ease for you so you shouldn't really have to enter the protocol of indel network protocols at the application layer so example page 2 p

21:22

next thing so this was a lot of fun so streams of flows of data that can be read from or written to and they are built upon the transports and protocols so what is the transport transport is concerned

21:34

about how still looms over the network where for article is a protocol what so what do we know that arise from the network so it turns out that it works and how to turn the raw bytes it's something we have such as the next request http called and really units work with with streams and perhaps protocols and things like that so my final thoughts of with DHT I've managed to get 100 % unit coverage and this because they think I is part of Python and testing is is normal using the built-in into library more on and so that's how you organize your code is this is a key factor that but this is in contrast with for example twist which has its own test run on its own unit test classes to work on the same way the as the 1 in the standard library which can be confusing and my DHT implementation is small and simple so area 871 lines of code as of this morning interdigitated itself this because they think it makes it easy of about 1 minute of it makes it easier to think about concurrent problems OK so uh underneath the abstractions made he to write a simple cut this short and competence on and finally on attention the difference between buyer and CPU-bound don't use a single value if you need to do something with lots of CPU overhead because of course that's going to block the event if you have of that

23:15

however that's the sweet spot so the future of the color changes for better in in part the of 5 carotenes with the a sink and the weights and to which we don't mention gets very his favorite

23:26

features that must be good I said if you remember his security the ieee church in earlier on I simply bought is taken at the front end and the correct the was in fact few 5 on this a proper native proteins and they're not to be actually computer-generated would like remain at the new basing syntax outside the codebook 5 everything but not of some internal yields from actually makes it easier to to get to the end of the and also the features that it awaits context manager and iteration protocols involved I think the best option it and so had fallen to his predecessor of the features that of have test this characters of the average purposes only on the Finnish but you know what few years the project is probably with the given B uh any questions you as anybody have a short question short question like HTK capture no questions that all OK thank you very much

Spracherkennung

Texterkennung

Bildinhalt

00:00

Gewicht <Mathematik>

Hash-Algorithmus

Modul

Flächeninhalt

Hypermedia

Hash-Algorithmus

Vorlesung/Konferenz

Projektive Ebene

Speicherabzug

Information

Softwareentwickler

Minkowski-Metrik

Tabelle <Informatik>

01:21

Resultante

Bit

Multiplexbetrieb

Relativitätstheorie

Modul

Code

Socket-Schnittstelle

Computeranimation

Client

Informationsmodellierung

Zustandsdichte

Festspeicher

Speicherabzug

Server

Vorlesung/Konferenz

Primitive <Informatik>

02:08

Tabelle <Informatik>

Bit

Hash-Algorithmus

Datennetz

Datenparallelität

Interaktives Fernsehen

Kontextbezogenes System

Ein-Ausgabe

Term

Code

Computeranimation

Message-Passing

Knotenmenge

Prognoseverfahren

Datennetz

Code

Mereologie

Hash-Algorithmus

Optimierung

Datenstruktur

Message-Passing

Funktion <Mathematik>

Tabelle <Informatik>

04:11

Folge <Mathematik>

Hash-Algorithmus

Punkt

Pauli-Prinzip

Fluid

Iteration

Zahlenbereich

Aggregatzustand

Element <Mathematik>

Code

Gerichteter Graph

Computeranimation

Eins

Übergang

Metropolitan area network

Loop

Informationsmodellierung

Knotenmenge

Algorithmus

Reelle Zahl

Hash-Algorithmus

Stichprobenumfang

Speicherabzug

Optimierung

Ereignishorizont

Peer-to-Peer-Netz

Tabelle <Informatik>

Softwaretest

Schlüsselverwaltung

Datennetz

Graph

Kategorie <Mathematik>

Güte der Anpassung

Systemaufruf

Einfache Genauigkeit

Knoten <Statik>

Kontextbezogenes System

Ereignishorizont

Quick-Sort

Konditionszahl

Speicherabzug

Ext-Funktor

Tabelle <Informatik>

Lesen <Datenverarbeitung>

07:12

Differential

Resultante

Subtraktion

Programmiergerät

Datenparallelität

Mathematisierung

Familie <Mathematik>

Oval

Code

Computeranimation

RFID

Task

Virtuelle Maschine

Datensatz

Informationsmodellierung

Task

Datennetz

Vorlesung/Konferenz

Optimierung

Addition

Sichtenkonzept

Datennetz

Mathematisierung

Systemaufruf

p-Block

Systemaufruf

Quick-Sort

Ereignishorizont

Rechenschieber

Programmfehler

Rechter Winkel

Ablöseblase

Reelle Zahl

Standardabweichung

Fitnessfunktion

08:53

Resultante

Programm

Datennetz

Stellenring

Turing-Test

Iteration

Ereignishorizont

Systemaufruf

Computeranimation

Singularität <Mathematik>

Iteration

Datennetz

Optimierung

Ereignishorizont

Verkehrsinformation

09:34

Metropolitan area network

Virtuelle Maschine

Task

Automatische Handlungsplanung

Vorlesung/Konferenz

Bridge <Kommunikationstechnik>

Quick-Sort

Ereignishorizont

Computeranimation

RFID

10:44

Resultante

Lineares Funktional

Befehl <Informatik>

Sichtenkonzept

Krümmung

Ereignishorizont

Code

Computeranimation

Task

Objekt <Kategorie>

Rechenschieber

Task

Verkettung <Informatik>

Datennetz

Code

Speicherabzug

Projektive Ebene

Generator <Informatik>

Ordnung <Mathematik>

Fehlermeldung

Lesen <Datenverarbeitung>

Zeichenkette

12:27

Resultante

Prozess <Physik>

Euler-Winkel

Gebäude <Mathematik>

Anwendungsspezifischer Prozessor

Wurm <Informatik>

Strömungsrichtung

Mailing-Liste

Computeranimation

Endogene Variable

Gewöhnliche Differentialgleichung

Task

Message-Passing

Knotenmenge

Einheit <Mathematik>

Code

Verweildauer

Endogene Variable

Mereologie

Vererbungshierarchie

Reelle Zahl

Kurvenanpassung

Große Vereinheitlichung

Lesen <Datenverarbeitung>

13:44

Resultante

Lineares Funktional

Logarithmus

Versionsverwaltung

Information

Quick-Sort

Computeranimation

Web log

Rechenschieber

Task

Objekt <Kategorie>

Metropolitan area network

Task

Loop

Verweildauer

Vererbungshierarchie

Speicherabzug

Mini-Disc

14:52

Resultante

Bit

Punkt

Wellenlehre

Computeranimation

Task

Puls <Technik>

Datenmanagement

Task

Perspektive

Code

Koroutine

Ereignishorizont

Lineares Funktional

Systemaufruf

Quellcode

Objektklasse

Ereignishorizont

Quick-Sort

Systemaufruf

Ranking

Objekt <Kategorie>

Generator <Informatik>

Funktion <Mathematik>

Loop

Gerade Zahl

Mereologie

15:55

Task

Einfügungsdämpfung

Task

Funktion <Mathematik>

Menge

Loop

Code

Anwendungsspezifischer Prozessor

Endogene Variable

Ereignishorizont

Textbaustein

Computeranimation

16:35

Lineares Funktional

Hash-Algorithmus

Schlüsselverwaltung

Ortsoperator

Datennetz

Abstraktionsebene

Computeranimation

RFID

Metropolitan area network

Knotenmenge

Menge

Hash-Algorithmus

Abstand

Schlüsselverwaltung

17:23

Spannweite <Stochastik>

Wort <Informatik>

Spezifisches Volumen

Schlüsselverwaltung

Computeranimation

17:48

Tabelle <Informatik>

Trennungsaxiom

Resultante

Subtraktion

Gruppenoperation

Interaktives Fernsehen

Zahlenbereich

Abgeschlossene Menge

Schlussregel

Peer-to-Peer-Netz

Computeranimation

Überlagerung <Mathematik>

Energiedichte

Knotenmenge

Spannweite <Stochastik>

Minimalgrad

Verkettung <Informatik>

Menge

Objektorientierte Programmiersprache

Information

Aggregatzustand

Tabelle <Informatik>

19:19

Knotenmenge

Ortsoperator

Hash-Algorithmus

Ablöseblase

Wiederkehrender Zustand

Extrempunkt

Schlüsselverwaltung

Computeranimation

Tabelle <Informatik>

19:42

Resultante

Metropolitan area network

Message-Passing

Knotenmenge

Menge

Mailing-Liste

Hill-Differentialgleichung

Computeranimation

20:09

Resultante

Nichtlinearer Operator

Protokoll <Datenverarbeitungssystem>

Datennetz

Benutzerfreundlichkeit

Abstraktionsebene

Ausnahmebehandlung

Kartesische Koordinaten

Datenfluss

Transportproblem

Maskierung <Informatik>

Computeranimation

Übergang

Homepage

Streaming <Kommunikationstechnik>

Knotenmenge

Arithmetische Folge

Speicherabzug

Idealer Punkt

Streaming <Kommunikationstechnik>

Aggregatzustand

Instantiierung

21:34

Mittelwert

Subtraktion

Gewicht <Mathematik>

Komponententest

Klasse <Mathematik>

Mathematisierung

Implementierung

Zentraleinheit

Code

Gerichteter Graph

Computeranimation

Streaming <Kommunikationstechnik>

Lesezeichen <Internet>

Einheit <Mathematik>

Programmbibliothek

Kontrast <Statistik>

Schnitt <Graphentheorie>

Gerade

Softwaretest

Datennetz

Protokoll <Datenverarbeitungssystem>

Abstraktionsebene

Teilbarkeit

Ereignishorizont

Flächeninhalt

Mereologie

Kantenfärbung

Overhead <Kommunikationstechnik>

Zentraleinheit

23:25

Softwaretest

Codebuch

Protokoll <Datenverarbeitungssystem>

Computersicherheit

Iteration

Kontextbezogenes System

Computeranimation

Endogene Variable

Konfiguration <Informatik>

Motion Capturing

Gewöhnliche Differentialgleichung

Message-Passing

Datenmanagement

Debugging

Zoom

Projektive Ebene

Gravitationsgesetz

Metadaten

Formale Metadaten

Titel

Lessons learned with asyncio ("Look ma, I wrote a distributed hash table!")

Inhaltliche Metadaten

Nicholas Tollervey - Lessons learned with asyncio ("Look ma, I wrote a distributed hash table!") This talk introduces the asyncio module. I'll cover what it's for, how it works and describe how I used it to write a real-world networked application (a distributed hash table). We'll explore the event loop, coroutines, futures and networking with examples from my code. This won't be an exhaustive exposition. Rather, attendees will grasp enough of asyncio to continue with their own studies. By the end of this introductory talk I hope attendees will want to learn more about asyncio and perhaps give it a try in their own projects.