Sessions at Erlang Factory SF Bay Area 2010 with video

Unscheduled

Most developers will be familiar with lex, flex, yacc, bison, ANTLR, and other related tools to generate parsers for use inside their own code. Erlang has its own pair, leex and yecc, for accomplishing most complicated text-processing tasks. For computer-friendly languages, however, context-free grammars and their parser-generators leave a few things to be desired. This talk is about how the seemingly simple prospect of parsing some text turned into a new parser toolkit for Erlang called Neotoma, and why functional programming makes parsing fun and awesome.

For a long time, and due to the lack of main FP concepts in most mainstream languages, we missed opportunities to abstraction and code expressiveness and conciseness. With today’s democratization of FP, Computational Abstraction is what will enable us to be less dependent on specific programming language syntax offering; creating libraries of control structures and composition forms that help find concise and expressive solutions to enterprise programming challenges (null, lists treatment, error handling), capturing elegantly important business concepts in code, and programming at the right level of abstraction. For a long time, and due to the lack of main FP concepts in most mainstream languages, we missed opportunities to abstraction and code expressiveness and conciseness. With today’s democratization of FP, Computational Abstraction is what will enable us to be less dependent on specific programming language syntax offering; creating libraries of control structures and composition forms that help find concise and expressive solutions to enterprise programming challenges (null, lists treatment, error handling), capturing elegantly important business concepts in code, and programming at the right level of abstraction.

While Erlang/OTP provides a rich set of abstractions and tools for developing distributed applications, the operation of a distributed Erlang system at scale presents many challenges and pitfalls that developers should be aware of at all phases of application development. In this talk, Andy will present a number of techniques for building operationally friendly Erlang systems that effectively use the OTP libraries, based on real-world experience at Mochi Media and Basho Technologies.

Erlang is great for implementing network protocols, so let's write a TCPIP stack top to bottom. The Enet library includes a port program for accessing an OS TAP device, a suite of protocol encoders and decoders and a network interface module that correctly responds to Ping.

This talk covers a simple port program, How the binary syntax saved my hard-drive, tcpdump in Erlang, grandiose ambition, and the highest ping times you've ever seen over a loopback adapter.

Over the last few years, I have been meeting "Erlang people" more and more often, and I was getting this clear impression that "you people" have some kind of magic ability to reason intuitively about concurrent systems in a way that I could not. That bothered me, so I wanted to learn Erlang. Being a language implementer, the most obvious way to do that is to just go ahead and implement an Erlang VM, right?

The result of this "little exercise" is Erjang, an open-source JVM-based Erlang VM, which I think has potential to be useful as a means to push Erlang into new areas of application, help grow the Erlang community, and most importantly make all those Java-heads out there willing to learn how to do concurrency right. In this "brave new world" of networked services and multi-core everywhere, there are a lot of people who are challenged in providing even basic reliability, efficiency and correctness. My hope is that Erjang's existence can influence some of those outside the Erlang community to do better.

In technical terms, Erjang reads .beam files and compiles them to Java's equivalent .class files which are then read into the running JVM. It runs off a plain Erlang/OTP distribution - it only requires the beam files from there; Erjang itself is written in Java. As off this writing, it can run some non-trivial erlang programs, but is not yet capable of booting OTP [will have to update this abstract as we go along - follow updates on my blog]. Comparing the BEAM virtual machine and Erjang, the most obvious differences are that (a) Erjang will not be able to provide [soft] real-time guarantees since it uses Java's garbage collector, and (b) it has limited support for native code and port drivers (other than file and network I/O). The upside is the new ways this allows us to deploy Erlang systems.

In this presentation I will talk about how Erjang works, what behaviour to expect from Erlang programs running on top of Erjang, what I learned along the way, and demonstrate [live] that Erjang runs well enough to be obviously useful.

Collecta is a real-time search engine that allows users and developers to access enormous of amounts of information at extremely low latencies. Our system is almost entirely Erlang powered, making use of ejabberd, Webmachine, CouchDB, Riak, RabbitMQ, as well as our own Erlang projects, and is deployed over more than a hundred machines.

I will talk about why Collecta choose to focus our development on the Erlang language and why our problem domain is well suited to Erlang's strengths. Search encompasses a large number of problems, but we find that Erlang excels at solving many of them. I'll also describe our general architecture as well as the reasoning behind several of our technology choices.

Remote Procedure Call (RPC) over public Internet has to be cryptographically protected for secure communication. In the current Distributed Erlang, this cannot be done in a firewall-friendly way.

This presentation provides an introduction of writing code using Secure Shell (SSH) protocol on Erlang with Erlang/OTP ssh module, and a description of an RPC method based on SSH subsystem implemented on Erlang.

We have developed our ProcessOne our own interoperable version of Google Wave server. It supports a large subset of the required feature (From Wave operational transformation engine to Wave storage). The default client protocol is XMPP, meaning our Wave server can work as an ejabberd XMPP server extension and can be use directly inside XMPP client.This talk will present the Wave protocol, our Wave server, developed in Erlang, and its architecture.

While it is not a fast language, for many purposes Erlang is considered "fast enough". But what happens when "fast enough" isn't fast enough? You don't have to rewrite your whole project in C or C++. This talk will cover the exercise of profiling an Erlang application. You will learn how to replace critical code sections with C functions using the new NIF (Native Implemented Function) interface. You will learn how to integrate a native 3rd party library using the linked in driver interface. And finally you will learn how to protect C drivers from race conditions in Erlang's highly concurrent environment and how to debug them in a running Erlang VM.

The talk will focus on how I (with the help of the entire Cloudant team) built our database service based on CouchDB on top of EC2. Specifically how we use Erlang, Chef, EC2 and other tools to build highly available and performant database clusters. This includes using Chef and Erlang's hot code upgrades to automate cluster-wide upgrades without restarting any services.

The main topics will include:

1. Getting high availability in the cloud
- How to think about and work with the cloud (and infrastructure you don't control)
- Expect and prepare for failure
- When EBS devices and EC2 instances go bad
- Cooking high availability into your system
- Removing single points of failure

2. Pitfalls we faced and how to work around them
- Performance
- Reliability

3. Automation and configuration management
- Including how we use Erlang and Chef to automate hot upgrades

Tracing and debugging large systems still remains a challenging and sometimes daunting task. There are few tools at hand, and even if some data is collected the sheer volume and hours required to analyse it makes one wince. However, there's a hidden gem in OTP which many developers and testers may not be aware of - the Inviso application, providing support for setting up and collecting real-time tracing in multi-node environments. Although still a work in progress, we've made an effort to provide a clean, functional and safe API to it. Together with a friendlier user-interface it is getting ready to assist developers who are building massively distributed systems.

But on-line tracing is one thing; post-mortem debugging is another. As a developer or support engineer you have probably spent enormous energy and time manually analyzing log files, trying to find out what went wrong, when and why.

Using Exago, now you can automatically parse and process log files, and check them against an abstract model of the system. In case of failure, it will report the abstract state where the error occurred, and the events that led to the point of failure.

In brief, this tool automates most of the difficult and tiresome task of manually inspecting log files, tasks that you would probably pass to the interns.

I'll discuss the projects goals, adoption, and inner workings of the popular document database. I'll show some JavaScript code, for those of you who'd like to Relax and learn to write offline capable, peer replicating CouchApps.

There is growing interest in embedded languages inside the Erlang platform, from erlang_js to erl-lua. As a fan of embedded languages, Chad's talk will explore the process of integrating Lua with Erlang as a built-in driver; calling into Lua from Erlang and vice versa. We will do this through exploration of the creation of a distributed UNIX systems management utility created in Erlang and Lua.

Most frameworks for process, file, and server monitoring treat distributed monitoring across multiple systems as an afterthought. Erlmon is built from the beginning to allow distributed monitoring of UNIX systems, all in Erlang. But the average system administrator doesn't want to learn Erlang syntax to monitor services in the network, which is where Lua comes in. We'll explore the creation of the erlmon utility, with a focus on the integration of the configuration system in Lua, and we'll discuss a case study of a real-world system using this monitoring tool.

Git is the distributed version control system that's taking the world by storm. The Linux kernel, Ruby on Rails, and now even Erlang/OTP have all chosen it as their VCS of choice. This talk will focus on mastering the handful of commands that comprise 90% of the functionality you will use on a day-to-day basis. I'll kick things off with the best method to install Git on your favorite platform. Next up is creating local repositories, staging/adding files (all about the index), making commits, and viewing history. From there you'll learn to share your repositories with others via GitHub, fetch and merge changes that others make, and resolve conflicts. I'll also introduce a few common workflows that you can use to streamline collaborative development. Each step of the way I'll demonstrate the commands and explain what's happening under the hood to help you establish a proper mental model that will enable you to master the more complex parts of Git with ease. Even if you already know Git, there should be something new for you in this talk. By the end of the session you will be well on your way to using Git to make your life easier than you ever thought possible!

Riak Search is a distributed data indexing and search platform built on top of Riak. The talk will introduce Riak Search, covering overall goals, architecture, and core functionality, with specific focus on how Erlang is used to manage and execute an ever-changing population of ad hoc query processes.

Scala is a statically-typed, hybrid functional and object-oriented language for the JVM. The Scala standard library includes an Erlang- inspired Actors library. In this talk, I'll discuss how Scala compares and contrasts to Erlang, highlighting the advantages and disadvantages of each language for particular needs. For example, we'll discuss the pros and cons of a rich type system and static typing in Scala. We'll discuss ways that Scala is perhaps more general purpose than Erlang, but not as powerful in the areas where Erlang excels.

Erlang combines ideas from functional programming, logic programming, distributed systems construction, operating systems design and packages these into a platform that can be used for building fault-tolerant scalable systems.

But where did these ideas come from? Which are the most important ideas?

This talk presents a personal view of what I think are the important ideas in Erlang. I'll talk about where the ideas came from and how they developed and changed with time.

I'll also talk about the failed ideas - the ideas we had that lived for a short while, even implemented, and then were removed from the language. The failed ideas are interesting because they often seem to be superficially correct, but on deeper inspection turn out to be wrong.

I'll talk a little bit about some of the ideas which are "good but not yet implemented" usually these represent ideas where behind the scenes their lurks a horribly difficult implementation problem.