Author: Praveen Potluru

JSON

Case classes are Scala’s preferred way to define complex data. Below is one way to represent JSON data

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

abstractclassJSON

caseclassJSeq(elems:List[JSON])extendsJSON

caseclassJObj(bindings:Map[String,JSON])extendsJSON

caseclassJNum(num:Double)extendsJSON

caseclassJStr(str:String)extendsJSON

caseclassJBool(b:Boolean)extendsJSON

caseobjectJNullextendsJSON

objectJSONAppextendsApp{

def show(json:JSON):String=jsonmatch{

caseJSeq(elems)=>"["+(elems map show mkString",")+"]"

caseJObj(bindings)=>

val assocs=bindingsmap{

case(key,value)=>"\""+key+"\": "+show(value)

}

"{"+(assocs mkString",")+"}"

caseJNum(num)=>num.toString

caseJStr(str)=>"\""+str+"\""

caseJBool(b)=>b.toString

caseJNull=>"null"

}

val data=JObj(Map(

"firstName"->JStr("John"),

"lastName"->JStr("Smith"),

"address"->JObj(Map(

"streetAddress"->JStr("21 2nd Street"),

"state"->JStr("NY"),

"postalCode"->JNum(10021)

)),

"phoneNumbers"->JSeq(List(

JObj(Map(

"type"->JStr("home"),"number"->JStr("212 555-1234")

)),

JObj(Map(

"type"->JStr("fax"),"number"->JStr("646 555-4567")

))

))

))

println(show(data))

}

Left hand-side of a For-expression generator can also be a pattern as shown below which lists all first-name and last-name of phone#s that start with 212:

1

2

3

4

5

6

7

8

val data:List[JSON]=...

for{

JObj(bindings)<-data

JSeq(phones)=bindings(”phoneNumbers”)

JObj(phone)<-phones

JStr(digits)=phone(”number”)

ifdigits startsWith”212”

}yield(bindings(”firstName”),bindings(”lastName”))

Here JObj(bindings) acts as implicit filter

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

val books:List[Book]=List(

Book(title="Book 1"

authors=List("Author-1","Author-2")),

Book(title="Book 2"

authors=List(”Bird,Richard”,”Wadler,Phil”)),

....)

{for{

b1<-books

b2<-books

ifb1.title<b2.title

a1<-b1.authors

a2<-b2.authors

ifa1==a2

}yield a1

}.distinct

distinct is needed to remove duplicate authors who are in results list more than twice

Partial Functions

Partial Function, as opposed to Total functions, provides an answer only for a subset of possible data, and defines the data it can handle by isDefined as follows:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

val divide=newPartialFunction[Int,Int]{

def apply(x:Int)=42/x

def isDefinedAt(x:Int)=x!=0

}

scala>divide(0)

java.lang.ArithmeticException:/by zero

scala>divide.isDefinedAt(1)

res0:Boolean=true

scala>if(divide.isDefinedAt(1))divide(1)

res1:AnyVal=42

scala>divide.isDefinedAt(0)

res2:Boolean=false

More common way of writing Partial Functions is using "case" statement which gives a default implementation of isDefined:

1

2

3

4

5

6

7

8

9

10

11

12

val divide2:PartialFunction[Int,Int]={

cased:Intifd!=0=>42/d

}

scala>divide2(0)

scala.MatchError:0(of classjava.lang.Integer)

scala>divide2.isDefinedAt(0)

res0:Boolean=false

scala>divide2.isDefinedAt(1)

res1:Boolean=true

Note that our exception changed to MatchError instead of ArithmeticException when using "case".

Partial-Functions are well handled by "collect" as shown below:

1

2

3

4

5

scala>List(41,"cat")map{casei:Int⇒i+1}

scala.MatchError:cat(of classjava.lang.String)

scala>List(41,"cat")collect{casei:Int⇒i+1}

res1:List[Int]=List(42)

Magic here is that collect expects a PartialFunction and invokes the isDefined method. If we define partial-function inline, the compiler knows that it’s a partial function and you avoid explicit PartialFunction trait.

Seq,Set and Map are also Partial-functions

1

2

3

4

5

6

7

8

9

10

11

val pets=List("cat","dog","frog")

scala>pets(0)

res13:java.lang.String=cat

scala>pets(3)

java.lang.IndexOutOfBoundsException:3

scala>pets.isDefinedAt(0)

res14:Boolean=true

scala>pets.isDefinedAt(3)

res15:Boolean=false

scala>Seq(1,2,42)collect pets//safely collect values of indexes

res16:Seq[java.lang.String]=List(dog,frog)

Checking for isDefined can be painful and luckily Scala supports lift method that converts partial-function to total-function which returns an Option

We can chain together partial-functions using orElse or andThen which are defined in PartialFunction trait just like lift.

1

2

3

4

5

6

// converts 1 to "one", etc., up to 5

val convert1to5=newPartialFunction[Int,String]{....}

// converts 6 to "six", etc., up to 10

val convert6to10=newPartialFunction[Int,String]{....}

scala>val handle1to10=convert1to5 orElse convert6to10

handle1to10:PartialFunction[Int,String]=

withFilter

Scala provides another variation of filter called withFilter which doesn't create a new collection like filter does. It acts like a view that filters the result to be passed on to subsequent calls to map/flatMap etc.

Translation of for is not limited to lists or sequences or even collections.
It is based solely on the presence of the methods map, flatMap and withFilter. This lets us use for syntax for our own types as well – you must only define map, flatMap and withFilter for these types.
There are many types for which this is useful: arrays, iterators, databases, XML data, optional values, parsers, etc.

As long as client interface to the database defines the methods map, flatMap and withFilter we can use for syntax database querying.
This is the basis of Scala database connection frameworks ScalaQuery and Slick.
Similar ideas underly Microsoft’s LINQ.

Streams

Streams are similar to Lists except for their Tails are evaluated on demand.

Streams are defined from a constant Stream.empty and a constructor Stream.cons.

1

val xs=Stream.cons(1,Stream.cons(2,Stream.empty))

They can also be defined like the other collections by using the
object Stream as a factory Stream(1, 2, 3)

The toStream method on a collection will turn the collection into a stream:(1 to 1000).toStream > res0: Stream[Int] = Stream(1, ?)

Stream supports almost all methods of List except for :: which always produces a List. #:: should be used instead to produce a Stream which can be used in Expressions as well as Patterns.

To find the second prime number between 1000 and 10000:((1000 to 10000).toStream filter isPrime)(1)

x #:: xs == Stream.cons(x, xs)

Even the implementation of Streams are very close to Lists with only major difference being the use of "Call-by-name" to declare second param of cons ro filter ops as follows which causes lazy-evaluation.

I was pretty eager to try out Raspberry Pi Zero after their recent addition of Camera connector in v1.3 version. Luckily I got hold of a Raspberry Pi Zero from Adafruit last week and here goes my setup..

One pain-point in setting-up Raspberry Pi Zero is its needed accessories (mini HDMI-to-HDMI, micro USB OTG cable etc, power cable, monitor, keyboard, mouse, wifi dongle etc..also RPi Zero doesn’t come with USB hub built-in as in its elder bigger brother Rasperry Pi 3) to get it up and running. Thanks to this great article by Andrew, now we can setup RPi Zero as a USB gadget so that I can simply SSH into it from my Macbook via USB.

Once SD card is flashed with latest Raspbian Jessie Lite image (Jessie full works equally good) , open up the boot partition Finder/Windows Explorer on your Mac/PC and add to the bottom of the config.txt file dtoverlay=dwc2 on a new line, then save the file.

Open up the cmdline.txt. Be careful, each parameter is seperated by a single space (it does not use newlines). Insert modules-load=dwc2,g_ether after rootwait.

Thats it ! Now hook-up RPi zero to Macbook/PC with micro-USB to USB cable and you can ssh into as

1

ssh pi@raspberrypi.local

Setting up WiFi dongle

Edit sudo nano /etc/wpa_supplicant/wpa_supplicant.conf as follows:

1

2

3

4

5

6

7

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev

update_config=1

network={

ssid="SSID-here"

psk="password-here"

}

Mounting USB drives

This blog lists key points I noticed while trying to shift gears to Functional Programming using Scala JVM language coming from a traditional Object-oriented programming background for over a decade. I’ve extensively used Coursera’s Scala course and online documentation/blogs to come-up with this content.

What is Functional programming

Mainstream languages like C, C++, Java etc were based on the idea of update in place. They presuppose that the way to solve a programming problem is to have procedures that read and write values to memory locations, be it directly visible to the programmer (by means of pointers) or not.

Other languages, like Lisp, are based on a different assumption: a computation is best expressed as a set of functions over immutable values. So, in order to produce any kind of repetitive computation, recursion is essential. This assumption has strong roots in lambda calculus and related mathematical formalisms.

Function evaluation strategies

Scala function can be evaluated either Call-by-name or Call-by-value. Call-by-value has the advantage that it evaluates every function argument only once. Call-by-name has the advantage that a function argument is not evaluated if the corresponding parameter is unused in the evaluation of the function body.

Scala uses Call-by-value generally, unless the type of function parameter starts with ‘=>’ in which Call-by-name is used.

Tail-recursion

Recursion has been looked-down upon in Imperative style programming but its the basic building block for Functional Programming. Stack-overflow issue still exists if you have too many recursive calls and thats where Tail-recursion comes into play. We can convert any Recursive function into Tail-recursion by using an intermediate function containing an Aggregator. This lets Compiler use the same stack-space for all recursive method calls instead of creating a new one for each Recursive call. Below is an example of factorial definition.

Higher-order functions and Currying

Functions are first-class citizens which gets treated just like any primitive types, i.e. you can pass it as a parameter to a function, return it from a function and define a var to hold a function.

Currying is when you break down a function that takes multiple arguments into a series of functions that take part of the arguments. Advantage of this is that it offers great amount of expressiveness and reusability to a function. Function arguments associate to the left in context of Currying. Basically, this lets you express your solution just as if you you write a mathematical equation.

We’ll develop motivation for Currying step-by-step:

1

2

3

4

5

6

7

def sum(f:Int=>Int,a:Int,b:Int)={

def loop(a:Int,acc:Int):Int=

if(a>b)acc

elseloop(a+1,f(a)+acc)

loop(a,0)

}

sum(x=>x *x,3,5)

1

2

3

4

5

def sumInts(a:Int,b:Int)=sum(x=>x,a,b)

def sumCubes(a:Int,b:Int)=sum(x=>x*x*x,a,b)

def sumFactorials(a:Int,b:Int)=sum(fact,a,b)

Since a and b are getting passed unchanged from sumInts and sumCubes into sum, we can get their definitions even shorter by getting rid of params as follows:

1

2

3

4

5

6

def sum(f:Int=>Int):(Int,Int)=>Int={

def sumF(a:Int,b:Int):Int=

if(a>b)0

elsef(a)+sumF(a+1,b)

sumF

}

Here sum is now a function that returns another function. Returned functions sumF applies the given function parameter f and sums the results. We can now define earlier 3 functions as follows:

1

2

3

4

5

def sumInts=sum(x=>x)

def sumCubes=sum(x=>x*x*x)

def sumFactorials=sum(fact)

These functions can in turn be applied like any other function:

1

sumCubes(1,10)+sumFactorials(10,20)

In above example, we can avoid sumInts, sumCubes & sumFactorials middlemen as follows:

sum (cube) (1,10)

sum(cube) applies sum to cube and returns ‘sum of cubes’ function

sum(cube) is therefore equivalent to sumCubes

This function is next applied to arguments (1,10)

Generally, function application associates to the left:

sum (cube)(1,10) == (sum (cube)) (1,10)

The definition of functions that return functions is so useful in functional programming that there is a special syntax for it in Scala. For example, the following definition of sum is equivalent to the one with the nested sumF function, but shorter:

1

2

3

def sum(f:Int=>Int)(a:Int,b:Int):Int=

if(a>b)0elsef(a)+sum(f)(a+1,b)

Thus Currying resembles the process of evaluating a function of multiple variables, when done by hand, on paper as follows:

For example, given the function f(x,y)=y/x

To evaluate f(2,3), first replace x with 2

Since the result is a function of y, this new function g(y) can be defined as g(y)=f(2,y)=y/2

Next, replace the y argument with 3, producing g(3)=f(2,3)=3/2

On paper, using classical notation, this is usually done all in one step. However, each argument can be replaced sequentially as well. Each replacement results in a function taking exactly one argument. Below is an example where we can use wildcard as a placeholder for unknown param that’ll be provided later:

1

2

3

4

5

6

7

8

scala&gt;def concat(w1:String)(w2:String)=w1+" "+w2

concat:(w1:String)(w2:String)String

scala&gt;val hello=concat("hello")_

hello:String=&gt;String=

scala&gt;hello("world")

res0:String=hello world

Example: MapReduce

Below is screenshot of IDE where product function is implemented with Currying principles. mapReduce generalizes this further more by supporting any operation like sum, product etc.

With above context, we can re-implement product in terms of mapReduce as follows:

We are not instantiating Trait directly, but rather creating a suitable object for the Trait to attach itself to so you can use the Trait’s functionality without needing to define a class that extends the Trait.

“AnyRef” is just alias of java.lang.Object and is the base class of all reference types

“AnyVal” is base type of all primitive types

“Nothing” is sub-type of every other type. It’s used to signal abnormal termination (for eg. ‘throw Exc’ expressionn is used to abort evaluation and its type is Nothing) and, also as element type of empty collections as in Set[Nothing]

“Null” is subtype of every class/ref inherited from Object/Any thus incompatible with AnyVal. Every reference class type also has null as a value.

As in Java, Polymorphism is achieved by both sub-classing and generics. Below is an example:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

traitList[T]{

def isEmpty:Boolean

def head:T

def tail:Lis[T]

}

classCons[T](val head:T,val tail:List[T])extendsList[T]{

def isEmpty=false

}

classNil[T]extendsList[T]{

defisEmpty=true

def head=thrownewNoSuchElementException("Nil.head")

def tail=thrownewNoSuchElementException("Nil.tail")

}

def singleton[T](elem:T)=newCons[T](elem,newNil[T])

Note that:

Type parameters doesn’t effect Scala during runtime as it uses Type-erasure(type info is used only by compiler and not preserved for runtime)

If we define a method param as val, it implicitly defines a val with same name in method definitionand assigns method param to it.

Generics in methods: singleton method can be invoked as follows with the type info:

1

2

singleton[Int](1)

singleton[Boolean](true)

Alternatively, we can let Scala deduce type info from the context thus letting us invoke singleton as follows:

1

2

singleton(1)

singleton(true)

Above code creates what is called as Immutable linked list. This is fundamental to Functional programming which is constructed using 2 basic elements:
Nil: the empty list
Cons: cell containing an element and remainder of listBelow is graphical representation of Immutable linked lists:

Functions as Objects

Function-types relates to Classes and function value relate to objects. Functions type A => B is just abbreviation for scala.Function1[A,B] which is define as follows:

1

2

3

4

packagescala

trait Function1[A,B]{

def apply(x:A):B

}

There are also traits with Function2, Function3 etc to support upto 22 params.

Anonymous function such as (x: Int) => x * x would be expanded as follows:

1

2

3

4

classAnonFun extendsFunction1[Int,Int]{

def apply(x:Int)=x *x

}

newAnonFun

This in turn gets converted to as follows using Java anonymous class syntax:

1

2

3

newFunction1[Int,Int]{

def apply(x:Int)=x*x

}

Function calls such as f(a,b) is expanded to f.apply(a,b). For eg:

1

2

valf=(x:Int)=>x*x

f(7)

would be translated to:

1

2

3

4

valf=newFunction1[Int,Int]{

def apply(x:Int)=x*x

}

f.apply(7)

ETA Expansion

Note that method such as def f(x: Int): Boolean = ... is not a function value as it ends in infinite expansion. But when name of method is used in place where function type is expected, then its converted to function value (x: Int) => f(x) which gets expanded as below. This is known as ETA expansion :

In summary, functions and methods are not the same in Scala (although most of the cases we can ignore this difference). A Scala method, as in Java, is a part of class whereas Function is a complete object (instance of class that implement one the Traits Function0,Function1,Function3..)

Pattern matching

Pattern matching provides a functional way of object decomposition. A case class definition adds feature to pattern match on classes. This modifier adds the benefits of

‘==’: does a structural equality comparison, i.e two case class instances with same params are equal

toString: method that gives string representation of class members

copy(): we can copy whole of a case class instance values into a new instance.

all params will be declared with ‘val’ by default. Note that if you use currying fashion, only the first param(s) will be provided with ‘val’

Since the concrete sub-class has no body, we will be using Pattern matching using ‘match’ keyword.

For eg,

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

trait Expr

//declare case classes

caseclassNumber(n:Int)extendsExpr

caseclassSum(e1:Expr,e2:Expr)extendsExpr

//This internally declares Number and Sum singleton objects that

//lets us simply invoke Number(2) and Sum(2,3)

// object Number { def apply(n: Int) = new Number(n) }

// object Sum { def apply(e1: Expr, e2: Expr) = new Sum(e1,e2)

//provide implementation for case classes

def eval(e:Expr):Int=ematch{

caseNumber(n)=>n

caseSum(e1,e2)=>eval(e1)+eval(e2)

}

Note match statement execution is sequential as it executes all cases starting from top to bottom one-by-one and skips after a matching case.

A good side-effect of such a matching is that it provides an alternative way to design functions. For example, consider the factorial function. If you choose the recursive version, usually, you would define it like this:

1

2

3

def fact(n:Int):Int=

if(n==0)1

elsen *fact(n-1)

Where-as a Pattern matching based solution is:

1

2

3

4

def fact(n:Int):Int=nmatch{

case0=>1

casen=>n *fact(n-1)

}

Lists

Lists are immutable and are recursive whereas Arrays are mutable and flat.

val l = List(1,2,3) is similar to using Cons (short form for constructor) operator as 1::2::3::Nil.
Note that operators ending in ‘:’ are right associative and thus above can be translated as Nil.::3.::2.::1

All operations on lists can be expressed in terms of ‘head’, ‘tail’ and ‘isEmpty’

Lists can also be used in pattern matching as follows:1 :: 2 :: xs Lists starting with 1 and then 2x :: Nil or List(x) Lists with length 1List() or Nil Empty listList(2 :: xs) List containing only element as another list that start with 2

Below is an example of a typical way to decompose lists with pattern-matching using Insertion sort as an example:

1

2

3

4

def isort(xs:List[Int]):List[Int]=xsmatch{

caseList()=>List()//check if list is empty

casey::ys=>insert(y,isort(ys))//if not empty, decompose list into its head and tail and then process

When is a List preferred to a Vector ?

List: if our operations involve having a Head and Tail of a sequence as these are done at constant-time with Lists whereas its complicated with a Vector

Vector: when we need bulk operations like Map, Filter, Fold

Vectors are created analogous to Lists including its operations. Only exception is we’ll replace :: with :+. x +: xs creates new Vector with leading element x followed by elements of xs and +: creates new Vector with trailing element x preceded by all elements of xs.

This blog describes how to stream video + audio to Youtube Live events using Raspberry Pi. There are lot of sites online that describe how to Live stream a webcam video to Youtube with Raspberry Pi but no where I found working solution for live streaming both Audio & Video to Youtube. Hence this blog..

Bit of background:

I work for a non-profit organization that needed a free 24/7 webcam streaming. They were using VLC in tandem with Quicktime broadcasting for last 10 yrs that worked great until the recent advent of new HTML5 and Mobile platforms. It became obsolete especially due to its need for Quicktime browser plugin and lack of support for Mobile platforms.

After some research, Youtube Live events came out to be the only free streaming solution. Initially I tried using Adobe Flash Live encoder to live stream to Youtube but its a User-interface driven tool and not a fire-and-forget type solution. This where Raspberry Pi fit the job.

Keyboard, Mouse, HDMI cable, Monitor. This is optional and only needed until you setup SSH. Alternatively, you can use this cool PiBakery project to burn SSH and WiFi info directly into your SD card thus avoiding needs for initial peripherals to boot up RPi.

You need to install the ffmpeg onto your RPi in order use above script. I’ve detailed its setup in my blog here.

For some reason, Audio starts falling behind the Video after about 20 mins. To give better user experience, I setup a Cron job to re-start above ffmpeg script code every 20 mins. Since Youtube does DVR of your live streaming, users won’t notice any difference.