Big Data is a new field that is growing every day. HTML5 and JavaScript applications are being used to showcase these large volumes of data in many new interesting ways. Some of the latest client implementations are being accomplished with libraries such as AngularJS. This is because of its ability to efficiently handle and organize data in many forms.

Making business-level decisions off of real-time data is a revolutionary concept. Humans have only been able to fathom metrics based off of large-scale systems, in real time, for the last decade at most. During this time, the technology to collect large amounts of data has grown tremendously, but the high-level applications that use this data are only just catching up.

Anyone can collect large amounts of data with today's complex distributed systems. Displaying this data in different formats that allow for any level of user to digest and understand its meaning is currently the main portion of what the leading-edge technology is trying to accomplish. There are so many different formats that raw data can be displayed in. The trick is to figure out the most efficient ways to showcase patterns and trends, which allow for more accurate business-level decisions to be made.

We live in a fast paced world where everyone wants something done in real time. Load times must be in milliseconds, new features are requested daily, and deadlines get shorter and shorter. The Web gives companies the ability to generate revenue off a completely new market and AngularJS is on the leading edge. This new market creates many new requirements for HTML5 applications. JavaScript applications are becoming commonplace in major companies. These companies are using JavaScript to showcase many different types of data from inward to outward facing products.

Working with live data sets in client-side applications is a common practice and is the real world standard. Most of the applications today use some type of live data to accomplish some given set of tasks. These tasks rely on this data to render views that the user can visualize and interact with. There are many advantages of working with the Web for data visualization, and we are going to showcase how these tie into an AngularJS application.

AngularJS offers different methods to accomplish a view that is in charge of elegantly displaying large amounts of data in very flexible and snappy formats. Some of these different methods feed directives' data that has been requested and resolved, while others allow the directive to maintain control of the requests. We will go over these different techniques of how to efficiently get live data into the view layer by creating different real-world examples. We will also go over how to properly test directives that rely on live data to achieve their view successfully.

Techniques that drive directives

Most standard data requirements for a modern application involve an entire view that depends on a set of data. This data should be dependent on the current state of the application. The state can be determined in different ways. A common tactic is to build URLs that replicate a snapshot of the application's state. This can be done with a combination of URL paths and parameters.

URL paths and parameters are what you will commonly see change when you visit a website and start clicking around. An AngularJS application is made up of different route configurations that use the URL to determine which action to take. Each configuration will have an associated controller, template, and other forms of options. These configurations work in unison to get data into the application in the most efficient ways.

AngularUI also offers its own routing system. This UI-Router is a simple system built on complex concepts, which allows nested views to be controlled by different state options. This concept yields the same result as ngRoute, which is to get data into the controller; however, UI-Router does it in a more eloquent way, which creates more options. AngularJS 2.0 will contain a hybrid router that utilizes the best of each.

Once the controller gets the data, it feeds the retrieved data to the template views. The template is what holds the directives that are created to perform the view layer functionality. The controller feeds directives' data, which forces the directives to rely on the controllers to be in charge of the said data. This data can either be fed immediately after the route configurations are executed or the application can wait for the data to be resolved.

AngularJS offers you the ability to make sure that data requests have been successfully accomplished before any controller logic is executed. The method is called resolving data, and it is utilized by adding the resolve functions to the route configurations. This allows you to write the business logic in the controller in a synchronous manner, without having to write callbacks, which can be counter-intuitive.

The XHR extensions of AngularJS are built using promise objects. These promise objects are basically a way to ensure that data has been successfully retrieved or to verify whether an error has occurred. Since JavaScript embraces callbacks at the core, there are many points of failure with respect to timing issues of when data is ready to be worked with. This is where libraries such as the Q library come into play. The promise object allows the execution thread to resemble a more synchronous flow, which reduces complexity and increases readability.

The $q library

The $q factory is a lite instantiation of the formally accepted Q library (https://github.com/kriskowal/q). This lite package contains only the functions that are needed to defer JavaScript callbacks asynchronously, based on the specifications provided by the Q library. The benefits of using this object are immense, when working with live data.

Basically, the $q library allows a JavaScript application to mimic synchronous behavior when dealing with asynchronous data requests or methods that are not thread blocked by nature. This means that we can now successfully write our application's logic in a way that follows a synchronous flow.

ES6 (ECMAScript6) incorporates promises at its core. This will eventually alleviate the need, for many functions inside the $q library or the entire library itself, in AngularJS 2.0.

The core AngularJS service that is related to CRUD operations is called $http. This service uses the $q library internally to allow the powers of promises to be used anywhere a data request is made. Here is an example of a service that uses the $q object in order to create an easy way to resolve data in a controller. Refer to the following code:

Here, we can see that the phoneService function uses the $http service, which can request for all the phones. The phoneService function creates a new request object, that calls a then function that returns a promise object. This promise object is returned synchronously. Once the data is ready, the then function is called and the correct data response is returned.

This service is best showcased correctly when used in conjunction with a resolve function that feeds data into a controller. The resolve function will accept the promise object being returned and will only allow the controller to be executed once all of the phones have been resolved or rejected.

The rest of the code that is needed for this example is the application's configuration code. The config process is executed on the initialization of the application. This is where the resolve function is supposed to be implemented. Refer to the following code:

Directives take over once the controller executes its initial context. This is where the $compile function goes through all of its stages and links directives to the controller's template. The controller will still be in charge of driving the data that is sitting inside the template view. This is why it is important for directives to know what to do when their data changes.

How should data be watched for changes?

Most directives are on a need-to-know basis about the details of how they receive the data that is in charge of their view. This is a separation of logic that reduces cyclomatic complexity in an application. The controllers should be in charge of requesting data and passing this data to directives, through their associated $scope object.

Directives should be in charge of creating DOM based on what data they receive and when the data changes. There are an infinite number of possibilities that a directive can try to achieve once it receives its data. Our goal is to showcase how to watch live data for changes and how to make sure that this works at scale so that our directives have the opportunity to fulfill their specific tasks.

There are three built-in ways to watch data in AngularJS. Directives use the following methods to carry out specific tasks based on the different conditions set in the source of the program:

Watching an object's identity for changes

Recursively watching all of the object's properties for changes

Watching just the top level of an object's properties for changes

Each of these methods has its own specific purpose. The first method can be used if the variable that is being watched is a primitive type. The second type of method is used for deep comparisons between objects. The third type is used to do a shallow watch on an array of any type or just on a normal object.

Let's look at an example that shows the last two watcher types. This example is going to use jsPerf to showcase our logic. We are leaving the first watcher out because it only watches primitive types and we will be watching many objects for different levels of equality.

This example sets the $scope variable in the app's run function because we want to make sure that the jsPerf test resets each data set upon initialization. Refer to the following code:

This run function sets up our data object that we will watch for changes. This will be constant throughout every test we run and will reset back to this form at the beginning of each test.

Doing a deep watch on $rootScope.data

This watch function will do a deep watch on the data object. The true flag is the key to setting off a deep watch. The purpose of a deep comparison is to go through every object property and compare it for changes on every digest. This is an expensive function and should be used only when necessary. Refer to the following code:

Doing a shallow watch on $rootScope.data

The shallow watch is called whenever a top-level object is changed in the data object. This is less expensive because the application does not have to traverse n levels of data. Refer to the following code:

During each individual test, we get each watcher service and call its run function. This fires the watcher on initialization, and then we push another test object to the data array, which fires the watch's trigger function again. That is the end of the test. We are using jsperf.com to show the results. Note that the watchCollection function is much faster and should be used in cases where it is acceptable to shallow watch an object. The example can be found at http://jsperf.com/watchcollection-vs-watch/5. Refer to the following screenshot:

This test implies that the watchCollection function is a better choice to watch an array of objects that can be shallow watched for changes. This test is also true for an array of strings, integers, or floats.

This brings up more interesting points, such as the following:

Does our directive depend on a deep watch of the data?

Do we want to use the $watch function, even though it is slow and memory taxing?

Is it possible to use the $watch function if we are using large data objects?

The directives that have been used in this book have used the watch function to watch data directly, but there are other methods to update the view if our directives depend on deep watchers and very large data sets.

Directives can be in charge

There are some libraries that believe that elements can be in charge of when they should request data. Polymer (http://www.polymer-project.org/) is a JavaScript library that allows DOM elements to control how data is requested, in a declarative format. This is a slight shift from the processes that have been covered so far in this article, when thinking about what directives are meant for and how they should receive data. Let's come up with an actual use case that could possibly allow this type of behavior.

Let's consider a page that has many widgets on it. A widget is a directive that needs a set of large data objects to render its view. To be more specific, lets say we want to show a catalog of phones. Each phone has a very large amount of data associated with it, and we want to display this data in a very clean simple way.

Since watching large data sets can be very expensive, what will allow directives to always have the data they require, depending on the state of the application? One option is to not use the controller to resolve the Big Data and inject it into a directive, but rather to use the controller to request for directive configurations that tell the directive to request certain data objects. Some people would say this goes against normal conventions, but I say it's necessary when dealing with many widgets in the same view, which individually deal with large amounts of data.

This method of using directives to determine when data requests should be made is only suggested if many widgets on a page depend on large data sets.

To create this in a real-life example, let's take the phoneService function, which was created earlier, and add a new method to it called getPhone. Refer to the following code:

this.getPhone = function(config) {
return $http.get(config.url);
};

Now, instead of requesting for all the details on the initial call, the original getPhones method only needs to return phone objects with a name and id value. This will allow the application to request the details on demand. To do this, we do not need to alter the getPhones method that was created earlier. We only need to alter the data that is supplied when the request is made.

It should be noted that any directive that is requesting data should be tested to prove that it is requesting the correct data at the right time.

Testing directives that control data

Since the controller is usually in charge of how data is incorporated into the view, many directives do not have to be coupled with logic related to how that data is retrieved. Keeping things separate is always good and is encouraged, but in some cases, it is necessary that directives and XHR logic be used together. When these use cases reveal themselves in production, it is important to test them properly.

The tests in the book use two very generic steps to prove business logic. These steps are as follows:

Create, compile, and link DOM to the AngularJS digest cycle

Test scope variables and DOM interactions for correct outputs

Now, we will add one more step to the process. This step will lie in the middle of the two steps. The new step is as follows:

Make sure all data communication is fired correctly

AngularJS makes it very simple to allow additional resource related logic. This is because they have a built-in backend service mock, which allows many different ways to create fake endpoints that return structured data. The service is called $httpBackend.

Testing bbPhoneDetails

To showcase how to use $httpBackend, we have created tests for the bbPhoneDetails directive. The bbPhoneDetails directive makes requests for its own information. This information could be very large, which means special precautions need to be taken when requesting for many phones on the same page. This potentially large data is being separated by individual requests for each individual directive.

The bbPhoneDetails directive has a small set of requirements. Refer to the following requirements:

Ability to request for data based on a configuration object

If this configuration object changes, then request for new information

Handle all error cases, with regards to requests, correctly

To write tests that prove these requirements, we start by creating a simple describe block that contains all of the services we will need. This describe block also contains our first look at how to use the $httpBackend service.

The preceding described block is the parent-level closure that will hold all of our tests. The first beforeEach block calls the bbPhoneListApp module to inject its context into the scope of the test. The second beforeEach block contains the most important piece of code, with regards to how bbPhoneDetails accomplishes its requirements. This is the $httpBackend service. The third beforeEach clause defines the link functions that are used, which contain the compiled directives.

In the following tests, we will be using the bbPhoneDetails directive to make different requests. These different requests will expect different responses depending upon the actual request. To accomplish this functionality, we are using the whenGET method provided by the $httpBackend service. This method takes a string as a variable that will match a request that can be made in an it clause. If a match is made, then it will respond with the specified data, which will serve as the constant that will prove our tests are successful.

it('should contain the correct scope parameters based upon theconfiguration file', function(){
successPhoneLinkedDOM = successPhoneLinkFn(scope);
//apply needed, because the directive is watching the config for changes.
//the directive's watch would never fire if this apply was not present.
scope.$apply();
//flush function will execute the $httpBackend functions that have
//successfully matched. This will throw an error if nothing
//matches.
$httpBackend.flush();
var phoneScope = successPhoneLinkedDOM.isolateScope();
expect(phoneScope.phone.age).toBe(1);
expect(phoneScope.phone.id).toBe("xxx-xxx-xxxx");
expect(phoneScope.phone.imageUrl).toBe("testPhone.jpg");
expect(phoneScope.phone.name).toBe("Amazing Phone");
expect(phoneScope.phone.snippet).toBe("This is a Super Duper Phone");
});

This test proves that the bbPhoneDetails directive is making the correct requests based on the configuration object it is using as input. Since we are using the flush() function, provided by $httpBackend, we can show that specific requests are being made. This is because all requests that are made during the execution of this test will not execute their respond function until the flush() function is fired. In this specific case, the directive requests details for the test-phone.json file. The data is retrieved and used correctly inside the directive, and this is proven by testing each attribute of the phone for accuracy.

The next test case that we will write will be for the second logical pathway that this directive could use, depending on how the request it makes plays out. This will be the error scenario that could take place if a file was requested for, which did not exist. Refer to the following test case:

The error scenario is proven by using the $httpBackend service to return a 404 error code when a specific request is made. The directive should handle this error and any other error correctly.

Since we are using the AngularJS promise system inside of the directive, we can ensure that if a 404 error is handled correctly, then all other error scenarios will work.

The last requirement that must be met occurs when the directive's configuration object changes in any way. This should force a new request that will subsequently update the directive's scope objects with the new data. Refer to the following test case:

This last test proves that the directive updates itself and makes new requests whenever its configuration object changes. The test is essentially the first test minus any expect functions plus more logic that changes configObj.url and checks to make sure that the updates were made correctly. The key to these tests is to make sure to call flush(), so that all the pending requests can be processed accordingly.

Now that the requirements have been laid out correctly, let's move on to writing the directive in order to see how we can actually make these tests pass.

Writing the bbPhoneDetails directive

Now, we can create a directive that uses an isolated scope and takes a configuration object that can be watched for changes. Any time that the configuration object changes, we can go ahead and make a request for the large data set. Refer to the following code:

The link function used by bbPhoneDetails watches its configuration object for changes. Once it makes the change, it then calls the getPhone method provided by the phoneService function. This method returns a promise object, which is created by the $http service. This promise object is then resolved once the data has either been requested successfully or an error has occurred. The final result of all the directives working in unison is the same as when all the data is requested at once. The difference is that now the data requests can be scaled in manageable sections. A live example can be found at http://plnkr.co/edit/mBm7zjGIrBt6GLn9Mo0r?p=preview.

Remember, the objective of this article is to show how directives do not have to watch huge amounts of data. This is about scaling; even though the demo does not have a large amount of data being used, it does take this possibility into consideration. In some cases, separating the XHR requests into smaller chunks is faster than requesting for all of the data at once and helps create faster digest cycles. This also yields a better user experience, as the user does not have to wait for all of the data before the page finally loads. Now, they can see a portion of the page with lite details, and then they can be shown some type of loading message that the details are being obtained for each widget.

Working with D3

There are many external libraries that are meant for graphing and displaying large amounts of data in a nice consolidated, organized view. Some examples of libraries are flot.js, datatables.js, heatmap.js, and many more. We are going to describe how to work with one of the most popular visualization libraries.

D3 is very popular and has many different features. AngularJS and D3 work wonderfully together if used correctly. D3 has wonderful view-level techniques built inside that work perfectly with the data-binding abilities of AngularJS.

AngularJS watches the model for changes and calls functions depending on whether the data has changed. The directives written in this article will focus on calling the D3 function to perform most of the very intrinsic DOM alterations. The unique portion will be how these directives are tested for accuracy in an AngularJS context and how they know when to call their D3 related functions.

We are going to introduce Protractor to test the D3 directives. Since all of the directives created in this article will require some type of live data feed, we are going to showcase how to get this real data into the view while testing.

Protractor is an End-2-End testing framework built on top of the Selenium driver to allow AngularJS applications a simple way to test bindings and check whether interactions are working as expected. More details can be found at https://github.com/angular/protractor.

The YouTube views bar chart

In the Black Belt demo, there is a media element page. This media element page has a video player that is fed by a bootstrap typeahead directive. The typeahead directive calls a function in the controller that hits YouTube's API upon every keystroke. The returned data is a set of data objects that contain information about videos, in relation to what was typed into the typeahead directive.

Let's build a simple bar graph widget to show a visualization of this YouTube data and put it into the uiTypeahead drop-down list. This will help our users decide which video they would like to watch.

This bar chart is going to rely on D3 for all of its major DOM manipulations. The purpose of the example is to create a real-life example that uses live data. This directive is not in charge of the requests because the uiTypeahead directive takes care of that.

All the bar chart has to do is know when it should update its DOM with scaled bars that resemble how many views each video has in relation to the other results. To build the YouTube bar chart, we need to alter the uiTypeahead template in order to render and feed our bbBarChart directive. This will allow the uiTypeahead directive to communicate its data to our bbBarChart directive. The communication will be through an isolate scope set inside of bbBarChart.

This is as simple as adding one extra li element to the typeahead template, as shown in the following code:

The bbBarChart function watches a set of data for changes. Specifically, it watches data coming from YouTube. This data is not in the exact format we need, to showcase which video has more views in a bar chart, so we massage the data a bit before we pass it to the redraw function.

Massaging data before a DOM is created is a normal part of working with client-side technologies. Sometimes, it is not possible to change the server, so the data must be altered before we can show it in a view of our choice. In this specific case, we are just taking the title and the number of views from the data object to be used in the bbBarChart directive, which renders the data to the view.

D3 takes care of the dirty work. This makes it possible to have a dynamic bar chart that scales perfectly every time we update our search parameters and new data is fed into the directive. This demo can be found at http://angulardirectives.joshkurz.net/dist/#/mediaelement. Refer to the following screenshot:

E2E tests for bbBarChart

To test the bar chart correctly, we are going to employ Protractor. This is a new library that is specifically used to write E2E tests for AngularJS. It works with the Selenium Web Driver and by default, works with Jasmine as well. This will allow us to spin up a browser window and actually type AngularJS into the input and make sure that all is working correctly, all with one command.

Once Protractor is set up, all we have to do is write a simple test that queries YouTube and checks to makes sure that bbBarChart is creating the correct output based upon what is being typed in. In our controller, we have specified a maximum results limit when querying YouTube. This limit is 30, and this is how we will ensure that the bar chart creates the correct number of bars. This test will also prove that by clicking on a specific bar, we are able to load YouTube videos into the bbMediaPlayer directive. Refer to the following test:

This is a simple test that will prove our d3BarChart directive is working and is also working with other directives properly. Some of the individual expectations could be proven in a unit test and would probably be better suited for a unit test. This was a brief explanation of Protractor and how it can be used. Using E2E tests in this manner can be very powerful. It is also a great spectacle to see your site running in an automated fashion.

Summary

In this article, we learned that using live data in HTML5 applications is a must for production-level apps. What is important, is the data and how this data is displayed. There are many different ways to harness large amounts of data on the server, and these methods are becoming more commonplace in mom and pop start-up companies. This means that the deciding benefactor is no longer who can yield the most data, but what can be done with this data.

AngularJS offers many ways to request data and feed the view layer data that can be transformed into different visualizations. These methods include using combinations of services, directives, and controllers to manage how data should be requested and when it should be requested. Usually, the directive's job is to determine when its data changes and to update its view accordingly.

Alerts & Offers

Series & Level

We understand your time is important. Uniquely amongst the major publishers, we seek to develop and publish the broadest range of learning and information products on each technology. Every Packt product delivers a specific learning pathway, broadly defined by the Series type. This structured approach enables you to select the pathway which best suits your knowledge level, learning style and task objectives.

Learning

As a new user, these step-by-step tutorial guides will give you all the practical skills necessary to become competent and efficient.

Beginner's Guide

Friendly, informal tutorials that provide a practical introduction using examples, activities, and challenges.

Essentials

Fast paced, concentrated introductions showing the quickest way to put the tool to work in the real world.

Cookbook

A collection of practical self-contained recipes that all users of the technology will find useful for building more powerful and reliable systems.

Blueprints

Guides you through the most common types of project you'll encounter, giving you end-to-end guidance on how to build your specific solution quickly and reliably.

Mastering

Take your skills to the next level with advanced tutorials that will give you confidence to master the tool's most powerful features.

Starting

Accessible to readers adopting the topic, these titles get you into the tool or technology so that you can become an effective user.

Progressing

Building on core skills you already have, these titles share solutions and expertise so you become a highly productive power user.