Ivar Conradi Østhus

Menu

Custom build in dojo 1.7.

Introduction

In this blog-post I will explain how to do a simple custom build with dojo’s build system. Dojo is a great JavaScript framework and you can read more about it at dojotoolkit.org.

I write this from the experience gained from upgrading our application from dojo 1.6 to dojo 1.7. Dojo’s upgrade to version 1.7 is the biggest change in the dojo core we have seen for a long time, now introducing AMD (asynchronous module loading). This upgrade also affected the build system, which is new for dojo 1.7. We found that we where unable to reuse our current build-set-up after upgrading. I wrote about this because dojo still lacks some easy and simple examples on how to do custom build.

Why do we need custom build?

Dojo 1.7 introduced AMD which heavily increases the speed modules, classes and dependencies are loaded. Now your application only need to load dependencies required to satisfy the specific request and can wait with other dependencies until they actually are needed. Modules can even be downloaded to the browser in parallel (huge speed gain). This change in dojo will make many dojo applications start up much faster, especially if they use many modules.

Still, when the application grows, and you get hundreds or even thousands of JavaScript files it will be many HTTP GET request required to download all these small JavaScript files which builds up a typical application. We know by experience that many small files is more costly to download, than if we concatenate them into larger files which are downloaded. This gets even worse if we first have to download module A to find out that it requires module B which again requires module C. We have no way of knowing that we need to download module C before we have downloaded both module A and module B. This is fine in development, but not in production.

Layers to the rescue

What we do in our application is to combine related JavaScript files into layers. We try to see the application from the user-perspective and combines related scripts. We combine our modules in such a way that all files needed to perform a specific action is built into a layer. We also use a core-layer where we put core functionality shared by many different modules (the overlap).

Core-layer – Contains files/modules used many places in our application. Definitions in this layer is excluded from the other layers.

Function-layer-x – Contains files/modules required to perform action x in the application. We have many “x” layers.

I have placed dojo-src under scripts and my own company package inside the “mycompany” folder. It is important to not mix the dojo-source with my own source files. This makes it much easier to upgrade dojo later.

I use index-dev.html during development. In release I switch to index-release.html. I will explain the difference between these files later.

The built files are placed in the relese directory with the build script.

The package profile

The mycompany folder is the folder containing all the files for the “mycompany” package. To help the build system to build this layer I have a package profile, telling the build system important information about this package, such as which files are valid AMD modules, which files are test files and configuration files to only
copy during the build phase.

This profile is heavly based on dojo’s profile for the dojo package, found under dojo/dojo.profile.js

The layer build profile

To perform the actual build I use a profile on a higher layer (application layer) which tell the build system about the different packages used in the application, and what layers to build. This can be packages we have built our self, such as the mycompany package, our third party packages, such as the dijit package provided by dojo.

In the release profile I have defined which packages I have used and the location of them. I have also included the package “mycompany”. This package holds all of the company specific code. The example also shows that I have included two layers:

mycompany/layers/core – This layer contain the core functionality

mycompany/layers/example – Our example widget. This includes all modules required for ExampleWidget (all transitive dependencies), but excludes dependencies found in the core layer.

Executing the custom build

I execute the build from the “mywebapp/scripts/dojo-release-1.7.1-src/util/buildscripts” folder by using the build.sh or build.bat script:

./build.sh profile=../../../release.profile.js -r

The build system will place the built files into the release directory inside scripts because I have defined the relaseDir inside our relase.profile.js. The build-script will decide whether to use node or Java. Node is preferred because it can run the analysis and build process in parallel with multiple threads, while Rhino only supports one thread, which makes it much, much, much slower….

Development mode vs. release mode

Development mode

To enable us to develop without having to build the layers all the time I use a special html-file for starting the application in dev mode named index-dev.html. In this mode we just load required modules asynchronously as they are needed. This is supported by dojo out of the box and we basicly requieres stuff just when we need them. It us much easier to debug if we load all modules individually during development.

I have measured some stats in dev mode (no layers):

57 individual HTTP GET requests

840ms before onLoad is fired

As seen from the numbers I actually end up with 57 individual request to fetch all resources required for my little application. This is fine in development because it is easy to change code, and we get better debugging in the browser.

Release mode

After I have performed the build I will have a release catalog containing our built javascript, including my layers. This folder contains the built layers which I want to use in my application. In production I will use index-release.html where I have changed the code a bit to also handle the built layers. The important part is to notice the extra require part in the start-app script:

As you can see I require both layer-modules first, which includes all the required JavaScript files in my application. This way I reduces the number of HTTP GET requests to a minimum. I still keep all my files in the release folder. This enables easy fallback for dojo to load dependencies asynchronously if I for some reason should forget (or exclude) a reuired JavaScript in the layer.

Stats in release mode (with layers):

5 individual HTTP GET requests

355ms before onLoad is fired

This is significantly better performance and better suited for production.

Can different layers include the same modules?

Can I have the same JavaScript bundled into different layers? Yes you can. Dojo will actually handle this very well. You should try though to avoid this when possible, and always evaluate whether these modules can be part of a “core” layer because the layers will be larger if multiple layer modules include the same scripts. This basically comes down to a trade off, we want to limit the size off the core layer and at the same time limit the size of a function layer.

One solution can of course be to have multiple “core layer” for different function groups. It all comes down to you having to find which layering that gives the best performance for your application.

Don’t put everything in one layer

Even though fewer HTTP GET’s are good we must make sure to not put everything inside ONE layer. Putting it in one layer can give us one HUGE file, which basically gives our application an undesired long startup time. We can also gain something from loading a few modules in parallel. We should evaluate the nature of the application when we decide how to define our layers. We should try to group stuff we need together inside modules. It is also important to discover the overlaps between layers and try to create “overlap” layers.

Summary

This ends my blog about how to build custom layers in dojo 1.7.1. I have placed all my files on github: https://github.com/ivarconr/dojo-custom-build. Please post any comments or questions you may have. In the next blogpost I will explain how we are using custom build as part of our maven build process to ensure that our layers are built each time we build our Java Webapplication with Maven.

Advertisements

Like this:

LikeLoading...

Related

Post navigation

68 thoughts on “Custom build in dojo 1.7.”

i have been at this(*build profile*) for long but i’m gonna try your example where you did the two require makes perfect sense one to require the file(layer) and looking at the layer code it creates a cache mapping the other dependencies built into the layer. i hope this works.

Firstly, thanks thanks thanks.. i spent an entire day lookin into Dojo docs and got nowhere. This tutorial is awesome. I have one follow up question. In the compressed layer file, one of my classes is defined as follows

define([“dijit”, “dojo”, “dojox”, “dojo/require!foo/bar/baz”]

this leads to a calls to

http:/host:port/release/js/dojo/require.js

Failed to load resource: the server responded with a status of 404 (Not Found)

Should’nt this be loaded as part of the dojo layer?

PS: My entire project works perfectly fine in dev build. All the code is Pre 1.7(sync), just migrating to 1.7

The “dojo/require” module (legacy) is not equal to “require” (the new way to require dependencies) and is not part of the built dojo.js module. It is part of the old synchronous loading system used in dojo < 1.7. If you want to use it, you have to build it in to one of your own layers to avvoid the additional get (just add it as an include on one of your layers or you can set the module mapping for dojo correctly).

I am curious of the source file for this definition. Can you show me that, or extract it into a simple scenario?

Ahh.. you are absolutely correct. NLS files are not included in the layers. In pre 1.7 you could specify to the build-system that which nls-files you want to bundle (via the localeList option). I have investigated this a lot (reading all documentation and source code for the buildsystem) and found that this is currently not supported in the new build system. I have found some references to a “locales” array, but it has never been used (might be that they plan to include this later?). For our application I have found this to be a minor issue, and we just load the NLS files async from the Google CDN. Personally I would prefer to do a layer-build for a specific language and disabling the NLS feater, as many users don’t need it.

Looking at my build logs, “dojo/cldr/nls/number” is a dependency; However, at runtime, “dojo/cldr/nls/en/number” is looked for which is not available. Do you know why that would happen? I’ve not specified any locale in djConfig.

Hi,
by “more serious” i meant more complex.
i used to work mostly with jquery and pure javascript, the concepts of builds, layers and AMD are new to me and it is quite hard to find good tutorials for beginner.

Fabulous! Thanks for this useful example, it really helped me get started with the new builds.

I’m stuck trying to get layers and custom modules to work with the parser and I must be missing something. I can get parseOnLoad to work with dijits (Button, Dialog, whatever), but not with mycompany.widget.Example. Any ideas? I appreciate any hints or help.

This is how I modified your example to try to use the parser. The programmatic widget shows up, but when it tries to load the declarative/parsed widget I get an error saying “Could not load class ‘mycompany.widgets.Example”.

1. when i build a simple example with a someprofile.js, i get a release folder created which is 14MB or greater.
im looking for a single JS file or so…. which is less than 1MB. Is there any other way we could achieve it.
the Build Arguments which i pass is
build.bat profile=../../dij.profile.js action=release ##### without profile.js as file name also i tried #######

2. If i have to Deploy the web app which i develop in DOJO to Mobile using phoneGap ….. how should i make a build to make a single JS file which contains all that the app requires….?

You only have to include the layer files in your application (which can be 1 single js-file). All your included resources should be inside there. You can even build in a custom dojo-base in to that layer so that you only include what is needed by your application. This will keep the application size to an absolute minimum.

My html requires dojo.js. I notice that the browser attempts to include selector/acme.js, which I didn’t move to the web server – I thought I’d only need dojo.js. When the page loads, the javascript console reveals the error “_5c4 is not a function” when attempting to do something with digit.byId.

When I instead load the full Dojo from ajax.googleapis.com, everything works perfectly.

Thanks, that’s a good point. I just thought I could have a smaller build if I went totally custom. We don’t have any custom code to include, so I thought this would be trivial. In fact, it was trivial for my custom 1.3 build, but now we want to upgrade to 1.7.2 in order to take advantage of AMD modularization.

One more thing – I notice that with my command line build that the browser attempts to include selector/acme.js, which I didn’t move to the web server. I thought I only needed to move dojo.js.

Anyway, can you help with some direction on using the pre-built core and only bundling the necessary digit stuff? I’d like to host all the dojo-related .js locally, and obviously I want to minimize the number of file requests the browser makes.

Brilliant, thanks for all of your help. I was able to get a working dojo.js file down to the size of 182 K. I was wondering if you had any suggestions on how to go even smaller? Here’s the profile I used:

It sounds a little big for so few dependencies, but your profile looks correct to me. The dojo-core is 137K, so I guess you got a full dojo bundled. I need to investigate a little verify if this is correct..

One tip is to ensure that your javascript file(s) are gzipped by the server before it is delivered to the client.

Seems like full dojo core is included when you use the “boot:true” option. This is also the option you should use in order to build a “custom dojo build”, according to the official tutorial [1], so I guess you can’t opt out of a full dojo core.

I might have missed something in the documentation and you will find all the details about the flags you can use for each layer on [2].

Thank you for your help.
I am managing dojo for a few time and this is the first time I make a build.
I have read many profile examples but I am having some troubles, i.e. I dont know what I am doing wrong.

The build make the next output:
release
release/layers // << within the layers
release/layers/nls // << within localeList js layers
release/dojo
release/dojo/dijit // << all the dir content within all themes
release/dojo/dojo // << all the dir content (within _firebug dir…)
release/dojo/dojox // << all the dir content
release/dojo/snet // << all the dir content
release/dojo/build-report.txt

In this part, one question is how I can left the dojo/dojo dir with the minimal content (and if posible within 1 file)
and another question is if there is posible to left in the build only 1 theme.

The real problem is:
I have to put the layers dir into dojo dir because dojo needs to find the layers and the nls layer locale file.
And it gives me another error (in firebug): "Could not load class 'dijit.Menu"

2. When dojo moved from dojo 1.6 to dojo 1.7 they lost the possibility of including nls-files into the custom layers. You can resolve this issue in two ways:
a) just use their CDN when including dojo. All files you haven’t included in a layer, such as a nls file, will be automatically required from the proivded CDN.
b) or you can of course host the source-files yourself if you want. You must specify where dojo is located though. It will assume a dojo, dijit and a dojox folder structure from that location.

3. I really need more details to be able to help you resolve the “Could not load class ‘dijit.Menu'” issue.

Thank you for your reply.
The problem is that within the build directory structure created (see above) dojo can not find
– layers/core.js
– layers/menu.js
When I move the layers dir into dojo dir (so: relese/dojo/layers/) dojo find them,
but gives an error “Could not load class ‘dijit.Menu”, and I dont know how to resolve it.

As I said the build’s dir structure dont have the layers dir into dojo dir (profile->releaseName), so dojo cant find the layers.
The build output structure should be:
release
release/dojo
release/dojo/layers //<< in dojo
release/dojo/layers/nls //<< in dojo
release/dojo/dijit
release/dojo/dojo
release/dojo/dojox
release/dojo/snet
release/dojo/build-report.txt

I think you should not put your custom layers and own packages inside the release/dojo folder. This sort of mix up different concerns, It would be better to tell dojo where to find your own custom built modules. This can be done with this code snippet:

Why it can’t find dijit.menu is a mystery for me. I have built your menu layers and after inspecting the layer-module it seems to have included dijit/menu correctly. I really need a more concrete example/some extracted code that shows the problem to help you with that issue.

Because you where building your own custom dojo (I don’t see you have any reason for doing that?) you also have to add the kernel and loader to be able to register a module-path.

In the index.html I load first the base layers:
“layers/core” and “layers/menu”

before requiring “dijit/menu”. If you see in the network logs you will see that only the layers are loaded.

I also noticed that you had cyclic dependencies in your layers-definition. This makes the build-system complain, and you should watch the build-logs carefully. You must never have an error in the build log, and the warnings should be at a minimum.

I have also demonstrated how to use “”dojo/domReady!”, which is an AMD-plugin. This is the new way to require that the dom is loaded, and should be prefered over the legacy dojo.ready.

Thanks, Ivar, for your very useful post.
I have a doubt and I appreciate your help. I followed step-by-step your example, and I was able to make my build.
I have 3 layers: base (for modules of dijit and dojox), core (my core modules), widget (my widgets). At header of my pages I put:

If your need a variable to be global just drop the “var” assignment, or in a browsert you can always assign it to the window object to make it global because the window object is always the global scope in all browsers.

That said I would find another way to access it, as globals breaks the whole idea of using AMD. What do you try to achieve? Make a signleton? If that’s the case your should redesign your “manager” module to only export one instance of the manager object, and you can then just require that one object every place you need it.

Thanks a lot! Yes, it is a singleton..I’m researching how to use singleton pattern with AMD, but your tip was valueable. One thing is strange yet: if I call require to a specific module (e.g “manager/Lookup”) out of the code:
require([“Manager/layers/base”,”Manager/layers/core”,”Manager/layers/widget”], function() {
});

Sorry to bother you – I need your help. I’m trying to have a release build for my Dojo project,
and I seem to have hit a wall.

Everything worked fine in development mode, where all dojo modules and custom
application modules loaded separately. Only there were more than 400 HTTP requests
to load the application, so it took a while.
I could build a release layer containing all dojo, dojox and dijit components that my
application was using, and the application loaded OK, while the custom modules were
still loading one by one. That was good, but not good enough.

Next, I built another layer aggregating the custom components.

The “bootstrap” component is called geno.Base, and it has the openWidget
method that needs to be invoked, and that method initializes the rest of application.

Thank you for your reply! I recognized that when I copied-and-pasted my question, most of the text (namely, file snippets) was deleted – probably, because of HTML tags that were there… and my carefully crafted question appeared to be very sloppy…

I did read the doc you pointed me to, and I reviewed it again just to make sure I comply. I do. Or I think I do 🙂

Anyhow, I did your (1) and (2).
For (1), I have the following in my index.html file (is that what you meant?):
<script type=”text/javascript” src=”/dojoroot/release/dojo/dojo/dojo.js”></script>
<script type=”text/javascript” src=”/dojoroot/release/dojo/dojo/geno.js”></script>
to include both layers that build script generated for me.
For (2), I did the inline script like this:

As we have a java-app installed as a war, we have chosen to trigger all the builds with maven and only includes the built layer in the final war. You can have a look at my boiler-plate project for this at github (Does only make sense if you are on a java/maven plattform)

Hello, I do believe your web site might be having browser
compatibility issues. When I take a look at your website in Safari,
it looks fine however when opening in I.E., it’s got some
overlapping issues. I simply wanted to give you a quick heads up!
Apart from that, excellent blog!

Hi , it is indeed a good article. I need some help on migrating legacy < 1.7 code base, our code base is so huge and it will take lot of time to convert to AMD format. Is the new build process backward compatible , I mean will it be able to parse the old format of dojo classes?