26 November, 2012

I am working on a modular ASP.NET MVC application, one of the problems is communication among modules. At the beginning, with just 3 - 4 modules, it might seem possible to call methods among modules directly. As the system grows, number of modules increases and dependencies among modules become more complex, such solution become impractical.

I didn't need anything fancy, just a simple publisher/subscriber pattern implementation for the single application, so I decided to write my own 'library'.

Publisher - subscriber pattern

Implementation

The core of the implementation is the MessageBus class. A publisher obtains an instance of the MessageBus (from IoC container), sends a message and the MessageBus distributes the message to appropriate subscribers.

Subscribers need to implement the IMessageSubscriber<TMessage>
interface, so they can process messages of the desired type.

The actual implementation of MessageBus is pretty simple, the difficult part - managing references to subscribers is done by an IoC container (encapsulated in IDependencyResolver). An IoC container is usually already in place in MVC applications, so why reinvent a wheel ...

21 July, 2012

I am building a modular ASP.NET MVC application and this task brings a lot of interesting challanges. One of them is to design a data access layer.

Background

We are using Entity Framework in our solutions and for this project we'd like to take advantage of new features of the Entity Framework, especially Code First development and data migrations.

Any module in our application might want to save some data to the database - in order to simplify communication with database, the application core provides bunch of classes responsible for wiring up all the stuff like connection strings, entity configurations, etc. With that infrastructure in place, we can use the same, simple code to access data anywhere in our application ... and keep in mind that our domain model is spread across multiple modules and possibly across multiple assemblies. How have archived that?

Implementation

If you are using Entity framework Code First, the most of the actions involve DbContext class. It provides methods for adding new entities, retrieving existing ones from database, saving changes and much more. Typically you create derived class with properties of DbSet<TEntity> type to expose collections of entities and EF will do all the work.

Entity framework internally uses a conceptual model to map entities and tables. In order to create it EF needs types of entities to include in the model. It the previous example it's done by exposing DbSet<Comment>, but this solution isn't applicable in our case because we don't know types of the entities types at compile time. Fortunately there are other ways - the one we have chosen uses DbModelBuilder class and creating DbContext from the model produces by the builder. The whole process is encapsulated in the DbContextBuilder.Build method.

EntityCatalog property contains a collection of IEntityConfig objects, that instructs DbModelBuilder which entities should be included in the model and how these entities should be mapped to the database.
To avoid any manual use of reflection or manual registration of the modules, EnityCatalog property is populated by MEF.

Along with model classes every module includes a configuration class that implements IEntityConfig interface and is used to setup DbModelBuilder, if denoted with Export attribute, this class is automatically discovered by MEF.

For the sake of simplicity this example loads model class only from the current assembly. but trust me it's working with multiple assemblies as well.

Usage

DbContextBuilder class implements singleton pattern (I know I could have done better, but it's prototype ...) so its instance is available across application. In my controllers I can instantiate a new instance of the DbContext by calling DbContextBuilder.Instance.Build().

19 July, 2012

Just three months after Microsoft released the whole ASP.NET stack under an opensource license, there is another surprise. As announced today on Scott Hanselman's blog and Scott Gu's blog Microsoft is releasing the Entity Framework under Apache 2.0 license.

31 May, 2012

I have been stuck with javascript application development in last few weeks. Even though Javascript isn't my the most favorite language, it's interesting experience that brings new challenges.

We are developing a web application that should work on personal computers and mobile devices (iPhone, iPad, Android phones and tablets, ...). The core of our application is the same for both platforms, but UI is little different. We'd like to give users the best possible experience and in the case of our application it means working with touch events a lot.

Using touch events isn't difficult, but debugging application with these events can be really painful.

Opera Dragonfly is your friend

Without proper tooling it would be almost impossible do debug this kind of user input intensive application on mobile devices. Thanks to guys from Opera you can have such tools for free.

Every desktop Opera browser contains Opera Dragonfly - an extension that allows developers to inspect DOM of the page, monitor network traffic, monitor resources used by the page and debug Javascript. It isn't anything special - almost all browsers have some kind of developer console, but Opera has another ace in it's sleeves - remote debugging and Opera mobile emulator. Yes, Google Chrome and Firefox mobile offers some kind of remote debugging support, but Opera wins with it's ease of use. Configuration of the remote debugger is really easy and you got all the functions of the Opera Dragonfly - even breakpoints in javascript code.

Step by step instructions

Configure Dragonfly to accept remote connection by clicking icon in the toolbar

Configure Opera Mobile to connect to Dragonfly by visiting page "opera:debug", entering correct connection settings (if you are running Dragonfly and Emulator on the same computer, IP address is 127.0.0.1 or localhost) and clicking connect.

Navigate to the page you want to debug

That's it. Now you have Opera Mobile Emulator talking to the Dragonfly and vice-versa. You can set breakpoints in javascript files, inspect DOM elements by selecting them directly in the DOM emulator, monitor network traffic and lot of other things :-)

Of course you can use real physical device with Opera Mobile instead Opera Mobile emulator, but I found the emulator more convenient during development. From my experience emulator is better for application development and physical device is better for testing overall user experience.

All products will continue to be fully suported and developed by Microsoft, but will be accepting contributions from the community as well. Also being released under Apache 2.0 opens a lot of new possibilities - e.g. building your custom version of the projects.

26 March, 2012

In the previous post I have shown, how to do simple filtering of the OSM data with SpatialLITE library. Today I will go a step further and do something more complicated - extract data from OSM file within specific bounding box.

It seems to be pretty straightforward task so, how hard can it be?

There is one problem, we need to solve - some ways and relations near edges of the bounding box would spread across bounding box boundaries - part of the entity would lie inside the bounding box and the other part outside. We need to decide what to do with these split entities. Should we include them? Should we exclude them?

Split entities

Basically there are four way how to deal with them:

exclude ways and relations that have any part outside the bounding box - this leaves isolated nodes in the output - see picture

include ways and relations that lies partially outside the bounding box, but do not add any nodes that lies outside the bounding box - this leaves ways and relations incomplete - again see picture

include all entities with all nodes that lies partially inside the bounding box - this produces most the most complete map, but on the other hand you will end up with many entities outside the bounding box

clip entities at the edges of bounding box

The option 4 might seem as the best solution, but we would need to add new nodes on the edges of the bounding box, split ways into two, or do other complicated (with OSM data model) things.

Options 1 - 3 look very similar to each other and it depends on your needs, which one you should choose. But from the performance point of view option 3 is significantly more complex. Entities in OSM files are sorted by entity type and then by ID, so with options 1 and 2 we can create output file in single pass, but with option 3 it is impossible. During first pass we find all nodes in the bounding box, and all ways and relations that contain any of the selected nodes. In the second pass we need to add nodes that are outside bounding box but belong to the included ways and relations.

Tracking IDs

For all options we need to track IDs of entities inside bounding box (or you can keep all entities in the memory - but with millions of entities it might be impossible). There are couple ways of doing that - for example Osmosis is using either bit board or list of integers. I'd like to try something else - as mentioned above OSM files are sorted by entity type and then by ID, another observation tells us that entities with consecutive ID often lies near each other. We can take advantage of that - we will store IDs as list of Range objects.

struct IdRange {
public int From;
public int To;
}

Every IdRange structure will represent consecutive range of IDs. If there are at least 2 Ids in IdRange object in average, this representation of used nodes will use less memory then simple list of IDs.

When it comes to the memory consumption the results are worse for SpatialLITE. .NET applications are usually a little bit memory intensive and this sample application isn't exception. While peak memory consumption of Osmosis was 53MB, SpatialLITE took whole 84MB of the memory. It seems that the biggest memory consumption was caused by PbfWriter, which is weird and I will have take a look on it. The IdTracker, that should occupy the most of the memory, actually reached size of only 19MB. An internal stats from the IdTracker shows, that there are approx. 4.5 IDs in every Range object - so it isn't actually bad idea.

11 March, 2012

I’m a big fan of the OpenStreetMap project, so when I released the first version of the SpatialLITE library last week, classes for working with OpenSteetMap data couldn’t be missing in the project. Right now the library supports OSM XML files (without compression) and OSM PBF files, both for reading as well as writing. And how can you work with OSM data? How does SpatialLITE library compare with other tools in terms of speed? Let’s find out …

... err wait, some introduction might be neccessary ...

IEntityInfo vs. IOsmGeometry

There are two different representation of the OSM entities in the library – IEntityInfo and IOsmGeometry objects.

IEntityInfo represents a lightweight object that contains only data for the particular entity – relationships with other entities are described only by IDs of related entities.

WayInfo class

On the other hand a collection of IOsmGeometry objects represents tree of interconnected objects that implement IGeometry interface. This allows you to access properties of related entities directly, perform spatial analysis with methods from SpatialLite.Core library or use any other methods from the library that accepts IGeometry parameters. IOsmGeometry objects are pretty powerful, but the price is obvious – all objects must be held in the memory, and with large files it might easily become OutOfMemoryException kind of problem.

The decision whether use IEntityInfo objects or IOsmGeometry objects is up to you. Is some cases it might be better to stick with the simple IEntityInfo objects and sometimes you will need more complex IOsmGeometry objects. Fortunately it is possible to switch between these two representations of the OSM entities.

Right now there are two formats supported – OSM XML and OSM PBF. Both formats support reading as well as writing – so there is OsmXmlReader, OsmXmlWriter, PbfReader and PbfWriter.

Readers accepts a stream or a file in their constructors and provide forward only reading capabilities – pretty much the same behaviour you find in build-in readers (e.g. BinaryReader). Because of the structure of files it is impossible to create IOsmGeometry objects in the reader and thus readers returns IEntityInfo objects. If you need to work with IOsmGeometry objects, you can use OsmDatabase class that encapsulates process of creating full fledged objects from data read by IOsmReader.

The Principle of the writers is pretty much the same – they also accepts a stream or a file in the constructor and then provide forward only, writing capabilities. Both IOsmGeometry and IEntityInfo objects contains all necessary information for serialization, so both object types are accepted as parameter of the Write Method.

Putting it together

OK, now when we have covered basics, let’s go back to the title of this post – filtering of OSM data. For now we would implement filer that is able to process just nodes.

Ingredients

1 IOsmReader

1 IOsmWriter

1 expression to determine whether node should be filtered out or not

For the purpose of this demo we would like to find all guide posts in the OSM file. In OSM a guide post is represented by a node with the information:guidepost tag. Writing an expression to choose such entities is simple:

I wanted to find out how fast the SpatialLITE library is - Osmosis application was chosen as competitor, because it is probably the most popular tool for processing OSM data. Both tools were used to perform the same task: Extract all nodes with information:guidepost tag from the OSM files (5.25 GB OSM XML file, 245 MB PBF file).

Let's see the results ...

SpatialLITE (XML)

SpatialLITE (PBF)

Osmosis (XML)

Osmosis (PBF)

Run 1

5:01

1:30

5:58

1:46

Run 2

4:48

1:34

6:40

1:44

Run 3

4:56

1:29

6:01

1:43

Speed comparison of SpatialLITE and Osmosis (all times in minutes)

Not bad, I guess :-)

OK, it might not be fair comparison for the Osmosis, it has many additional features and can perform significantly more complex filtering, but it shows that the reader / writer classes in the SpatialLITE library are anything but slow.