Wednesday, May 19, 2010

Intro

Now that Xtext is at 1.0 RC1 I thought it was time to start using more of all the new features for Eclipse b3. One of the features I wanted to add was to support time stamps in a nice way in the editor. Internally, a time stamp is naturally stored as a java.util.Date so there is never a question about the exact UTC it is representing. When editing however, you may want to use some other format (if not copying an actual timestamp, you may want to use something like 'feb 10, 11:00:00am' .

The issue is that the reference to 'feb 10, 11:00:00am' in the source text has no time zone information, and the name of the month may not be in english etc. In order for the source to be valid everywhere, it would be required to fully specify the date format used, as well as the timezone and store this in the source. I choose a middle ground where the editor understands the more human friendly formats and offers to help to convert it to a format that is always possible to parse.

All of this may not be all that interesting, but it gave me opportunity to try some features of Xtext that I had not used. The rest of this blog is about my first iteration of the implementation, and it shows some Xtext techniques like:

Using an ecore data type in the grammar

A Date value converter

Overriding the SyntaxErrorMessageProvider

Providing a quick fix for a ValueConverterException

The Grammar

This simply declares that a language element 'Entity' has a 'timestamp'. The TIMESTAMP rule declares that it returns an ecore:EDate. Luckily we don't have to state more than the import of ecore to make use of it in our language. Also in our favour is that EDate is declared in ecore. If this was for a datatype not in ecore, we would need to create a model containing the definition of the data type. As this was not the case here, we can move on to the data converter.

Date Value Converter

This is almost boiler plate code, but there are some interesting details. Here is the converter method.

01@ValueConverter(rule ="TIMESTAMP")02publicIValueConverter<java.util.Date> TimestampValue() {03return newAbstractNullSafeConverter<Date>() {0405@Override06protectedString internalToString(Date value) {07SimpleDateFormat fmt =newSimpleDateFormat("yyyyMMddHHmmssZ");08fmt.setTimeZone(TimeZone.getTimeZone("UTC"));09return'"'+ fmt.format(value)+'"';10}1112@Override13protectedDate internalToValue(String string, AbstractNode node)throwsValueConverterException{14string = string.substring(1, string.length()-1);1516// First choice, if a timestamp string, use it.17try{18// Allow non UTC strings since they are fully qualified with offset and can thus19// be parsed by anyone.20SimpleDateFormat fmt =newSimpleDateFormat("yyyyMMddHHmmssZ");21fmt.setTimeZone(TimeZone.getTimeZone("UTC"));22returnfmt.parse(string);23}24catch(ParseException e) {25// ignore and try timestamp format26}27// Second choice - if using java default for the locale28// Needs special processing as it probably does not contain TZ in the string)29try{30// try the default locale style of Date Time and see if it parses31DateFormat.getDateTimeInstance().parse(string);32// if this parsed, it is not likely that the default is the full33// format with timezone offset, so flag this as a special error :)34// that is fixable35// Although simple, it makes sense from a user perspective, a time in36// local format can be entered and transformed to a timestamp.37throw newValueConverterException("Not in timestamp format", node,newNonUTCTimestampException());38}39catch(ParseException e) {40DateFormat fmt = DateFormat.getDateTimeInstance();41String defaultFormat =(fmtinstanceofSimpleDateFormat)42?((SimpleDateFormat)fmt).toLocalizedPattern()43:"Default format for the locale";44throw newValueConverterException("Not in valid format: Use 'yyyyMMddHHmmssZ' or "+ defaultFormat +45"Parse error:"+ e.getMessage(), node,null);4647}48}49};50}

The code first tries to convert the string entered by the user using the wanted timestamp format. If this fails, an attempt is made to use the default format. If this works, we know we have source text that (most likely) does not have the correct time zone information in it, and we want to offer a quick fix to convert the format. But how can that be done — the ValueConverterException does not allow us to specify a 'diagnostic code' that allows a quick fix to detect the particular problem. The ValueConverterException is also final (in the 1.0RC1 release at least), so the only option is to use a marker Exception as the cause (In this case NonUTCTimestampException).

The final attempt to convert (again using the preferred timestamp format) is there simply to catch the error (it could have been remembered from the first attempt).

As you will see later, the design can be improved further by supplying the actual format that was used to successfully parse the entered timestamp in the marker exception, but I left that for a later iteration.

Note that the error message includes the two valid formats as feedback to the user in case the entered text was unparsable. It would be easy to try several formats.

Overriding the Syntax Error Message Provider

The default SyntaxErrorMessageProvider is a class that hands out SyntaxError instances describing a problem occuring in a particular context. In my case I just wanted to add handling of the ValueConverterException with my special non-UTC cause Exception.

As you can see, this is straight forward, simply return a SyntaxErrorMessage with a diagnostic code (a static string) that I called IBeeLangDiagnostic.ISSUE_TIMESTAMP__NON_UTC. At this point, non of the new code (except the data value conversion is in effect, and a bit of magic is needed to make it kick in.

Xtext makes good use of google guice dependency injection. In addition to the standard guice, there is also advanced so called 'polymorphic dispatching'. This means, that even if it is not apparent in the guice module Xtext generates for a DSL that something can be bound to a specialized class, it is still just as easy to bind almost anything by simply adding a method.

This is pretty much bolier plate code for a quick fix (when generating a DSL with Xtext, there is a sample that shows ow it is done). The code above simply converts the source string using the default format in the value converter, turning it into a timestamp in the correct format. It the replaces the string in the input text.

An improvement would be to pass the date format used in the 'Issue' (it is possible to pass data with a diagnostic code), but I did not look into how to do this with the SyntaxError class yet.

A big thank you to Sebastian Zarnekow at Itemis for pointing me in the right direction

Sunday, May 9, 2010

The Eclipse b3 Aggregator is based on and part of the Eclipse b3 project. Eclipse b3 provides a versatile and adaptable framework supporting build, assembly and deployment processes. It supports a rich set of use cases. One of those - the aggregation of repositories - is the focus of the b3 Aggregator tool.

The Eclipse b3 Aggregator combines repositories from various sources into a new aggregated p2 repository. It can also be configured to produce a hybrid p2/Maven2 repository. There are many situations where using aggregated repositories is a good solution, here are some examples:

Projects want to provide convenient access to their products - Installation instructions requiring the user to visit several repos for a complete install are not uncommon. An aggregated repo for all those locations provides a convenient one-stop-shop strategy. The aggregation can perform mirroring of all consumed p2 repos or selectively provide indirection via a composite repo.

Organizations or teams want control over internally used components - It may be necessary to have gated access to relevant/"blessed" p2 repos where an organizational "healthcheck" has been performed prior to internal distribution. Furthermore, internally used aggregated repos can provide a common basis for all organizational users (i.e. for both IDE distribution as well as for content used when building internal applications).

Increase repository availability - by aggregating and mirroring what is used from multiple update sites into internally controlled servers.

Distributed Development Support - an overall product repository is produced by aggregating contributions from multiple teams.

Owners of a p2 repo for a given project may not be in position to host all required or recommended components due to licensing issues - Buckminster's SVN support can serve as an example here, as it requires components available in the main Eclipse p2 repo as well as third-party components. Hence users have to visit several repos for a complete install.

The b3 Aggregator is focused on supporting these specific requirements, and it plays an important role in the full scope of the b3 project. The Aggregator is however used in scenarios outside of the traditional "build domain" and this has been reflected in the user interface which does not delve into the details of "building" and should therefore be easy to use by non build experts.

Functional Overview

The b3 Aggregator performs aggregation and validation of repositories. The input to the aggregator engine (that tells it what to do) is a b3aggr EMF model. Such a model is most conveniently created by using the b3 Aggregator editor. This editor provides both editing and interactive execution of aggregation commands. The editor is based on a standard EMF "tree and properties view" style editor where nodes are added and removed to form a tree, and the details of nodes are edited in a separate properties view. Once a b3aggr model has been created it is possible to use the command line / headless aggregator to perform aggregation (and other related commands). (Note that since the b3aggr is "just and EMF model", it can be produced via EMF APIs, transformation tools, etc. and thus support advanced use cases).

The model mainly consists of Contributions; specifications of what to include from different repositories, and Validation Repositories; repositories that are used when validating, but that are not included in the produced aggregation (i.e. they are not copied). The model also contains specification of various processing rules (exclusions, transformation of names, etc.), and specification of Contacts; individuals/mailing-lists to inform when processing fails.

Here are some of the important features supported by the b3 Aggregator in Eclipse 3.6M7:

p2 and maven2 support — the aggregator can aggregate from and to both p2 and maven2 repositories.

Maven2 name mapping support — names in the p2 domain are automatically mapped to maven2 names using built in rules. Custom rules are also supported.

Mirroring — artifacts from repositories are mirrored/downloaded/copied to a single location

Selective mirroring — an aggregation can produce an aggregation consisting of a mix of references to repositories and mirrored repositories.

Cherry picking — it is possible to pick individual items when the entire content of a repository is not wanted. Detailed picking is supported as well as picking transitive closures like a product, or a category to get everything it contains/requires.

Pruning — it is possible to specify mirroring based on version ranges. This can be used to reduce the size of the produced result when historical versions are not needed in the aggregated result.

Categorization — categorization of installable units is important to the consumers of the aggregated repository. Categories are often choosen by repository publishers in a fashion that makes sense when looking at a particular repository in isolation, but when they are combined with others it can be very difficult for the user to understand what they relate to. An important task for the constructor of an aggregation is to be able to organize the aggregated material in an easily consumable fashion. The b3 aggregator has support for category prefixing, category renaming, addition of custom categories, as well as adding and removing features in categories.

Validation — the b3 aggregator validates the aggregated result to ensure that everything in the repository is installable.

Blame Email — when issues are found during validation the aggregator supports sending emails describing the issue. This is very useful when aggregating the result of many different projects. Advanced features include specifying contacts for parts of the aggregation which is useful in large multi layer project structures where issues may related to the combination of a group of projects rather than one individual project - someone responsible for the aggregation itself should be informed about these cross-project issues. The aggregator supports detailed control over email generation including handling of mock emails when testing aggregation scripts.

Documentation

The b3 aggregator documentation is available here on the Eclipse Wiki.

Wednesday, May 5, 2010

Until there is migration documentation, my experiences of migrating the Eclipse b3 project from Xtext 0.8 (~M4) to 1.0 nightly (> M6) may be of value to others. I did this by first migrating to M6, and then using the nightly - this so I had a state to roll back to in case the nightly would fail me completely.

Migrating to 1.0 M6 version

Merge of o.e.xtext.ui.common and o.e.xtext.ui.core into o.e.xtext.ui

Almost everything that was either in ui.common, or ui.core is now in just ui - all that is needed is to change the imports, and update any dependencies to the two merged bundles with the new bundle. (See below for some additional changes).

New StructureAfter running the mwe workflow, I got 4 new packages in my dsl project with the suffix ".ui". These packages contained the corresponding classes found in the existing packages without the ".ui" suffix. I moved/merged my code over to the new packages and deleted the old packages.UI Module changeThe UIModule for my DSL had to change to the following signature and constructor:

since the JavaScopingFragment no longer exist. Don't know if the ImportURIScopingFragment is what I want, but I had to pick one.

Converting Java Strings

Strings.convertFromJavaString now has an extra boolean argument useUnicode which should be set to true to process \\uXXXX escapes. (I did set it to true). I use this method in some terminal converters.

Changes in plugin.xml

A manual merge of all changes in plugin.xml_gen to my plugin.xml (basically changes related to use of "ui" in package names) was required.

Manifest change

Manifest file needed update as the activator is in a different package:

has been dropped from DefaultAutoEditStrategy. It was only there to block an easteregg (typing 42 displays a funny comment about 'the meaning of life' - but the easter egg and method seems to both be gone in the nightly).

Serialization

I have not done much with b3 serialization yet, so required changes were small. I needed to add a single method:

I had attached commands to the popup menu that should appear (and they did in 0.8 M4) over the editor's outline. But this stopped working. I am waiting on some wisdom from the Xtext gurus on this...This was a temporary issue with plugin.xml changes not taking effect. After a restart and clean build it now works just like before.

Syntax highlight has changed, and I am trying to figure out how it works now...It stopped working because I forgot to move things over to the new UIModule (as described above).

Summary

All in all, the migration was quite painless. Knowing that changes were to take place in several of the services, I only have minimal implementations (the default, or just a few lines to fix something glaring) in many places (in wait for the 1.0 release and new documentation). If you have a lot of code and using everything "to the hilt" in 0.7.2, you may want to wait for the official release and the documentation.

I will update this article as I find more things that needs to be changed, or if I changed something in error.

Tuesday, May 4, 2010

The Helios release of Buckminster has the following new and noteworthy features available in 3.6M7

Support for Git - uses, updates, or clones git repository as needed

Headless JUnit and EclEmma (code coverage) support

Comprehensive documentation available - introduction, examples, and reference. Download PDF, 250 pages, includes descriptions of the new features described here.

Graphical dependency visualizer - resolutions can be viewed and navigated/filtered with a Zest based viewer

Much improved target platform support - using new features in PDE to automatically manage/materialize target platform config

Provisioning and management of API baseline

New EMF based editors for MSPEC and RMAP - much easier to use than editing XML

Reader type for Project Set (.psf) files - makes it easy to integrate or migrate projects that are using .psf files to describe where the source is

p2 repository size reduction to 1/3 using improved pack200 support

OmniVersion support - the support for non OSGi version has been changed to use the p2 OmniVersion implementation for increased flexibility - backwards compatible with Buckminster version-type, version scheme used in earlier versions.

Qualifier generator using Build Identifier - use a property to control the content of a version qualifier

LDAP style filters on RMAP providers, CQUERY advisors, and MSPEC nodes - makes it possible to parameterize more things, reduces the need for multiple slightly different copies of these files.

Smart version range generation for feature 'includes' - heuristics result in natural choices

Support for category.xml files - the new PDE mechanism for categorizing result in p2 repository is supported

Headless 'install JRE' support

Better defaults often renders the MSEPC unnecessary - automatic materialization to Target Platform for binaries often removes the need to use a MSPEC.

Using new p2 API, p2 'pure' reader, and using separate p2 agent - reduces risk of contamination of the running instance's p2 data.