Categories

Meta

Tag: Java

Before cracking on to step 2 in the master plan, where we rewrite the app in javafx instead of swing, it might be possible to tidy up how I build releases for the users. Currently I take the jar (previously created by eclipse and now by gradle) and then run the javabuilder command from java 1.8 with a whole bunch of parameters to create the install package for each platform, e.g. for mac it’s this:

This results in a great dmg in outdir/bundles that has the usual drag to install experience for mac. It’s also signed, and for the life of me I have no recollection how it gets hold of my developer certificates from the keychain, probably a smart default/convention (on the ToDo list to investigate).

What I’d really like is to have gradle do all this for me, without me trying to remember each time what I need to do for Mac and Windows. This is what the plugin at https://bitbucket.org/shemnon/javafx-gradle/overview promises, I think! This is what we’re discovering here, don’t want to gripe as it may be my lack of context but documentation seems very sparse.

First step was to download the 32MB repository and investigate the contents. Then following the readme and building a couple of the samples, it all seemed to make sense. What is really useful is the FullyExpressed sample, which shows most of the configuration you can specify if the built in defaults (called convention these days) aren’t suitable.

Then, when adding the plugin to my test gradle project for Drum Score Editor, it got messy. So the only way I could see to integrate is was to copy the file javafx.plugin into the gradle project directory and reference it in the build.gradle using plugin from: syntax. Once through this, I was simply not understanding the errors encountered when trying a gradle build, it seemed to be a clash between jar files / directives in the build.gradle. To save you the same pain, this is the point I realised that if I add the javafx plugin, I had to remove the java plugin from the build.gradle. From there, once I’d specified as a minimum the mainClass in the javafx closure in build.gradle, I could at least get a complete successful compile and deploy cycle, i.e. it produced the mac app, dmg, pkg etc. Short lived success though – running the app gave:

LSOpenURLsWithRole() failed with error -10810 for the file /Users/alanwhite/Development/gradletest/build/distributions/DrumScoreEditor.app.

I remember in the dim and distant past dealing with such obscure errors when trying to figure out in the bad old days, prior to javapackager, how to create the mac app. This one appears to say it can’t find a java runtime, but I learned a while back that there’s lots of misleading clues in this space. Turns out I’d mistyped the mainClass in the build.gradle – only telling you here in case anyone googles the error and maybe avoids a wasted hour tracking it down!

Time to move the last pieces of functionality explicitly stated in the old build scripts above, the name of the app is fairly simple, that’s a directive you can see in the final build.gradle below, however icons require some explanation, all became clear when I found this blogpost from the author of the javafx-plugin http://speling.shemnon.com/blog/2013/01/28/adding-icons-with-the-gradle-javafx-plugin/. Don’t be put off by the age of some of these posts btw (it’s 2015 I’m writing this), I’ve seen comments about javafx-plugin not being maintained. This will be a problem when it stops working, right now, I’m getting close to liking it by this point!

Having found my original icons that made the iconset in use on the Mac builds, after copying them into src/deploy/package and prefixing them with shortcut_, a gradle build generated the icns needed and buried them in the app.

The last piece of customisation is the developer id for signing the app. Not sure how the prior method works, but I think now it works by providing my name (as specified in the developer cert), by concatenating with the well known strings Apple use, it pulled it out of the keychain. Very happy with the progress here, next – integrating the Windows build.

Windows

First the good news, taking all the work performed on the Mac, adding to a git repo, pushing and pulling down to the windows machine and “it just worked”. gradlew did what is was supposed to in loading the correct version of gradle etc – really useful feature.

The problem was my build was failing when trying to build the installer. I learnt a lesson here after an hour or so poking around the web and trying different things. Important lesson: if something doesn’t work use “gradlew build –debug”. The output is verbose but all I had to work on before this was error 2 on exec of iscc.exe. Nice, huh? With the debug output I could see that iscc.exe was complaining the icon being used for setup was too large.

A bit more digging and it looks like javafx-gradle combines all the different sized pngs in src/deploy/package into a multi-layered ico file so windows can choose the best fit resolution. Useful except if you’ve got the full complement of sizes required for a full Mac iconset, the generated ico is bigger than the 100Kb that Inno Setup supports (taken from the source code on github).

By trial and many errors, I thought the best solution I could come up with for maintaining the conventional directory structures and minimal customisation to the build.gradle file was to use the icon configuration statement and put it inside the windows platform specific section, which overrides the convention if invoked. That way there’s no need to complicate the Mac build with additional statements, nor compromise on file locations.

This however wasn’t actually why it worked, I’d clumsily also deleted a 512×512 png at the same time as it turns out from checking the javafx-plugin source code that the icons statement doesn’t apply within the platform specific scope statements. Grr, controlled testing needed.

I’d like to have Android and iOS versions of Drum Score Editor, I’ve been asked many times. To do this, the recommended way according to each of the vendors of the platforms (Google, Apple, Microsoft) is to buy and learn their proprietary implementation languages to get the best seamless experience, completely rewrite your app and have it feel like it belongs on their platform, naturally. Drum Score Editor is 25,000 lines of Java, I’m not rewriting that several times!

Microsoft Windows: a different language, C++, different Windows APIs, a different App Store, licensing and purchasing technologies

Android: well it’s written in Java, but not as we know it Jim. A different build environment, yet another set of app store and purchasing technologies

Unfortunately that means lots of different skills, and lot’s of code rewriting multiple times; skills I can’t afford to employ or learn, and time I’d lose for little functional progress as Drum Score Editor is my hobby, not some multi-million dollar IT conglomerate.

However, there’s new technologies becoming available from the open source world that might make the concept of one piece of software for all platforms possible. Drum Score Editor is written in a language called Java and it’s user interface built using the Swing libraries. There’s a new way of writing desktop apps in Java called JavaFX. Once written in JavaFX, you can use various technologies to make your app installable, and runnable on Mac OS X, Windows, iPads and Android tablets. Some work will need to be done for each platform to make the app feel natural, but the bulk of the effort remains common.

Sidebar: I don’t think I’ll ever get away from the differing app stores and technologies, I do like the reach the app store concept gives, e.g. the free 1.97 version of Drum Score packaged for the Mac App Store had 1,077 downloads. Neat, but to replicate on all platforms, and provide an in-app purchase for the studio workflows is a lot of work.

Looking into the technologies needed to get closer to having the bulk of Drum Score’s source code being the same across all platforms and to reduce the amount of effort involved in packaging and releasing it, the first thing I need is a cross-platform build system. Gradle is touted as the way to go with the best integrations for Android and iOS, as well as being designed with Java in mind in the first place. Currently I use Eclipse’s built in build mechanism to produce an executable jar, and then use the javabuilder tool to create installers for Mac and Windows. Gradle appears to consume all of that, and integrate with Eclipse.

So ….. the plan!
Step 1: convert to Gradle, ensuring I can retain the Eclipse IDE, source code control in Git, and build Mac and Windows packages as before.
Step 2: convert Drum Score Editor to JavaFX, leveraging the development and packaging system in step 1, producing Mac and Windows packages
Step 3: produce iOS package, and see how usable it is & introduce platform specific code in Drum Score Editor for the iOS platform
Step 4: repeat step 3 for Android

Step 1: Convert to gradle
A little tricky figuring it out. There’s lot’s of documentation but it doesn’t seem to be complete, it’s hard to pick through it, and in some cases various stack exchange answers were the only source. Nevertheless if you’re used to working open source you’ll be used to picking around for the ultimate truths.

I wanted to leverage the convention over configuration paradigm as much as possible, without changing actual program source code.

My settings.gradle file looks like this:

rootProject.name = 'DrumScoreEditor'

This ensures the produced jar file with the application in has the same name, rather than the name of the directory you’ve put the code in to build it.

The dependencies statement is needed because I always copy locally the exact versions of 3rd party libraries I’m using. We can argue about best practises but this is to reduce my overheads. I do a lot of development disconnected from the network so need to be self-contained and as few distractions due to changing versions of these libraries. This effectively tells the compiler where to look for the 3rd party libraries. This is one area where I had to change the project structure.

The sourcesets resources directive is because the resources, i.e. images, that the app loads are currently in a well known location in the source tree, and this ensure they’re copied over into the equivalent location in the class tree that’s bundled into the final jar. I could have tried moving them, so the convention would work but that means a source code change – could be simple as I used a public static constant for the path, not taking the distraction on just yet.

The jar statement is all about how to configure the produced jar file. Here we say bundle all 3rd party libraries in the jar, and then specify the entry point to the program.

At this stage ensure you can run your jar, after ‘gradle build’ it’s in build/libs. A ‘java -jar DrumScoreEditor.jar’ and all was eventually well, once I’d got the build.gradle file looking as above.

A quick ‘gradle eclipse’ and the necessary files were created to allow a relatively drama free import into eclipse using File->Import, Existing Project Into Workspace. Ran first time, due to having resolved all the runtime libraries first as per the build.gradle.

Tonight I have 2 hours free. It’s time to pull together the various snippets of info and manual tasks I’ve been following to build my app that runs on Mac OS X and Windows desktops and engineer a script I can just run each time.

The tools: first javafxpackager. https://docs.oracle.com/javafx/2/deployment/packager.htm

I’ve used this partially to build the windows executable but am using manually built ant script for Mac OS and manually signing everything in the bundle messy. Let’s sort that first.

Mac OS X

My compile / edit / test cycle is in Eclipse, and when I’m happy I export a runnable jar with the required Apache libs embedded, so my starting point for packaging is the jar. Might be nice one day to have a task in Eclipse that just does the packaging, anyway no distractions tonight.

I’m using jdk 7 still on the Mac and javafxpackager is nowhere to be found. First hurdle. The hunt begins, firstly Oracle tell me it’s included in the version of the jdk I’m running, 7.0.6. Aha it’s been renamed javapackager – nope that’s not there either. Sod it, let’s update to the latest jdk 7 – not ready to go to 8 yet. Hmm, still no packager. Pokes about a bit and sees in /Library/Java/JavaVirtualMachines there’s a few versions of the sdk now including 7u71, the most recent. Checks it out and the 7u6 I think I’m running and lo and behold there’s javafxpackager in both. OK so where is Mac OS X picking the SDK up from, or more importantly where _should_ it be picking it up from.

And voila more learnings! Short answer is ensure JAVA_HOME is set preferably using /usr/libexec/java_home to choose. Now javac reports it’s using the jdk I just installed. OK but still no javafxpackager in the default path. Henceforth it shall be known as $JAVA_HOME/bin/javafxpackager ….

Back to the job in hand, how do I go from a runnable jar to a signed .app with an embedded jre and a suitably signed installer pkg? javafxpackager -deploy appears to be the way, or it’s equivalent using ant, which might be more extensible and usable going forward. $JAVA_HOME/lib/ant-javafx.jar is your friend at this point. 30 minutes reading and researching later I think the 2 hours tonight isn’t going to be enough …..

Prototyping using the command line, the following generated a usable .app

Seemed to strip out the Quicktime libraries that Apple reject your app store submission for, but can’t find a way to get it to sign everything for me yet, nor produce a .pkg (signed or otherwise). Using -native installer produced a well put together dmg with a typical ‘drag to /Applications’ image – nice but doesn’t help for the app store.

Lost 6 hours of my life tracking down this issue, and of course in the end it turns out to be my user error, although as always as a developer you challenge whether it needed to be this hard to debug.

Objective

What was I trying to achieve was to create mac app bundle to put in a dmg to distribute my java desktop app. In Eclipse, generated a Mac app bundle using the export task.

Into Finder, double clicked the app and nothing happens. A quick search around Console, shows:

A quick google makes suggestions like there’s no main entry point defined in the jar file

This however is a rat-hole, there is indeed no main entry point defined in the manifest but a few more searches reveal the package isn’t invoked using java -jar xx.jar, it’s invoked using java -cp xx.jar mainEntryPoint. So we check the Info.plist in the app bundle, and find the main class is defined. So what’s going wrong?

Set up the build.xml file from the various entries on the java.net and Oracle articles referenced above and running ant causes a no such file exception. It’s looking for an Info.plist file to be in the build directory. Odd, this isn’t mentioned anywhere!

If you’re not an experience Ant user (and I am not!), what this means is I forgot a target statement in the build.xml file, here’s the file I used at this stage which worked ….. or so I thought, it certainly allowed Ant to build the app bundle.

Frustratingly, trying to launch the generated .app bundle, I got exactly the same errors. What was going on here? I checked the Mac OS X Gatekeeper security level, it’s set to allow apps downloaded from anywhere so this should work!

Solution

There are many articles on various things which cause the 10810 error, from process table full to permissions and missing libraries.

What a huge diversion all these ratholes are you can go down. This is all so sketchy, returning errors meaning “unknown error”, meaningless “return code 1” in the syslog.

Bottom line is: do not try and follow the examples, work stuff out. Mine was failing because ……. I had assumed the main method name needed specific in the mainclassname statement, as is shown in the Oracle example. As soon as I made it the class name, and not the method it worked. i.e.

Should I post this? After all it was my error in specifying it wrongly. Yes, because if anyone else is searching for answers in this murky, poorly documented space, made messy by Apple divorcing Java without a properly focused handing of the baton to Oracle….. this might just help!

Introduction

We saw that leveraging the Swing approach reduces the amount of code involved, however patterns emerge in the code that show how Swing is encapsulating the underlying AWT routines, i.e. we see similar code moving from a generic controller to the TransferHandler. By using Swing containers rather than our own Model/View combination, we also reduce code.

Both the applications have limited functionality, what this article does is demonstrates how to add a visual icon of the widget being transferred, so the user can see exactly what the canvas would look like where they drop the widget.

Custom Component

When using the custom component (see DragginCustom2 in the github repo), the changes are all in the controller. When we’ve decided we’re starting a drag operation we create an image of the widget.

The next challenge is the user could grab the widget anywhere within it, so we need to work out the offset of the mouse pointer from the top of the widget, so the image can be displayed correctly without it jerking to the mouse pointer location.

This offset is also used to place the widget in the right place when the user releases the mouse and drops it in place. If we didn’t take the offset into account the widget would be added with it’s top left corner where the mouse pointer is, not where it is visually on the screen.

CanvasDragController

Point to note, in the dragGestureRecognized method, once we’ve identified the widget over which the mouse drag was started, we create a BufferedImage and ask the widget to paint itself in that buffer.

Notice how the bounds location of the widget are set to 0,0 temporarily. This is because we want it to be drawn starting at the top of the buffer, not the same distance in as it is from the canvas.

Also we need to store locally the dragImageOffset when the drag is initiated. That value is needed later in the drop method but the AWT system doesn’t store it anywhere, you have to take care of that.

Swing Component

The changes happen in the TransferHandler, full source is available in the DragSwing2 app in the github repo. Pretty much identical logic as the custom example above but just in different places.

The same issues apply about ensuring we work out the offset from where the mouse was dragged to the top left of the component, and using that when dropping the widget on the canvas.

CanvasTransferHandler

There are some nuances to using a drag image with this approach, particularly where you can call the setDragImage method and accompanying setDragImageOffset. We call it to set the image when the transferable is created, the only other option is when Swing asks for the drag options supported by the canvas, this didn’t seem a logical choice.

Although the TransferHandler approach allows the dragImageOffset to be retrieved when the drop is taking place, i.e. in the importData method, it bizarrely has no ability to tell us where the mouse was when get a chance to specify the drag offset! So we override the exportAsDrag method to squirrel away the mouse location at the time, so it can be used later.

Introduction

This article focuses on using the TransferHandler approach to drag and drop in a Java Swing based app. Many developers avoid the Swing TransferHandler as it’s not particularly well understood, however used appropriately it can save a lot of work. In order to compare and contrast the differences between using AWT drag and drop and Swing TransferHandler, we’re using exactly the same app as the first post in this series, and with reference to the previous point, we’re nearly 100 lines of code shorter overall – with considerably more functionality available to use which we won’t cover until a future post – keeping it working the same as the DragginCustom example at this stage.

The Example Application

The use case for the application here is starting very simply as being able to drag and drop boxes around in a window. These boxes are derived from Swing JPanel, to demonstrate fully how to do this if you’re using pure Swing components.

We will build this into a complete document layout application, for example think of having a sheet of paper and you want to manually (using drag and drop) move the various areas on the page around, areas could be widgets containing text, or images or musical symbols as in the main app I maintain (drumscore.whiteware.org).

Class Structure

DragginSwing contains the main entry point for the app, scheduling creation of all graphical resources on the Swing event dispatch thread (EDT, don’t worry about it if you haven’t heard of it before, just trust me at this stage the technique here is how to make sure your app remains thread safe and responsive).

In the constructor for DragginSwing, which is derived from a Swing JFrame, the main window container for the app, we create the resources needed i.e. the main Canvas which draws the CanvasWidgets, and the CanvasDragController which controls all actions on the CanvasWidgets on the Canvas.

In MVC terms the Canvas is the View and the Model in this case, and CanvasDragController is not surprisingly the Controller.

Once these objects are created the constructor exits and hands over execution to the Swing event driven model, which calls methods in the CanvasDragController as needed.

Don’t be concerned about the declaration of the DataFlavor object at the top of the DragginCustom object at this stage, it’s just so there’s a globally available definition of a data type throughout the file, in a production application the various classes involved would be in separate files for ease of maintenance so a public static could be declared in the right place.

We’ll move to that in a later post in the series, right now all the classes here are private classes within the DragginSwing class.

Canvas and CanvasWidget

These are the 2 classes which make up the main body of the application. In the version of this app based on custom components we had a Model to hold the widgets, however we’re leveraging the Swing component model here where the component stores the data like a model and knows how to render it like a View.

The CanvasWidget is derived from the Swing JPanel component also, in order to demonstrate how to store components within components, the standard containment hierarchy in Swing, and leverage the TransferHandler for drag and drop.

We’ll cover the controller class once we’ve taken a deeper look at the view and the widgets the model stores.

Canvas

The Canvas object serves as the view and the model, storing all the widgets as well as being responsible for the overall rendering of those widgets. One major difference with the design compared to the custom component described in the DragginCustom app in the prior article, is that Swing doesn’t always tell it’s view to update when changes happen to the components it’s storing, i.e. you can delete a component from the JPanel but you also have to explicitly tell it the data has changed and it should repaint.

class Canvas extends JPanel {
public Canvas() {
setLayout(null);
}
}

Very simply put, as we’re leveraging more of the Swing goodness, the Canvas is a lot simpler than when dealing with customer components. All we are doing is setting a null layout manager in the Canvas class, as we’re explicitly controlling where our CanvasWidgets are placed.

CanvasWidget

This is a very simple object whose only job is to draw a rectangle when and where it’s told to, and in one of two different colours, depending on whether it’s moving property is set.

CanvasDragController, the CanvasTransferHandler and the CanvasWidgetTransferable

The CanvasDragController is responsible for handling all user input directed at the view it’s registered on; interpreting that user input; and eventually manipulating the contents of the model appropriately. It leverages the CanvasTransferHandler derived from the Swing TransferHandler for much of this, making the CanvasTransferHandler an extension of the controller functionality, but provided mostly by Swing instead of coding it ourselves.

A CanvasWidgetTransferable is an object created by the CanvasTransferHandler to store all the details of the CanvasWidget it has identified to be dragged around. We’ll see why it’s important to take a copy of the widgets we want to drag and drop as we examine the way the TransferHandler approach works.

CanvasWidgetTransferable

This class can be regarded as a suitcase into which all the data that defines a particular CanvasWidget gets stuffed when it’s time for it to be moved.

In the constructor of the CanvasWidgetTransferable we take the bounds of the supplied CanvasWidget and store that rectangle, in a production app we’ll probably want to store more than just the bounds, if it’s a container in it’s own right, i.e. if it’s derived from JPanel it’s probably because you’ve got other components within it that you’ll want to preserve and have transferred with the object.

This data can be retrieved in a number of different formats, these are identified by what AWT refers to as DataFlavors. We allow the stored bounds to be retrieved in two different formats, either as a Rectangle object, or as a string representation of the rectangle object.

As per the prior post, the reason for allowing the data to be obtained as a DataFlavor.stringFlavor as well as the Rectangle we want, is that it means we can drop any CanvasWidgetTransferable onto any application (including non-Java apps) that can have text dragged to them – useful as your objects stored in the Transferable become more complicated.

The class is line for line identical between this Swing TransferHandler based app and the AWT based app – reason for this if it’s not clear is that Swing TransferHandler abstracts the underlying AWT architecture and simplifies it’s use, when used appropriately.

CanvasDragController

The controller is where all the users inputs are interpreted, and actions taken as a result, usually against the model. In our case all we’re doing is having the mouse gestures which happen while the cursor is over the Canvas directed to us, so we can initiate a Swing TransferHandler based drag and drop when appropriate.

Our app starts by having a single CanvasWidget in the top left of the window. If a user grabs it with the mouse and drags it elsewhere, our app immediately changes the colour of the grabbed CanvasWidget, so it appears in a lighter colour in it’s original location.

The thinking here is this allows the user to know where they’ve dragged it from until they’ve decided where to put it. Once we’ve told the TransferHandler to start the drag operation all interaction is controlled through our CanvasTransferHandler until the user releases the mouse.

Should the drag operation be cancelled, or fail for whatever reason the CanvasTransferHandler ensures the original CanvasWidget has it’s colour reset to show it’s staying there.

As described in the first post in the series the transferable is put onto the system clipboard so other apps or objects controllers will get told if the user drags the mouse over their components. With the transferable being formed the way it is, with DataFlavors, the other components controllers can examine if they understand the format and can use it to create a new dropped object.

The CanvasDragController starts by declaring it wants all the mouse gestures that happen over the Canvas, and that the Canvas should have the CanvasTransferHandler associated with it for all drag and drop actions.

As the CanvasTransferHandler is used, the CanvaseDragController itself is much simpler, with some of the functionality migrating to the transfer handler, and some being handled for us, by the underlying Swing classes.

CanvasTransferHandler

This is the centre-piece of the drag and drop operation, when using the Swing approach. The way to think about this is this is a two-part operation, and the methods in the TransferHandler are dedicated to each part. The first part is to export the data that’s to be transferred into a package that the system can drag around. The second part is to import that data if the user dropped it on this component.

Part 1 – Export

If a drag operation is requested on the component this is associated with, the TransferHandler is consulted to see if the drag operation is a valid one, this is determined in the getSourceActions method.

Swing uses the createTransferable method to have the contents of the object packaged for transfer. Once the drag and drop operation completes, Swing invokes the exportDone method to say everything’s done and to allow the source of the drag to do any tidy up operation which might be needed now there’s no further need for the package.

Part 2 – Import

While the mouse is being moved over the component this is associated with, the TransferHandler repeatedly calls canImport to see if we’ll accept the transfer if the user releases the mouse.

If they do drop the package here, Swing call the importData method so you can interpret the contents of the package and add it as appropriate.

Conclusion

We now have a working app, and like the custom component based one it has a few shortcomings – try running this app on Mac OS X, then on Microsoft Windows, then Linux. The behaviour should be the same imho, i.e. an image of the object being dragged appears under the mouse pointer. The API allows a way to fix this behaviour and we’ll look at that.

Also in the real world, we’ll need to cope with the user scaling the Canvas, i.e. zooming in or out. The standard approach to that in Java is an AffineTransform but we’ll need to figure out how to apply that transform to the TransferHandler, as it only knows the actual sizes of the components in the panel – will it tell us truly when over a zoomed component? This is problem that currently there’s no answer to and a true shortcoming in the TransferHandler design.

Another challenge is those CanvasWidgets aren’t always the same size, in fact the user may want to be able to change their size using the mouse, expectation in a regular user interface today is that you grab the corner of an object and drag it and we’ll show how to integrate that with the potential to launch a Swing drag and drop from the controller.

What if the app design has to allow the user to select one or more CanvasWidgets and then apply any action to the subset of CanvasWidgets that are selected? Maybe drag them all, or change all their sizes, while zoomed. How would they use the mouse to select the CanvasWidgets they wanted, yep by dragging the mouse around them, once again complicating the use of the drag gesture on the Canvas.

You’ve got the bones of a fairly complete drawing app if you have all that.

Introduction

Java Swing drag and drop isn’t trivial, and descriptions of it to be found out on the net are sometimes contradictory or incomplete. Even the official Java tutorial starts by saying avoid using the underlying AWT drag and drop functionality and use the Swing TransferHandler approach, and immediately goes on to show examples using AWT.

The reason for all this confusion is there is no one size fits all with the current Swing TransferHandler functionality.

So this is the first in a series of blog posts about different use cases and approaches for drag and drop. It confused me horribly trying to learn the different ways of doing things when writing a very complete desktop specialist music manuscript editing program. This is mostly what I’ve learned, and I’m hoping publishing this will help others out and also help me learn more should those who know better chime in!

The Example Application

The use case for the application here is starting very simply as being able to drag and drop boxes around in a window. These boxes are not derived from Swing components, to show absolute clarity of how to do this if you’re using a potentially large number of graphical constructs that don’t need the Swing overheads and function. This is often the case where you need the graphical items to be moved around (and resized, as shown in a later post) and you need to interact with them in a multitude of ways, but it’s still recommended at an overall app level to leverage the Swing framework to get the app up and running and all the other goodness Swing brings, e.g. menus, actions, multi-threading model and so on.

We will build this into a complete document layout application, for example think of having a sheet of paper and you want to manually (using drag and drop) move the various areas on the page around, areas could be widgets containing text, or images or musical symbols as in the main app I maintain (drumscore.whiteware.org).

This is a reasonably complete example of using java drag and drop where you’re using custom widgets on your drawing canvas and not other Swing components.

Class Structure

DragginCustom contains the main entry point for the app, scheduling creation of all graphical resources on the Swing event dispatch thread (EDT, don’t worry about it if you haven’t heard of it before, just trust me at this stage the technique here is how to make sure your app remains thread safe and responsive).

In the constructor for DragginCustom, which is derived from a Swing JFrame, the main window container for the app, we create the resources needed i.e. the main Canvas which draws the CanvasWidgets stored in the CanvasModel, and the CanvasDragController which controls all actions on the data in the CanvasModel.

In MVC terms the Canvas is the View, the CanvasModel is the Model, and CanvasDragController is not surprisingly the Controller.

Once these objects are created the constructor exits and hands over execution to the Swing event driven model, which calls methods in the CanvasDragController as needed.

Don’t be concerned about the declaration of the DataFlavor object at the top of the DragginCustom object at this stage, it’s just so there’s a globally available definition of a data type throughout the file, in a production application the various classes involved would be in separate files for ease of maintenance so a public static could be declared in the right place.

We’ll move to that in a later post in the series, right now all the classes here are private classes within the DragginCustom class.

Canvas, CanvasModel and CanvasWidget

These are the 3 classes which make up the main body of the application. As described above Canvas is the view and is responsible for rendering the data on the screen. CanvasModel is the model and is responsible for storing all the data for the app and is used by the view.

Of course the CanvasWidgets themselves could be more complex combinations of classes which make up an MVC model themselves but we’re not doing that here, at this stage, as there’s enough to take in already.

We’ll cover the controller class once we’ve taken a deeper look at the view, the model and the widgets the model stores.

Canvas

The Canvas object serves as the view and implements the Observer interface so that it can be notified by the model when data has changed and it needs to refresh the on screen display.

The update() method is invoked when the model notifies the view of data changes, and as we’re only observing the CanvasModel we were given in the Canvas constructor, we don’t need to go checking who invoked us, it’s safe to assume all we need to do is tell Swing to schedule painting the screen again, by calling JPanel.repaint().

For Swing aficionados note we’re not setting a null layout manager in the Canvas class, even though we’re explicitly controlling where our CanvasWidgets are stored. This is because we’re not even telling Swing about our graphical content for this pane, we do this by not using the JPanel.add method, and storing them in our separate model.

So if we’re not using Swing to do the rendering of our canvas why are we bothering to tell Swing about it? The Canvas is derived from JPanel, this is where in the class hierarchy we break from pure Swing to using our own graphical items, but there are some things we have to respect in the underlying Swing architecture.

Calling repaint() means that at the right point in the processing of events in the framework, Swing will call the JPanel standard routines for updating the display, these include paint(), paintComponent(), paintChildren(), paintBorder() and a number of other required methods to maintain the logical structure. We override the paintChildren() method and do our drawing there. By not calling super.paintChildren() in our override we effectively prevent Swing from managing the display contents and it’s all ours but within the framework which still allows things like borders should we want them.

CanvasModel

A very simple model implementation, which wraps a list of CanvasWidgets. It provides add and remove convenience methods and the getWidgetList method so the contents can be iterated, e.g. in the view for updating the screen.

The getWidgetAt method is provided to identify which of the CanvasWidgets in the list sit under a given point. This method is used exclusively at the moment by the controller to figure out if a mouse gesture was made over one of our widgets.

We provide the notifyModelChanged() method to standardise how we tell other objects who’ve registered an interest in the contents that a change to the contents has been made.

CanvasDragController and the CanvasWidgetTransferable

So that’s all pretty standard stuff up till now, this is where the magic happens, as they say at Disney World! CanvasDragController is the controller and is responsible for handling all user input directed at the view it’s registered on; interpreting that user input; and eventually manipulating the contents of the model appropriately.

A CanvasWidgetTransferable is an object created by the CanvasDragController to store all the details of the CanvasWidget it has identified to be dragged around. We’ll see why it’s important to take a copy of the widgets we want to drag and drop as we examine the way CanvasDragController works.

CanvasWidgetTransferable

This class can be regarded as a suitcase into which all the data that defines a particular CanvasWidget gets stuffed when it’s time for it to be moved.

In the constructor of the CanvasWidgetTransferable we take the bounds of the supplied CanvasWidget and store that rectangle as the bounds are effectively the only unique piece of data in every CanvasWidget.

This data can be retrieved in a number of different formats, these are identified by what AWT refers to as DataFlavors. We allow the stored bounds to be retrieved in two different formats, either as a Rectangle object, or as a string representation of the rectangle object.

So why do we allow two different ways of retrieving the data? Surely only the Rectangle object is of any use as that’s what Java will understand.

The reason for allowing the data to be obtained as a DataFlavor.stringFlavor is that it means we can drop any CanvasWidgetTransferable onto any application (yes even non-Java ones) that can have text dragged to them.

This is really useful for testing when your transferable becomes much more complicated. If things aren’t working out as you expect you can drag and drop into a text editor which will request the string format, and then you can examine exactly what’s been created in the transferable. This helps isolate if your issues are in the creation of the transferable or the retrieval.

If you’re now asking why bother, as Eclipse has a great debugger that could examine this, go ahead and try using that debugger while dragging an object around and let me know how you get on!!

CanvasDragController

The controller is where all the users inputs are interpreted, and actions taken as a result, usually against the model. This, as you can imagine, can lead to fairly lengthy logic in controllers. No exception to that rule here, the CanvasDragController is the lengthiest of the classes. As we grow the app in later posts, you’ll see how we structure the controller so it doesn’t grow to be a single monolithic, unmaintainable chunk of code.

So what does ours do, and how? Our app starts by having a single CanvasWidget in the top left of the window. If a user grabs it with the mouse and drags it elsewhere, our app immediately changes the colour of the grabbed CanvasWidget, so it appears in a lighter colour in it’s original location.

The thinking here is this allows the user to know where they’ve dragged it from until they’ve decided where to put it. Once the drag operation is started we’re in the frameworks hands until the user releases the mouse to drop the CanvasWidget at the new location. Our controller is then notified and it creates a new CanvasWidget at the dropped location and deletes the old one.

Should the drag operation be cancelled, or fail for whatever reason the controller ensures the original CanvasWidget has it’s colour reset to show it’s staying there.

Questions? Yes, like why on earth doesn’t it just change the x,y co-ordinates of the original CanvasWidget instead of all this complexity? Answer: what if the user is dropping it on a different Canvas from the one it came from? That’s one of the big plus points in using the provided drag and drop framework, it allows the objects to be dropped not only on other instances of a canvas, but also in other non-Java apps.

When the framework takes the transferable is puts it into a system clipboard and other apps or objects controllers will get told if the user releases the mouse over their components. With the transferable being formed the way it is, with DataFlavors, the other components controllers can examine if they understand the format and can use it to create a new dropped object.

The CanvasDragController starts by declaring the Canvas it’s been passed as being a component that should be watched for mouse drag gestures, and should such a gesture be detected, the CanvasDragController is to be told about it.

In the constructor it also declares the same Canvas as potentially having objects dragged to it (i.e. dropped on it), and should that ever happen, yep, the CanvasDragController wants to know about it.

The DragGestureListener interface defines methods which will be invoked if the framework detects a drag gesture has happened while over a component it’s been told to watch out for.

The DropTargetListener interface defines methods which will be invoked if the framework detects a drop gesture over a component it’s been told could have objects dropped on it.

We also implement a 3rd interface, the DragSourceListener. This defines methods which are called, at different stages in any ongoing drag operation, for the benefit of the component that the drag originated from, in case it wants to take any action. The only reason we do this in our app is to tidy up once we know a drop operation has completed elsewhere, i.e. delete the source CanvasWidget as we’ve created a new one elsewhere – because the drag operation in this example is a ‘move’, not a ‘copy’. We’ll show how to add that in the next posts.

Eagle-eyed Java coders will note that the CanvasDragController could have 4 or 5 fewer lines of code. The constructor could store the Canvas and it’s CanvasModel in variable accessible by all the methods in the class, instead of extracting them from the event parameter passed in each method.

I don’t do that in order to show that you can examine the components the events come from using the getComponent() method on each event parameter on the various drag and drop methods. I also prefer that as it localises the variables and you can see where their values originate from.

Conclusion

We now have a working app, it has a few shortcomings, try running this app on Mac OS X, then on Microsoft Windows, then Linux. The behaviour should be the same imho, i.e. an image of the object being dragged appears under the mouse pointer. The API allows a way to fix this behaviour and we’ll look at that.

Also in the real world, we’ll need to cope with the user scaling the Canvas, i.e. zooming in or out. The standard approach to that in Java is an AffineTransform but as you’ll notice above, there’s no MouseAdapter involved so we have to compensate and scale the values the DragGestureListener receives.

Another challenge is those CanvasWidgets aren’t always the same size, in fact the user may want to be able to change their size using the mouse, expectation in a regular user interface today is that you grab the corner of an object and drag it but wait … aren’t we already having Java interpret drags as being to move the object, how do we make it do both?

I’ve not finished yet, what if the user wanted to be able to select one or more CanvasWidgets and then apply any action to the subset of CanvasWidgets that are selected? Maybe drag them all, or change all their sizes, while zoomed – maybe. How would they use the mouse to select the CanvasWidgets they wanted, yep by dragging the mouse around them, once again complicating the use of the drag gesture on the Canvas.

You’ve got the bones of a fairly complete drawing app if you have all that.

What if in your app, you absolutely must have the CanvasWidgets as JPanels in their own right, and they have to be part of the component hierarchy and so they have to be added to the JPanel the Canvas is derived from, and therefore you can’t override paintChildren() either. We’ll take a look at that next, and then look at the compromises each approach brings.