Java development with an iPhone touch pad for the Atari 2600 from an urban hip-hop perspective

Blackberry Secrets

I really want to follow up on my Groovy Maven Blackberry work and so far this is all I can do. Yesterday I managed to actually run it for the first time in over a year or two since I wrote it. My earlier blog post was completely inaccurate, indicating that all you needed is the plugin source. It has a HEAVY dependency on RAPC, and I am not authorized to distribute RAPC. (I am also not authorized to distribute breakfast cereal to to my children because I’m not as calorie conscious and portion aware as I should be… says the wife.) In order to actually use my plugin you have to create one of various “rapc-compile-tools” bundles and list it as a dependency of the plugin. It’s very important the you be version aware when building and listing the “rapc-compile-tools”. Let me elaborate.

My plugin looks in its CLASSPATH under a folder “/Blackberry-utils/” for the following files:
“rapc.jar”, “net_rim_api.jar”, “SignatureTool.jar”, “sigtool.csk”, “sigtool.db”, “sigtool.set”
In order for the magic to work you must create a Maven jar artifact which includes a “/Blackberry-utils/” folder holding these files. The plugin expands the files from the dependency into the system’s temp folder and then uses all of these files to actually perform the compile to cod format. It does a bunch of other (probably unnecessary) hickery-doodle to create a jar identical to what will be in the final cod.

What you need to do is download a version of the JDE. Then create a “Blackberry-utils” folder somewhere on your hard drive. Copy each of the above listed files into this folder then jar the folder. You can find most of them in your JDE download which can be expanded by running the installer. (See my earlier post about running the Simulator on Mac for details on running Windows installers on Mac.) The sigtool files are created after you build and receive a RIM certificate. (If you don’t have a RIM certificate you can fake these files by renaming empty text files but they must be present for the plugin to work.) Finally deploy the jar to your Artifactory (or other) Maven repository. If you don’t use a local hosted Maven repository then you can manuallly create a pom file place a copy in the jar (I forget exactly where it goes but I think it belongs under /META-INF) and place both the pom file and the jar in a Maven conventionally formed path under “$HOME/.m2/repo”. The final path will probably look like “$HOME/.m2/repo/your/group/artifactid/version”. It’s to your benefit to install Artifactory or something equivalent to simplify both this deployment and future license restricted dependencies like javax.mail and commercial APIs.

Its a rather involved process to create the rapc-compile-tools and I hope to simplify in the future. Also be version aware and you may want to create multiple versions of the rapc-compile-tools to support various Blackberry devices and OS versions. For instance, you may want to build version 4.3.0 for older Blackberry curve devices and 4.5.0 and 4.7 for the newer Storm, Bold, and Tour devices. What you DO NOT want to do is compile against 4.7 and deploy to a Storm running an older 4.5 OS. remember your target audience on blackberry is far less likely to update their firmware than an iPhone user. You’re also likely to run into users sporting the 4.2.1 and older OSes so use the lowest possible version of the RIM apis possible to build your application.

Success at the 11th hour… literally! It is now 11pm and I get the Blackberry Storm simulator to pop up on my Macbook Pro Leopard install under X11. To say that this is chalenging is an understatement. However with a little patience you too can read the remainder of this blog post and and have the Blackberry Storm staring at you. (Also, wait for my followup when I revisit running RAPC under Maven.) The biggest challenge most people have running Blackberry sims on OSX is the particular sim itself. I’ve discovered that not all sims are created equal. Particular sims load while others kinda load while still others crash and burn. Follow the directions here and Download the exact version of the sim mentioned and you should be ok. Or, if you’re like me you won’t find the exact version. The example assumes Storm sim version 4.7.0_4.7.0.46 but I couldn’t find it. Instead I was successful with version 4.7.0.76. Before that I got really close with the ol’ Sprint 8830-4.2.2.123. This one loads but gives me “Access violation reading from 0×00000024”. Many say you can get rid of this by running clean.bat but I couldn’t find it in my install. I also read that dedicating the process to a single core could help (Linux tip) but I couldn’t find a CPU affinity tool that runs on Mac. I remembered that I would get these kinds of errors in Windows when I had JAVA_HOME set to point to JDK6 or JDK5 depending on the sim. So I unset my JAVA_HOME but still got the error. Also the newer Storm version 5.0 gives me:
fixme:win:EnumDisplayDevicesW ((null),0,0×33ed40,0×00000000), stub!
fixme:file:MoveFileWithProgressW MOVEFILE_WRITE_THROUGH unimplemented
X Error of failed request: BadRequest (invalid request code or no such operation)
Major opcode of failed request: 128 (Apple-DRI)
Minor opcode of failed request: 7 ()
Serial number of failed request: 4685
Current serial number in output stream: 4685

I have a feeling there are a few other sims that work (try BlackBerry_Simulators_4.6.1.79_8350i.exe also listed in the example) but YMMV.

…And you’ll need to install MDS as well if you want any network access. Fortunately that’s not difficult at all. Just make sure you have JDK6 if you’re running a later version of MDS and recode the run.bat as an equivalent run.sh.

So I finally got the debugger attached to a Blackberry simulator. I can step through simple hello world apps in most simulators. I can’t debug our main app yet. I figured out how to hack a user supplied jad into a Blackberry project created with the Eclipse plugin. Its not straight forward. You can’t just add a Jad to the project. You can’t use the import command. You have to manually edit the “.jdp” file. I only know to do this because I remembered the rapc command line syntax. I also learned from my earlier JDE hacking, how to edit this file to add additional files to the project. I’m having trouble getting something more complicated loaded on the Storm simulator. It must be because I didn’t add the 4.7 component pack. Instead I tried to change the library variable created by the 4.5 pack to point to an existing net.rim.api.jar that I pulled out of a JDE 4.7 install. It’s late and the downloads for the rest of the component packs are taking forever. Why does the Blackberry update site want to authenticate you once for each component pack you download?

In response to a response to my white screen of death post I give you detailed visual directions on how to properly void your warranty, or if you have a company issued device like I do, how to assure nobody in your company ever trusts you next to the coffee maker much less with $300+ devices. That said, proceed at your own risk/discretion/arrogance/ignorance:

[click here to for the Subversion link to the Groovy Maven Blackberry Tools]

*Update* You do need to download and install the JDE or the Eclipse JDE plugin to be able to build a dependency for my plugin. I incorrectly stated this is optional below because it had been over a year since I used the tool myself. I am writing a followup post to elaborate on the inaccuracies.

…Because I couldn’t think of a dumber name. I’ve been sitting on this code for quite a while now. I’ve also been very busy. Busy learning/reading/writing JQuery/Java/Javascript for the MapQuest Local and Gasprices pages (my official means of keeping my house warm), busy fiddling with RubyCocoa, Objective-C and regular Cocoa on the side as my after hours pet-project, and busy trying to keep two crappy cars on the road while my hot water heater explodes leaking rusty wetness from years of neglect all through my finished basement while I sip Bud Ice a flight above completely oblivious to the disaster. (I’ve also been busy trying to put together some demo stuff for the kids at school… Cliff loves the kids like Martin.) What I haven’t been busy with is following up on the cool Blackberry stuff I used to do earlier this year. I’ve been watching my hit counters and page statistics climb up from the thirteen people that originally knew about me to a whopping 15.5 and I’ve been watching as those extra 2 1/2 people like to hang around my Blackberry posts. I’ve been feeling sorry for these 2 1/2 persons… particularly the 1/2 guy because his legs and lower torso will probably never grow back as he claws his way through the internets using that one good arm in a desperate attempt to pull his head and chest up to my home page expecting to finally see something worthwhile. (Thank you dearly Mr. Half-man-half-missing! I couldn’t have made 15.5 without you!!!) As a result I bring you today’s half-assed attempt at sharing my work.

I just committed three, count them, three maven plugins each written in Groovy over a decade ago (in computer years which is roughly 6 human months). These plugins are intended to make your mobile development life easier and, if you prefer Mac OSX, possible. I haven’t actually looked at the source code since they’d been written I just thought it was finally time to throw them out in the wild.

Maven Gant
So far I have a gant plugin which is something we use internally to launch Gant from Maven. I caution that I’ve found issues with the Groovy Maven support in that not all of the Groovy goodness is available such as the RootLoader and other things that I can’t remember hitting my head on. However it works for basic Gant builds and I think it makes its way completely through our complicated Maven/ant build wrapped in a Gant coating after we take care to explicitly file system load some extra Ant task jars. (Something about Maven calling Gant calling Maven calling Ant to load the Maven Ant tasks just doesn’t work right.)

Midlet Maven
I threw together a MIDlet helper Maven plugin to do OTA uploads to a slightly modified version of the Antenna OTA server. It should work with the official OTA server as well and if it doesn’t drop me a line and I can point to the single line of code that needs to be changed. I think it should do a PUT instead of a POST for the upload, my modified OTA server accepts either and I should just fix the dang file or post the modified OTA or both instead of rambling about it but since it’s easier to ramble than to do any real work, there ya’ go.

Blackberry Maven
This is my latest committed version of the Blackberry plugin for Maven. It’s the very same tool I used for compiling .cod files on my Mac way back. Alls yuh haff tuh do is install this Blackberry tools bundle in a secret location on your hard drive… or even better install a Maven proxy repo then pull the bundle from their. It’s simple yuh see… alls yuh haff tuh do is download something like Artifactory, unzip it, oh yeah… you’ll need something like Jetty to run the war file that you unzip… oops better grab Tomcat because I remember a lingering problem with Artifactory on Jetty, so yeah, get Tomcat right? Then configure it as a Windows startup service. Then maybe enable the admin and manager console cuz you’ll need to admin and manage it. Then maybe fiddle with the server.xml to get it on port 80, then mebbe you wanna put Artifactory in the ROOT context b/c you wanna be able to just browse the machine with a simple URL. Now why were we configuring Tomcat again? Oh yeah, the Artifactory thing that will eventually hold the extra bundle that the dumb rapc compiler depends on… You know what? Keep it simple. Copy/paste the repository folder from the Groovy svn location over top of your repository under ~/.m2. (Be careful on OSX because this is a replace copy instead of an append copy. Mac users should just drag the net folder from within the repository folder into their ~/.m2/repository). From there you should be able to use Maven to compile cod files from Java source. If everything works well enough, you should also be able to upload to an Antenna OTA server hosted in the clouds and then install to your device. Unfortunately the USB loader doesn’t quite work under OSX so unless you have parallels installed (in which case all the OSX specifics is moot) you’ll need to do it this way.

DANGER!!!
The Blackberry plugin supports only Blackberry OS 4.3.0 and higher. While it will generate cods that run on earlier revisions of the OS you will be subject to the all too painful verification error. I was in the middle of making it compatible with earlier revisions when I hit a few snags, particularly with the rapc tools. To be perfectly honest I STRONGLY reccommend a different approach if you want to build for Blackberry OS < 4.3.0.

These tools are sitting on the Groovy Subversion repository. You DO NOT NEED TO INSTALL GROOVY. You also do not even need to download/install the JDE. The only tool you need to use are the tools and compile to rapc files is Maven. Download/install Maven, check out from the subversion link, move the repository from the checked out folder into your $HOME/.m2 folder, and run “mvn install” in each folder then you’re all set! If there are any Maven/Groovy people out there that wish to help me build these tools out feel free to jump in.

Such a painful problem it is to create Blackberry software from anything other than their proprietary JDE. There are options, however. There’s some blackberry Ant tools (an independent project and also there’s a few things in the Antenna project). Then there’s RIM’s rapc docs, which aren’t too informative. Finally there’s the post I made a while ago detailing some of the inner workings of rapc. In the post and also in the comments I hint at ways to get images and other resources into the final cod file. To save you the pain of reading there I’ll write about it here. Once you figure out how to compile with RAPC and at least get a working cod file, you need to compile again. The trick is to pickup the jar file that RAPC creates next to your cod and stick images and resources in there. Finally, remove some rapc generated .csi and .cso files before you pass the jar back to rapc. (If you forget to remove these files rapc will complain and list them in the error output, though you might not understand what it’s trying to tell you. Just know if you get an error recompiling jar and you see mention of a file ending in .cs-whatever that you’ll need to remove the file(s) in question.) Here’s some pseudo-code since I’m too lazy/busy/uncertain/tired/vitamin-B-deficient/loaded-with-excuses to post the real solution:

What’s really going on in rapc, Blackberry’s crazed compile tool? Let me begin by telling you that everything you’ve read until this point is not entirely accurate. There are those that will tell you that you have to run preverify.exe prior to running rapc. Then there are those who will tell you that the bb-ant-tools are sufficient. There are people that complain that the RIM compiler has bugs in it. (This may be true but I’ve so far been able to fix broken builds by using all of the advice I’m including here.) I’ve also found a blog that claims you can build from non-Windows operating systems. I’ve read in some places that either WTK or the JDE must be in the PATH so that rapc can find a preverifier. While some of these tips, tools, tutorials work they all have limits. Eventually you’ll find your application either throws random NPE’s, verification errors, or infinitely reboots the device! Now pay attention!

Now let me explain my project. I have a little bit of dependency injection going on via source generation. Then I also have some external jar dependencies. All of this creates a bit of a challenge for basic rapc invocation. In order to generate my midlet cod file I need to:

Run a source generator

Include the generated source in my list of things to compile

somehow include my external jar dependencies in the compile.

Include resources in my compiled output

Pick up my suit from the cleaners

Get the entire thing to run on my Mac?

While the last bullet is entirely optional and irrelevant, I include it because its the only way to get completely familiar with rapc. That is, if I can meet/greet/beat the challenge of cross platform compilation of Blackberry then I’ll really understand the intracacies of compiling with rapc. Now onto the juice.

How Rapc works… I think…
There are various approaches to compiling Blackberry applications that I know of.

Compile your app against the MIDP and CLDC APIs (include either WTK or MPowerplayer or someother party’s midp and CLDC jars on your bootstrap classpath for javac… making sure you use javac1.3 or below), preverify with JDE preverify.exe (found in %JDE_HOME%/bin), jar and finally compile with rapc.

Compile your application source files entirely with Rapc.

The first option has problems because there’s the possibility that javac will generate class files that will cause verification errors or (as my team discovered recently) the white screen of death when loaded onto the device. The second option has problems because it’s very tricky and prone to the same errors unless you are very dillegent. It took a lot of head banging and trial/error to come up with these details so read closely.

How do we use rapc? This is the question that still plagues me. There’s little documentation around the nuances you’ll encounter when invoking the rapc compiler from the command line. The rapc compiler is an executable (.exe) file in the JDE install directory living under the bin subfolder. First thing to note is the rapc.exe just delegates everything to rapc.jar. That means the same exact command line parameters you would use with rapc.exe will work with rapc.jar. (Call the jar with the “java -jar” command.) The most important thing to understand is that you cannot mix and match. That is you cannot use the output of javac/WTK preverify to fuel rapc. Furthermore you need to pay strict attention to the version of the JDE you use because it must be the same (or earlier) version of your target platform. For eg., if you intend to target Blackberry devices running a 4.2.2 OS you should use the compiler tools from the JDE version 4.2.2 or earlier. If you don’t pay attention to this step your product will work just fine until it’s time to ship… at which time you’ll experience weird, impossible to track/reproduce on a developer workstation, errors. There are other important tools in the JDE that you’ll need as well. The preverify tool is required (unless you’re using JDE 4.3.0 and targeting OS 4.3+) as well as the SignatureTool.jsr. Finally there is the net_rim_api.jar which lives under the lib folder in the JDE install directory. The next big thing to understand is the command line options.

import
Use the import= to specify the location of the net_rim_api.jar. This flag takes the relative or absolute path of not only the rim APIs but any other jar or cod dependencies your application has. (I haven’t tried cod dependencies but the docs say you can list cod files.) Be careful because this is only for resolving dependencies. Anything you list here will not be included in the final output cod. What this means is any jar files listed here must either be compiled as separate cod libraries or their source files somehow inlined in the final compiled. Dependencies listed here are scope as provided. In other words it is assumed they will be made available at runtime by either the RIM platform or by some pre-installed package.

codename
(codname for 4.3.0+)
This parameter threw me for a minute when I attempted to roll back to an earlier version of the RIM compile toolchain. I believe (and I need to double check and update this fact later) that rapc version 4.3.0 and higher expect a codname flag while versions prior to 4.3.0 expect a codename flag. The value you specify here will be the value used in the output codfile name. Paths can be relative to the working directory (the folder rapc is called from) or possibly absolute paths. For instance if you give build/MyApp as the value then rapc will spit all of its output into the build folder relative to the working directory giving you a final MyApp.cod file there. (you’ll also have MyApp.jar, MyApp.csi, MyApp.debug and other files in this folder.)

midlet
This is the application type. Use this flag if you plan to develop a J2ME spec compliant midlet. Blackberry support another application type, cldc, which provides RIM specific features (more on that in possibly another post) and is specified by the mutually exclusive -cldc flag. A You have to -midlet or -cldc to specify which kind of application you are building.

cldc
Use this flag if your are building a CLDC application that adheres to RIM’s proprietary launching interfaces. CLDC apps offer abilities to run in the background and be started at device boot time. There are other hidden gems here that I’d have to get into with another post.

jad
Use this parameter if you have a JAD file that includes your application’s meta data. This parameter is optional as you may substitue a .rapc file to include the meta data. Application meta data includes things like the icon to display on the desktop, the midlets contained in your aplication bundle, runtime properties, etc. Referr to the J2ME midlet spec for more detail.

rapc file
This would be the path to a “.rapc” file that includes meta data about your application. I’ll probably follow up with another post to explain the details behind this proprietary RIM spec. While a JAD file also includes meta data there are certain properties that can be included in this file that cannot be included in the JAD file and vice-versa.

class or java files
The last parameter should specify either a jar file containing the compiled classes or be a list of the .java files you intend to compile. Here’s where rapc gets really interesting. If you precompile your “.java” files you had better be certain you run the RIM preverify tool against these class files prior to invoking the compiler. Also I will repeat that you must use the same (or earlier) version preverify as your target platform. For eg., if you intend to target Blackberry devices running a 4.2.2 OS you should use both RIM’s rapc and preverify tools from the JDE version 4.2.2 or earlier. If you don’t pay attention to this step your product will work just fine until it’s time to ship… at which time you’ll experience weird, impossible to track/reproduce on a developer workstation, errors. (I said this before didn’t I?) Rapc is very picky here and the behavior varies between versions. It first calls out to javac assuming that the Java compiler is in the PATH. It then passes the output from javac to preverify. Note that in JDE 4.3.0 both “.java” source compiling and preverification is done from within the rapc.jar while earlier versions of rapc call out to the command line using the command “preverify” (not “preverify.exe”… this is an important distinction) and passing standard preverify parameters. Finally it creates the final “.cod” that will run on the device. What is included in the cod file depends on what is passed on the command line and what’s available from the working directory. For instance, your application icon (and I found this through trial and error) must be locatable from the working directory unless you specify an absolute path in the jad file. If you compile a jar file (not recommended for rapc prior to 4.3.0) then everything in that jar file (images, multimedia files, application resources, etc.) will be included in the final cod file. If you compile “.java” files directly from rapc then no application resources will make it into the final output cod file, only the compiled .class files will be there (along with the application icon specified in the jad if it can be found from the working directory). It took a lot of banging my head to figure this out. Finally instead of a jar file or space delimited list of “.java” files, you can specify a text file that lists (new line delimited style) all of the “.java” files you wish to pass to the compiler. This is a potential work-around to command line size restrictions that may be present on some versions of Windows. This file should be given with an @ prefix. For eg. if your have a MyApp.txt file then the last parameter for rapc will be @MyApp.txt. This is the preferred method of giving files to rapc because as your application grows so will the list of files.

One last got’cha I found was with trying to use rapc.jar directly instead of rapc.exe. If you are running a JDE earlier than 4.3.0 then you’ll get into trouble here because there is a secret parameter passed from rapc.exe that RIM won’t tell me about. Because the command line syntax for the jar is identical to the command line syntax for the “.exe” you can almost get away with it but because of a secret exchange happening between the Jar and exe you’ll get undefined behavior and weird errors in certain circumstances. This last gotcha is what makes it impossible to compile on an OS other than Windows while targeting a RIM OS earlier than 4.3.0.

In the end I’ve determined that it is not possible to accurately build a blackberry application on an operating system other than Windows. Also while there may be some truth to the myth of bugs in rapc I believe it all boils down to a bunch of nasty assumptions the RIM developers made during the design. (they assumed everyone would be so thrilled with their Swing based IDE that they would throw away Eclipse/Idea/Netbeans and burn their Macintosh laptops in celbration of the obviously superior RIM tool chain.) I do believe that if you pay close attention to the details outlined above you can diagnose most preverification and random erros you may be suffering from. I plan to revist this topic in the future as I’ve taken a break from mobile and begun work on some other things. However I do not plan to let rapc defeat me entirely. Keep an eye out because I’ll likely be updating this very same post with more details.