Tag Archives: J

Post navigation

The other day I attempted to browse a J script described in an old blog post only to find that my employer’s network monkeys had blocked the file sharing service. I’ve railed about IT control freaks in the past. They will not rest until it’s impossible to do useful work. I fumed and grumbled until I perceived a bigger problem. I have so many references to program code in this blog that it’s getting tedious tracking them down. Wouldn’t it be nice if my hacks were neatly organized in one coherent repository?

Let me introduce jacks. jacks, or “J-hacks”, organizes the J related code referenced in this blog into a single GitHub repository. Most of the scripts in jacks are one-offs but some have proven so useful that it makes sense to store them in a repository and track changes. From now on jacks will be the first place to look for code from this blog. You pull the contents of jacks into a new Git repository with the commands:

It took me a few moments to settle on the name “jacks.” I considered “jokes” because programmers often take their code too seriously and “jocks” because J programmers are wild out of control convention eschewing code jocks but jacks won out when I remembered the refrain “jack be nimble, jack be quick, jack jump over” whatever coding problem is pissing you off.

I have resolved my DHTMLX standard edition row data extraction problem. The standard edition does not serialize grids or track user cell changes. You have to pay for such luxuries. Because I’m a foul software Grinch and this is just an exploratory hack I had to roll my own. I am posting the relevant JavaScript because I could not find similar examples. Here is how you can fetch rows from standard edition DHTMLX grids and save them as JSON in a hidden textarea element. Eric Iverson suggested hidden textareas and they work like a charm.

Grids are the most important GUI user object. It’s hard to think of a user-friendly data munching application that doesn’t have a grid beating at its heart. Consequently, any serious GUI interface contender must support grids. My previous post showed how to use MathJax with JHS. MathJax is an impressive and important JavaScript library; it clearly demonstrates the potential of CHJ1 GUI interfaces but let’s face it, mathematical typesetting will not win many consulting contracts. Grids won’t seal the deal either but their absence is a huge “next” signal. To support serious business and technical applications JHS needs grids.

Fortunately, the JavaScript world is grid saturated. The difficulty is not finding a grid but choosing among dozens of candidates. For this demo I Googled around and found DHTMLX. According to this probably biased article the DHTMLX grid performs well on large inputs and, more importantly, there is an open source version.

You have to start somewhere so I opted to use DHTMLX to build a simple CSV file editor. The CSV files I am going to edit are TAB delimited text files. Each file has a fixed number of columns with column names in the first row. Here is an example TAB delimited file. The idea is to load the file data into the grid. Tweak a few rows and save the result. By increasing the size of the CSV file we can gauge the performance of the grid. Let’s get started.

Using the DHTMLX grid requires some preparation.

Create a local directory and edit J’s ~config/folders.cfg to reference the directory with the name GridDemo. jpath '~GridDemo' should return the full directory path.

The main J script is ~GridDemo\GridDemo.ijs. Start JHS and load this file.

load '~GridDemo/GridDemo.ijs'

Then browse to this site.

http://127.0.0.1:65001/GridDemo

If all goes well you will see the following GridDemo page after pressing the Edit Grid button.

Screenshot of GridDemo running on Chrome

To load and edit files enter their fully qualified names in the Input and Output boxes and press Edit Grid. To edit a cell double-click it. To save changes press Save Grid.2 There are more sophisticated ways to pick files on JavaScript pages. It’s easy to pop up standard host OS file dialogs but it’s not particularly easy to determine host directory paths. This post outlines the demons web programmers must slay to select host files. JHS circumvents these difficulties by asking the J server, which is a typically a local console process, to do the dirty work. JavaScript’s access to local files is limited for security reasons but J has no such restrictions. Use the force Luke!

Three test files t100rows.txt, t1000rows.txt, and t5000rows.txt are included with the demo. On my test machines load times vary from fractions of a second for the smaller files to nine seconds for the largest. This is competitive with the basic C# grid control and fast enough for serious work.

In subsequent posts I will explore JavaScript/JHS graphics options and start the process of integrating, grids, graphs and MathJax with JHS.

With the release of J 7.01 Jsoftware “deprecated” COM. J 6.02 can run as a COM automation server but J 7.01 cannot.1 Throwing COM under the bus is hardly radical. Microsoft, COM’s creator, has been holding COM’s head underwater for years. Many .Net programmers cringe when they hear the word “COM” and the greater nonwindows2 world never really accepted it. COM is a complex, over-engineered, proprietary dead-end. Yet despite its bloated deficiencies a lot of useful software is COM based. So with COM going away, at least for J programmers, the hunt is on for viable replacements and while we’re hunting let’s rethink our entire approach to building J GUI applications.

J GUI applications are traditional desktop applications. They’re built on native GUIs like Windows Forms and GTK and when done well they look and act like GUI applications coded in other languages. This is all good but there is a fundamental problem with desktop GUIs. There are many desktop GUIs and they do not travel well. Programmers have spent many dollars and days creating so-called cross-platform GUIs but, if you wander off the Windows, Mac and Linux reservation, the results are not particularly portable. And, as portable GUIs rarely outperform native alternatives, programmers tend to stick in their tribal silos. GUI programming is a bitch, has always been a bitch and will always be a bitch. It’s time to divorce the desktop GUI bitch.

All divorces, even the geeky GUI variety, are hard. When you finally cut the knot you’re not entirely sure what you’re doing or where you’ll end up. All you know is that there is a better way and with respect to J GUI applications I believe that JHS is that way. JHS leverages the large CSS, HTML and JavaScript (CHJ) world and in recent years some impressive browser-based applications have emerged from that world. The application that changed my mind about JavaScript and browser-based applications in general is something called MathJax.

MathJax typesets mathematics. It renders both LaTeX and MathML using fully scalable browser fonts. This is better than what WordPress does. The following Ramanujan identity taken from MathJax examples renders on WordPress as an image.

MathJax renders the same expression with scalable fonts and supports downloading the expression as LaTeX or MathML text. This is pretty impressive for browser JavaScript. I wondered how hard it would be to use MathJax with JHS and was pleased to find it’s easy peasy.

Writing a basic JHS application is a straightforward task of setting up three JHS J nouns HBS, CSS and JS.3HBS is a sequence of J sentences where each sentence yields valid HTML when executed. JHS generates a simple web page from HBS and returns it to the browser. MathJaxDemoHBS is:

It’s not hard to use JHS as a general application web server. JHS provides many common controls right out of the box but to compete with desktop applications it’s necessary to supplement JHS with JavaScript libraries like MathJax. In coming posts I will explore how to use industrial strength JavaScript grids and graphics with JHS.

Despite seven decades of programming experience documenting software remains a challenge. There are many reasons for this sorry state of affairs with the most important being that programmers simply do not agree on the need for documentation. As pathetic as this sounds it’s not without merit. It all depends on what you call “documentation.”

Writing technical documents for management, marketing or users usually results in excruciating rounds of Dilbertian critiques. Everyone understands your code better than you do. If you provide too much detail, you get complaints. If you use unfamiliar words, you get complaints. If you point out limitations, assumptions or caveats, you get complaints. If you assume basic 8th grade reading levels, you get complaints. If you use nonstandard fonts or unauthorized style templates, you get complaints. No wonder many programmers hate “documentation” and blow off the entire problem by making ludicrous claims about “self documenting code.” The self documenting cabal may have fooled management but they’re not fooling the rest of us. The need for illuminating program documentation is as pressing today as it was for ENIAC coders in the 1940’s and, when in it comes to illuminating documentation, the best overall approach was pioneered by Donald Knuth over twenty-five years ago and goes by the moniker literate programming.

Providing basic literate programming support in JOD has been on my to-do list for ages. I’ve held off until recently because I have never been happy with my mark up options. JOD directly supports simple J scriptdoc compatible leading comment block formatting. For example many of my J verbs start with a comment block like:

Even if you can’t spell J I bet you have a good idea about what this “program” does and, if you doubt my claims, I’ve left you with some examples to try the next time you find yourself in J. Stupid comments may be for losers but telling comments, especially example laden ones, really help! And, if you really find comments distracting, JOD has a deal for you!

compj purges pesky comments and reduces tedious long identifiers like mask to pure compact J. Getting rid of comments is trivial, putting them back in: not so much! JOD’s simple comment block formatting has been very effective but it’s hardly literate programming.

Literate programming requires more muscle. Knuth used his own TeX. TeX and LaTeX are certainly up to the job, as are many HTML and XML approaches. Unfortunately, all these mark up formats suffer from “distracting taggyness.” I can tolerate LaTeX but HTML and XML drives me nuts. Yes, there are perfectly fine editors for all these formats, but remember, we are inserting the resulting text into code that we will be looking at for the rest of our miserable coding lives! We need a mark up format that’s stable, readable, versatile, easy to use and, this is very important, easy to ignore! Markdown is such a format. It’s almost ideal for program comments and is capable of much more. I’ve started using markdown in JOD and it’s already paying its way.

jodliterate.ijs is a J utility script that can generate semi-literate LaTeX documents directly from JOD groups. It uses a version of pandoc with J syntax highlighting, see Pandoc based J Syntax Highlighting for details. I consider jodliterate semi-literate because it’s completely at the mercy of the programmer. If you don’t store coherent markdown text fragments in JOD all you get is a nice syntax highlighted listing. But, if you actually write about your group, jodliterate can produce essential documents. jodliterate.pdf is an example of this tool being used on itself. Self reference always makes an excellent test case. jodliterate will be included in the next JOD release. Until then you can download the J script from this directory. As always referenced files are available in the files sidebar. Enjoy!

Some Pandoc conversions are better than others. Pandoc does a better job of turning markdown into LaTeX than LaTeX into markdown. It’s also better at converting HTML into LaTeX than LaTeX into HTML. Pandoc works best when converting markdown, the simplest of its inputs, to other formats. In fact Pandoc does such a good job of converting markdown to HTML, HTML+MathJax, LaTeX or PDF that many writers are now saving their source documents as markdown text knowing they can easily produce other formats as needed.

As handy as Pandoc’s markup conversions are this nifty tool also supports syntax highlighting for over a hundred programming languages. Unfortunately, my favorite language J is not on Pandoc’s list of highlighted languages. [1] Where have I run into this problem before? Luckily for me Pandoc is an open source tool and Pandoc’s author has made it easy to add new highlight languages.

Pandoc is a Haskell program. I’ve been aware of Haskell’s existence for years but until I decided to take on this specialized Pandoc hack I had never studied or used the language. Usually when you set out to modify a large program in an unfamiliar programming language you’re in for what can only be described as an f’ing educational experience. It’s a testament to the quality of the Haskell’s global libraries and standard tools that a complete Haskell novice can effectively tweak large Haskell programs. Here’s what you have to do.

Install the Haskell Platform. The Haskell Platform is available for all the usual suspects. I’ve used both the Windows and Linux versions. I almost installed the Mac version on my wife’s Mac but resisted the urge.

Get with the Cabal. Cabal is the main Haskell package distribution and build utility. Cabal comes with the Haskell Platform and is easily accessed from the command line. Type cabal --help in your favorite shell to view the program’s options.

Spend sometime playing with Hackage. Hackage contains a large set of Haskell packages including all the source code required to build Pandoc.

After installing the Haskell Platform and familiarizing yourself with Cabal try building Pandoc. This will thoroughly exercise your Haskell system. Instructions for building Haskell packages are here. After reading the package build instructions run the following in your command shell:

$ cabal update
$ cabal install pandoc

This will download, compile and install a number of Haskell packages. Where Cabal puts the packages depends on your operating system. Cabal saves Linux packages in a hidden local directory. On my machine they ended up in:

/home/john/.cabal/lib

If you managed to build Pandoc you’re now ready to add a new highlighting language. Pandoc uses the highlighting-kate package for highlighting. highlighting-kate works by reading a directory of Kate editor xml language regex based definition files and generating custom language parsers. We want to generate a custom J parser so we need to download highlighting-kate source and add a Kate xml definition file for J.

You can find such a J Kate file on the J Wiki here. Download this file by cutting and pasting and save it as j.xml. Now do the following.

Run the Pandoc version command pandoc --version of the Pandoc you just built to determine the version of the highlighting-kate package you need.

Use Cabal to unpack the required highlighting-kate package. This downloads the required package and creates a temporary subdirectory in your current directory that contains package source code.

If all goes well a Pandoc executable will be written to this subdirectory.

~/temp/pandoc-1.9.4.2/dist/build/pandoc

You can check the new executable by running pandoc --version. The result should display J in the list of supported languages.

Now that we have a Pandoc that can highlight J we’re almost ready to blog gaudy J code. However before doing this we need to install some custom CSS. Custom CSS is not available on free WordPress.com blogs. To apply custom coloring schemes get the custom package and learn how to use WordPress’s custom CSS editor. As daunting as this sounds it’s no problemo for my limited purposes. To enable tango style Pandoc syntax highlighting on your WordPress blog paste tango.css into the custom CSS editor, check the “Add my CSS to CSS stylesheet” button and then press the “Save Stylesheet” button. Now your WordPress blog will be sensitive to the HTML span tags generated by Pandoc.

To show that all this hacking works as intended you can check out the Pandoc generated versions of this blog post. I’ve posted the original markdown source with PDF, LaTeX and HTML versions. All these files are available via the files sidebar. You can generate the HTML version with the command:

To get other versions simply change the file extension of the output -o file.

Bonebridge puzzle in MYST IVClick for “Haven Age” Walkthrough

Finally we are ready to post syntax highlighted J code. The following J verb bonebridge generates all “likely” lock combinations for the MYST IV Bonebridge puzzle in Pandoc’s tango style. At one time I was a big fan of MYST computer games. I always enjoyed being lost in a beautiful puzzle which, if you discard the beautiful bit, is a pretty accurate description of my programmer day job.

bonebridge=:3 : 0
NB.*bonebridge v-- lists totem symbol permutations for boneNB. bridge.NB.NB. The solution to this MYST IV puzzle is similiar to the bookNB. shelf puzzle in Tomanha but requires far more exploration ofNB. the age.NB.NB. You are confronted with 5 bones on the lock. All the bonesNB. move independently. You can see the settings for 4 bones. OneNB. bone has a broken display. The four visible bones have 8NB. symbols on them in the same order. The 5th bone also has 8NB. symbols and you can "safely" infer they are in the same orderNB. as the visible bones.NB.NB. Four bone symbols match symbols found on totem polesNB. distributed around the age. There is a 5th totem pole butNB. fruit eating mangrees obscure the totem symbol and I haveNB. never seen it. The totem poles are associated with ageNB. animals. In addition to the totem poles there is a chart inNB. the mangree observation hut that displays a triangularNB. pattern of paw prints. The paw prints define an animalNB. ordering. The order seems to be how dangerous a particularNB. animal is; big scary animals are at the top and vegetariansNB. are at the bottom.NB.NB. Putting the clues together you infer:NB.NB. a) the bridge combination is some permutation of fiveNB. different symbolsNB.NB. b) two possible symbol orders are given by the paw chartNB.NB. c) you know 5 symbols and the 4th is one of the remaining 4NB.NB. If this is the case the number of possible lock settingsNB. shrinks from 32768 to the ones listed by this verb.NB.NB. monad: bonebridge uuIgnoreNB.NB. bonebridge 0NB. known in paw order
known=.s:' square triangle hourglass yingyang'
unknown=.s:' clover cross xx yy'NB. all possible lock permutations
settings=.~. 5 {."1 tapl known,unknown
assert.((!8)%!8-5)=#settings
NB. possible ordering - we don't knowNB. what the fifth symbol is but itNB. occurs in the 3rd slot
order=. 8#s:<''
order=. known (0 1 6 7)} order
order=. unknown (2 3 4 5)} order
NB. keep unknown only in 3rd slot
settings=. settings #~-.+./"1 (0 1 3 4{"1 settings)e. unknown
settings=. settings #~(2 {"1 settings)e. unknown
NB. strict row sequence adverb
srsm=. 1 :'*./"1 u/&> 2 <\"1 y'NB. retain strictly increasing and strictly decreasing rows
grade=. order i. settings
settings #~((< srsm)"1 grade)+.(> srsm)"1 grade
)

[1] J has its own syntax highlighting tools but they are not part of a document generation system. Pandoc’s highlighters elegantly feed into many output formats making them far more useful.

On July 24, 2012 I gave a short talk, Writing Portable J addons, at the 2012 J Conference in Toronto Ontario. I try to avoid “death by PowerPoint” with my presentations. Take a peek at the following and see if I succeeded.