This month, my series on Language Translation returns to lexical analysis, and I present the amazing new, improved Canon tool.

Parts of this tool are identical (or nearly so) to code presented in my third, fourth and fifth installments, and I will not repeat these parts this month (although they are, of course, included on the code disk).

Specifically, the tool is a filter program; I developed a skeleton filter program in my third installment. It uses no fewer than six state machines for lexical analysis and parsing; lexical analysis and state machines were the subject of my fourth part. And it uses the balanced binary tree routines I developed in my fifth part to implement a symbol table.

B. What the Tool Should Do

The Canon tool functions as follows: the program reads in a dictionary of substitutions, then reads input files, performs the substitutions as required, and writes the result. The difference between this Canon tool and the standard MPW Canon is that is will not perform substitutions within comments or strings.

The tool is controlled by the MPW command line. It takes several possible options, which may be in any order.

B(1). The Dictionary File

The dictionary file must be named on the command line, with the -d <file name> option. If no dictionary is named, the tool will abort.

The dictionary files format is simple: each substitution is specified on a separate line, with the identifier (according to the languages definition of identifier) to be replaced first, followed by its replacement (which must also be an identifier). For example:

blip blop

specifies that the identifier blip should be replaced by the identifier blop whereever it occurs.

There is a second form of substitution, which consists of only one identifier. All identifiers in the input that match the dictionary identifier will be replaced by the dictionary identifier. This can be used to force canonical capitalization.

Finally, the dictionary can include line comments. The tool will ignore everything between a # sign and the end of the line. It also ignores blank lines.

B(2). The Input Files

Input files may be specified by simply naming them on the command line.

The input files should be either Pascal or C source files. The tool will read them according to their filename extensions: if the file name ends in .p, it will be treated as a Pascal file, and as a C file if it ends in .c.

If there are several input files, some .p and some .c, the first one named on the command line controls. If no input file has either a .p or a .c extension, then Pascal is the default.

If there are no input files named on the command line, the tool will read from standard input. The language will be Pascal.

The -p and -c options override all of the above language rules and force the language to Pascal or C, respectively. If there are more than one such option specified, the last one controls.

B(3). Other Command Options

The -o <file name> option names an output file. If no output file is named, the tool will write to standard output.

The -s option will make the tool case-sensitive. The default is case-insensitive.

B(4). Example

Here is an example of the Canon command line:

Canon -d dict file1 file2 -p > dummy

tells Canon to read the input files file1 and file2, performing substitutions from the dictionary file dict. The input will be treated as Pascal source, and the output will be written to standard output, which is in turn redirected to the file dummy.

C. Designing the Tool

You may have formed the impression that I like table-driven software. This program has no fewer than eight tables in it: two for character translation, one character classification table, four lexical analyzers and a parser. These are all kept in the resource fork.

Driving a program with tables makes the coding simpler. The price you pay is that the logic is hidden in a table, and consequently rather obscure. If you lose your notes, you may have to re-write the whole table to make a minor change! Assuming you hang onto your notes, however, tables make your program easy to change.

After I had written this program, I realized that I had forgotten about strings. Sure, I had a version of Canon that did not make substitutions within comments but it still made them within quoted strings. So I added that at the last minute; I added a few lines and columns to the lexical tables in the resource fork, and changed two constants in the code. That was it.

C(1). Main Routine

The main routine reads the command line, sets appropriate flags, reads the dictionary into the symbol table, and finally filters the input file(s).

It reads the command line in two passes. The first pass is for setting flags; the second does the work. I need to set the flags before reading any files because I need to know the source language before I read the dictionary file.

After the first pass, the routine reads in the dictionary, opens the output file (if any), and then goes into the second pass. The second pass reads and filters each input file, writing the result to the output file (or standard output).

C(2). Case Sensitivity

We want the tool to be case-insensitive unless the command line option -s is used. This will require some modifications to last times symbol table routines (the only place where string comparisons occur).

One approach would be to transliterate the key strings before calling strcmp. I want to minimize changes to the symbol table routines, though, since I dont intend to reprint them in this article.

Another way, the way I have chosen, is to write a case-insensitive version of strcmp. Then all I need to do is change the name of the call in the insert and lookup routines.

Probably the most efficient way would be to use the first method in insert and the second in lookup. Since all the comparisons in lookup are between a key string and keys in the table, and the table would already be case-insensitive, Id need only a half-case-insensitive comparison routine for lookup.

Of course, I still need to allow for case-sensitive lookup, if the -s flag is set. What Ill do is have two transliteration tables, one converting uppercase to lower, and the other a straight identity table. Ill set a global pointer to point to the appropriate table for my comparison routine to use.

C(3). Parsing the Dictionary

The first thing the program has to do is read in the dictionary. It does this in two phases: a lexical analyzer breaks the dictionary into tokens (identifiers, carriage returns, and errors), then a parser finds substitutions in the token stream.

C(3)(a). Lexical Analyser

There are two lexical analyzers, one for Pascal and one for C, because C allows underscores in identifiers. (Another, and probably better, way to do this is to have one lexical analyzer and two character tables.) In the interests of brevity, I will limit the discussion to the Pascal version; the C version is identical, except that it adds underscore whereever letter appears.

The first piece shows the the pound sign is a line comment character; after reading a pound sign, we scan to the next carriage return and then go back to state 0. We push the return character back onto the input, though, since it isnt part of the comment.

The second segment reads an identifier. Again, the character that ends the identifier isnt part of it, so it goes back onto the input. The lozenge thing indicates that we are going to return a token (i.e., accept states). The example lexical analyzer in installment 4 was the whole program, and it never returned anything. This one is called by a parser, and it returns one token each time it is called.

The dictionary parser is a simple hand-made thing, and does not use YACC (that would be like swatting a fly with a hammer). A dictionary is a list of lines; a line may be blank, or it may contain a one-ID specification, or a two-ID specification. That is,

line -> CR
line -> ID CR
line -> ID ID CR

Heres a state machine to implement that grammar:

Figure 2: Dictionary Parser

Recall that the parser gets its data by calling the lexical analyzer, and thus receives only three tokens: ID, CR and ERR. State 3 is the error recovery state; it reads to the end of the line, and then goes back to state 0 for the next line. Returns to 0 from state 0, 1 and 2 correspond to the three lines of the grammar above. In the latter two cases, the specification is added to the symbol table.

To make substitutions in the input file, we begin with a lexical analyzer that finds all the identifiers. Again, there are two versions, one for Pascal and one for C. I will discuss the C version only; Pascal does not allow underscores in identifiers, and the two languages have different comment constructs. See the fourth installment of this series for a lexical analyzer that reads Pascal comments.

The first segment reads comments, and is identical to the comment-reader presented in the fourth installment. The next two read strings. (The second segment is also present in the Pascal version, for compatibility, even though Pascal doesnt use quotation marks for anything.) Canon does not do any syntax checking, and will read strings that go beyond the end of the line.

The fourth segment reads identifiers. When it finds one, the lozenge means look it up and see if theres a substitution to be made. This scanner, unlike the dictionary scanner, doesnt return anything; it runs until it finds the end of file, making substitutions as appropriate.

Here is the state table (which uses the same class table as the dictionary lexical analyzer):

What follows does not include all the code for the tool. Parts of it are scattered through my last two articles; refer back to those if you need to see it all. Alternately, the entire source is included on the MacTutor source code disk.

This is the string comparison routine to use in place of strcmp in the symbol table insert and lookup routines. The only other change I made to those routines was to rename the local variable compare difference (to avoid conflicts with this routine name).

The routine functions just like the C routine: it returns a negative number if string1 is less than string2, positive it string1 > string2, and zero if theyre equal. The actual number returned is simply the difference between the first pair of different characters. CASETABLE is a global pointer to the appropriate transliteration table.

I parse the dictionary in two steps: lexical analysis and parsing. The gettoken routine breaks the input into tokens, which the parser routine fits together into substitution specifications.

D(3)(a). Lexical Analyser

This routine is similar to the lexical analyzer I used in my fourth article. The major difference is that it returns tokens as it finds them, rather than keeping control from the beginning to the end of the file. It knows that it has found a token when it gets a negative state number; it converts it into the token number that the parser expects, and returns it. (This is probably unduly complex; I should have just let the parser use negative token numbers and avoided the conversion.)

The first half of this routine is set-up work. In addition to loading its own state machine, the parser also fetches gettokens state machine. Its easier to do the work once, here, than to repeat it each time I can gettoken. The it also opens the dictionary file, reads it in, and so on. Eventually, it gets to do some parsing, and this should look familiar.

There is one complication: gettoken will not only return a token number but will, in the case of an identifier, also return the tokens text. I dont want to overwrite one identifier when I read the next, so I pass a pointer to one string at the beginning of the line, and then a pointer to a second string when I want to read the next identifier.

This routine should be familiar by now, except for when it finds an identifier. The state table flags identifiers with a state of -1; when the routine reaches that state, it looks up the identifier in the symbol table and performs any required substitution. In all other cases (things other than identifiers, or identifiers with no substitution), the routine simply copies the input to the output.

The tool, as I have presented it here, is not quite perfect. It is very slow. I ran it using the cannon.dict file that comes with MPW; after first finding all the duplicate lines, it took 22 minutes just to load the dictionary! I was stunned.

The problem, it turned out, was the createnode routine. There are over 3200 lines in the dictionary file, and createnode calls NewPtr three times for each line, for a total of almost 10,000 calls to NewPtr. And NewPtr is very slow. When I re-wrote the tool to reduce the 10,000 to a few dozen, the time to load the dictionary dropped to 16 seconds. (Yes, Im bragging...)

I chose not to present the faster version in this article, because I feel it confuses the issue. The changes I made are not related to the topic, and make the code more complicated. Instead, Ive included both versions on the source code disk, and Ill now give a quick description of the differences between the two.

I got rid of two-thirds of the NewPtr calls by leaving the data where I found it. In the above version, I read the file into memory, then find identifiers in the data and copy them into strings, which I pass to createnode. Createnode in turn copies these strings into its data structures. In the faster version, I find identifiers in the data and write nulls at their ends, then pass pointers to createnode, which simply copies the pointers into the appropriate node fields. So in addition to 6000 NewPtrs, Ive saved 12,000 string copies.

The complication is writing the null character. There are times when you dont want to overwrite the following character right away. Suppose its a return character...

I reduced the remaining 3000 calls to a handful by allocating the nodes in large arrays. I put new nodes in the free slots of the array until it fills up, with no calls to NewPtr. Once the array is full, I have to use NewPtr to create a new one, but since I use a large array size, this doesnt happen very often.

Community Search:

MacTech Search:

Software Updates via MacUpdate

EtreCheck 3.1.5 - For troubleshooting yo...

EtreCheck is an app that displays the important details of your system configuration and allow you to copy that information to the Clipboard. It is meant to be used with Apple Support Communities to... Read more

WALTR 2 2.0.8 - $39.95

WALTR 2 helps you wirelessly drag-and-drop any music, ringtones, videos, PDF, and ePub files onto your iPhone, iPad, or iPod without iTunes. It is the second major version of Softorino's critically-... Read more

Carbon Copy Cloner 4.1.12 - Easy-to-use...

Carbon Copy Cloner backups are better than ordinary backups. Suppose the unthinkable happens while you're under deadline to finish a project: your Mac is unresponsive and all you hear is an ominous,... Read more

Dropbox 16.3.27 - Cloud backup and synch...

Dropbox is an application that creates a special Finder folder that automatically syncs online and between your computers. It allows you to both backup files and keep them up-to-date between systems... Read more

Microsoft OneNote 15.29 - Free digital n...

OneNote is your very own digital notebook. With OneNote, you can capture that flash of genius, that moment of inspiration, or that list of errands that's too important to forget. Whether you're at... Read more

Spotify 1.0.44.100. - Stream music, crea...

Spotify is a streaming music service that gives you on-demand access to millions of songs. Whether you like driving rock, silky R&B, or grandiose classical music, Spotify's massive catalogue puts... Read more

SpamSieve 2.9.27 - Robust spam filter fo...

SpamSieve is a robust spam filter for major email clients that uses powerful Bayesian spam filtering.
SpamSieve understands what your spam looks like in order to block it all, but also learns what... Read more

VueScan 9.5.62 - Scanner software with a...

VueScan is a scanning program that works with most high-quality flatbed and film scanners to produce scans that have excellent color fidelity and color balance. VueScan is easy to use, and has... Read more

Fantastical 2.3.2 - Create calendar even...

Fantastical 2 is the Mac calendar you'll actually enjoy using. Creating an event with Fantastical is quick, easy, and fun:
Open Fantastical with a single click or keystroke
Type in your event... Read more

PCalc 4.4.4 - Full-featured scientific c...

PCalc is a full-featured, scriptable scientific calculator with support for hexadecimal, octal, and binary calculations, as well as an RPN mode, programmable functions, and an extensive set of unit... Read more

Latest Forum Discussions

Christmas is fast approaching and that means it's time to prepare for Santa's yearly pilgrimage around the globe. Christmas Eve is an exciting time as parents help their kids get ready to welcome Santa. You've got the cookies and milk all planned... | Read more »

Galaxy on Fire 3 and four other fantasti...

Galaxy on Fire 3 - Manticore brings the series back for another round of daring space battles. It's familiar territory for folks who are familiar with the franchise. If you've beaten the game and are looking to broaden your horizons, might we... | Read more »

The best apps for your holiday gift exch...

What's that, you say? You still haven't started your holiday shopping? Don't beat yourself up over it -- a lot of people have been putting it off, too. It's become easier and easier to procrastinate gift shopping thanks to a number of apps that... | Read more »

MyTona, based in the chilly Siberian city of Yakutsk, has brought a little festive fun to its hidden object game Seekers Notes: Hidden Mystery. The Christmas update introduces some new inhabitants to players, and with them a chance to win plenty of... | Read more »

PINE GROVE 1.0
Device: iOS Universal
Category: Games
Price: $1.99, Version: 1.0 (iTunes)
Description:
A pine grove where there are no footsteps of people due to continuous missing cases. The case is still unsolved and nothing has... | Read more »

Niantic teases new Pokémon announcement...

After rumors started swirling yesterday, it turns out there is an official Pokémon GO update on its way. We’ll find out what’s in store for us and our growing Pokémon collections tomorrow during the Starbucks event, but Niantic will be revealing... | Read more »

3 reasons why Nicki Minaj: The Empire is...

Nicki Minaj is as business-savvy as she is musically talented and she’s proved that by launching her own game. Designed by Glu, purveyors of other fine celebrity games like cult favorite Kim Kardashian: Hollywood, Nicki Minaj: The Empire launched... | Read more »

Clash of Clans is getting its own animat...

Riding on its unending wave of fame and success, Clash of Clans is getting an animated web series based on its Clash-A-Rama animated shorts.As opposed to the current shorts' 60 second run time, the new and improved Clash-A-Rama will be comprised of... | Read more »

Price Scanner via MacPrices.net

Never Settle for Low Performing Wifi With iOS...

AppYogi Software has announced the release of WiFi Signal Strength Status App 1.0, the company’s new utility developed exclusively for macOS. WiFi Signal Strength Status App features a unique, single... Read more

DietSensor, Inc., a developer of smart food and nutrition applications designed to fight diabetes and obesity and help improve overall fitness, has announced the launch of its DietSensor app for... Read more

Best Buy has dropped their price on the 64GB Apple TV to $159.99 including free shipping. That’s $40 off MSRP.
32GB Apple TVs are on sale right now for $98 on Sams Club’s online store. That’s $51 off... Read more

12-inch Retina MacBooks, Apple refurbished, n...

Apple has restocked a full line of Certified Refurbished 2016 12″ Retina MacBooks, now available for $200-$260 off MSRP. Refurbished 2015 models are available starting at $929. Apple will include a... Read more

Apple has Certified Refurbished 13″ MacBook Airs available starting at $849. An Apple one-year warranty is included with each MacBook, and shipping is free:
- 13″ 1.6GHz/8GB/128GB MacBook Air: $849 $... Read more

Apple refurbished iMacs available for up to $...

Apple has Certified Refurbished 2015 21″ & 27″ iMacs available for up to $350 off MSRP. Apple’s one-year warranty is standard, and shipping is free. The following models are available:
- 21″ 3.... Read more

MacTech is a registered trademark of Xplain Corporation. Xplain, "The journal of Apple technology", Apple Expo, Explain It, MacDev, MacDev-1, THINK Reference, NetProfessional, Apple Expo, MacTech Central, MacTech Domains, MacNews, MacForge, and the MacTutorMan are trademarks or service marks of Xplain Corporation. Sprocket is a registered trademark of eSprocket Corporation. Other trademarks and copyrights appearing in this printing or software remain the property of their respective holders. Not responsible for typographical errors.

All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.