Archives

All posts for the month August, 2015

A lot of the posts leading up to this were exploratory of the Action! language and were occurring as I was writing this program. This program is something I’ve wanted to write for a while. In this post I present a utility to read information about dBase II/III and FoxBASE database files from the PC. Something never intended for the Atari 8 bits. I’m going to create several utilities to read and write to this format. This first one just displays information about the DBF file itself and is thusly named DBFINFO:

If file is valid (cursory evaluation at this point)

If file has an associated memo file

Date Last Updated

Number of Records

Header Size

Data Record Size

Each field name, type, length, and applicable decimal precision

Ability to dump the output to the printer

To do this I created a few databases with a set number of fields and various numbers of records. I then used documentation for the dBase III file format, a hex editor, and careful scrutiny of the byte streams to decipher the files. The documented format I found may have been correct, but it didn’t completely match the test files I created. I’ve lost the link to the page I got it from and subsequent searches have yielded what appear to be fairly accurate mappings. Maybe the original link was for a different dBase version. At any rate, it was only a few bytes off in a couple of places and only took careful examination to resolve. I’m not going to cover the structure definition in this post, but I will include the links I do have at the conclusion.

With an accurate file mapping, I started writing code to read a file. The program here is the result. This program includes a little bit of everything I’ve covered in the past Action! posts. While the code is fully documented, I am going to break it down in this post. This time the breakdown will follow the complete source – I think that makes more sense than having it before like I have done before. And since this uses several files via INCLUDE, those are listed as well (but not broken down because their contents have previously been broken down in posts).

In addition to code I wrote it uses two code libraries from the Action! Toolkit: PRINTF.ACT and CHARTEST.ACT. The source code INCLUDEs them from D5. Alter the drive for the drive you have the Action! Toolkit loaded into.

It also uses the Action! runtime library. I copied the runtime library SYS.ACT to the same drive (D2) as the program source. I did this because it requires a modification. For a successful compile the PRINTF routine has to be commented out of the runtime (SYS.ACT) because it conflicts with the PRINTF from the toolkit. Normally I have the runtime in disk 4 (D4), but here you see it included from disk 2 (D2).

Source Breakdown : DBFINFO.ACT

Excerpts from the complete source above are further explained here. In this first section all the includes are pulled in. The first is the Action! Runtime package. This one is a copy and has PrintF commented out because I want the PrintF from the Action! Toolkit instead as it has padding built-in. The runtime package lets the program run without the Action! cartridge installed. Then I pull in two parts from the Action! Toolkit; the expanded PrintF routine and some Character Test routines. Following that I pull in the custom library routines I’ve written (broken down by category):

; Inc Action Runtime
INCLUDE "D2:SYS.ACT"
; Inc Action Toolkit parts
INCLUDE "D5:PRINTF.ACT"
INCLUDE "D5:CHARTEST.ACT"
; Include library
INCLUDE "D3:DEFINES.ACT"
INCLUDE "D3:LIBIO.ACT"
INCLUDE "D3:LIBMISC.ACT"
INCLUDE "D3:LIBDOS.ACT"

In this section I declare the dBase Header record, or all of it minus the actual field definitions anyway. There is a memo flag which indicates if the file has an associated memo field file; the year, month, and day of the last update; the number of data records; the size of the file header; and the data record size including one byte for the deletion flag:

These variables are used for storing the last encountered error (bgErr), and the address of the original Error vector. I change it to customize error handling:

; Global vars
BYTE bgErr=[0]
CARD cErrO

This is the custom quit routine meant to clean up environment things the program changes. I had grander plans for this but haven’t fleshed it out entirely yet. It just resets the Error vector back to the original. In its current state, it could just as well be a single line of code at the end of the program instead of a PROCedure:

This is the custom error routine. It accepts the error encounter as a byte parameter. It sets the global variable bgErr so it can be referenced later by other parts of the program. It then displays the appropriate error message. Non-DOS (i.e.; program custom) errors are high numbers like 255. Since this program is only designed to read dBase II/III, and FoxBASE files, that is currently the only custom error:

This routine reads a string of x bytes (bL) from the specified device (bD) and stores the result in the given string (pS). When it encounters a NULL byte (0), it starts padding the string with space until all bytes required are filled:

This is the start of the long (too long) main routine. The printing should be broken down into a separate function which would make it a lot shorter. In the next version perhaps. This declares a DBHead variable (dHd) to store the dBase header info (minus field definitions). It then declares 3 character strings to store the database file name (cDBF), string storage for a field name (cF), and small buffer (cB). It then declares a handful of bytes for various things; bI is an input byte; bFn tracks the field number; bLp is for looping; bTp is the field type; bFL is the field length; bFD is the field decimal precision; bHT is the header terminator; bRT is the record terminator. Then there are two cards; cI is an input card; cM is used to hold the number of bytes consumed by all records via calculation. Then there are two last bytes; bPr is the send to printer flag, and bSD holds the SpartaDOS detection flag:

This sections saves the original error vector and assigns it to the custom one. It then sets up the screen for output. I used plain ASCII for all these blog posts because ATASCII doesn’t display properly.

This section reads the first byte of the file. This is the signature byte. It tells what flavor of dBase created it. Then it displays the name of the dBase flavor based on the value. And then it optionally prints the same information:

This section decides if the program can continue. It is capable of reading dBase II, dBase III, and FoxBASE files. If the signature byte is not one of those, call the error routine and drop to the endif later in the code to exit cleanly, otherwise start the next section:

This reads the next 20 bytes from the file. The result is discarded. There are some additional flags in a couple of bytes but they are of no interest for this program:

; Read 20 bytes filler
EatByteD(4,20)

This calculates the # of field definitions it needs to parse from the file header. It’s the header size minus 32, all divided by 32. It then displays a header line for the subsequent field information and optionally prints it:

This prints the field information that was just read in, and optionally prints it. It starts with the field number and name. The number is printed using Action! Toolkits PrintF to take advantage of padded output at 3 characters wide. This also displays, and optionally prints, the field type in a short descriptive form. And it displays and optionally prints the field length and decimal precision:

This checks if the number of data records is 0 or not. The record terminator is in different positions based on the number of records. This byte is read to validate if the file is structure properly:

; Check record terminator
; 0 record file
if dHd.cNDR = 0 then

If the number of data records is 0, read the next byte which is the record terminator; hex 1A:

; Get record terminator, expect $1a
bRT=GetD(4)

If the number of data records is greater than 0, read all the bytes for all data records inclusive of each records deletion flag (number of data records times the data record size). The output is not needed for this program so it is discarded. Then read the record terminator:

This checks if the header terminator was read correctly. If the value is not right the file structure is likely corrupt or otherwise invalid. If the value is not hex 0D, display and optionally print an error:

This checks if the record terminator was read correctly. If the value is not right the file structure is likely corrupt, truncated, or otherwise invalid. If the value is not hex 1A, display and optionally print an error:

This concludes the check of the files signature byte to ensure the file can be parsed; from far above:

fi

This cleans up by closing the printer on device 5, if opened; and the dBase file (cDBF) on device 4:

; Close printer if opened
if bPr = TRUE then
Close(5)
fi
; Close DBF file
Close(4)

This concludes the initial device open success check for the dBase file (cDBF) on device 4 from far above:

fi

In this final section the custom quit routine is called to restore the original Error vector. Then if SpartaDOS was detected a call to my library routine SDx is made. This causes the program to JMP to DOSVEC so it will return to the SpartaDOS command prompt:

; Restore Error handler and clean up
DIQuit()
; If SpartaDOS, exit through DOSVEC
if bSD=1 then
SDx()
fi
RETURN

Results

I ran a PC DOS based version of a similar program as a validation against each of the files. I’ll present screenshots of both the PC DOS and my Action! version against each file.

The first is a zero (0) record DBF file (RECFLD.DBF) – essentially just a database structure:

The second is a single (1) record DBF file (RECFLD1.DBF):

The third is a triple (3) record DBF file (RECFLD3.DBF):

The last is a one hundred (100) field DBF file structure (no records – RECFLD99.DBF). Because there are more lines than will fit on the screen I redirected the PC output to this file: PC Output PDF

…

For the 99 record (100) one I also send the output to the printer: A8 Output PDF

All of the above showed databases with fields names of 8 characters. Here are screenshots from both the PC and Atari showing field names with 10 characters, just to prove it works:

I’ve included an ATR image as well. It has the program source, the libraries, the compiled program, and sample databases. Download the ATR. Remove the “.key” from it and add “.ATR”. It is not a Keynote file! ATR and zip are not allowed hosting types here so I added “.key” to it.

And finally there is this thread on AtariAge forums talking about dBase for Atari. Looks like dBase II was released for the Atari ST only.

Links

Here are some links discussing the dBase file format where I discovered what most of the structure is:

Like this:

I’ve been programming in one context or another nearly all my life. Sometimes for fun, sometimes professionally. Even now, though programmer is not my title, I find myself programming a lot, writing programs in interpreted languages like Perl, and scripted programs in command shells like Korn/BASH. I think programming even at a simple level like scripts in Korn (modern), or BASIC (retro, possibly some modern) is an essential skill for anyone working with computers.

Needless to say the early days on the Atari 8 bit were instrumental in teaching me logic sequences that are necessary to write programs. These are so ingrained I often think in “code” especially when solving problems. In my career, when I was strictly a programmer I was exposed to a lot of other programmers writing styles, many of which I could not stand for one reason or another. I didn’t want to write the code that someone else couldn’t stand so I immersed myself in best practices books.

Two books in particular played huge roles in how I code today. Though they are older books now, they still have some great fundamental teachings. One was geared a bit toward programming in C, but much of its teachings can be directly applied to other languages. When you see any of my code, with very few exceptions, the principles I learned in these books are apparent. After you finish reading this post you’ll understand why my variables seem to start with the same letters, and a bit more about my coding style.

Both of these books are Microsoft Press books. They were both released in January of 1993. The first of which is “Writing Solid Code” by Steve Maguire. The second is “Code Complete” by Steve McConnell. I highly recommend both of these books to anyone that writes code in any language.

Here are the top three topics I learned from these books, not in any particular order of importance.

Variable Notation

One of the things you may have noticed in much of the code I’ve posted so far is how I name variables. I adopted a form of naming called Hungarian Notation in the course of reading these books. There are arguments for and against it’s use. In all the years I’ve been using it I’ve not run into any of the negatives, so I endorse it’s use. You can read more about it at this link. So why use it? In simplest terms it aids in readability, essential self commenting code.

My implementation of the notation doesn’t follow the specification exactly. In short I preface all variables with a lowercase first letter of its type, but I don’t apply the function/procedure naming in absolute terms. So a variable of type char would start with a c, one of integer would start with i. I apply to each language I code in using that languages primitive types. One thing I do that is slightly different is naming of globals. I will add a g, typically the second letter in the variable name to signify a global. The system has worked great for me, but your mileage may vary.

Comments

Likely this largest lesson I learned from these books was the value in comments. Comments in code have saved me from many hours of reverse engineering. Think you’ll write some code once and never touch it again. Ha! Good luck with that. The intrinsic value of well structured comments will be readily apparent the first time you need to touch:

Code you wrote x years ago, say 10 for example. Will you remember or know what that specialized function does, how it works, or what data or string syntax it expects to work with many years later without any comments. Possible, but not likely.

Code written by someone else. This I find to be the most irksome. Even if the author commented it, sometimes the comments are not descriptive enough.

Comment, comment, comment!

Debugging

Debugging can be a nightmare. It was especially hard for me to debug the 6502 Assembly code I was learning last year. I didn’t have the tools or experience to do it easily and ended up spending hours tracing through it on paper! Not ideal. The top 3 things these books taught me about debugging are:

If the language supports it, use assertions to validate variable values. In absence of assertions, use printed output.

If you have a debugging tool, use it to trace through code blocks.

If you have a complex calculation, validate it with an alternative algorithm.

Links

Here are all the links from the post in one convenient location, as well as some additional relevant ones.

Like this:

In this post I expand on using records. This time creating a record array that contains a character string, or more specifically two character strings. Action! won’t allow you to add a CHAR ARRAY as part of a record (TYPE) definition. You can get around this limitation by using BYTE POINTERS. You place the string as the last variable in the TYPE definition, but as a BYTE declaration.

To include multiple string variables, a series of bytes must be added to “reserve” space for the first string. For the tContact TYPE below, bFNm is first name, bLNm is last name, and bR1 through bR10 are the “reserve” bytes. None of these will be referenced via the record.field syntax, though you could for types other than CHAR ARRAY.

Pointer Calculation

Suppose you have a BYTE ARRAY name bArray for storage space, a BYTE POINTER named bpFirst for first name, and bpLast for last name. The first name is 10 bytes (11 with size), and last name is 15 bytes (16 with size). The last name offset into the record is therefore 11. The size of the entire record (first name and last name) is 27 bytes. First Name Record 1 will be stored at 0 offset into the storage array. Record 2 will be stored at 0+(1*RecordSize) into the storage array. Record 3 will be stored at 0+(2*RecordSize) into the storage array, etc. Last Name Record 1 will be stored at 0+FirstNameSize offset into the storage array (11 bytes). Record 2 will be stored at 0+(1*RecordSize)+FirstNameSize offset into the storage array. Record 3 will be stored at 0+(2*RecordSize)+FirstNameSize into the storage array, etc.

Program Source

The source is fairly well documented and I’ve given a good explanation of how it works so I won’t break it down line by line:

Like this:

I’m writing a utility program to do something useful, stay tuned, and in doing so I needed a routine to get a yes or no response from the user. I didn’t want the user to have to press the RETURN key, so I decided to read the keyboard without an input. I also wanted validation on the input so only valid keys would exit, in this case Y,y,N,n.

I created the following routine which returns a BYTE value. The value will be 1 if either Y is pressed. The value will be 0 if either N is pressed. It takes one parameter which is a BYTE value. 1 indicates the routine should echo Y or N to the display after the press. The source will not be broken down as it is commented well.

Like this:

This time I’m exploring inline Assembly language within Action!. This is more or less the equivalent to Atari BASIC USR routines. It turns out to be pretty easy to do.

In this example I use inline Assembly to execute the following code:

JSR $000A

This breaks down into 3 bytes of code. The instruction JSR is $6C. The address breaks down into two bytes (LSB first) $0A and $00. To add the instructions as inline Assembly, they get wrapped in square brackets with each byte separated by a space like this:

[ $6C $0A $00 ]

This code simply does a jump to the DOS entry point vector (DOSVEC). The code is useful with compiled Action! programs running from SpartaDOS. A normal return results in a system hang so the special exit is required to return to DOS.

The sample program I created tests to see if SpartaDOS is loaded. It will report if SpartaDOS is present. If it is, the program will use the special exit routine through DOSVEC jump, other wise it will exit normally.

Breakdown

This section brings in the Action! Runtime library so the compilation will build a stand-alone executable that does not require the Action! cartridge:

INCLUDE "D4:SYS.ACT"
MODULE

This is a function which checks memory location 3889 for value 0, 15 ,68, or 89. Any of these values indicate SpartaDOS is loaded. The function returns a BYTE value of 1 if SpartaDOS is detected, and 0 if it is not.

This calls the function IsSparta to test if SpartaDOS is loaded and stores the result flag in bSparta. Then it prints “DOS : ” without an EOL:

; Is it Sparta or not
bSparta=IsSparta()
Print("DOS : ")

This section tests the Sparta flag bSparta. If it is 1, it prints “SpartaDOS” then calls the procedure SDExit to exit via the JSR to DOSVEC. If it is not 1, it prints “Other” then exits normally (RTS instruction):

Like this:

In this post I expand on the Action! record data type. A record is a combination of multiple data types. It is declared as a TYPE, then a variable of that type is initialized just like it would be with other types such as BYTE, CARD, and INT. Why would you need this? To group common elements together such as a date, or an address.

Declaring a record type takes the following syntax (type definition followed by variable declaration):

Like this:

Manipulating bits can be useful for a variety of reasons. One good example is speed. Bit operations are primitive to the processor and are typically very fast. On older processors like the 6502 in the Atari, bit shifting to do division and multiplication can be significantly quicker than the arithmetic counterparts.

Another good example is storing flags. This can be especially useful if memory is tight. In a single byte 8 flags can be stored as opposed to storing them in individual bytes, a space saving of 7 bytes.

Action! provides the following bitwise operators:

AND (&) : Compares two bits. Both bits must be set (on) for the return value to be set (on)

OR (%) : Compares two bits. Either bit, or both, can be set (on) for the return value to be set (on)

XOR (!) : Compare two bits. When both bits are set (on) or not set (off) the return value is not set (off). When either bit is set (on) the return value is set (on).

LSH : Shift bits of a byte to the left. With Action! the shift occurs across both bytes of CARD and INT types. The number of shifts can be specified as well. Left shifting by one bit is the equivalent of multiplying by 2.

RSH : Shift bits of a byte to the right. With Action! the shift occurs across both bytes of CARD and INT types. The number of shifts can be specified as well. Right shifting by one bit is the equivalent of dividing by 2 (for positive values).

Sample Program

Here is a quick program to demonstrate a few of the operators. Comments to explain are again inline: