\documentclass[a4paper,12pt]{article}
% for: LaTeX DogWagger version=`2.0.0' fileTarget=`cache/cache.c' startComment=`//' noWarn=`yes'
% removed: include=`everything'
%========================================================================
% LaTeX Dogwagger takes LaTeX files and pulls out verbatim comments, %
% concatenating the text enclosed in these verbatim comments %
% into executable code. Copyright (C) J van Schalkwyk, 2005. %
% LaTeX Dogwagger is made available under the GNU Public Licence (GPL). %
% To NOT include a particular verbatim section, a comment line %
% containing the text: DogWagger dogsAllowed=`no' must precede *ANY* %
% line containing the text \begin{verbatim} ! %
%========================================================================
% NB. use PdfLatex to create a PDF document from this file! NOT include=`everything'
% We strongly recommend using the following line with PdfLatex
% \usepackage{pdfcolmk}
% (It fixes up problems with the LaTeX color package due to necessity for
% a PDF stack).
% We obtained it from:
% http://www.tug.org/tex-archive/macros/latex/contrib/oberdiek/pdfcolmk.sty
% you can put the pdfcolmk.sty package in the same directory as your .tex files
% You \emph{can} comment out the line but colour won't work that reliably.
\pagestyle{headings}
\usepackage{amssymb}
\usepackage[]{graphicx} % for pdflatex
% \usepackage[dvips]{graphicx} % for LaTeX -> dvipdf
\usepackage[pdftex]{color}
\usepackage{pdfcolmk}
\usepackage{longtable}
\usepackage{float}
\usepackage{times}
\usepackage[ %dvipdfm, %remove the dvipdfm for pdflatex!
bookmarks=false,pdfborder=0,colorlinks=true,%
linkbordercolor={1 1 1},% % white border is invisible
linkcolor=red]{hyperref} % THIS PACKAGE MUST BE THE *LAST* ONE!
% \hyphenation{}
\definecolor{MyGreen}{rgb}{0,0.7,0}
\begin{document}
% Copyright (C) Johan Michael van Schalkwyk, 2005 (J van Schalkwyk)
% This documentation and code is released under the Gnu Public Licence (GPL), a copy of which can
% be obtained at:
% \href{http://www.gnu.org/copyleft/gpl.html}{http://www.gnu.org/copyleft/gpl.html}.
% Please note the conditions of this licence.
\setlength{\topmargin}{0pt} \setlength{\voffset}{0.5in}
\setlength{\oddsidemargin}{0.5in} \setlength{\textheight}{585pt}
\title{Analgesia Database: \\ Caching library\\ ~ \\ \normalsize Version 0.90}
\author{J.M. van Schalkwyk}
\date{\today}
\maketitle
\tableofcontents
\newpage
\section{The C file: cache.c}
The menu structure of the pain database is somewhat hierarchical: we work by ward and
then select an individual patient. This suggests that if we had some way
of tweaking our tables so that only data for a particular patient were accessible, we
might speed SQL data access considerably. We find that this is the case --- performance
is far better if we temporarily constrain the PROCESS table to refer to a single patient
(say that patient with database ID number 1234),
and even better if we subsequently constrain the EPOCH table to refer only to such processes.
We do so by saying respectively:
% DogWagger dogsAllowed=`no' : don't DogWag this one.
{ \color{MyGreen}\footnotesize\begin{verbatim}
CACHE(PROCESS.Person.1234)
\end{verbatim}}
\ldots and \ldots
% DogWagger dogsAllowed=`no' : don't DogWag this one.
{ \color{MyGreen}\footnotesize\begin{verbatim}
CACHE(PROCESS:EPOCH.Process)
\end{verbatim}}
We can subsequently UNCACHE the tables in reverse order.
This library is very similar to that described in \emph{IdxLib.tex}.
\subsection{Rationale}
Examining things in more detail, the rationale
for the existence of this library is threefold:
\begin{enumerate}
\item Up to 90\% of PDA processor time is consumed by our SQL SELECT statements;
\item There are major problems with speed of SQL queries on the PDA, exacerbated by
our somewhat clumsy coding, and the sub-optimal file handling in PalmOS;
\item Our menus are pretty hierarchical, and in particular, once we've chosen a
particular patient, we could comfortably ignore the greater part of the database! The
caching program permits us to do this `ignoring'!
\end{enumerate}
Consequently I've decided to write caching routines, which cache files along the lines
of the following example:
\begin{enumerate}
\item We locate all PROCESSes pertinent to a particular
patient, and create a temporary PalmOS `file'
(aka database) called p-PROCESS which is similar to the permanent PROCESS table, but
contains only processes relevant to this particular patient.
\item Similarly, we use these identified processes to finger all observations which
are relevant, and we strip these out of the EPOCH table, placing them into a temporary
p-EPOCH table.
\item When in this \emph{caching scope}, we never access the permanent PROCESS and
EPOCH tables. We only access the far smaller temporary tables.
\item Sleight of hand: We will set things up so calls to access relevant tables are
via CACHE library whenever we are in the caching scope! Might even consider making
this standard, and the CACHE library decides!
\item We might perform similar tricks on all of the subsidiary tables. If we don't cache
\emph{all} dependent tables, then our software will detect an apparent loss of relational
integrity when it tries to match keys from dependent tables to our cached tables in
the course of searches through those dependent tables, but this isn't the major issue
it would appear to be, as those searches were doomed to fail anyway!
\item When outside caching scope, table interrogation is as usual (and slow). The
`temporary' tables are deleted on exiting caching scope.
\item Note that because of the clumsiness of PalmOS in finding files, it will probably
be wise to eventually `fix' things so that this library can locate local IDs. We don't do
so at present.
\item When no longer needed, we destroy the unwanted temporary files.
\end{enumerate}
\subsection{A caching upgrade}
\label{formats}
Initially, the CACHE instruction on the PDA (we have not yet created something similar in Perl)
was very simple. Submit the ID of the \emph{person} of interest, and the library routine
does all the rest; submit zero, and all cached files are removed, as is the ability to substitute
a p-file for the main database file.
Now we've created a more generic CACHE instruction. If we submit not only the person, but
an indication of the chain of databases, we can perform independent caching in several areas.
For example, we might say
% DogWagger dogsAllowed=`no' : don't DogWag this one.
{ \color{MyGreen}\footnotesize\begin{verbatim}
CACHE(PROCESS.Person.1234)
\end{verbatim}}
\ldots the implication being that we identify all PROCESSes referring to a certain
Person (with key value 1234).
We might then turn off caching by saying:
% DogWagger dogsAllowed=`no' : don't DogWag this one.
{ \color{MyGreen}\footnotesize\begin{verbatim}
UNCACHE(PROCESS)
\end{verbatim}}
With such an approach it would even be possible to maintain many cache threads simultaneously.
For example, if we are entering a particular menu while caching is on for a given person,
we might cache particular menu components thus:
% DogWagger dogsAllowed=`no' : don't DogWag this one.
{ \color{MyGreen}\footnotesize\begin{verbatim}
CACHE(MENUITEMS.miMenu.909)
\end{verbatim}}
\ldots would whittle down MENUITEMS to just those items which reference the given menu. We would
later UNCACHE just MENUITEMS to restore the status quo. CACHE(0) might still turn off all.
We would also be able to sequentially cache dependent menus. For example if we have a table
EPOCH which depends on PROCESS through its Process key, we say:
% DogWagger dogsAllowed=`no' : don't DogWag this one.
{ \color{MyGreen}\footnotesize\begin{verbatim}
CACHE(PROCESS:EPOCH.Process)
\end{verbatim}}
In order to implement the above, we would need some way of looking up which menus were
cached. This lookup should be implemented within the Cache library as a simple linked
list rooted in the `globals' within that library. In addition, during the actual caching
process, we need some sort of 'recursive' linking of lists as we pass down the
hierarchy (for example PROCESS : EPOCH : NONEVENT).
There is a further problem. If we have a `terminal' table such as NONEVENT (which isn't
referenced in turn by any other table), there's no point in meticulously storing
a linked list of all row keys identified on the offchance that
somewhere later we'll say `CACHE(NONEVENT:sometable)'!
The wrinkle we introduce is thus:
% DogWagger dogsAllowed=`no' : don't DogWag this one.
{ \color{MyGreen}\footnotesize\begin{verbatim}
CACHE(EPOCH:~NONEVENT.Epoch)
\end{verbatim}}
The tilde signals that we can save space and time by not storing the NONEVENT row keys. Clearly
a tilde in the wrong place (in a table with future dependencies) will
cause failure of referential continuity, so this
feature should be used with due caution.
\newpage
\subsection{Fine structural detail}
\label{structure}
For you to understand what we're doing, we must review some structural features of \emph{our}
database tables.
\begin{enumerate}
\item One of our database tables
is represented as a PalmOS `database' (which we'll call a file).
\item The first record in that file (record 0) contains header information about
our database.
\item The format of our database is fully described in the document \emph{PerlPgm.tex},
however we need to know that there is a 32 bit negative number at offset +8 which
contains the (negated) number of records in the file.
\item We also must know how to access the column descriptors. You:
\begin{enumerate}
\item First get the 2 byte number of columns at offset +0xE in record zero: As always, this
is big-endian (larger byte leftmost).
\item Examine each column in turn. \emph{Offsets} of the column details are located in
two byte numbers starting at +10h from the beginning of the record.
\item Column details are specified in our column descriptor format, also described
in \emph{PerlPgm.tex}. For our purposes the first two bytes contain the relative
offset of the column name, and the second two bytes the length of the column name,
a maximum of 15 characters. By `relative offset' we mean relative to the start of
the column descriptor.
\end{enumerate}
\item We use knowledge of the above formatting to retrieve the column number of the
relevant column within a database, for example we are interested in the column called
`Person' within the PROCESS database, and `Process' within the EPOCH database.
\item An important feature of our headers is that there is an additional pointer
above the last of the column descriptor offsets, which points to one above the very
last byte of the header. We can also therefore easily retrieve the length of
the header, without asking PalmOS.
\item Similarly, important data can be retrieved from each data row in any of our
database tables. At offset +8 we have the (positive) primary key for that line, a 32 bit
signed integer.
\item The value for a particular column in any row can be obtained by going to
+10h bytes from the start of the row, and then adding 2 bytes per column (The
first column is column zero). Get the two-byte value from the resulting offset.
The absolute offset (from the start of the row) of the column datum we want is that
two-byte value. Its length can be determined from the subsequent offset, and
the length of the entire row can be determined by examining the very last offset.
\end{enumerate}
\subsection{Includes}
Pretty standard includes:
% DogWagger sectionTitle=` (Section $[SECTION])'
{\footnotesize\begin{verbatim}
#include
#include
#include "../palmsql3A.h"
// for MAINCARD etc.
#include "cache.h"
#include "../err/ERRDEBUG.h"
#include "../console/CONSOLE.h"
#define MAXACTUALROWS 2000
// move this to palmsql3A.h
#define MAXTABLES 30
// braces and belt, is this a reasonable limit??
#define MAXIDNODES 500
// hmm. check this.
#define MAXCACHESTORBUF 32+4+MAXACTUALROWS*4
\end{verbatim}}
\subsection{Overview of Main functions}
\label{overview}
We have a few simple functions:
\begin{enumerate}
\item MAKECACHE initially needs to be given a particular Person identified in the PROCESS table.
This function will then whip once through the PROCESS table, and while so doing create
the p-PROCESS table.\footnote{We use the braces-and-belt approach of deleting all old
p-tables if they are detected.} At the same time as doing this, MAKECACHE must create an array
of process primary keys.
At future invocations,
MAKECACHE must then similarly abstract relevant rows from EPOCH into p-EPOCH, which will
shrink dramatically. It uses the array of process primary keys to determine which EPOCH
rows should be retained in p-EPOCH.
\item We might stop there, but it is tempting to use similar mechanisms on all files
dependent on EPOCH (etc). Note that if we do this, we must similarly retain an array
of epoch primary keys, and retain rows in other p-files based on these epoch primary keys
(and even if necessary, on down the line).
\item Note that (at present) only SQL \emph{queries} will use the p-files. However,
every time we alter a non-p file, we will need to update or reconstruct the relevant
p-file (that is, if it exists).
\item ISCACHING returns a value of 1 if caching is occurring, 0 if not, and under zero
if caching is disabled.
\item KILLCACHE deletes \emph{all} p-files.
\item CACHEFINDTABLE is simplicity itself. When the SQL library invokes its
CreateTableList function, it invokes CACHEFINDTABLE which if caching is off,
returns an unchanged table name, otherwise altering the name to p-TABLENAME.
CreateTableList must supply the name in a buffer with space for two extra bytes
so that the p- prefix can be inserted! Subsequent searches then simply use
the pared down form of the table!
\item CACHEINSERT meets the problem of a new row being inserted into e.g. PROCESS.
It simply inserts the same row at an appropriate place in p-PROCESS.
\item CACHEUPDATE is similar to CACHEINSERT, finding and \emph{replacing} the relevant
row in the cache file whenever a row is updated in the source.
\item DISABLECACHE disables all caching.
\end{enumerate}
\newpage
\section{Basic functions}
The following are fairly standard startup functions.
% DogWagger sectionTitle=` (Section $[SECTION])'
{\footnotesize\begin{verbatim}
Err start (UInt16 refnum, SysLibTblEntryPtr entryP)
{
extern void *jmptable ();
entryP->dispatchTblP = (void *) jmptable;
entryP->globalsP = NULL;
return 0;
}
\end{verbatim}}
During caching, we will store temporary data in a clumsy linked list
(rather than having a clumsy array) thus:
% DogWagger
{\footnotesize\begin{verbatim}
typedef struct {
Int32 key;
void * next;
} cacheNode;
\end{verbatim}}
The idea is that as we select out 32-bit primary keys, we will need to store them in a list
for future reference. See later usage.
Next, a simple data structure used for storing the name of a cached table. The convoluted
typedef is `just because' of the design of C. Otherwise we have trouble with self-referential
pointers.
% See http://www.embedded.com/showArticle.jhtml?articleID=9900748
% another option is to typedef struct TableNode but then the pointer breaks,
% and if we say struct TableNode * this introduces its own problems (need to cast..)
% we can turn the TableNode * into a void pointer, but this is nasty.
% DogWagger
{\footnotesize\begin{verbatim}
typedef struct TableNode TableNode;
struct TableNode {
Char * tname;
Int16 nlen;
cacheNode * data;
TableNode * next;
};
\end{verbatim}}
In the above structure, we store the name of the data table (and the length of the name), as
well as a cacheNode pointer to all the selected 32-bit keys, which will be stored in
the reverse order in which they are encountered.
Here are the globals:
% DogWagger
{\footnotesize\begin{verbatim}
typedef struct {
UInt16 CONSOLE;
UInt16 ERRLIB;
UInt16 DEBUGFLAGS;
UInt16 refcount;
Int16 ENABLED;
TableNode * TABLELIST;
} CacheLib_globals;
// include count in expectation of multiple openings!
\end{verbatim}}
We root our \emph{most recent} TableNode in the globals, and it in turn points to the
next most recent, and so on down the line.
% DogWagger
{\footnotesize\begin{verbatim}
Err CACHEOpen (UInt16 refnum)
{
SysLibTblEntryPtr entryP = SysLibTblEntry (refnum);
CacheLib_globals *gl = entryP->globalsP;
if (!gl)
{
gl = entryP->globalsP = MemPtrNew (sizeof (CacheLib_globals));
MemPtrSetOwner (gl, 0); // note use of sys fxs here, above
gl->CONSOLE = 0;
gl->ERRLIB = 0;
gl->DEBUGFLAGS = 0;
gl->refcount = 0;
gl->ENABLED = 0; // off at start.
gl->TABLELIST = 0;
}
gl->refcount ++;
return 0;
}
\end{verbatim}}
We have to allocate memory for our globals using MemPtrNew; ideally we should wrap
the system calls as usual (Here using \verb+c_+ rather than \verb+w_+).
% DogWagger
{\footnotesize\begin{verbatim}
Err CACHEClose (UInt16 refnum, UInt16 *numappsP)
{
SysLibTblEntryPtr entryP = SysLibTblEntry (refnum);
CacheLib_globals *gl = entryP->globalsP;
if (!gl) {
return 1; // not open
}
// clean up:
*numappsP = --gl->refcount;
if (*numappsP == 0) {
// here would free up open files, delete temp files ... ?
MemChunkFree (entryP->globalsP);
entryP->globalsP = NULL;
}
return 0;
}
Err nothing (UInt16 refnum) {
return 0;
}
\end{verbatim}}
\subsection{Some wrappers}
The following were lifted from \emph{Sql3Lib.tex}:
% DogWagger sectionTitle=` (Section $[SECTION])'
{\footnotesize\begin{verbatim}
Int16 c_MemPtrFree (MemPtr p)
{ Int16 ok;
if (p == 0)
{ return 0;
};
ok = MemPtrFree (p);
if (!ok) { return 1; };
return 0; // clumsy
}
\end{verbatim}}
% DogWagger sectionTitle=` (Section $[SECTION])'
{\footnotesize\begin{verbatim}
Int16 c_MemPtrUnlock (MemPtr p)
{Int16 ok;
if (p == 0)
{ return 0;
};
ok = MemPtrUnlock (p);
if (!ok) { return 1; };
return 0;
}
\end{verbatim}}
% DogWagger sectionTitle=` (Section $[SECTION])'
{\footnotesize\begin{verbatim}
MemHandle c_MemHandleNew (UInt32 size)
{ if (! size)
{ return 0;
};
return MemHandleNew (size);
}
\end{verbatim}}
% DogWagger sectionTitle=` (Section $[SECTION])'
{\footnotesize\begin{verbatim}
MemPtr c_MemHandleLock (MemHandle h)
{ if (! h )
{ return 0;
};
return MemHandleLock (h);
}
\end{verbatim}}
% DogWagger sectionTitle=` (Section $[SECTION])'
{\footnotesize\begin{verbatim}
Int16 c_MemHandleUnlock (MemHandle h)
{ Int16 ok;
if (h == 0)
{ return 0;
};
ok = MemHandleUnlock (h);
if (!ok) { return 1; };
return 0;
}
\end{verbatim}}
\subsection{A few simple utilities}
\subsubsection{Copy}
xCopy is straightforward (but clumsy):
% DogWagger sectionTitle=` (Section $[SECTION])'
{\footnotesize\begin{verbatim}
Int16 xCopy (Char * dest, Char * xsrc, Int16 cnt)
{ if (! dest)
{ return 0;
};
if (! xsrc)
{ return 0;
};
while (cnt > 0)
{ *dest++ = *xsrc++;
cnt --;
};
return 1;
}
\end{verbatim}}
\subsubsection{Same}
The following simple comparison returns just yes (1 = identical strings), or zero.
If the lengths differ, the strings cannot be the same, of course.
% DogWagger sectionTitle=`Same (Section $[SECTION])'
{\footnotesize\begin{verbatim}
Int16 c_Same (Char * p0, Int16 p0len, Char * p1, Int16 p1len)
{
if (p0len != p1len)
{ return 0;
};
while (p0len > 0)
{ if (*p0 != *p1)
{ return 0; //fail
};
p0len --;
p0 ++;
p1 ++; // clumsy.
};
return 1; // identical strings
}
\end{verbatim}}
\subsection{Debugging and error handling}
\subsubsection{Some memory management}
The following routines are minor modifications of memory handlers
specified in \emph{CProgMain.tex}.
% In these routines, refnum is at present not used,
% but might be used for debugging (with error writes) as in \emph{Sql3Lib.tex}.
% DogWagger sectionTitle=` xNew (Section $[SECTION])'
{\footnotesize\begin{verbatim}
Char * xNew (Int16 memsize)
{ MemHandle memH=0; // clumsy.
MemPtr memP=0;
memH = c_MemHandleNew(memsize);
if (! memH)
{ return 0;
};
memP = c_MemHandleLock(memH);
if (! memP)
{ return 0;
};
return ((Char *) memP);
}
\end{verbatim}}
% DogWagger sectionTitle=` (Section $[SECTION])'
{\footnotesize\begin{verbatim}
Int16 Delete (MemPtr memP)
{
if (! memP)
{ return 1; // ugly
};
if (! c_MemPtrUnlock (memP))
{ return 0; // fail
};
if (! c_MemPtrFree (memP))
{ return 0;
};
return 1; // success
}
\end{verbatim}}
\subsubsection{Write text to console}
Now, the routine to actually write text to the console:
% ConTx_c( refnum, "t", 1, 0);
% DogWagger
{\footnotesize\begin{verbatim}
void ConTx_c(UInt16 refnum, Char * txt, Int16 txlen,
UInt16 bugflag)
{
UInt16 CONSOLE;
SysLibTblEntryPtr entryP;
CacheLib_globals *gl;
UInt16 dbg;
entryP = SysLibTblEntry (refnum);
gl = entryP->globalsP;
CONSOLE = gl->CONSOLE;
dbg = gl->DEBUGFLAGS;
if (bugflag) // okay, might use &&
{ if (! (dbg & bugflag))
{ return;
};
};
if (! CONSOLE)
{ return; // fail
};
// now for some sanity checks:
if (txlen < 1)
{ ConWrite(CONSOLE, "", 3);
return;
};
if (! txt)
{ ConWrite(CONSOLE, "