It appears that the SELECT statement builds a list of all records in the file
and the READNEXT then takes record ids one by one from this list to be used in
the READ operation. This would require two passes through the file and, because
we are discussing a very large file, it is likely that the data must be
physically read from disk twice because the data cached by the READNEXT will
have been discarded from memory as later groups are processed.

This is not what actually happens. Instead, the SELECT builds a partial select
list that contains only the record ids in the first used group.
The READNEXT/READ processes these records and when a READNEXT finds the list
to be exhausted, a list of all records in the next used group is built.
This continues until the final group has been processed.

Because constructing each partial select list is interleaved with the READ
operations, there is a very high probability that the read can take
the record from cache memory, removing the need for a second physical disk
transfer. The application could potentially run twice as fast as would occur
if the entire file was selected before execution of the processing loop.
If the processing includes display of data derived from the records, there is
also the possibility of improved user perception as the first record will
appear on the screen almost immediately rather than after a complete scan of
the file.

Disadvantages of Partial Selects

The mechanism described above has some important side effects that developers
need to understand.
The select operation needs to maintain a pointer to its current position in
the file. The QMBasic language provides no concept of scope for this internal
pointer such that, if the program exits from the loop before the final group
has been selected, there is no way in which the partial select mechanism can
be terminated automatically.

The scanning process requires that the file remains open until the select
has been completed. Closing the file in the application is not sufficient. The
partial select list must also be terminated, typically by use of CLEARSELECT.
Any program that may only partially process a select list should always use
CLEARSELECT to ensure that it is not holding the file open.

For this pointer to remain valid from one partial select to the next, split
and merge operations within a dynamic hashed file must be suspended while
a select is in progress on that file. Once all select operations from all
processes are terminated, the file reconfiguration will catch up.

Writing a new record to the file in the processing loop may result in that
record being returned by a READNEXT on a subsequent group. It is usually
easy to code around this.

Completion

Some operations require that the scanning process completes building the list
of all remaining record ids in the file. These situations include:

READLIST to transfer the list to a dynamic array.

SELECTINFO() to count items in the list.

Using a select list as a string.

QMClient

Use of QMSelect() and QMReadNext() in a QMClient application uses the same
mechanism as described above. The API also includes the QMSelectPartial()
function to return a list of up to 100 record ids and QMNextPartial() to
continue this process. This can imporve performance by replacing many
QMReadNext() calls with just one operation whilst still gaining much of the
benefit of the underlying partial select mechanism.

The Virtual File System

The VFS also supports partial select lists though a VFS handler can always
choose to return the entire select list if the data store to which it
connects has no equivalent capability.
The skeleton VFS handlers provided with QM show how to implement this feature.