XDS: Files

The files generated by XDS are either ASCII type files that can be
inspected and modified by using a text editor, or binary, compressed
image files that can be looked at using the XDS-Viewer program.

All files have a fixed name defined by XDS, which makes it mandatory to
process each data set in a newly created directory to avoid name
clashes. Clearly, one should not run more than one XDS-job
simultaneously in the same directory.

Rotation data images are processed in 8 steps which are called in
succession by XDS. Results and diagnostics from each step are
documented in files with the name extension .LP for inspection by the
user. Information between the steps is exchanged by files, which allows
repetition of selected steps with a different set of input parameters
without rerunning the whole program. Note, that by rerunning a
processing step the earlier version of the output files from this step
will be overwritten. Thus, these older files should first be given
another name if their original contents are meant to be saved.

Information exchange between processing steps of XDS
Files with the extension cbf are binary, compressed images.

Each parameter name consists of a string of characters without
intervening blanks or exclamation marks and includes an equal sign as
its last character. The parameter value must directly follow the parameter
name and be on the same line. The parameter names cannot be abbreviated;
they are case sensitive, too. The parameters may be given in arbitrary order.
Characters in a line to the right of an exclamation mark are comment.

The file XDS_ASCII.HKL contains the result of data processing by XDS,
namely the corrected intensities of all reflections recorded in the
data images.
Output files generated by xscale adopt a similar layout but are named
as defined by the user.

The file consists of a header, reflection data records, and a line
marking the end of the data. Each line is at most 132 characters in
length. The data records consist of a fixed number of numerical items
that are separated by at least one blank. Header lines and the
terminator line are distinguished from the reflection data records by
the presence of the exclamation mark symbol '!'.

Header
The first header line defines the file format (always
FORMAT=XDS_ASCII), states whether symmetry equivalent reflections have
been merged or not (MERGE=FALSE or TRUE), and whether Friedel's law
holds true or not (FRIEDEL'S_LAW=TRUE or FALSE). The file XDS_ASCII.HKL
generated by the CORRECT-step of XDS has always MERGE=FALSE, whereas
the ouput file generated by xscale often has symmetry equivalent
reflections merged together (MERGE=TRUE).
Subsequent header lines specify experimental parameter values and other
information whose meaning is obvious or refers to an input parameter
value explained in the section
Input parameters. The amount of
information given in the header may vary and is subject to future
development.
However, the header always specifies the number of numerical items in
each reflection data record, which is the value of the parameter
called NUMBER_OF_ITEMS_IN_EACH_DATA_RECORD=. For the file generated
by CORRECT (XDS_ASCII.HKL) each reflection data record contains 12
items whose meanings are specified by the following keywords.

ITEM_H=1; the first number in a data record is the reflection index h.

ITEM_K=2; the second number in a data record is the reflection index k.

ITEM_L=3; the third number in a data record is the reflection index l.

ITEM_IOBS=4; the fourth number in a data record is the reflection
intensity.

ITEM_SIGMA(IOBS)=5; the fifth number is the error of the intensity. A
negative sign is attached to SIGMA(IOBS) to indicate a MISFIT (IOBS is
incompatible with symmetry equivalent reflection intensities). At
present, such MISFITs are ignored in the subsequent processing by
xscale.

ITEM_XD=6; X-coordinate (pixels) of reflection on detector.

ITEM_YD=7; Y-coordinate (pixels) of reflection on detector.

ITEM_ZD=8; centroid of image numbers that recorded the Bragg peak.

ITEM_RLP=9; reciprocal LP-correction factor that has been applied to
this reflection.

ITEM_PEAK=10; means that the tenth item contains the percentage of
observed reflection intensity. A value less than 100.0 indicates
that the reflection was either incompletely recorded or overlapping
with neighbouring reflections or included untrusted pixels in the
profile region. Note, that the reflection intensity given in ITEM_IOBS
always refers to the complete reflection regardless of the value
of ITEM_PEAK.

ITEM_CORR=11 means that the eleventh item contains the percentage of
correlation between observed and expected reflection profile.

ITEM_PSI=12 means that the twelvth item is the value of
Schwarzenbach and Flack's
psi-angle (°) which completely specifies diffraction geometry
with respect to the unit cell axes.

The last line of the header is identified by !END_OF_HEADER

Data records
one line per reflection, with the meaning of the data items in each
line being explained in the header. Each data record line consists of
numerical items which are separated by at least one blank. A negative
sign is attached to SIGMA(IOBS) to indicate a MISFIT.

The file (type ASCII) contains a list of strong observed diffraction spots
that is generated by the "COLSPOT" step. Information about a spot
is saved as a single line containing the items x,y,z,Intensity,iseg. The
items are separated by at least one blank character.

x,y denotes the spot position (pixel units) within the detector segment
plane #iseg recording the spot. The coordinates are already corrected
for spatial distortions

z is the centroid of image numbers that contribute to the spot

Intensity is an indicator for the spot intensity

iseg is the identifying number of the detector segment where the spot
is recorded. If the detector has only one segment this item is omitted
so that the file SPOT.XDS is compatible with older versions of XDS
(before release March 30, 2013).

The file is read subsequently by
"IDXREF" and indices are attached to each spot. Indices 0 0 0
are used to indicate a spot that could not be indexed. On return, the
original file contents is replaced by records
x,y,z,Intensity,(iseg),h,k,l
for each spot, where h,k,l are the reflection indices. Item iseg is only
present if the detector contains more than one segment.

Each file contains all diffraction parameters necessary for computing
the locations of all reflections occuring in the data images.

An initial set of parameters is determined by "IDXREF" and
saved on file XPARM.XDS. A final set of parameters is computed by
"CORRECT" and saved on file GXPARM.XDS. Both files are of
identical format, the meaning of the parameters is as described below.
In case a better solution has been obtained from the refinements in the
"CORRECT" step, one could replace XPARM.XDS by GXPARM.XDS and
rerun "INTEGRATE" and "CORRECT". Usually this is
not necessary, but this possibility may be useful in certain critical
cases.

The file comprises 11 lines of data values that code for

The first line only contains one keyword, XPARM.XDS, distinguishing the
new file format from the older versions (released before March 30, 2013)
which do not have this line.

This file contains the results from the INTEGRATE step. The file begins
with a self-explaining header. Each header record starts with a '!'.
The last header record is indicated by !END_OF_HEADER. The maximum length
of a header line is at most 512 characters which is sufficient to
accommodate long path names of the image directory.
The reflection records follow immediately after the header. A terminator
record, !END_OF_DATA, follows the last reflection record.

Each reflection record consists of 21 numerical data items
h, k, l, IOBS, SIGMA, XCAL, YCAL, ZCAL, RLP, PEAK, CORR, MAXC, XOBS,
YOBS, ZOBS, ALF0, BET0, ALF1, BET1, PSI, ISEG
that are output as a single line not longer than 200 characters.
The numerical items are separated by a blank and can be read in free-format.

h: h-index of the reflection.

k: k-index of the reflection.

l: l-index of the reflection.

IOBS: intensity of the reflection obtained by profile fitting. The
intensity is already LP-corrected assuming an unpolarized incident
beam. The final polarization correction is carried out in the CORRECT
step. IOBS refers to the complete reflection intensity regardless of
the value of PEAK. Thus, if I denotes the recorded intensity obtained
from the scaled images, IOBS=I*RLP/(PEAK/100).

SIGMA: e.s.d. of IOBS as obtained from profile fitting.

XCAL: calculated X-coordinate (pixel units) in the segment plane
of the detector where the reflection is expected to be recorded.

YCAL: calculated Y-coordinate (pixel units) of the reflection
in the segment plane of the detector.

This compressed file contains the last data image processed in the
INTEGRATE step enriched with ellipsoids around all expected diffraction
spots. The user is strongly encouraged to have a look at this image.
The clearly visible spots in this
control image should appear encircled, centered near the spot maximum
(reflections close to the rotation axis are not integrated).
Often, problems in data processing become immediately obvious by looking
at this control image.

In the CORRECT step, XDS checks for the presence of a file named
FILTER.HKL in the current directory. A file of this name can be
provided by the user to specify reflections on file INTEGRATE.HKL
that should be ignored in the CORRECT step and thus be excluded from
the final output file XDS_ASCII.HKL.
Typically, FILTER.HKL is generated by a special user program to handle
complicated experimental situations in which parts of the detector
become obscured.

Each reflection record of the file FILTER.HKL consists of 9 numbers
with at least one blank character between them (free format record) H K L X Y Z DX DY DZ. H K L are the integer reflection indices, X Y are the location (pixel units) in the segment plane
of the detector where the reflection is expected to be recorded, Z is the expected running number of the image containing
the Bragg peak, DX DY DZ are dimensions of the target area around X Y Z
used for decision making.
A reflection record h k l XCAL YCAL ZCAL on INTEGRATE.HKL
will be ignored if there exists at least one record on file FILTER.HKL
so that all of the relations are satisfied h=H, k=K, l=L, |XCAL-X|<DX, |YCAL-Y|<DY, |ZCAL-Z|<DZ

In the CORRECT step, XDS checks for the presence of a file named
REMOVE.HKL in the current directory. A file of this name can be
provided by the user to specify the indices of reflections that should
be excluded from the final output file XDS_ASCII.HKL. Each reflection
record of this file consists of the reflection indices h, k, l and
possibly other information that will be ignored. Up to 10000
reflections can be specified.

Reflections that do not obey Wilson's statistic often arise from ice
rings in the data images. These outliers are reported near the end of
the file CORRECT.LP. To suppress the unwanted reflections from the
final output file XDS_ASCII.HKL, the user copies them to a file named
REMOVE.HKL in the current directory and repeats the CORRECT step
(grep alien CORRECT.LP > REMOVE.HKL). NOTE, that for the repeat run
space group number and cell constants must be provided so that the
indices listed in REMOVE.HKL are consistent!

The files X-CORRECTIONS.cbf and Y-CORRECTIONS.cbf are generated by the
XYCORR step of XDS and contain look-up tables to correct the X and Y pixel
positions on the detector for spatial distortions. The spatial
corrections for a pixel at IX,IY (0<IX<=NX, 0<IY<=NY) are
found in the tables at address IA4=IX4+NXBY4*(IY4-1) where
IX4=1+(IX-2)/4, IY4=1+(IY-2)/4, and NXBY4= 1+(NX-2)/4, such that the
corrected pixel-coordinates are
IX("CORRECTED")=IX+I2XCOR(IA4)/10
IY("CORRECTED")=IY+I2YCOR(IA4)/10.
The other files are generated by the CORRECT step of XDS. All of the
files can be visualized using the XDS-Viewer program. The
spatial corrections reported by clicking the left mouse button are in pixel
units multiplied by 10.

The files DX-CORRECTIONS.cbf and DY-CORRECTIONS.cbf show systematic
discrepancies between observed and calculated spot locations.
Such systematic discrepancies could result from the use of incorrect
look-up tables X-CORRECTIONS.cbf, Y-CORRECTIONS.cbf during data
processing.

The files GX-CORRECTIONS.cbf and GY-CORRECTIONS.cbf consist of the original
look-up tables (X-CORRECTIONS.cbf, Y-CORRECTIONS.cbf) plus the systematic
discrepancies (DX-CORRECTIONS.cbf, DY-CORRECTIONS.cbf).
To salvage the data set, the new look-up tables
(GX-CORRECTIONS.cbf, GY-CORRECTIONS.cbf) can be renamed by the user
to replace the original files X-CORRECTIONS.cbf, Y-CORRECTIONS.cbf
and a complete data processing (with the exception of the XYCORR step)
can be carried out.

BLANK.cbf contains the dark current (non-Xray) background. The dark
current for a pixel at IX,IY (0<IX<=NX, 0<IY<=NY) is found
at address IA4=IX4+NXBY4*(IY4-1) in the table, where
IX4=1+(IX-2)/4, IY4=1+(IY-2)/4, and NXBY4= 1+(NX-2)/4.

These files contain an image of the INTEGER array IBKG(NX*NY) such
that IBKG(IX+NX*(IY-1)) is the background value at pixel position IX,IY
(0<IX<=NX,0<IY<=NY). NX,NY are the number of pixels along
the detector X- and Y-axis. (IX is the fast index.)

BKGINIT.cbf is generated in the INIT step and serves as input of the
DEFPIX step. DEFPIX recognizes shaded regions in the background table
BKGINIT.cbf and removes them from the set of trusted pixels.
Untrusted pixels are marked by -3.

BKGPIX.cbf is the background table resulting from the DEFPIX step. It
serves as starting background table for the INTEGRATE step.

This file contains an image of the INTEGER array IFRAME(NX*NY) such
that IFRAME(IX+NX*(IY-1)) is the ratio between the background at IX, IY
and the mean background for all pixels of the same resolution,
multiplied by 10000. NX,NY are the number of pixels along the detector
X- and Y-axis. (IX is the fast index.) This file is generated in the
DEFPIX step of XDS and should be inspected by the user with the XDS-Viewer
program. The image clearly shows shaded regions of the detector that
should be removed from the trusted detector area. Pixels in the shaded
region have a value below 10000, and in case the default setting was
unsatisfactory, the user may change the input parameter
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS= and repeat the DEFPIX step. As
a result from the DEFPIX step the initial pixel background table
BKGPIX.cbf will be obtained with untrusted pixels marked by -3.

This file, generated in the CORRECT step of XDS, contains an image of the
correction factors applied to the reflection intensities as function of
image number and detector region of their recorded Bragg peaks.
The correction factors are multiplied by 1000 and saved in the file
ABSORP.cbf as a CBF-compressed INTEGER array(IX+NX*(IY-1)), with
NX "fast" pixels (IX=1,NX; along image numbers) running from left to right
and NY "slow" pixels (IY=1,NY; enumerating the detector regions) pointing
upwards. This file is meant to be inspected by the user with the XDS-Viewer program
and replaces the numerical print-out within CORRECT.LP of previous versions
of XDS.

This file, generated in the CORRECT step of XDS, contains an image of the
correction factors applied to the reflection intensities as function of
image number and resolution of their recorded Bragg peaks.
The correction factors are multiplied by 1000 and saved in the file
DECAY.cbf as a cbf-compressed INTEGER array(IX+NX*(IY-1)), with
NX "fast" pixels (IX=1,NX; along image numbers) running from left to right
and NY "slow" pixels (IY=1,NY; along (2*sin(theta)/lamda)^2) pointing upwards.
This file is meant to be inspected by the user with the XDS-Viewer program and
replaces the numerical print-out within CORRECT.LP of previous versions of XDS.

This file, generated in the CORRECT step of XDS, contains an image of the
correction factors applied to the reflection intensities as function of
the detector coordinates of their recorded Bragg peaks.
The correction factors are multiplied by 1000 and saved in the file
MODPIX.cbf as a cbf-compressed INTEGER array(IX+NX*(IY-1)), with
NX "fast" pixels (IX=1,NX; along the detector's X-direction) running from
left to right and NY "slow" pixels (IY=1,NY; along the detector's Y-direction)
pointing upwards. This file is meant to be inspected by the user with the
XDS-Viewer program and replaces the numerical print-out within CORRECT.LP of
previous versions of XDS.