Daily Procedures

Raw data from all instruments are copied back to the ARI in
Liverpool within minutes of being taken, through the night. Rough "quicklook" reductions
are made using whatever calibration files (flats, fringe frames, etc) are available at that instant, so that users can perform sanity checks on their observations if they wish. The following morning, where a reduction pipeline is avilable, the data is automatically fully-reduced using the very latest calibration files, and loaded into the searchable Data Archive.

Data from all instruments (reduced if there is an automated pipeline and
raw if there is not) are normally available for download that morning. An email is automatically sent to the PI notifying them of the new
data (if you would also like an alert email sent to a Co-I please
contact the LT Support Astronomer).

Where Data are Stored Online

There are three web-based interfaces to allow PIs and CoIs
access to their data; Quicklook, Recent Data, and the
Archive. The main distinguishing feature between these
interfaces is the time-scale on which the data are made available. All
of these data sources are password protected
for as long as the data are still within their proprietary period, after which they become
publicly available.

In addition to the three main user interfaces discussed above, a
small amount of data which for one reason or another are not available
in the main searchable data archive, may be found in Unloaded Data. In most cases these
are early comissioning data for particuilar instruments obtained when
a reduction pipeline was not yet available or do not meet the quality
control requirements for ingestion to the main archive. Most of the
data included here are unreduced raw files direct from the
instrument. Also included here are some very early (2004) LT data
obtained before the robotic systems were comissioned. Most LT users
will never want access to any of these files.

IN SUMMARY - the three main repositories for LT data are
Quicklook, Recent Data, and the main LT Archive. For the
majority of users, the most commonly used interface is
Recent Data, though all three are listed below.

In some cases observations may only be partially
reduced (i.e. the pipeline may not always use the most up-to-date calibration files
or may use reduction algorithms optimised for speed rather than
accuracy). However, data products should certainly be good enough to check
the overall quality of any data acquired.

For each night you will find:

JPEG images and reduced FITS files for all targets observed on that date.

The recent data archive contains exactly the same data as
are loaded into the main searchable LT data archive. Data are
simply held here for a few weeks for more convenient
download. (Note that data uploaded to this Recent Data archive are
also uploaded to the main LT Searchable Archive, described below,
at the same time).

For new instruments without a working data reduction pipeline, the
raw data may be held online here indefinately or until they can
be reduced and loaded into the main data archive.

Data are stored in subdirectories named by the local civil
date at the start of that night. A full night's data are stored
together in the same subdirectory, i.e. the system does not
create a new directory at midnight.

For each night you will find

JPEG images and reduced FITS files for all targets observed on that date.

A gzipped tar file containing all reduced data bundled together for easy download.

All raw data files (usually also contained in a gzipped tar file).

An observing log.

Reduced FITS file names end in "_1.fits"
if they have undergone only basic instrumental reductions (bias subtraction,
flat-fielding, etc.)
or "_2.fits"
if they include derived results such as object
catalogues, extracted 1D spectra etc;
unreduced FITS file names end in "_0.fits".
Please see the individual instrument web pages for further details.

This is the main repository for long term storage of ALL
reduced LT data. There is currently no time limit on storage.

Contains all reduced LT data taken since September 2004. If you
need to recover data obtained for you before that date, please
look at the Unloaded Data or contact us.

Only contains those data which have been through the automated
reduction pipelines. For instruments where no pipeline is
currently in operation, alternative temporary measures may be in
place to provide the data to users (see Recent Data above).

Reduced FITS file names end in "_1.fits"
if they have undergone only basic instrumental reductions (bias subtraction,
flat-fielding, etc.)
or "_2.fits"
if they include derived results such as object
catalogues, extracted 1D spectra etc;
unreduced FITS file names end in "_0.fits".
Please see the individual instrument web pages for further details.

Observating Logs & Error Codes

Observing logs are available on a per proposal basis so users have
access to a full list of all observations taken on a particular
night for a single proposal. These may be found in Recent Data.

If there have been any problems during the passage of data through
the Data Pipeline, error codes are added to the observing log, and
more detailed codes are written to custom FITS headers in the FITS
data file (see below). These are used to alert users to more specific flags in
the FITS header. The observing log codes are cumulative and are found
in the log at the end of the line for each image. For example, if a
problem is encountered in both overscan subtraction and bias frame
subtraction, the flag would be set to -6. (i.e., -2 + -4 )

OBSERVING LOG CODES FOR THE CCD PIPELINE

Code #

Error String

-2

Overscan subtraction

-4

Bias frame subtraction

-8

Trimming off the overscan

-16

Flat fielding

-32

Dark frame subtraction

-64

Fringe subtraction

Downloading Data & Proprietary Period

Data are available for immediate download over HTTP. Consult
the Where Data are Stored section above to
determine the access method best suited to you. Recent Data uses a static directory structure so
that users always know where to look for their data. The Data Archive will automatically create hypertext
links to all data that match submitted search criteria.

To access FITS data products, users will need the password that was
provided to the PI in the notification email when his or her data were
first loaded into the archive. The password is specific to the
proposal ID (not the user ID) and any future or past data obtained for
the proposal should be accessible using the same password. PIs may
therefore want to circulate the proposal password with their CoIs so
that they may access the data.

Data usually remain proprietary to the PI for one year after the
end of the semester within which the data were acquired. Formal
rules are defined by the LT's data access
policy.

Calibrators and Standards

All standard calibration data (Photometric, polarimetric standards
etc) taken with the LT are public and available for use by all
observers.

As with all other data, once reduced these data can be accessed
directly from the Searchable Data Archive or, in the short term, the
Recent Data pages. Each set of calibrators has its own specific
proposal ID. The table below lists the proposal ID users should
interogate for each calibration standard.

Feedback Form

We welcome your thoughts and suggestions regarding your LT data and overall experience as a user.
Please feel free to complete this Feedback Form at any time, or to contact either
LT Staff or, if you prefer, a member of the LT User Group.