Filtering Lightcurves

Synopsis:

It may be necessary to filter your data to remove periods of
anomalous background levels, such as the flares seen in ACIS
observations. The CIAO package includes the dmextract tool to calculate the lightcurve of
a dataset (or region within a dataset), and the
dmgti tool to create a
GTI file given a table and a set of limits.

Purpose:

To analyze the light curve of the ACIS imaging background in
order to clean the dataset of periods of anomalous
background rates.

The algorithm used to detect flares is simple, so it may not
be sufficient in all cases. If you intend to use
the ACIS blank-sky background files, the cleaning
described in this thread is not conservative
enough. Instead, use the alternative method described in
the Removing ACIS
Background Flares thread.

To avoid any background variations due to sources in the
field we first filter out regions of high/variable emission
from the dataset. There are a number of ways to create a
list of regions that should be excluded - for instance you
may wish to use the output from one of the source detection programs.
In this thread we shall use visual estimation, as an example.

We can now create a lightcurve of this background region
using the CIAOdmextract
tool. The best way to bin the lightcurve
depends on the dataset and your scientific objectives;
for this example we use a bin length of 200 seconds:

The lc_sigma_clip() routine provided by the
lightcurves.py
script performs an iterative
sigma-clipping algorithm, removing those points that fall
outside +/-3 sigma from the mean at each iteration until all
data points are within +/-3 sigma (the number of sigma used to clip
the data defaults to 3 but can be changed).
This algorithm is robust
but not perfect; it can easily "overclean" a noisy
lightcurve and should not be used blindly.
The routine can print out the limits used and can also
create a GTI file that encodes these limits.

The top plot shows the light curve of lc_c7.fits,
where the time axis is displayed on the x axis as a
relative value (the first bin in the light curve is used as
the 0 point). Points that are in green are those that passed
the sigma-clipping and the horizontal dotted line shows the
mean value of these points (the numerical value is given in the
plot title).

The bottom plot shows a histogram of the count rate values; the
green version again shows the data that passes the sigma-clipping
algorithm.

Since there is a large flare, we re-run the analysis picking
only those periods of the light curve with a "low" count
rate. The limit used here is for demonstration purposes
only; the value you should use depends on your science
goals:

Note that this count-rate filter has been unable to
remove all the flare periods. You can try lowering the
maximum rate - e.g. to 0.7 - but you may find
that for such strongly-flaring data that it is hard to
select a "clean" set of times.

By default, deflare filters out the times where
the background count rate exceeds the
±3σ
about the mean. These regions are indicated in red on
the lightcurve that is displayed in
Figure 5 for a lightcurve of ObsID
3103.

Lightcurve filtered using
lc_sigma_clip with the default
values, including clipping any points
outside ±3σ
about the mean.

It is possible to filter the lightcurve more
aggressively, by rejecting times with count rates
that are close to the mean after each iteration.
To reject times with count rates exceeding
±1.6σ
about the mean (corresponding to a confidence level of
~90%), set the nsigma parameter:

The lightcurve is filtered by clipping any
count rates outside of ±1.6σ
from the mean. This
is done by setting the deflare
parameter nsigma=1.6, which is
equivalent to setting the
lc_sigma_clip parameter, sigma=1.6.

The output time periods can be used to filter the event
list. The lc_sigma_clip routine can create a
GTI file directly,
or you can use the limits it calculates to
create the file yourself using the
dmgti tool, or use them
directly within a DM filter
expression, as described below.

updated for CIAO 3.3: default value of
dmextracterror and bkgerror
parameters is "gaussian"

01 Dec 2006

updated for CIAO 3.4: dmgti uses the value of the
TIMEPIXR header keyword to adjust start and stop times (users
may see a small shift in the time filter when compared to CIAO
3.3 because of this); kernel parameter removed from
dmgti; ChIPS version

18 Jan 2008

updated for CIAO 4.0: analyze_ltcrv.sl v1.6 (plotting
routines have been removed from the script; plotting will be
replaced once it is updated for the new ChIPS
syntax); filename and screen output updated for reprocessed
data (version N004 event file); slsh version 0.8.2-0/S-Lang
version 2.1.3

updated for CIAO 4.1: analyze_ltcrv.sl has been replaced
by lightcurves.sl and the routine analyze_ltcrv()
by lc_sigma_clip();
added a Python version of the
script; plotting capabilities
have been restored;
the routine can now
generate a
GTI file directly.

06 May 2009

check the version of the CIAO scripts package instead
of the individual script

11 Feb 2010

updated for CIAO 4.2: added an example of filtering on count
rate

05 Mar 2010

added a subsection on using deflare
script; updated script tarfile version to 05 Mar 2010