2.5. Rapid Response Mode

For the 2017APR semester, a rapid response mode was introduced. This is designed to allow the
ATCA to respond almost instantly to a request for a NAPA over-ride.
This section describes how to use this mode.

2.5.1. Who can use the rapid response mode?

The ATCA rapid response mode is only available to proposers who have submitted a
NAPA proposal for the current semester. If you are interested in being granted
permission to use the rapid response mode for your NAPA proposal, please talk
to Jamie Stevens (<Jamie.Stevens@csiro.au>).

2.5.2. How does the rapid response mode work?

The rapid response mode is designed to take a user-made schedule, and get it onto the telescope
as soon as possible after the request has been made. Getting the telescope to stop one schedule
and start another is very easy; determining when "as soon as possible" actually starts is the
tricky part. The rest of this section describes the process which takes place to perform an
over-ride observation.

You submit a NAPA science case to the ATNFTAC, which gives you a ranking grade for the semester.

You ask for, and are supplied with an authorisation token which will allow you to
request an over-ride through the rapid response service.

At some point during the semester, you decide to trigger your NAPA proposal.
You (or more likely your software) creates a CABB schedule that
observes both the target of interest and a nearby phase calibrator.

You submit this schedule to the ATCA rapid response web service.

The service checks that you are an authorised user, and rejects the request if not.

The minimum amount of time that you have requested is still available to be
scheduled for your project, considering the amount of time you requested from
the TAC, and the amount of time you have used before this request. If you do
not have sufficient time, the request is rejected.

A start time is determined where

the source is above the ATCA lower elevation limit,

the project that must be over-ridden has a lower grade than the
NAPA,

and that the schedule can be executed for at least the minimum required
amount of time while still meeting the above conditions.

The search for this suitable start time is limited to either 100 hours after the
request submission, or to a user-specified duration, if it is less than 100 hours.
The 100 hour limit is designed to allow for an automated response even for long
weekends where the proposers may not be available for a manual over-ride, while
the user-specified duration can indicate that an observation that can't be made
within some time frame would no longer be scientifically interesting.

If a suitable start time cannot be found, the request is rejected.

An estimate is made of when the flux-density and bandpass calibrators should
be observed, given the start time. The schedule is altered to insert those scans.

If the request has not been rejected by this point, the request is approved.

At the start time determined in the previous step, several additional checks
are made.

All scheduled scans will be above the ATCA lower elevation
limit at their expected start time, for at least the minimum required time.

The project currently being observed has a grade that is lower than the grade
of the NAPA.

If any of these conditions is not met, then the start time of the project is delayed
from between 5 seconds and 10 minutes, at which point each of the checks are made again.

When all checks from the previous step are satisfied, a final set of schedule alterations
are made.

The schedule is shortened if necessary, to fit in the time available at this start
time. The amount of time scheduled will always at least the minimum required.

If the user has requested to use the same frequency configuration as the project
being over-ridden, and

the band currently being observed (or that was last observed) is not 7mm
or 3mm,

the array is not currently split between two bands,

and the project that is currently being observed (or that was last observed)
is not C999,

then the over-ride schedule is altered to use the same frequency configuration
as is currently listed. Otherwise, the frequency configuration is left as specified
in the request.

If the slew from the current position to the first scan of the schedule is less
than 20 seconds, and the delays need to be reset because the user-specified
frequencies are being used instead of the currently configured frequencies,
an additional “Normal” scan of 40 seconds is inserted at the
beginning of the schedule. This scan is named “setup” so it can
easily be identified and discarded during later data reduction.

The first scan in the schedule is altered, in the following ways.

The command focus default is added, to ensure the focus
is set correctly.

If the scheduled frequencies are not the same as those that were observed
before the over-ride, the command corr reset delays is
added, and the tv-channels are set to be 1020 to 1022 in each
IF.

The altered schedule is copied into the right place, caobs detects
this new schedule file and immediately begins executing it. Over-ride schedules are
started at scan 1, and continue through until the last scan, and will not loop.

While the schedule is being executed, an automatic monitoring system will watch that
the schedule is progressing, and will attempt to automatically restart the schedule
in the event that it gets stopped.

2.5.3. Can I over-ride any scheduled block?

All scheduled slots that have a project code starting with “C” can be over-ridden
by the rapid response mode, provided that your TAC score is sufficiently
high and CABB is in the correct mode.

Blocks of time scheduled for reconfigures, maintenance and VLBI cannot be over-ridden.

In the 2017APR semester, it is not possible to over-ride time scheduled for legacy projects,
but this will change in the 2017OCT semester, so that any over-ride project that has a score
higher than or equal to the lowest score given to a scheduled project may over-ride successfully.

2.5.4. How can I make a schedule file automatically?

As part of the project to develop this rapid response mode, a library for creating and
manipulating CABB schedules was developed. It can be found
on GitHub, and is
free for use.

Because many of the existing libraries that astronomers use to deal with transient
event streams are written in Python, the choice was made to focus first on providing
a Python library for schedule manipulation. This library is targetted toward Python
2.7.

Documentation on how to use the library in your own code is part of the GitHub repository.

2.5.5. What is this authorisation token that I need?

To ensure that only those projects that have been pre-approved can perform over-rides on
the ATCA, any request for over-ride time needs to be accompanied by
a valid authorisation token. This can only be obtained by emailing
<Jamie.Stevens@csiro.au>.

There are two types of authorisation token available. The first is a test token, which
allows you to submit a request, and go through the entire process of the over-ride, but
not actually get time on the telescope. This is designed to be used while you are trying
to develop and perfect your automated trigger routines. You may issue as many requests
as you like with a test token.

The second token will allow your request to allocate time on the telescope. This type of
token will not be issued until you have demonstrated that your code can properly submit
a test request. You will need to work with Jamie Stevens during this process.

Authorisation tokens are valid only for a specific project and semester, and will be
rejected if used outside those bounds.

The authorisation token itself is a
JSON Web Token.
You will be supplied with a normal ASCII file that will need to be
passed back to the service whenever you make a request.

2.5.6. But how do I make a request?

All requests to trigger an over-ride observation are submitted through a web service.
The exact format of the request won't be documented here, and we suggest that you use
the Python code we provide when you are building your trigger response software.

The library which makes it easy to submit over-ride requests can be found
on GitHub, along
with documentation and examples.

2.5.7. How can I find out what the service is doing?

If your request is unable to be scheduled given the bounds you provided, you will receive
your rejection in the response from the web service.

If the web service doesn't reject your request, your schedule will enter the queue to be
executed. At this point, two emails will be sent to this effect. As with all emails sent by
the rapid response service, they will be from Jamie.Stevens@csiro.au and will go to all
the members of the NAPA team (as specified on its OPAL coversheet), along with all the
members of each project that will need to be displaced.

The first email will have the subject “ATCA rapid response mode triggered for Cxxxx
”, where of course the real NAPA project code will be specified instead of “
Cxxxx”. The second email will have the subject “Your ATCA rapid response
over-ride request has been queued for execution.”.

The first email will look something like this.

The ATCA rapid response mode was requested by the NAPA project C008 at UTC 2017-03-31 04:14:14.

This over-ride has been scheduled to begin at UTC 2017-03-31 04:14:14 and end at approximately UTC 2017-03-31 04:27:06.

This over-ride has been assigned the ID xaap6yuacy10ltg9lc3g.

This will displace time from the project C007.

The observations will be automatically started without any user intervention. While the observations are running, please do not intervene unless advised to by observatory staff.

You will receive another email when the observations have started, and another when the over-ride observations have ended. Messages to the same effect will also appear in the ATCA Portal chat.

Please send any feedback about this system by replying to this email.

As you can see, the time of the request, and the expected start and end times are all listed, along
with the unique ID assigned to the over-ride, and a list of all the projects that will be displaced.

The next email will look similar to the following.

This email is from the ATCA rapid response system.

What follows is a log of the progress of your requested over-ride experiment.

Project code: C008

Over-ride ID: xaap6yuacy10ltg9lc3g

2017-03-31_04:14:14: Your experiment has been placed in the queue, and should start at 2017/3/31 04:14:14. You will receive another email at that time.

If you have any concerns about this experiment, please reply to this email.

This second email introduces the time-stamped log of the progress of the over-ride. All subsequent
email communication will contain this log, as it was when the email was sent. The log will have
entries for all the important actions made by the rapid response service, and entries that will be
useful for later data reduction.

At the start time, the service will either start or delay the schedule, and in either case an email
will be sent. If the schedule is started, the email will have the subject “An ATCA rapid
response over-ride is now being observed”, while if the schedule is delayed the subject will
be “Your ATCA rapid response over-ride request has been delayed”; the reasons for the
delay will be included in the log. If the schedule is started, the name of the schedule will be given
in the email, and you may load it in the CABB web scheduler to see how the service has altered it.

No emails will be sent while the schedule is running, except if the service encounters a problem.
In that case, you will get an email with the subject “There has been a problem while
executing your ATCA rapid response over-ride”.

When the schedule finishes, the service will send out one final email, with the subject
“ATCA rapid response over-ride is now complete”. This email contains the full
log, which will include the name of the RPFITS files that contain the data.

If for some reason the service is unable to perform the scheduled over-ride (for example, if the
antennas are weather stowed for an extended period of time), then the schedule might expire and
be removed from the queue. If this happens you will get an email with the subject
“Your ATCA rapid response over-ride request has expired”.

2.5.8. What is the recommended strategy for a rapid response schedule?

The schedule you make depends on what type of observation you are trying to perform. What follows
are some general recommendations for rapid response schedules.

Select the frequencies that will give you the best science, rather
than ask for the frequency configuration that is being used when the telescope is over-ridden.
There is no actual need to have a perfectly calibrated array when doing an observation with
the ATCA. As long as the delays can be measured while observing a strong
calibrator, you will be able to correct for those delays in Miriad.
If you specify the frequencies you want, the service will ensure that the delays are set back
to their defaults at the start of your observation, which will always produce a dataset that
can be successfully calibrated.

To ensure that the observations can be monitored effectively with default delays, the
tvchannel range is limited to 1020 - 1022 in each IF. This 3 MHz of
bandwidth near the band centre is sufficient to measure the system temperature and to show
the calibrator's flux density in vis. You will not need to
specify this when creating the schedule, as the service will automatically do this for you.

If measuring very early prompt emission is important to your
science, ensure that you visit the target source first, before your first scan on the
gain calibrator.
Observing your target for a few minutes before going to the gain calibrator should not
be a problem, as the gains can be reliably extrapolated at cm wavelengths. Also, if you are
observing in the 16cm band, there is a very good chance you will have a source in your field
that is bright enough to allow for self-calibration.

Keep your scans short so the service can easily cut it to fit the
earliest available slot.
If authorised to do so, the service can automatically shorten the schedule you provide in
order to fit it into the earliest possible available slot; how it does this is described in
Section 2.5.9. If you keep your scans short (ie. don't have
hour long scans between visits to your calibrator) then the shortener will act much more
reliably.

Do not include a flux density calibrator in your schedule.
The service will automatically add the appropriate flux density calibrator to your schedule,
depending on frequency and sidereal time.

2.5.9. How does the service shorten my schedule?

The service shortens a schedule in four stages.

The hard end time of the available slot is determined. This may be because the source has
set, or a project has started with a score higher than that of your project. All scans that
end after that hard end time are immediately deleted from the schedule, except if it is the
flux density calibrator scan that the service added.

As mentioned in Section 2.5.8, this step can cause problems
if your scans are too long. Consider a scenario where the scan that ends after the hard end time
is scheduled to run for 1 hour. Even if the hard end time is only 10 minutes from the scheduled
end of this scan, the entire scan is cut, and you have lost a potential 50 minutes of observing
time.

If the schedule is still too long at this point, all scans on the nominated calibration source
may be shortened to have a dwell length of 1 minute. This is done from the first scan and
continues until all the calibrator scans have been shortened, or the schedule now finishes before
the hard end time.

If the schedule is still too long at this point, then the scan on the flux density calibrator
is shortened by 1 minute.

If the schedule is still too long at this point, then the source scans are shortened.
The service calculates the amount of time that needs to be cut for the schedule to fit in
the slot, and then subtracts as much time as possible from the last remaining source scan.
If the last remaining source scan is longer than the cut time, then the scan survives,
otherwise it, and the following calibrator scan, are removed entirely. This process repeats
until the schedule finishes before the hard end time.

If at the end of this process, the schedule length falls below the specified minimum useful
time required, the start time will be delayed and the schedule restored to its original
specification.