Reference complex gain calibration refers to determining the "system" amplitude and phase with high precision as a function of IF band (i.e. integrated over each 500 MHz IF band). A typical reference calibration would cover 34 bands, 2 polarizations, and 13 antennas. The routines (on pipeline machine) to retrieve and analyze the data are in the module refcal_anal.py. Start from a time range during which the reference calibration observation occurred (check the [http://ovsa.njit.edu/phasecal/ phasecal results] to see which 1-hr scan had stable phases)

Reference complex gain calibration refers to determining the "system" amplitude and phase with high precision as a function of IF band (i.e. integrated over each 500 MHz IF band). A typical reference calibration would cover 34 bands, 2 polarizations, and 13 antennas. The routines (on pipeline machine) to retrieve and analyze the data are in the module refcal_anal.py. Start from a time range during which the reference calibration observation occurred (check the [http://ovsa.njit.edu/phasecal/ phasecal results] to see which 1-hr scan had stable phases)

Line 82:

Line 83:

</pre>

</pre>

This will create summary plots of the amplitude and phase, will write the two corresponding *.png files into the /common/webplots/refcal folder, and will print as a text string the lines needed to add an entry. Just cut from the terminal and paste into the wiki editor at the appropriate place to preserve time order.

This will create summary plots of the amplitude and phase, will write the two corresponding *.png files into the /common/webplots/refcal folder, and will print as a text string the lines needed to add an entry. Just cut from the terminal and paste into the wiki editor at the appropriate place to preserve time order.

+

+

== Getting Band 4 Phase (2017-Dec-24) ==

+

[[File: Band4_phase_diff.png | thumb | 600px | '''Figure 1:''' LoRX - HiRX phase differences 2017-12-24. The blue points are the phase differences for common frequencies between pcal_lo_all.fsq and pcal_hi_all.fsq, for parallel feeds (XX and YY), and the orange points are the same for the crossed feeds (XY and YX). Note that the XX and XY points are identical (except at high frequencies where noise is greater), while the YY and YX points differ by what looks like a frequency-independent constant. The green curves are quadratic fits to the blue points. The phase differences, and their fits, are nearly identical on each of the 13 antennas, which was not necessarily expected. Most of the slope here is a delay difference of about 5.9 ns.]]

+

+

Because the refcals have up to now been done with the high-frequency receiver (HiRX) on Antenna 14, we are only measuring the reference phase on bands 5-34, even though band 4 has been present in the solar data. However, the shape of the phase vs. frequency is very regular, suggesting that if we know this shape (which could change over time, so it should be remeasured from time to time) then we can predict the Band 4 phase based on the phases of the higher bands.

+

+

To do this, we measure the phase differences between the low-frequency receiver (LoRX) and HiRx for those frequencies that are in common. It happens that there are many frequencies in common, since the LoRX covers 1-6 GHz and the HiRX covers 3-18 GHz. In fact, the pcal_lo_all.fsq frequency sequence measures bands 1-12 (1-7 GHz) and the phases seem to be coherent from 6-7 GHz. This provides 38 frequencies in common between LoRX and HiRX, from which to determine the phase differences.

+

+

The procedure, then, is to measure the LoRX and HiRX phases near the meridian passage of a strong calibrator (so that all of the feeds are approximately parallel and no feed-rotation-correction is needed). We first observe the source with the LoRX receiver using pcal_lo_all.fsq, and then observe the source with the HiRX receiver using pcal_hi_all.fsq. This is part of another procedure where we then rotate the Ant 14 feed by 90 degrees so that it is crossed with respect to the others. It seems that the XX and YY phases of the parallel feeds should be the same as the XY and YX phases of the crossed feeds, and therefore we should get the same phase differences in the two cases.

+

Here is an example of obtaining the phases of such observations and plotting the phase differences, with corresponding parabolic fits:

The resulting plot is shown in '''Figure 1''' above. Curiously, the XX and YY phase differences (blue points) differ by about -0.5 radians, while the XX and YX phase differences (blue in top row vs. orange in bottom row) differ by +0.5 radians. Meanwhile, the XX and XY phase differences are essentially identical. This relates to the X vs. Y delays in some manner that has yet to be understood.

+

+

The phase differences can be extrapolated to the band4 phases by

+

for p in np.array(ps)[::2]:

+

print ('{:8.2f}'*8).format(*np.polyval(p,band4_fghz))

+

+

for p in np.array(ps)[1::2]:

+

print ('{:8.2f}'*8).format(*np.polyval(p,band4_fghz))

+

which results in the following tables (I put the headings in by hand):

+

+

XX Phase Differences for Band 4

+

2.87 2.89 2.90 2.92 2.94 2.96 2.97 3.00

+

------ ------ ------ ------ ------ ------ ------ ------

+

1 -0.596 -0.562 -0.528 -0.494 -0.460 -0.426 -0.392 -0.358

+

2 -0.574 -0.541 -0.508 -0.475 -0.442 -0.408 -0.375 -0.341

+

3 -0.567 -0.535 -0.502 -0.469 -0.436 -0.403 -0.370 -0.337

+

4 -0.617 -0.583 -0.549 -0.514 -0.479 -0.444 -0.409 -0.374

+

5 -0.664 -0.629 -0.594 -0.559 -0.523 -0.488 -0.452 -0.416

+

6 -0.653 -0.619 -0.584 -0.549 -0.514 -0.479 -0.444 -0.409

+

7 -0.593 -0.558 -0.524 -0.489 -0.454 -0.419 -0.384 -0.349

+

8 -0.565 -0.532 -0.499 -0.466 -0.433 -0.399 -0.366 -0.332

+

9 -0.720 -0.686 -0.651 -0.617 -0.582 -0.548 -0.513 -0.478

+

10 -0.542 -0.510 -0.478 -0.446 -0.413 -0.381 -0.348 -0.315

+

11 -0.660 -0.626 -0.591 -0.557 -0.522 -0.487 -0.452 -0.417

+

12 -0.521 -0.490 -0.460 -0.429 -0.397 -0.366 -0.335 -0.303

+

13 -0.793 -0.753 -0.714 -0.674 -0.634 -0.595 -0.555 -0.515

+

+

YY Phase Differences for Band 4

+

2.87 2.89 2.90 2.92 2.94 2.96 2.97 3.00

+

------ ------ ------ ------ ------ ------ ------ ------

+

1 -1.016 -0.980 -0.944 -0.909 -0.873 -0.837 -0.801 -0.765

+

2 -0.988 -0.952 -0.916 -0.880 -0.844 -0.808 -0.772 -0.736

+

3 -1.012 -0.977 -0.942 -0.906 -0.871 -0.836 -0.800 -0.765

+

4 -1.008 -0.973 -0.939 -0.904 -0.869 -0.834 -0.799 -0.764

+

5 -1.065 -1.027 -0.990 -0.953 -0.916 -0.878 -0.841 -0.803

+

6 -0.924 -0.890 -0.856 -0.823 -0.789 -0.755 -0.721 -0.687

+

7 -0.584 -0.561 -0.538 -0.515 -0.492 -0.468 -0.444 -0.420

+

8 -1.038 -1.003 -0.967 -0.931 -0.895 -0.859 -0.823 -0.787

+

9 -1.079 -1.045 -1.010 -0.975 -0.939 -0.904 -0.869 -0.834

+

10 -1.155 -1.117 -1.079 -1.041 -1.003 -0.964 -0.926 -0.888

+

11 -1.046 -1.010 -0.975 -0.939 -0.903 -0.867 -0.831 -0.795

+

12 -0.781 -0.750 -0.719 -0.688 -0.657 -0.625 -0.594 -0.562

+

13 -1.163 -1.123 -1.082 -1.042 -1.001 -0.960 -0.919 -0.879

+

+

=== Implementation in refcal_anal.py (2018-Mar-05) ===

+

[[File: delay_curve.png | thumb | 600px | '''Figure 2:''' The band-averaged phase differences in high-frequency bands between 2018-02-05 and 2018-02-07 (reference day). The orange line is a linear fit to the points, forced to pass through (0,0). The green points are the fit values for band 4, which are used to estimate this day's (2018-02-05) band 4 phases. Note that the scatter in ant 6 was because it was not tracking that day.]]

+

The refcal_anal.py routine has been updated to include the band-averaged band 4 phase calibration. This implementation has the following two components.

+

+

:'''1.''' On the day we take the measurements with the above-mentioned LoRX-HiRX observation sequence: It calculates the phase differences between low-frequency and high-frequency measurements over bands 5 - 10, extrapolate it to band 4, and subtract this extrapolated difference from the actual band 4 phases measured by low-frequency receiver to obtain the "reference band 4 phases" as if they were measured by high-frequency receiver. This have to be saved in the SQL database with the special keyword '''lohi=True'''.

+

+

:'''2.''' On regular days when we do NOT take measurements with LoRX: It calculates the "estimated band 4 phases" using the closest earlier reference band 4 phases calculated in step #1. First, it calculates the phase differences in high-frequency bands between the target day and the reference day (a "delay curve"). This should be fit with a linear curve passing through zeros like in '''Figure 2'''. We use this fit to obtain the phase difference in band 4 between the target day and the reference day. Knowing the "reference band 4 phases" from #1, we estimate the band 4 phases on arbitrary day.

+

+

To do the first task of saving reference refcal, follow the example lines of code below (in pipeline):

The last plot created will have the calculated band 4 phases added to the usual refcal phase summary plot. There is no special keyword needed for the second task - the routine now automatically calculates the estimated band 4 phases for every refcal, and produces a new plot showing the delay curve like Figure 2, calculated with respect to the most recent reference refcal. If this delay curve starts to show large scatters, it is likely that we need a new LoHi scan to obtain the new reference refcal. It is also important to note that the new reference refcal is needed '''after every delay center reset''', since resetting the delay center introduces some offsets in delay curves and therefore causes the band 4 estimation algorithm to fail.

+

+

The estimated band 4 phase errors from this algorithm is currently ~20 degrees. We also would like to eventually obtain the phases at every frequencies instead of band-averaged values.

Procedure for Analysis

Reference complex gain calibration refers to determining the "system" amplitude and phase with high precision as a function of IF band (i.e. integrated over each 500 MHz IF band). A typical reference calibration would cover 34 bands, 2 polarizations, and 13 antennas. The routines (on pipeline machine) to retrieve and analyze the data are in the module refcal_anal.py. Start from a time range during which the reference calibration observation occurred (check the phasecal results to see which 1-hr scan had stable phases)

import refcal_anal as ra
from util import Time
#example time range of the refcal scan that took place BEFORE the observation
trange=Time(['2017-04-08T14:00','2017-04-08T15:00'])

Now read the data from that timerange

out=ra.rd_refcal(trange)

Note that there are optional arguments that can be given, such as projid='PHASECAL' (the default) and srcid='1229+060', but these are generally not needed.

To correct for differential feed rotation (for refcal data after 05:00 UT on 2017-Jul-01), you can use:

out_corr = ra.unrot_refcal(out)

Take a look at the phases (averaged over each band):

ra.graph(out_corr)

By default it displays only bands 5, 11, 17, 23 and polarization XX. To change the bands and polarization to display, use the "bandplt" and 'pol' parameters (better to check more bands).

ra.graph(out_corr, bandplt=[5,13,19,25], pol=1)

Observations within this time range will be used for averaging to get the reference phases/amplitudes. Data will be flagged if the SNR is less than minsnr, which can be supplied. The default minsnr is set to 1.0. We can further determine a finer selection of time range within this scan that has clean and stable phases if needed, by setting the new timerange in "trange" below.

refcal=ra.refcal_anal(out_corr,timerange=trange,scanidx=[0])

It will generate four plots. First and second are phases and amplitudes of the data similar as those produced by ra.graph(), but with the selected time range for averaging highlighted (one can use "bandplt" and "pol" to choose bands and polarization to display as well). The 3rd plot is the averaged phase vs. frequency and the 4th is the averaged amplitude vs. frequency. Those data will be our "analyzed" reference calibration data saved in the returned dictionary (named "refcal" here). The dictionary contains the actual complex values (refcal['refcal']), a flag array with the same shape (refcal['flag']), and some timestamps, e.g., refcal['t_mid'] is the middle of the time range used for averaging. Both refcal['refcal'] and refcal['flag'] have a shape of (15, 2, 34) -- which corresponds to # of antennas, # of polarizations, and # of bands.

If you determine some antenna, band, polarization need to be flagged/unflagged, they have to be manually changed in the returned refcal['flag'] array. In this example, Antenna 12 was not tracking, and the algorithm did not pick up all of them, so we have to do this manually -- setting refcal['flag'][11, :, :]=1

refcal['flag'][11]=1

If satisfied with the results (by looking at the plots), we can go ahead and send them to the SQL database

import cal_header as ch
ch.refcal2sql(refcal)

To retrieve the refcal data back from SQL database based on, e.g., a solar data one wish to calibrate. We can provide a time:

t=Time('2017-04-07T19:40')
refcal=ra.sql2refcal(t)

The resulted refcal have three keys: 'amp' (15 x 2 x 34 array), 'pha' (15 x 2 x 34 array), and 'timestamp' (in jd). The refcal data found will be the closest one PRIOR to the supplied observation time.

Updating the Reference Calibrations Table

The table below lists the known good reference calibrations. To facilitate adding entries to this table, do the following, where the time is after the desired refcal, but before any later refcal:

t=Time('2017-06-23T19:40') # retrieves the refcal at 13:07 UT on that day
refcal=ra.sql2refcal(t)
ra.graph_results(refcal, savefigs=True)

This will create summary plots of the amplitude and phase, will write the two corresponding *.png files into the /common/webplots/refcal folder, and will print as a text string the lines needed to add an entry. Just cut from the terminal and paste into the wiki editor at the appropriate place to preserve time order.

Getting Band 4 Phase (2017-Dec-24)

Figure 1: LoRX - HiRX phase differences 2017-12-24. The blue points are the phase differences for common frequencies between pcal_lo_all.fsq and pcal_hi_all.fsq, for parallel feeds (XX and YY), and the orange points are the same for the crossed feeds (XY and YX). Note that the XX and XY points are identical (except at high frequencies where noise is greater), while the YY and YX points differ by what looks like a frequency-independent constant. The green curves are quadratic fits to the blue points. The phase differences, and their fits, are nearly identical on each of the 13 antennas, which was not necessarily expected. Most of the slope here is a delay difference of about 5.9 ns.

Because the refcals have up to now been done with the high-frequency receiver (HiRX) on Antenna 14, we are only measuring the reference phase on bands 5-34, even though band 4 has been present in the solar data. However, the shape of the phase vs. frequency is very regular, suggesting that if we know this shape (which could change over time, so it should be remeasured from time to time) then we can predict the Band 4 phase based on the phases of the higher bands.

To do this, we measure the phase differences between the low-frequency receiver (LoRX) and HiRx for those frequencies that are in common. It happens that there are many frequencies in common, since the LoRX covers 1-6 GHz and the HiRX covers 3-18 GHz. In fact, the pcal_lo_all.fsq frequency sequence measures bands 1-12 (1-7 GHz) and the phases seem to be coherent from 6-7 GHz. This provides 38 frequencies in common between LoRX and HiRX, from which to determine the phase differences.

The procedure, then, is to measure the LoRX and HiRX phases near the meridian passage of a strong calibrator (so that all of the feeds are approximately parallel and no feed-rotation-correction is needed). We first observe the source with the LoRX receiver using pcal_lo_all.fsq, and then observe the source with the HiRX receiver using pcal_hi_all.fsq. This is part of another procedure where we then rotate the Ant 14 feed by 90 degrees so that it is crossed with respect to the others. It seems that the XX and YY phases of the parallel feeds should be the same as the XY and YX phases of the crossed feeds, and therefore we should get the same phase differences in the two cases.
Here is an example of obtaining the phases of such observations and plotting the phase differences, with corresponding parabolic fits:

The resulting plot is shown in Figure 1 above. Curiously, the XX and YY phase differences (blue points) differ by about -0.5 radians, while the XX and YX phase differences (blue in top row vs. orange in bottom row) differ by +0.5 radians. Meanwhile, the XX and XY phase differences are essentially identical. This relates to the X vs. Y delays in some manner that has yet to be understood.

The phase differences can be extrapolated to the band4 phases by

for p in np.array(ps)[::2]:
print ('{:8.2f}'*8).format(*np.polyval(p,band4_fghz))

for p in np.array(ps)[1::2]:
print ('{:8.2f}'*8).format(*np.polyval(p,band4_fghz))

which results in the following tables (I put the headings in by hand):

Implementation in refcal_anal.py (2018-Mar-05)

Figure 2: The band-averaged phase differences in high-frequency bands between 2018-02-05 and 2018-02-07 (reference day). The orange line is a linear fit to the points, forced to pass through (0,0). The green points are the fit values for band 4, which are used to estimate this day's (2018-02-05) band 4 phases. Note that the scatter in ant 6 was because it was not tracking that day.

The refcal_anal.py routine has been updated to include the band-averaged band 4 phase calibration. This implementation has the following two components.

1. On the day we take the measurements with the above-mentioned LoRX-HiRX observation sequence: It calculates the phase differences between low-frequency and high-frequency measurements over bands 5 - 10, extrapolate it to band 4, and subtract this extrapolated difference from the actual band 4 phases measured by low-frequency receiver to obtain the "reference band 4 phases" as if they were measured by high-frequency receiver. This have to be saved in the SQL database with the special keyword lohi=True.

2. On regular days when we do NOT take measurements with LoRX: It calculates the "estimated band 4 phases" using the closest earlier reference band 4 phases calculated in step #1. First, it calculates the phase differences in high-frequency bands between the target day and the reference day (a "delay curve"). This should be fit with a linear curve passing through zeros like in Figure 2. We use this fit to obtain the phase difference in band 4 between the target day and the reference day. Knowing the "reference band 4 phases" from #1, we estimate the band 4 phases on arbitrary day.

To do the first task of saving reference refcal, follow the example lines of code below (in pipeline):

The last plot created will have the calculated band 4 phases added to the usual refcal phase summary plot. There is no special keyword needed for the second task - the routine now automatically calculates the estimated band 4 phases for every refcal, and produces a new plot showing the delay curve like Figure 2, calculated with respect to the most recent reference refcal. If this delay curve starts to show large scatters, it is likely that we need a new LoHi scan to obtain the new reference refcal. It is also important to note that the new reference refcal is needed after every delay center reset, since resetting the delay center introduces some offsets in delay curves and therefore causes the band 4 estimation algorithm to fail.

The estimated band 4 phase errors from this algorithm is currently ~20 degrees. We also would like to eventually obtain the phases at every frequencies instead of band-averaged values.