* If you are on our AWS cloud server "virgo" (means for accessing the server are discussed in [[#Using Software on AWS server|Section 3.1.1]]), it is located at /common/data/eovsa_tutorial/IDB20170821201800-202300.4s.slfcaled.ms.

* If you are on our AWS cloud server "virgo" (means for accessing the server are discussed in [[#Using Software on AWS server|Section 3.1.1]]), it is located at /common/data/eovsa_tutorial/IDB20170821201800-202300.4s.slfcaled.ms.

−

+

For completeness, the pre-slfcalibration visibility data can be downloaded from [https://drive.google.com/open?id=1ZMCY9Y3FTv-m0QFyBBwrvqaSuS5pAc3l this Google Drive link].

+

''To reduce the amount of data for this tutorial, the time resolution has been reduced by a factor of 4, to 4 s per sample. The actual time resolution available is 1 s.'' If you are on virgo, please copy it over to your own working directory that you created earlier under the guest account (e.g., ~/bchen/).

''To reduce the amount of data for this tutorial, the time resolution has been reduced by a factor of 4, to 4 s per sample. The actual time resolution available is 1 s.'' If you are on virgo, please copy it over to your own working directory that you created earlier under the guest account (e.g., ~/bchen/).

Browsing and Obtaining EOVSA data

Browsing EOVSA data in RHESSI Browser

Currently the most convenient way for browsing EOVSA data is through the RHESSI Browser. First, check the "EOVSA Radio Data" box on the data selection area (top-left corner). Then select year/month/date to view the overall EOVSA dynamic spectrum. Note if a time is selected at early UTC hours (e.g., 0-3 UT), it will show the EOVSA dynamic spectrum from the previous day. Also note that EOVSA data were not commissioned for spectroscopic imaging prior to April 2017. An example of the overview EOVSA dynamic spectrum for 2017 Aug 21 is shown in the figure on the right.

The overview EOVSA dynamic spectra are from the median of the uncalibrated cross-power visibilities at a few short baselines, which are not (but a good proxy of) the total-power dynamic spectra. The effects of spatial information blended in the cross-power visibilities can be clearly seen as the "U"-shaped features throughout the day, which are due to the movement of the Sun across the sky that effectively changes the length and orientation of the baselines. Flare times can be easily seen in the EOVSA dynamic spectra, which usually appear as vertical bright features across many frequency bands. More information can be found on this page.

We are currently working on a pipeline to create quicklook full-disk images for implementation at the RHESSI Browser in a way similar as the RHESSI quicklook images. Please stay tuned.

Once you identify the flare time, you can find the full-resolution (1-s cadence) uncalibrated visibility files (in Miriad format) at this link. Each data file is usually 10 minutes in duration. Name convention is YYYYMMDD/IDBYYYYMMDDHHMMSS, where the time in the file name indicates the start time of the visibility data.

Calibrating EOVSA data

Calibration: Calibrating EOVSA data is an involved process. We are currently working on a semi-automatic pipeline for calibrating the visibility data. At this moment, however, please contact the EOVSA team if you wish to have calibrated visibility data for a specific event. In the near future, we envision the pipeline-calibrated visibility data to be available for download on a regular basis.

Self-calibration: For improving the quality of the imaging, self-calibration is usually needed. This is still a work-in-progress, but we have had some successful practice in self-calibrating some flare events. Please refer to Self-Calibrating Flare Data for examples and detailed discussions on our current practice for self-calibrating EOVSA flare data.

Analyzing EOVSA data

Software

We have developed two packages for EOVSA data processing and analysis:

Using Software on Amazon Cloud Server

NOTE: the guest account is ONLY INTENDED FOR THE EOVSA TUTORIAL, WHICH WILL BE DISABLED SHORTLY AFTER THE RHESSI WORKSHOP (you will be notified prior to that). If you wish to continue using the server for testing purposes after the tutorial, please contact Bin Chen to set up a personal account.

DISCLAIMER: We are NOT RESPONSIBLE for any loss, damage, or leak of your personal data on the server.

Close setup window and click the new session's icon, which will log you in.

Startup SunCASA and sswIDL

Once you are connected to virgo, you will have access to SunCASA and GSFIT (included in the sswIDL installation). To not interfere with others (who share the same "guest" account), please create your own directory and work under it. For easier identification, please kindly use the initial of your first name and your full last name as the name of your directory (here "bchen" is used as an example).

Please follow this link for details regarding installation of GSFIT on your own machine (which supports multiple platforms). This will take you to another page.

Example Data and Scripts

In this tutorial, we will demonstrate steps for spectral imaging (self-)calibrated visibility data for a C-class flare on 2017 Aug 21 at ~20:20 UT (Section 3.3), followed by a tutorial on visualizing and analyzing the resulting spectral imaging cube (Section 3.4).

To reduce the amount of data for this tutorial, the time resolution has been reduced by a factor of 4, to 4 s per sample. The actual time resolution available is 1 s. If you are on virgo, please copy it over to your own working directory that you created earlier under the guest account (e.g., ~/bchen/).

Example scripts for doing the analysis covered in this tutorial are available as this Github repository. On Virgo, the scripts can be found under /common/data/eovsa_tutorial/.

Spectral Imaging with SunCASA

EOVSA data is handled in CASA tables system, known as a Measurement Set (MS). The actual visibility data are stored in a MAIN table that contains a number of rows, each of which is effectively a single timestamp for a single spectral window and a single baseline. Within SunCASA, in addition to everything already available in CASA, you have access to additional tools that allow you to explore and utilize the new radio dynamic spectroscopic imaging data from EOVSA. We also had success in using SunCASA for analyzing data from the Karl G. Jansky Very Large Array. The steps are very similar to those described here.
To start SunCASA, use:

suncasa

Within SunCASA, you are under the IPython environment. Everything you know about (I)Python should be applicable here. The installation comes with frequently used packages including Matplotlib, Numpy, SciPy, AstroPy, SunPy. However, it is not very intuitive to add (compatible) Python packages within (Sun)CASA. If you choose to do so, you may want to check this page on how to install AstroPy (which is not originally shipped with CASA) within CASA. This method is generally applicable to adding other packages (but not thoroughly tested). If you need some specific packages for your analysis, and it does not require direct interaction with (Sun)CASA, we recommend you to use the standard Python environment. On Virgo, we have installed Anaconda 3, which can be accessed by, e.g., typing "ipython" in a terminal window.

from suncasa.utils import dspec as ds
import matplotlib.pyplot as plt
## define the visbility data file
msfile = 'IDB20170821201800-202300.4s.slfcaled.ms'
## define the output filename of the dynamic spectrum
specfile = msfile + '.dspec.npz'
## select relatively short baselines within a length (here I use 0.15~0.5km),
## and take a median cross all of them (with the domedian parameter)
## alternatively, you can use the "bl" parameter to select individual baseline(s)
uvrange = '0.15~0.5km'
## this step generates a dynamic spectrum and saves it to specfile
dspec=ds.get_dspec(vis=msfile, specfile=specfile, uvrange=uvrange, domedian=True)
## dspec is a Python dictionary that contains the resulting dynamic spectrum.
## A copy is saved in "specfile" as a numpy npz file.
## Other optional parameters are available for more selection criteria
## such as frequency range ("spw"), and time range ("timeran")
## Use "ds.get_dspec?" to see more options
## define the polarizations to show (here I use XX and YY)
pol='XXYY'
## The following command displays the resulting cross-power dynamic spectrum
ds.plt_dspec(specfile, pol=pol) # alternatively you can use dspec as the input

Now you should have a popup window showing the dynamic spectrum. Hover your mouse over the dynamic spectrum, you can read the time and frequency information at the bottom of the window.

Quick-Look Imaging

Imaging EOVSA data involves image cleaning, as well as solar coordinate transformation and image registration. We bundled a number of these steps ino a module named qlookplot, allowing users to generate an observing summary plot showing the cross power dynamic spectrum, GOES light curves and EOVSA quick-look images. Now let us start with making a summary plot of EOVSA image at the spectral window 5 (5.4 GHz).

The empty panels are there as we only selected one polarization for imaging/spectroscopy. Feel free to explore by adjusting the above parameters, e.g. use a different time range ("timerange" parameter) and/or frequency range ("spw" parameter).

With qlookplot, it is easy to engage solar data from SDO/AIA in the summary plot. Setting plotaia=True in the qlookplot command will download SDO/AIA data at the given time to current directory and add it to the summary plot.

The resulting radio image is a 4-D datacube (in solar X-pos, Y-pos, frequency, and polarization), which is, by default, saved as a fits file msfile + '.outim.image.fits' under your working directory. The name of the output fits file can be specified using the "outfits" parameter. In this example, we combine all selected frequencies (specified in keyword "spw") into one image (i.e., multi-frequency synthesis). Therefore the third axis only has one plane. The fourth axis contains polarization. In this example, however, only the first one is populated ("XX"). Here is the relevant information in the header of the resulting FITS file.

By default, qlookplot produces a full sun radio image (512x512 with a pixel size of 5"). If you know where the radio source is (e.g., from the previous full-Sun imaging), you can make a partial solar image around the source by specifying the image center ("xycen"), pixel scale ("cell"), and image field of view ("fov"). Here we show an example that images a 8-s interval around 20:21:14 UT using multi-frequency synthesis in 12-14 GHz and a smaller restoring beam. The microwave source is show to bifurcate into two components, which correspond pretty well with the double flare ribbons in SDO/AIA.

The output FITS file as a 4-d cube is saved to msfile + '.outim.image.fits' under your working directory. The 30 spectral windows used for spectral imaging are combined in the output FITS file. So NAXIS3 (frequency axis) is 30.

Batch-Mode Imaging

This section is for interested users who wish to generate FITS files with full control on all parameters being used for synthesis imaging. We provide one example SunCASA script for generating 30-band spectral imaging maps, and another for iterating over time to produce a time series of these maps.

Producing a 30-band map cube for a given time

An example script can be found at this Github link. If you are on the AWS server Virgo, it is under /common/data/eovsa_tutorial/imaging_example.py. First, download or copy the script to your own working directory and cd to your directory.

Second (optional), change inputs in the following block in your copy of the "imaging_example.py" script and save the changes. This block has definitions for time range, image center and FOV, antennas used, cell size, number of pixels, etc.

The output is a combined 30-band FITS file saved under "outdir". The naming conversion is outimgpre + YYYYMMDDTHHMMSS_ALLBD.fits. In this example, it is "./images/EO20170821T202115.000_ALLBD.fits". Here is the detailed information of the axes of the output FITS file.

From this point, you can use your favorite language to read the fits files and plot them. The last block of the example script uses SunPy.map to generate the plot shown on the right. For SSWIDL users, please refer to Section 3.3.3.3.

Producing a time series of 30-band maps (a 4-d cube)

An example script can be found at this Github link. If you are on the AWS server Virgo, it is under /common/data/eovsa_tutorial/imaging_timeseries_example.py. The script enables parallel-processing (based on a customized task "ptclean3" -- do not ask me why we have "3" in the task name). To invoke more CPU processes for parallel-processing, change "nthreads" in the preambles from 1 to the number of threads you wish to use.

Caution: This script is very computationally intensive. Please do not try to run this script on the AWS server during the RHESSI 18 live tutorial, as it has limited resources that everyone must share.

The output fits images and a summary file imresfile are saved under "outdir" (in this example "./image_timeseries"). The naming convention of output fits images is the same as the previous section. The summary yields the time, frequency, and path to every image.

The last block of the example script uses SunPy.map to generate a series of plots and wraps them as a javascript movie.

Plotting Images in SSWIDL

All the output files are in standard FITS format in the Helioprojective Cartesian coordinate system (that most spacecraft solar image data adopt; FITS header CTYPE is HPLN-TAN and HPLT-TAN). They are fully compatible with the SSWIDL map suite which deals with FITS files. We have prepared SSWIDL routines to convert the single time or time-series FITS files to an array of SSWIDL map structure. The scripts are available in the Github repository of our tutorial. For those working on Virgo, local copies are placed under /common/data/eovsa_tutorial/. Two scripts are relevant to this tutorial:

; cd to your path that contains casa_readfits.pro and casa_fits2map.pro. If on Virgo, just add the path
IDL> add_path, '/common/data/eovsa_tutorial/'
IDL> fitsdir = '/common/data/eovsa_tutorial/image_series/'
IDL> fitsfiles = file_search(fitsdir+'*_ALLBD.fits')
; The resulting fitfiles contains all multi-frequency FITS files under "fitsdir"
; The following command converts the fits files to an array of map structures.
; "casa_fits2map.pro" also work well on a single FITS file.
; (Optional) keyword "calcrms" is to calculate rms and dynamic range (SNR) of the maps in a user-defined "empty" region
; of the map specified by "rmsxran" and "rmsyran"
;;; Here we load 10 time frames as a demonstration
IDL> casa_fits2map, fitsfiles[45:54], eomaps [, /calcrms, rmsxran=[260., 320.], rmsyran=[-70., 0.]]

The resulting maps have a shape of [# of frequencies, # of polarizations, # of times]

Spectral Fitting with GSFIT

In this section we discuss using GSFIT (now a package of SSWIDL) to perform spectral fit based on the resulting FITS images produced from the previous steps. We will use the FITS images produced in Section 3.3.3.2 as the input. On Virgo, we prepared an IDL save file located at /common/data/eovsa_tutorial/21August2017.sav. In addition, this folder contains three more demo files that may be used to demonstrate the GSFIT functionality.
For those intending to use their own machines for this part of the tutorial, these IDL fits files may be downloaded from this location

GSFIT GUI Application

The GSFIT GUI application may be launched as follows

IDL> gsfit [,nthreads]

where nthreads is an optional argument that indicates the number of parallel asynchronous threads to be used when performing the fit tasks. By default, GSFIT launches with only one thread, but the user may interactively add or delete threads as needed at the run-time up to the number of CPUs available on the system (Note, if you are working on the AWS server, please use only one thread). After some delay while the interface loads, the GUI below should appear.

GSFIT GUI appearance on Windows OS

GSFIT GUI appearance on Linux/Mac OS

A detailed description of the GSFIT functionality is provided in the page linked below.

GSFITCP Batch Mode Application

GSFITCP is the command prompt counterpart of the GSFIT GUI microwave spectral fitting application. Once started, GSFITCP is designed to run in unattended mode until all fitting tasks assigned to it are completed and log on the disk in an user-defined *.log file, the content of which may be visualized using the GSFITVIEW GUI application. When run remotely on an Linux/Mac platform, the GSFITCP may be launched on a detached screen, which allows the remote user to logout without stopping the process in which GSFITCP runs.

GSFITCP is launched using the following call:

IDL> gsfitcp, taskfile, nthreads, /start

where taskfile is a path to a file in which a GSFIT task has been previously saved, as explained in the GSFIT Help page, nthreads is an optional argument indicating the number of parallel asynchronous threads to be used, and the optional keyword /start, if set, requests immediate start of the batch processing.

A detailed description of the GSFITCP functionality is provided in the page linked below.

GSFITVIEW GUI Application

The GSFITVIEW GUI application may be launched as follows

IDL> gsfitview [,gsfitmaps]

where the optional gsfitmaps argument is either the filename of an IDL *.sav file containg a GSFIT Parameter Map Cube structure produced by the GSFIT or GSFITCP applications, or an already restored such structure.

A detailed description of the GSFITVIEW functionality is provided in the page linked below.