tag:blogger.com,1999:blog-4994845427978586964Mon, 05 Mar 2018 22:54:59 +0000SCUBA-2cadcsmurfstarlinkdata processingorac-drpicardJSAcalibrationACSISDRJLSFCFstauCUPIDastronomy softwareteleconOMPfaqopacitiesACSIS/HARPGAIAWVMclumpsdrip feedjcmtminutesnanahopetwitterPipelines and ArchivesThe latest happenings in JCMT data pipeline development, as well as our interfaces with the JCMT data archive at CADC.http://pipelinesandarchives.blogspot.com/noreply@blogger.com (Frossie)Blogger136125tag:blogger.com,1999:blog-4994845427978586964.post-4183957911110306809Sat, 07 Feb 2015 02:15:00 +00002015-02-10T07:00:38.337-10:00Better fonts in postscript output from KAPPA<div dir="ltr" style="text-align: left;" trbidi="on">Until now, the fonts available for the annotation of axes within KAPPA, <i>etc, </i>were very limited (basically those supplied by PGPLOT). As of 6th February, anyone using a Starlink installation that has been rsynced from Hilo or built from a git checkout will have access to several higher quality postscript fonts when producing postscript output.<br /><br />The environment variable PGPLOT_PS_FONT can be set to one of the following four values:<br /><br /><ol style="text-align: left;"><li>"Times"</li><li>"Helvetica"</li><li>"Courier"</li><li>"NewCentury"</li><li>"Zapf"</li></ol><div>to produce axis annotations in the corresponding font family. In addition, the usualt PGPLOT font number can be set to control whether the font is italic and/or bold., using the following font numbers:</div><div><ul style="text-align: left;"><li>"font=1": Normal</li><li>"font=2": Italic</li><li>"font=3": Bold</li><li>"font=4": Bold and Italic</li></ul><div>For example, here is a postscript image created using the default PGPLOT font with the following command:</div></div><div><br /></div><div>%&nbsp;<span style="font-family: Courier New, Courier, monospace;">display $KAPPA_DIR/m31 mode=perc \</span></div><div><span style="font-family: Courier New, Courier, monospace;">&nbsp; &nbsp; &nbsp; &nbsp; percentiles=</span><span style="font-family: 'Courier New', Courier, monospace;">\[2,98\] style=^sty</span></div><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-iNDXIw1Uvpg/VNVxvdHMl4I/AAAAAAAAFDA/UCWw1aakrbU/s1600/pgplot.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img src="http://1.bp.blogspot.com/-iNDXIw1Uvpg/VNVxvdHMl4I/AAAAAAAAFDA/UCWw1aakrbU/s1600/pgplot.png" height="296" width="400" /></a></div><div>If you first define the PGPLOT_PS_FONT environment variable to "Times" you get the following:</div><div><br /></div><div><span style="font-family: Courier New, Courier, monospace;">% setenv PGPLOT_PS_FONT Times</span></div><div><span style="font-family: 'Courier New', Courier, monospace;">% display $KAPPA_DIR/m31 mode=perc \</span></div><span style="font-family: Courier New, Courier, monospace;">&nbsp; &nbsp; &nbsp; &nbsp; percentiles=</span><span style="font-family: 'Courier New', Courier, monospace;">\[2,98\] style=^sty</span><br /><div><span style="font-family: 'Courier New', Courier, monospace;"><br /></span></div><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-EDxgDb4hv90/VNVy8RUVN9I/AAAAAAAAFDM/Chu-QVWkAbQ/s1600/times.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img src="http://4.bp.blogspot.com/-EDxgDb4hv90/VNVy8RUVN9I/AAAAAAAAFDM/Chu-QVWkAbQ/s1600/times.png" height="297" width="400" /></a></div><div><span style="font-family: inherit;">In the above command the text file </span><span style="font-family: Courier New, Courier, monospace;">sty </span><span style="font-family: inherit;">&nbsp;contains</span></div><div><span style="font-family: Courier New, Courier, monospace;"><br /></span></div><div><div><span style="font-family: Courier New, Courier, monospace;"># Set the default line width and colour for all graphical elements.</span></div><div><span style="font-family: Courier New, Courier, monospace;">width=3</span></div><div><span style="font-family: Courier New, Courier, monospace;">colour=black</span></div><div><span style="font-family: Courier New, Courier, monospace;"><br /></span></div><div><span style="font-family: Courier New, Courier, monospace;"># &nbsp;Override the above defaults for specific graphical elements.</span></div><div><span style="font-family: Courier New, Courier, monospace;">colour(border)=green</span></div><div><span style="font-family: Courier New, Courier, monospace;">colour(ticks)=green</span></div><div><span style="font-family: Courier New, Courier, monospace;">width(border)=3</span></div><div><span style="font-family: Courier New, Courier, monospace;">width(ticks)=3</span></div><div><br /></div></div><div><span style="font-family: inherit;">If, in addition, the Font attribute is set to 4 (bold and italic), you get the following:</span></div><div><span style="font-family: inherit;"><br /></span></div><div><br /><div><span style="font-family: Courier New, Courier, monospace;"><br /></span></div><div><span style="font-family: 'Courier New', Courier, monospace;">% display $KAPPA_DIR/m31 mode=perc \</span></div><span style="font-family: Courier New, Courier, monospace;">&nbsp; &nbsp; &nbsp; &nbsp; percentiles=</span><span style="font-family: 'Courier New', Courier, monospace;">\[2,98\] style="'^sty,font=4'"</span></div><div><span style="font-family: 'Courier New', Courier, monospace;"><br /></span></div><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-SHqBhUG1tus/VNV0B6HdJpI/AAAAAAAAFDU/CczWq-DPLAM/s1600/times_bi.png" imageanchor="1"><img src="http://2.bp.blogspot.com/-SHqBhUG1tus/VNV0B6HdJpI/AAAAAAAAFDU/CczWq-DPLAM/s1600/times_bi.png" height="297" width="400" /></a></div><div><span style="font-family: inherit;">And dont forget, you no longer need to use&nbsp;<i>psmerge</i> to combine postscript plots! See <a href="http://pipelinesandarchives.blogspot.com/2013/04/getting-hard-copy-graphics-from.html">this previous blog post</a>.&nbsp;</span></div></div>http://pipelinesandarchives.blogspot.com/2015/02/better-fonts-in-postscript-output-from.htmlnoreply@blogger.com (David Berry)0tag:blogger.com,1999:blog-4994845427978586964.post-6091132947469848267Fri, 28 Nov 2014 12:17:00 +00002014-11-28T02:42:52.863-10:00Converting HEALPix Maps to Tangent Plane Projection<div dir="ltr" style="text-align: left;" trbidi="on">Most astronomers are used to dealing with images that use the tangent-plane projection to map the spherical sky onto the flat image plane. However, the Advanced Data Products (ADPs) within the JCMT Science Archive (JSA) use a <a href="http://fits.gsfc.nasa.gov/wcs/mhg_20050615.pdf">HEALPix projection</a> instead, to facilitate the stitching together of adjacent tiles on the sky. The HEALPix projection differs from the tangent-plane projection in several ways:<br /><br /><ol style="text-align: left;"><li>The pixels are not quite square, with aspect ratios roughly in the range 1.0 to 1.4 depending on where on the sky the pixel is located (although all pixels are of equal area).</li><li>In the north and south polar regions (absolute Declination above 41.8 degrees) the RA and Dec axes are not orthogonal - the angle between east and north is about 60 degrees.</li><li>Maps that straddle the boundary between the polar and equatorial regions suffer from a discontinuous distortion as shown in the following image, a tile taken from a SCUBA-2 observation of M31.</li></ol><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-4YIKsPk7lmQ/VHhU9ZmwBEI/AAAAAAAAE-8/_WgClPQjKrE/s1600/pgplot2.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img src="http://4.bp.blogspot.com/-4YIKsPk7lmQ/VHhU9ZmwBEI/AAAAAAAAE-8/_WgClPQjKrE/s1600/pgplot2.gif" height="320" style="border: none;" width="400" /></a></div><div>In addition, JSA ADP maps are rotated so that north (in the equatorial region) is at 45 degrees to the image X axis.</div><div><br /></div><div>If you would prefer to see your maps in the usual tangent-plane projection with square pixels and north upwards, you can use the <a href="http://starlink.jach.hawaii.edu/devdocs/sun258.htx/node139.html">JSAJOIN</a> command within the Starlink <a href="http://starlink.jach.hawaii.edu/devdocs/sun258.htx/sun258.html">SMURF</a> package. This command pastes together one or more input maps in HEALPix projection, and then resamples the result onto a tangent-plane projection (the <a href="http://starlink.jach.hawaii.edu/devdocs/sun258.htx/node140.html">JSASPLIT</a> command does the opposite). &nbsp;For instance if the original HEALPix maps are listed in text file <span style="font-size: x-small;">"<span style="font-family: Courier New, Courier, monospace;">tiles</span>",</span><span style="font-family: inherit;"> then running the following command:</span></div><div><span style="font-size: x-small;"><br /></span></div><div>&nbsp; &nbsp; <span style="font-family: Courier New, Courier, monospace;">% jsajoin ^tiles tan_map.sdf</span></div><div><span style="font-size: x-small;"><br /></span></div><div><span style="font-family: inherit;">will paste all the HEALPix maps into a new tan-projected map in file "</span><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">tan_map.sdf".</span><span style="font-family: inherit;">&nbsp;By default the output projection is centred on the centre of the available input data, and the pixels are square with size equal to the nominal size of the input pixels. However, the details of the output projection can be specified explicitly, or a template map can be provided that defines the require output projection. See the <a href="http://starlink.jach.hawaii.edu/devdocs/sun258.htx/node139.html">documentation</a> for further details.</span><br /><span style="font-family: inherit;"><br /></span><span style="font-family: inherit;">The JSAJOIN command is actually a script that uses various other SMURF and KAPPA commands to do the work. In particularly, it uses the SMURF JSAPASTER command to paste together the individual HEALPix maps into a single HEALPix map, and then uses the KAPPA <a href="http://starlink.jach.hawaii.edu/devdocs/sun95.htx/node428.html">WCSALIGN</a> command to resample the combined HEALPix map onto a tangent-plane projection. It is of course possible to use these commands directly from the command line yourself if you want more control than is provided by the JSAJOIN script &nbsp;- for instance if you want to use different values for the resampling parameters. If you want to see how how JSAJOIN chooses it's resampling parameters, search for string "</span><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">wcsalign</span><span style="font-family: inherit;">" within file </span><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">$SMURF_DIR/jsajoin.py.</span></div></div>http://pipelinesandarchives.blogspot.com/2014/11/converting-healpix-maps-to-tangent.htmlnoreply@blogger.com (David Berry)0tag:blogger.com,1999:blog-4994845427978586964.post-8119852255336383586Tue, 19 Aug 2014 20:32:00 +00002014-08-19T10:32:45.483-10:00Recovering the number of chunks used by makemap - take 2.<div dir="ltr" style="text-align: left;" trbidi="on">The way in which the chunking behaviour of MAKEMAP is recorded, &nbsp;described in previous post ("<a href="http://pipelinesandarchives.blogspot.co.uk/2014/08/recovering-number-of-chunks-used-by.html">Recovering the number of chunks used by makemap</a>" ), has been changed. &nbsp;The SMURF extension of the output map created by MAKEMAP no longer contains an NCHUNK item. Instead, the FITS extension of the map contains a pair of headers as follows:<br /><br /><ul style="text-align: left;"><li>NCONTIG: An integer giving the number of contiguous chunks of data within the supplied input time-stream data files. This number does not include any chunking performed because of low memory and will usually be 1 if a single observation is supplied as input.</li><li>MEMLOW: A boolean value which is TRUE if the data was split into chunks due to the computer having insufficient memory.</li></ul></div>http://pipelinesandarchives.blogspot.com/2014/08/recovering-number-of-chunks-used-by_19.htmlnoreply@blogger.com (David Berry)0tag:blogger.com,1999:blog-4994845427978586964.post-1689511887107128677Thu, 14 Aug 2014 12:39:00 +00002014-08-19T10:39:03.394-10:00 Recovering the number of chunks used by makemap<div dir="ltr" style="text-align: left;" trbidi="on"><i>This post has been superceded - see the new post "<a href="https://pipelinesandarchives.blogspot.com/b/post-preview?token=6xew8UcBAAA._Mbf0QwqK13Te3m-3Wqefw.EO9YxVLctZBw0L7hfeM-cQ&amp;postId=8119852255336383586&amp;type=POST">Recovering the number of chunks used by makemap - take 2</a>".</i><br /><br />As of today, the number of chunks of continuous time-series data used by <span style="font-family: Courier New, Courier, monospace;">makemap</span> is recorded in the output map. It is stored in the NCHUNK item within the SMURF extension of the output map. So for instance, it can be viewed by doing:<br /><br /><span style="font-family: Courier New, Courier, monospace;">&nbsp; &nbsp;% kappa</span><br /><span style="font-family: Courier New, Courier, monospace;">&nbsp; &nbsp;% setext mymap.sdf smurf get nchunk loop=no</span><br /><br /><div>If NCHUNK has a value of 1, then all the input time-stream data was processed as a single chunk. If NCHUNK is larger than 1, then either the computer had insufficient memory to process the data as a single chunk, or the data was not taken as a single continuous observation.</div><div><br /></div><div>An rsync of the Hilo stardev system is required to use this new feature.</div></div>http://pipelinesandarchives.blogspot.com/2014/08/recovering-number-of-chunks-used-by.htmlnoreply@blogger.com (David Berry)0tag:blogger.com,1999:blog-4994845427978586964.post-8499208097856861461Fri, 25 Jul 2014 18:25:00 +00002014-07-25T08:27:27.011-10:00astronomy softwareDRSCUBA-2smurfstarlinkA new Starlink release contains notable updates to the SCUBA-2 configuration files.<br /><div style="text-align: justify;">The latest Starlink release - 2014A - has been made public. For details please read the release notes provided at:&nbsp;<a href="http://starlink.jach.hawaii.edu/starlink/2014A" target="_blank">http://starlink.jach.hawaii.edu/starlink/2014A</a></div><br /><div style="text-align: justify;">As part of this new release we want to highlight one significant update and a couple of new additions to the SCUBA-2 reduction arsenal of config files.<br /><br /></div><h3>Updates to bright_extended config file</h3><div style="text-align: justify;">This config file 'dimmconfig_bright_extended.lis' has always been intended for reducing data containing bright extended sources. It has remained untouched for a couple of years now despite advances in our understanding of SCUBA-2 reduction of bright regions. The config file now contains the following parameters/links:</div><div style="text-align: justify;"><br /></div><span style="font-family: Courier New, Courier, monospace;">&nbsp; &nbsp;^$STARLINK_DIR/share/smurf/dimmconfig.lis</span><br /><span style="font-family: Courier New, Courier, monospace;"><br /></span><span style="font-family: Courier New, Courier, monospace;">&nbsp; &nbsp;numiter=-40</span><br /><span style="font-family: Courier New, Courier, monospace;">&nbsp; &nbsp;flt.filt_edge_largescale=480</span><br /><span style="font-family: Courier New, Courier, monospace;">&nbsp; &nbsp;ast.zero_snr = 3</span><br /><span style="font-family: Courier New, Courier, monospace;">&nbsp; &nbsp;ast.zero_snrlo = 2</span><br /><span style="font-family: Courier New, Courier, monospace;"><br /></span><span style="font-family: Courier New, Courier, monospace;">&nbsp; &nbsp;ast.skip = 5</span><br /><span style="font-family: Courier New, Courier, monospace;">&nbsp; &nbsp;flt.zero_snr = 5</span><br /><span style="font-family: Courier New, Courier, monospace;">&nbsp; &nbsp;flt.zero_snrlo = 3</span><br /><span style="font-family: inherit;"><br /></span><br /><div style="text-align: justify;"><span style="font-family: inherit;">In previous Starlink releases i.e.&nbsp;</span>Hikianalia, the&nbsp;<span style="font-family: inherit;">bright_extended configuration file only contained the following:</span></div><span style="font-family: Courier New, Courier, monospace;"><br /></span><span style="font-family: 'Courier New', Courier, monospace;">&nbsp; &nbsp;</span><span style="font-family: Courier New, Courier, monospace;">numiter = -40</span><br /><span style="font-family: 'Courier New', Courier, monospace;">&nbsp; &nbsp;</span><span style="font-family: Courier New, Courier, monospace;">ast.zero_snr = 5</span><br /><span style="font-family: Courier New, Courier, monospace;"></span><br /><span style="font-family: 'Courier New', Courier, monospace;">&nbsp; &nbsp;</span><span style="font-family: Courier New, Courier, monospace;">flt.filt_edge_largescale = 600</span><br /><span style="font-family: inherit;"><br /></span><br /><h3><span style="font-family: inherit;">New 'FIX' config file</span></h3><span style="background-color: white; font-size: 16px;"><span style="font-family: inherit;">Two new config parameter files have been added&nbsp;</span></span><span style="background-color: white; font-size: 16px;">These are intended to be used with one of the existing dimmconfig files. They provide new values for selected parameters aimed at solving a particular problem ("blobs" in the final map, or very slow convergence).</span><br /><br /><ul><li><span style="background-color: white; font-size: 16px;"><span style="font-family: Courier New, Courier, monospace;">dimmconfig_fix_blobs.lis&nbsp;</span></span></li><ul><li><span style="background-color: white; font-family: inherit; font-size: 16px;"><span style="font-family: inherit; font-size: small; text-align: justify;">These parameters attempt to&nbsp;</span><span style="font-size: small; text-align: justify;">prevent smooth bright blobs of emission appearing in the final map. It does this by 1) identifying and flagging samples that appear to suffer from ringing a soft-edged Butterworth filter in place of the normal hard-edged filter, and 3) rejecting samples for which the separate sub-arrays see a markedly different common-mode signal.</span></span></li></ul></ul><ul><li><span style="background-color: white; font-size: 16px;"><span style="font-family: Courier New, Courier, monospace;">dimmconfig_fix_convergence.lis</span></span></li><ul><li>The parameters defined by this file attempt to aid the convergence&nbsp;<span style="background-color: white;">process, and should be used for maps that will not converge within&nbsp;</span><span style="background-color: white;">a reasonable number of iterations.</span></li></ul></ul>http://pipelinesandarchives.blogspot.com/2014/07/a-new-starlink-release-contains-notable.htmlnoreply@blogger.com (Harriet Parsons)0tag:blogger.com,1999:blog-4994845427978586964.post-6572168931721630622Wed, 23 Jul 2014 11:01:00 +00002014-07-23T01:03:11.328-10:00Spotting chunking caused by insufficient memory<div dir="ltr" style="text-align: left;" trbidi="on">It has been shown that better maps are created if all available time stream data are processed in a single chunk to create a single map. If the available memory is too small to allow this to happen, the data will be split into multiple chunks and a separate map made from each chunk, which are all co-added at the end to make the final map.<br /><br />Thus it can be important to spot whether or not your map was "chunked" or not. Until recently the only option was to trawl through the makemap screen output looking for the warning about insufficient memory. Now, however, you also have an option to tell makemap to abort immediately with an informative error message if the data would be chunked due to insufficient memory (no output map is created in this case). To do this, add "memcheck=1" to your config.<br /><br />Note, this only affects what happens if the data is processed in chunks due to lack of memory - if chunking is caused by the time-series data being intrinsically discontiguous then a map is still created even if memcheck is set to 1.<br /><br />It will be necessary to rsync the stardev system from Hilo to use this feature. See <a href="https://www.blogger.com/%C2%A0http://starlink.jach.hawaii.edu/starlink/rsyncStarlink">starlink.jach.hawaii.edu/starlink/rsyncStarlink</a></div>http://pipelinesandarchives.blogspot.com/2014/07/spotting-chunking-caused-by.htmlnoreply@blogger.com (David Berry)0tag:blogger.com,1999:blog-4994845427978586964.post-5362874849221581036Tue, 08 Jul 2014 21:15:00 +00002014-07-08T11:15:18.638-10:00calibrationorac-drpicardCalibrating SCUBA-2 data in surface brightness units<p>In most cases the default calibration for SCUBA-2 data processed by the ORAC-DR pipeline is mJy beam<sup>-1</sup>. The exception is the recipe for extended sources, REDUCE_SCAN_EXTENDED_SOURCES, which calibrates data in mJy arcsec<sup>-2</sup>. <p>Unfortunately there was an error in an earlier version of this recipe which meant that the FCF was applied incorrectly. The corrected method is available now with an update of ORAC-DR (either from github or via rsync from JAC). If you have data processed with this recipe (either by running it yourself, or downloading processed products from CADC) then re-calibrating the data is easy: simply divide by the pixel area using KAPPA <code>cdiv</code>. <p>There is a new PICARD recipe for easy calibration of maps produced by running <code>makemap</code> by hand. CALIBRATE_SCUBA2_DATA allows data to be calibrated in in per-beam and surface brightness units. With no parameters, this recipe will calibrate data in mJy beam<sup>-1</sup>. For surface brightness calibration, set the recipe parameter USEFCF to 1 and FCF_CALTYPE to ARCSEC, and the recipe will then use the default ARCSEC FCF for the wavelength of the given data. <p>The recipe can also convert the calibration from one type to another. If your data are already calibrated in mJy beam<sup>-1</sup>, they can be given to CALIBRATE_SCUBA2_DATA with the FCF_CALTYPE recipe parameter above, and the recipe will create a new file (with suffix <code>_cal</code>) with units of mJy arcsec<sup>-2</sup>. The value and units of the FCF are written into the FITS header of the calibrated file. <p>The companion recipe, UNCALIBRATE_SCUBA2_DATA, will undo the current calibration, reverting the units to pW in the output file (which has a suffix of <code>_uncal</code>). <p>Using ORAC-DR or PICARD to perform the (un)calibration is preferred to simply multiplying your data by the FCF as they also set the units correctly for the output files(s), and write the value of the FCF used into the FITS header of the file. <p>However, there is one note to highlight: the recommended way to calibrate data (either from raw or when changing from per beam to per square-arcsec) is to calibrate the individual observations first, and then coadd those (re)calibrated files. Calibrating or re-calibrating coadds will fail because the coadding step was recently updated to remove FITS header entries that differ between the input files. These usually include the UTDATE which is used by the ORAC-DR calibration system. A future upgrade will provide a workaround though the recommendation to calibrate individual observations stands.http://pipelinesandarchives.blogspot.com/2014/07/calibrating-scuba-2-data-in-surface.htmlnoreply@blogger.com (AndyG)0tag:blogger.com,1999:blog-4994845427978586964.post-4747492168312285796Fri, 06 Jun 2014 11:29:00 +00002014-06-06T01:30:41.818-10:00Change to FLT Ringing Filter<div dir="ltr" style="text-align: left;" trbidi="on">If you use the AST.SKIP and FLT.RING_BOX1 configuration parameters to prevent artificial blobs of emission appearing in your SCUBA-2 maps, then you may have seen cases where &nbsp;the centres of real bright sources are badly affected, as in the following example spotted by Harriet:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-ErCRqMAovWQ/U5GdBOGgolI/AAAAAAAAEyE/Xbc1gEPl2T0/s1600/ringing-problem.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-ErCRqMAovWQ/U5GdBOGgolI/AAAAAAAAEyE/Xbc1gEPl2T0/s1600/ringing-problem.png" height="400" width="396" /></a></div><br />This issue may also cause complete holes to appear around bright sources. These problems are caused by the fact that the ringing filter is applied on the initial "pre-iterations" specified by AST.SKIP, as well as on all subsequent "real" iterations. If the ringing filter is restricted to just the "real" iterations, the following map is produced (left uses the same scale as above, right is a close up of the central region):<br /><br /><br /><center><table><tbody><tr><td><a href="http://1.bp.blogspot.com/-UIJ6w5HMLgE/U5GgzrmbSXI/AAAAAAAAEyQ/2Ys6r7Use0U/s1600/ringing-problem2.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-UIJ6w5HMLgE/U5GgzrmbSXI/AAAAAAAAEyQ/2Ys6r7Use0U/s1600/ringing-problem2.png" height="200" width="198" /></a></td><td><a href="http://4.bp.blogspot.com/-5_VZNw7Ry9Q/U5Gg1kKJ-8I/AAAAAAAAEyY/UbMtvfKohpo/s1600/ringing-problem3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-5_VZNw7Ry9Q/U5Gg1kKJ-8I/AAAAAAAAEyY/UbMtvfKohpo/s1600/ringing-problem3.png" height="199" width="200" /></a></td></tr></tbody></table></center>The stardev system at Hilo has been updated to include this fix.<br /><br /><i>For those interested in the details...</i> The problem is caused by the fact that no FLT masking can be done on the first pre-iteration, since there is no map available at that point to use as a mask. Therefore bright sources cause heavy ringing on the first iteration. This ringing is then detected and the corresponding samples flagged by the ringing filter. This means that the bright sources are blanked out in the map created at the end of the first iteration. On the second pre-iteration, this map is used to locate the bright sources that should be masked out when estimating the FLT model - but the bright source is not present in this map and so is not masked out on the second iteration either, leading to heavy ringing again. And round it goes doing the same thing on each iteration. Restricting the ringing filter to "real" iterations solves the problem because bright sources have been masked out in the FLT model by the time the ringing filter is used for the first time, and so have not caused the heavy ringing that would otherwise have been detected and flagged by the ringing filter. In fact, in may have been sufficient just to inhibit the ringing filter on the first pre-iteration, rather than on all pre-iterations, but this would have been of little benefit since blobs only start to develop once the AST model is included in the iterative scheme (i.e. once all the pre-iterations have completed).</div>http://pipelinesandarchives.blogspot.com/2014/06/change-to-flt-ringing-filter.htmlnoreply@blogger.com (David Berry)0tag:blogger.com,1999:blog-4994845427978586964.post-4660317121253383531Thu, 05 Jun 2014 11:10:00 +00002014-06-05T20:34:07.706-10:00Interpreting SCUBA-2 map Quality Arrays<div dir="ltr" style="text-align: left;" trbidi="on">SCUBA-2 maps created using "AST-masking", such as is done by &nbsp;<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">dimmconfig_bright_extended</span><span style="font-family: inherit;">, will contain a Quality array indicating the background pixels that were masked (<i>i.e.</i> forced to zero). In itself this is fairly simple - background pixels have a non-zero Quality value and source pixels have a Quality value of zero. However, it is also possible to have independent masks for the FLT and COM models, in addition to the AST mask. For instance, if the "<a href="http://pipelinesandarchives.blogspot.co.uk/2013/07/automatic-flt-masking.html">auto-FLT masking</a>" scheme is used the Quality array contains both a FLT and an AST mask and so some care needs to be used when interpreting the Quality array.</span><br /><span style="font-family: inherit;"><br /></span><span style="font-family: inherit;">Each value in the Quality array is restricted to taking integer values between 0 and 255, and so can be thought of as 8 separate bits. E</span><span style="font-family: inherit;">ach "bit plane" within the Quality array holds a single mask - AST, FLT or COM. &nbsp; The </span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="http://starlink.jach.hawaii.edu/docs/sun95.htx/node415.html">showqual</a></span><span style="font-family: inherit;"> command can be used to find out which mask is held by which bit plane</span><span style="font-family: inherit;">:</span><br /><span style="font-family: inherit;"><br /></span><br /><pre><span style="font-size: x-small;">% kappa<br />% showqual fred.sdf<br /> AST (bit 1) - "Set iff AST model is zeroed at the output pixel"<br /> FLT (bit 2) - "Set iff FLT model is blanked at the output pixel"</span><br /></pre><br />This means that the AST mask is stored in bit 1 (the least significant bit), the FLT mask is stored in bit 2, and there is no COM mask. Note, if a map was produced using FLT masking but no AST masking, then the FLT mask would be stored in bit 1.<br /><br />The decimal integer value of any element of the Quality array is equal to the binary value formed from the bits listed by <span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">showqual.</span><span style="font-family: inherit;">&nbsp;So in the above case the maximum Quality value is 3 (</span><span style="font-family: inherit;">the decimal equivalent of binary "11" - both bits set). Remembering that a bit is set (</span><i style="font-family: inherit;">i.e.</i><span style="font-family: inherit;"> is 1) for background pixels and cleared (</span><i style="font-family: inherit;">i.e.</i><span style="font-family: inherit;"> is 0) for source pixels, it follows that the four possible decimal Quality values in the above case (0-3) are:</span><br /><span style="font-family: inherit;"><br /></span><span style="font-family: inherit;">0 - neither bit set, so the pixel is inside both the AST and the FLT mask (a source pixel).</span><br /><span style="font-family: inherit;"><br /></span><span style="font-family: inherit;">1 - bit 1 set but not bit 2, so the pixel is outside the AST mask but inside the FLT mask (a border-line pixel).</span><br /><span style="font-family: inherit;"><br /></span><span style="font-family: inherit;">2 - bit 2 set but not bit 1,&nbsp;</span><span style="font-family: inherit;">so the pixel is inside the AST mask but outside the FLT mask (a border-line pixel).</span><br /><span style="font-family: inherit;"><br /></span><span style="font-family: inherit;">3 - both bits set, so the pixel is inside neither mask (a background pixel).</span><br /><div><span style="font-family: inherit;"><br /></span></div><div><span style="font-family: inherit;">In the following screen shot of a Quality array displayed using GAIA, the black areas have value zero and are thus inside both masks, the dark brown areas have value 2 and are thus inside the AST mask but outside the FLT mask. The light brown areas have value 3 and are inside neither mask. In this&nbsp;</span>particular<span style="font-family: inherit;">&nbsp;case, there are no &nbsp;areas with a quality value of 1, so the FLT mask is contained entirely within the AST mask.</span></div><div><span style="font-family: inherit;"><br /></span></div><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-Ar3gOj_mOks/U5BE60PEsMI/AAAAAAAAExk/b6PMPGbSvlw/s1600/qual.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-Ar3gOj_mOks/U5BE60PEsMI/AAAAAAAAExk/b6PMPGbSvlw/s1600/qual.png" height="263" width="320" /></a></div><div><span style="font-family: inherit;"><br /></span></div><div><span style="font-family: inherit;">There are several commands within </span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">kappa</span><span style="font-family: inherit;"> that manipulate Quality arrays in various ways. For instance, the </span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="http://starlink.jach.hawaii.edu/docs/sun95.htx/node401.html">setbb</a></span><span style="font-family: inherit;"> command allows pixel data values to be set bad if the associated Quality value has a specified collection of set bits. Thus:</span></div><div><span style="font-family: inherit; font-size: x-small;"><br /></span></div><div><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">% setbb fred 1</span></div><div><span style="font-family: inherit;"><br /></span></div><div><span style="font-family: inherit;">will set all pixels bad except for those inside the AST mask. Likewise,&nbsp;</span></div><div><span style="font-size: x-small;"><br /></span></div><div><div><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">% setbb fred 2</span></div><div><span style="font-family: inherit;"><br /></span></div><div><span style="font-family: inherit;">will set all pixels bad except for those inside the FLT mask. &nbsp;Note, the change made by </span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">setbb </span><span style="font-family: inherit;">is temporary - it can be undone by doing:</span></div><div><span style="font-size: x-small;"><br /></span></div><div><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-small;">% setbb fred 0</span></div></div><div><span style="font-family: inherit;"><br /></span></div><div><span style="font-family: inherit;">To display the</span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"> fred.sdf </span><span style="font-family: inherit;">map and then overlay the AST mask in blue and the FLT mask in red, do:</span></div><span style="font-size: x-small;"><br /></span><br /><pre><span style="font-size: x-small;">% display fred mode=perc percentiles=\[2,98\]<br />% setbb fred 1<br />% contour fred clear=no mode=good labpos=! style='colour=blue'<br />% setbb fred 2<br />% contour fred clear=no mode=good labpos=! style='colour=red'<br />% setbb fred 0</span><br /></pre><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-UITZsKAx4Hs/U5BPQaRoDKI/AAAAAAAAEx0/LGfHp9nMRmc/s1600/masks.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-UITZsKAx4Hs/U5BPQaRoDKI/AAAAAAAAEx0/LGfHp9nMRmc/s1600/masks.png" height="342" width="400" /></a></div></div>http://pipelinesandarchives.blogspot.com/2014/06/interpreting-scuba-2-map-quality-arrays.htmlnoreply@blogger.com (David Berry)0tag:blogger.com,1999:blog-4994845427978586964.post-7381360675745024244Fri, 04 Apr 2014 07:04:00 +00002014-04-03T21:04:09.287-10:00Map Variances when using FLT.FILT_EDGE_LARGESCALE_LAST<div dir="ltr" style="text-align: left;" trbidi="on">The FLT.FILT_EDGE_LARGESCALE_LAST makemap configuration parameter allows you to create a map in which the background regions (i.e. the regions zeroed out by the AST mask) are filtered on a shorter scale than the source regions, thus producing a pleasingly flat background without removing structure from the source regions.<br /><br />However, when using this option care needs to be taken when interpreting the variances stored with the map. Setting&nbsp;FLT.FILT_EDGE_LARGESCALE_LAST &nbsp;causes the filter size to be changed on the last iteration. Whilst this does not affect the data values in the source regions much (since the source signal has been extracted into a separate model by the time the final iteration occurs), it <i>does</i>&nbsp;affect the variances. This is because the variances are formed from the time-stream residuals that are left after removal of the astronomical signal estimated on the penultimate iteration, and all these residuals - no matter where they are on the sky - are filtered using the smaller filter size given by &nbsp;&nbsp;FLT.FILT_EDGE_LARGESCALE_LAST.<br /><br />The upshot of all this is that the basic scheme described above results in artificially low variances in the source regions. &nbsp;To avoid this, a change was introduced into SMURF on 13th February so that now, if the FLT.FILT_EDGE_LARGESCALE_LAST parameter is set, the variances stored in the map are the ones generated on the penultimate iteration (that is, the final iteration that used the basic filter size rather than the special last-iteration filter size). &nbsp;This should give more realistic variances for the source regions, but at the cost of &nbsp;unrealistically <i>high</i> variances in the background regions.</div>http://pipelinesandarchives.blogspot.com/2014/04/map-variances-when-using.htmlnoreply@blogger.com (David Berry)0tag:blogger.com,1999:blog-4994845427978586964.post-1770886205870830849Wed, 22 Jan 2014 19:21:00 +00002014-02-03T15:22:01.481-10:00orac-drpicardSCUBA-2CHECK_RMS: comparing SCUBA-2 map noise with the ITCThere's a new SCUBA-2 analysis tool in town, and its aim is to help observers get a feel for whether their maps are reaching the intended sensitivity levels. It comes in two forms: <ul><li>PICARD recipe: SCUBA2_CHECK_RMS</li><li>Offline pipeline recipe REDUCE_SCAN_CHECKRMS</li></ul>At its core, the "CHECK_RMS" analysis estimates the RMS and NEFD for a given observation to compare with values from the SCUBA-2 integration time calculator (ITC). To obtain access to these new recipes, you must be using <code>/stardev</code> in Hilo or update your own version using <code>rsync</code> (see <a href="http://starlink.jach.hawaii.edu/starlink/rsyncStarlink">http://starlink.jach.hawaii.edu/starlink/rsyncStarlink</a>). <h4>PICARD recipe: SCUBA2_CHECK_RMS</h4> <p>As usual, the PICARD recipe is designed to be run on processed data. To run it, type: <pre><br /><code>% picard -log sf -nodisplay SCUBA2_CHECK_RMS myfiles*.sdf</code><br /></pre>in the directory containing the files to be analyzed. You may notice a warning (cyan text) about missing NEP data: that doesn't affect the main analysis and can safely be ignored. However, if run in Hilo, the recipe can query the archive of log files generated by the QL pipelines for NEP data. These NEP data are used to determine an effective noise and NEFD from the timeseries noise calculations, which can then be compared with the other values. <p>The recipe can be given calibrated or uncalibrated maps: the default FCF will be applied to uncalibrated data. Note that these maps must be made from a <b>single</b> complete observation, either by hand with <code>makemap</code> or the offline pipeline. Single observations must be used because the recipe relies on FITS header information that will be incorrect for coadded files. Note that maps made by the SUMMIT pipeline should not be given to this recipe; the SUMMIT pipeline will soon start recording the same CHECK_RMS data itself. <p>The input images are cropped to a circle of radius 90 arcsec from which the median noise and NEFD (the values <code>RMS_MAP</code> and <code>NEFD_MAP</code> in the log file) are calculated. The mean exposure time is estimated over the same map area (<code>TEXP</code>). It's important to note that the <code>RMS_MAP</code> is derived from the variance component in the map, and not from the map itself. This means that the results should be largely independent of the chosen <code>makemap</code> config parameters, and are more likely to reflect changes in the number of samples that go into the map rather than any differences in spatial structure introduced by applying different high-pass filters in the map-maker. <p>The map size and scan type are read from the FITS header and used to define the observation type for the ITC. The RMS corresponding to this observation type is calculated from the SCUBA-2 ITC (<code>RMS_ITC</code>); the integration time is used to convert that to a corresponding NEFD (<code>NEFD_ITC</code>). However, it is important to note that if you have chosen a non-standard map size for a PONG then this recipe will be unable to calculate values from the ITC, rendering it somewhat less useful. <p>If the NEP data are available (in Hilo only, and for dates after 20120515), the average NEP for a given observation is calculated from the archived QL pipeline log file (<code>log.nep</code>) corresponding to the date and wavelength of the input map. The FCF (either from the file or the standard value) and mean transmission over the observation is used to convert this to a zenith NEFD (<code>NEFD_NEP</code>). The ITC scaled elapsed time is used to convert that to an RMS noise (<code>RMS_NEP</code>). <p>The results are written to a log file called <code>log.checkrms</code>, the format of which is described below. <p>The following recipe parameters are supported: <ul><li>KEEPFILES - if set to 1, the <code>_crop</code> files created during processing will be retained on disk, otherwise all intermediate files will be deleted. <li>MAP_RADIUS: radius (in arcsec) of the region used for calculating the map noise and NEFD. Default is 90 arcsec. Note that poor results can be derived if the map radius is too small and it may be more useful to use a larger radius for larger maps.</li><li>STATS_ESTIMATOR: Statistical estimator for the NEFD and RMS calculated from the input map. May be mean or median - default is median.</li></ul> <h4>Offline pipeline: REDUCE_SCAN_CHECKRMS</h4> <p>An alternative to the PICARD recipe, this recipe performs all the necessary steps on the raw data to do a complete self-consistent analysis for each observation, from calculating the timeseries noise to processing the data and comparing results with the ITC. The steps performed are: <ul><li>Calculate timeseries noise from the entire observation; <li>Determine various values from the FITS headers (elapsed time, average airmass, elevation, optical depth, transmission, date etc); <li>Make a map using the given config file (default if not specified); <li>Calculate the noise (from the variance component), NEFD and exposure time within a 3-arcmin diameter circle at the map centre; <li>Use the mapping parameters and elapsed time to derive the expected RMS and NEFD from the ITC; <li>Write results to log file. </ul>Since a custom <code>makemap</code> config file can be given (see below), it is possible to use this recipe as your main data reduction recipe (if the standard processing is sufficient), to avoid re-reducing data. Note that the recipe does not yet do any coadding, so currently output maps will have to be combined with <code>MOSAIC_JCMT_IMAGES</code>. <p>To use it, run it like any other pipeline recipe. Initialize the pipeline in the usual way: <pre><br /><code>% oracdr_scuba2_XXX -cwd</code><br /></pre>Add the names of the relevant files to a text file (say, <code>myfiles.lis</code>), and run the pipeline with: <pre><br /><code>% oracdr -log sf -nodisplay -loop file -files myfiles.lis REDUCE_SCAN_CHECKRMS</code><br /></pre> <p>All of the recipe parameters supported by the default <code>REDUCE_SCAN</code> recipe can be given to this recipe, the most important of which is likely to be <code>MAKEMAP_CONFIG</code> for specifying an alternative config type for <code>makemap</code>. If not given, the default config file (<code>dimmconfig.lis</code>) is used. Note that the same config file will be used for all data so be sure to use an appropriate config. <h4>Log file format</h4> <p>The log file, called <code>log.checkrms</code>, produced by each method contain the following entries for each map (observation): <ul><li><code>UT</code> - UT date including day fraction <li><code>Source</code> - object name, upper case with spaces removed <li><code>Obs</code> - observation number <li><code>FILTER</code> - filter (wavelength) <li><code>telapsed</code> - elapsed time of observation (sec) <li><code>texp</code> - mean exposure time, derived from EXP_TIME NDF component (sec) <li><code>trans</code> - mean line-of-sight transmission <li><code>nep_av</code> - mean NEP for current observation (W/sqrt(Hz)) <li><code>nep_av_err</code> - uncertainty in <code>nep_av</code> (W/sqrt(Hz)) <li><code>rms_nep</code> - RMS derived from <code>nefd_nep</code> (mJy/beam) <li><code>nefd_nep</code> - NEFD derived from <code>nep_av</code> and scaled elapsed time (mJy sqrt(sec)) <li><code>rms_map</code> - RMS noise in map, obtained from median of error component (mJy/beam) <li><code>nefd_map</code> - NEFD derived from combination of variance and exposure time images (mJy sqrt(sec)) <li><code>rms_itc</code> - RMS noise estimated by SCUBA-2 integration time calculator (ITC) (mJy/beam) <li><code>nefd_itc</code> - NEFD derived from <code>rms_itc</code> and scaled elapsed time (mJy sqrt(sec)) <li><code>itc_obstype</code> - observation type used by the ITC to derived RMS <li><code>rms_ratio</code> - ratio of <code>rms_map</code> to <code>rms_itc</code><li><code>El</code> - mean elevation in degrees <li><code>CSO</code> - mean zenith optical depth at 225 GHz, derived from WVM <li><code>Tau</code> - mean zenith optical depth at the current wavelength (given by FILTER above) <li><code>Radius</code> - radius in arcsec of image used in analysis <li><code>pixscale</code> - pixel scale in arcsec <li><code>f</code> - ITC f parameter <li><code>project</code> - project ID </ul> <p>Note that ITC-derived values will be undefined if the PONG map size is non-standard. <h4>So what should I look for?</h4> <p>The log file contains all the results of the analysis. It's worth comparing <code>RMS_ITC</code> with the number you determined at the time your proposal was written. It's important to be aware that the PICARD and ORAC-DR recipes estimate values from the ITC using the actual observed airmass/opacity, rather than an average obtained from the source declination. This will undoubtedly lead to differences in the ITC-derived parameters. <p>Perhaps the most important number is the <code>rms_ratio</code> which is the ratio of the <code>RMS_MAP</code> to the <code>RMS_ITC</code>. Ideally this value should be 1, though in practice that's rarely the case. Sometimes it's down to the region used for calculating the RMS being too small (in this case, increaseIf the results are very far from the expected values then contact your Friend of the Project to see if it's possible to work out why. <h4>SUMMIT pipeline</h4> <p>While not yet released, the SUMMIT pipeline will also perform its own CHECK_RMS analysis and will write a corresponding <code>log.checkrms</code> with the same format as for the other methods. <p>A few things to keep in mind when this is released: <ul><li>The NEP and derived values will be undefined (NaN) in the log file, as the SUMMIT pipeline does not have access to the QL log file data. <li>An entry is written each time a new image is created, which may yield multiple entries for a given observation. However, each entry should be self-consistent. </ul>This blog entry will be updated with the relevant documentation when the SUMMIT pipeline starts to produce these data. http://pipelinesandarchives.blogspot.com/2014/01/checkrms-comparing-scuba-2-map-noise.htmlnoreply@blogger.com (AndyG)0tag:blogger.com,1999:blog-4994845427978586964.post-4337815603374442965Tue, 21 Jan 2014 15:28:00 +00002014-01-21T05:28:30.330-10:00Temporary problems with SKYOOP<div dir="ltr" style="text-align: left;" trbidi="on">If you use the SKYLOOP script to create SCUBA-2 maps, you need to be aware that some issues have been reported recently with SKYLOOP that are under active investigation. One is that it can use more memory than is allowed by your system, with the result that it can be killed without warning in order to protect the rest of your system. No error or&nbsp;warning&nbsp;messages are issued when this happens - it just suddenly stops for no apparent reason. One way to avoid this to limit the amount of memory used by makemap. To do this run skyloop as follows:<br /><br />% skyloop &nbsp;extra="maxmem=120000"<br /><div><br /></div><div>where the number is in units of MiB and should be changed to suit your system.</div><div><br /></div><div>Sadly, even with this setting, things can sometimes still go wrong. This is because makemap is currently under-estimating the amount of memory it needs.</div><div><br /></div><div>A second issue is that if you use skyloop to create a map from a single contiguous chunk of data, and then use makemap to create a second map from the same chunk of data, there are significant differences between the two maps. In theory, the two maps should be the same - skyloop should produce better results when making maps from multiple chunks of data, but they should give the same results for single chunk maps. This indicates that something - maybe several things - is wrong with skyloop.&nbsp;</div><div><br /></div><div>I'm currently investigating both these issues and I am making good progress but have not yet completely solved them. So in the mean time, beware of using SKYLOOP and watch this space...</div><div><br /></div><div>David</div><div><br /></div></div>http://pipelinesandarchives.blogspot.com/2014/01/temporary-problems-with-skyoop.htmlnoreply@blogger.com (David Berry)0tag:blogger.com,1999:blog-4994845427978586964.post-4601872472964229882Fri, 17 Jan 2014 03:19:00 +00002014-01-16T17:20:53.750-10:00calibrationFCFspicardSCUBA-2starlinkUpdates to SCUBA2_CHECK_CAL: 2-component fittingHappy New Year to everyone reading this (or Hau‛oli Makahiki Hou as we say in Hawai‛i), and welcome to 2014. <br /><br />The PICARD recipe SCUBA2_CHECK_CAL has been updated to use a two-component gaussian profile to fit the beam when determining an FCF from a calibration observation, as long as the signal-to-noise ratio exceeds 100. Previously, a single-component fit was used which was not constrained to be a gaussian. (If the signal-to-noise ratio is not high enough it will fall back on the old behavior.) This two-component fit is based on the FWHM and relative amplitudes of the two components derived in Dempsey et al. (2013).<br /><br />This change has no effect on the ARCSEC FCFs, but results in a small (~1%) but consistent reduction in the BEAM FCFs. The effect should be small enough to be negligible, and with this change we now have slightly higher confidence in the resulting FCFs than with the old one-component fits. BEAMMATCH FCFs are likewise unaffected by the change, as testing showed they were best fit with profiles not forced to be gaussian.<br /><br />You can set the behavior of SCUB2_CHECK_CAL manually using the FIT_GAUSSIAN parameter in your config file. The default value of 2 uses a two-component gaussian fit (when S/N &gt; 100), a value of 1 uses a one-component gaussian fit, and a value of 0 recovers the old behavior of a one-component fit not constrained to be a gaussian.<br /><br /><div><span style="font-family: inherit;">At the moment, to use this feature you will need to update your Starlink to the current development version. Linux users can simply rsync Starlink from JAC following the instructions at&nbsp;</span><a href="http://starlink.jach.hawaii.edu/starlink/rsyncStarlink">http://starlink.jach.hawaii.edu/starlink/rsyncStarlink</a></div>http://pipelinesandarchives.blogspot.com/2014/01/updates-to-scuba2checkcal-2-component.htmlnoreply@blogger.com (Daniel Berke)0tag:blogger.com,1999:blog-4994845427978586964.post-4727541437655625685Tue, 22 Oct 2013 15:52:00 +00002013-10-22T05:52:37.698-10:00How to loose fewer samples with NOI.BOX_SIZE<div dir="ltr" style="text-align: left;" trbidi="on"><div>At the end of the first iteration, when the common-mode signal (COM model), low frequency drift (FLT model), and the astronomical signal (AST model) have all been removed from the data, the remaining residuals are used to estimate the variance of the high frequency noise in each bolometer (the NOI model). The reciprocal of these variances are used as weights on the second and subsequent iterations, and are applied to the time-series values that are binned to form the map.</div><div><br /></div><div>By default, the NOI model contains a single variance value for each bolometer , and so all samples from a single bolometer get the same weight in the map. However, the NOI.BOX_SIZE parameter can be used to force each bolometer to have <i>multiple</i> variance estimates, with one value for each box of samples. See <a href="http://pipelinesandarchives.blogspot.co.uk/2013/02/scuff-removal-using-noiboxsize.html">this post</a>.&nbsp;</div><div><br /></div><div>Whilst this is often in an improvement over the default situation in that it allows for varying noise levels, it has two potential issues:</div><div><ol style="text-align: left;"><li>All samples in a box are given the same variance value, thus introducing the potential for steps in the weights and corresponding artifacts in the map.</li><li>The noise in each box is found by taking the FFT of the data and finding the average power in the 2 to 10 Hz band. Using an FFT is problematic on short time streams, as any missing values (such as caused by steps, jumps, common-mode flagging, etc) have a proportionally larger effect, causing the FFT to be less representative&nbsp;of the data. A requirement that the time stream should have a majority of unflagged values in order to be usable means that a lot of data is often rejected because a reliable FFT cannot be obtained.</li></ol><div>An alternative scheme that avoids these two issues has been added to <span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">makemap</span>, which is enabled by setting "NOI.BOX_TYPE =1" in your configuration. This option causes the noise at a sample to be set equal to the variance of the neighbouring sample values - those that are less than half a box size (i.e. NOI_BOX_SIZE / 2) away from the sample. This scheme does not involve taking an FFT and produces a continuously varying estimate of the variance and so avoids steps in the weights used to make the map. It also is minimally affected by flagged samples and so results in fewer data samples being flagged as unusable (as shown by the "NOISE" entry in the samples statistics at the end of each iteration).</div></div></div>http://pipelinesandarchives.blogspot.com/2013/10/how-to-loose-fewer-samples-with.htmlnoreply@blogger.com (David Berry)0tag:blogger.com,1999:blog-4994845427978586964.post-7029354776256402306Wed, 09 Oct 2013 08:25:00 +00002013-10-08T22:25:49.968-10:00Weighting different observations in makemake and skyloop<div dir="ltr" style="text-align: left;" trbidi="on">Both makemap and skyloop can be used to produce maps from multiple overlapping observations. Up to now, it has not been possible to control the weights used for each observation when forming the total co-added map. Instead, fixed weights equal to the reciprocal of each pixel variance were used. The new CHUNKWEIGHT configuration parameter can now be used to specify a weight for each observation. The weights determined from the pixel variances are multiplied by these user-specified weights, which default to unity.<br /><br />The weights can be specified in two ways:<br /><ol style="text-align: left;"><li>The CHUNKWEIGHT configuration parameter can be set to an arbitrary algebraic expression in which the variable names are the names of FITS keywords. The keywords must exist in the input data and have numerical values. The expression should be enclosed in double quotes. As an example, you could include this line in your configuration file:<br /><br />&nbsp;&nbsp;&nbsp;<span style="font-family: Courier New, Courier, monospace;">chunkweight = "AMSTART * WVMTAUST"</span></li><br />The values of the FITS keywords are read from the observation header and substituted into the expression to get the weight to use for the observation.<br /><br /><li>A comma-separated list of explicit numerical weights can be supplied, enclosed in parentheses. If insufficient values are given in the list, the last value is replicated as often as is necessary. <br /><br />&nbsp;&nbsp;&nbsp;<span style="font-family: Courier New, Courier, monospace;">chunkweight = (1.0,2.0,3.0)</span></li><br /></ol><div><span style="font-family: Courier New, Courier, monospace;"><br /></span></div></div>http://pipelinesandarchives.blogspot.com/2013/10/weighting-different-observations-in.htmlnoreply@blogger.com (David Berry)0tag:blogger.com,1999:blog-4994845427978586964.post-3617787713865854613Tue, 08 Oct 2013 13:06:00 +00002013-10-08T03:06:32.378-10:00ACSIS/HARPdata processingHoles in heterodyne mapsHARP has only twelve operational receptors since May 20.&nbsp; If one or two also suffer from interference and fail quality assurance, then there can be locations in the reduced spectral cubes where no valid data are present.&nbsp; Such missing spectra (shown in blue) will often be arranged in a grid pattern as in the example below.<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-DjjnG2nf9-U/UlPxTMgm6nI/AAAAAAAAAEA/_CeTOvBIObw/s1600/holeyharp.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="377" src="http://1.bp.blogspot.com/-DjjnG2nf9-U/UlPxTMgm6nI/AAAAAAAAAEA/_CeTOvBIObw/s400/holeyharp.png" width="400" /></a></div><br />This can create issues if you wish to integrate flux or simply improve the cosmetic appearance.<br /><br />One approach is to smooth the data but this degrades all the spectra.&nbsp; Instead use KAPPA::FILLBAD. This replaces just the bad values with a smooth function derived from neighbouring values, derived by iterating to a solution of Laplace's equation.<br /><br />Just using FILLBAD's default values will duly fill in the holes.&nbsp; You might prefer not to smooth in the spectral axis by setting Parameter SIZE to [<i>s</i>,<i>s</i>,0] where <i>s</i> is the initial scale length in pixels.&nbsp; The zero means do not smooth along the third axis. The scale length should have a value about half the size of the largest region of bad data to be replaced.&nbsp; Since the largest bad regions apart from the cube peripheries are two pixels across, a command like this<br /><blockquote class="tr_bq">fillbad in=holey out=filled&nbsp; size="[1,1,0]"</blockquote>is appropriate.&nbsp; The graphic below shows the same region as before, but with the holes filled in.<br /><br /><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-0TPfNRmsAKk/UlP0DTqx8NI/AAAAAAAAAEM/NE9a-U9PRZ0/s1600/polyfillaharp.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="381" src="http://4.bp.blogspot.com/-0TPfNRmsAKk/UlP0DTqx8NI/AAAAAAAAAEM/NE9a-U9PRZ0/s400/polyfillaharp.png" width="400" /></a></div><br />FILLBAD does what it says on the tin.<br /><br />There will be some smoothing, but it's not obvious.&nbsp; Below is a KAPPA::CLINPLOT graphic for a part of the filled cube.&nbsp; Each spectrum is a pixel in the image above. Can you identify the three spectra which were previously bad?<br /><br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-uhMDWp54Ws0/UlP_Pe7j_HI/AAAAAAAAAEc/5FcokU9dXqs/s1600/filled_clinplot.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="287" src="http://3.bp.blogspot.com/-uhMDWp54Ws0/UlP_Pe7j_HI/AAAAAAAAAEc/5FcokU9dXqs/s400/filled_clinplot.png" width="400" /></a></div><br /><br />http://pipelinesandarchives.blogspot.com/2013/10/holes-in-heterodyne-maps.htmlnoreply@blogger.com (Malcolm Currie)0tag:blogger.com,1999:blog-4994845427978586964.post-4885178130268156032Fri, 19 Jul 2013 00:58:00 +00002013-07-18T14:58:02.925-10:00astronomy softwaredata processingFCFsorac-drSCUBA-2smurftauWVMInclusion of CSO Tau Fits for Determining FCFsDuring the first few months of 2013 the WVM at JCMT had several periods of instability where it was unreliable for determining the value of tau. Because of this, members of the science team collaborated to produce a collection of smooth polynomial fits to the tau values from the 225-GHz tau meter at the CSO for the affected nights, which can be used to perform extinction correction in place of the WVM tau values.<br /><br />In the latest version of starlink (which you can get by rsyncing from /stardev at the JAC) makemap will now first check to see if the date of the observation is one of the (somewhat sporadic) dates when the WVM was unstable. This occurs as long as the <b>ext.tausrc</b> parameter is set to "auto," which it is by default. If the date is one of the affected dates, makemap will look for an available fit to the CSO data in the file specified by the parameter <b>ext.csofit</b>, which is set up by default to refer to the collection of CSO fits produced at the JAC. If makemap cannot find a fit for an observation in the specified path it will print a warning that no fit or WVM data is available and refuse to reduce the observation, though this shouldn't ever happen in normal operation.<br /><br />If you have observations taken between January 19 and May 14 of this year, using the latest version of starlink rsynced from /stardev at the JAC will help ensure that you get the best extinction data available.<br /><br />The graph below shows a comparison between the FCFs derived from WVM data and those derived using the new CSO tau fits over the period of January-May 2013. The blue diamonds represent FCFs from the WVM, and the tan circles are FCFs from the CSO tau fits.<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-f9CuEyR5Lns/UedNOohoFWI/AAAAAAAAC8M/FPMGd1-SXjg/s1600/ComparisonWVM-CSOfits.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="318" src="http://1.bp.blogspot.com/-f9CuEyR5Lns/UedNOohoFWI/AAAAAAAAC8M/FPMGd1-SXjg/s400/ComparisonWVM-CSOfits.png" width="400" /></a></div><br />http://pipelinesandarchives.blogspot.com/2013/07/inclusion-of-cso-tau-fits-for.htmlnoreply@blogger.com (Daniel Berke)0tag:blogger.com,1999:blog-4994845427978586964.post-2233706659024736641Mon, 15 Jul 2013 16:09:00 +00002013-07-15T06:09:47.156-10:00Removing linear striations in maps<div dir="ltr" style="text-align: left;" trbidi="on">You've probably seen several maps that show the faint linear striations visible in the following 850 map.<br /><br /><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-Eo0UtaWTiVI/UeQbCkTUiXI/AAAAAAAAEQk/pW00or5ExYA/s1600/striations1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="317" src="http://4.bp.blogspot.com/-Eo0UtaWTiVI/UeQbCkTUiXI/AAAAAAAAEQk/pW00or5ExYA/s320/striations1.png" width="320" /></a></div><br />Well, in at least some cases, these striations seem to be due to bad bolometers that have a very different common-mode signal to the others. The map-maker has an algorithm that looks for and rejects such bolometers, but by default the rejection criteria are fairly &nbsp;weak, so few samples are rejected. Changing the &nbsp;<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="http://www.starlink.ac.uk/docs/sun258.htx/node157.html">com.corr_abstol</a> </span><span style="font-family: inherit;">value from its default of 0.2 to 0.8 produces the following map:</span><br /><span style="font-family: inherit;"><br /></span><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-EleU_2K6PGM/UeQce_lYxQI/AAAAAAAAEQ0/0g7NCSa7npw/s1600/striations2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="317" src="http://1.bp.blogspot.com/-EleU_2K6PGM/UeQce_lYxQI/AAAAAAAAEQ0/0g7NCSa7npw/s320/striations2.png" width="320" /></a></div><span style="font-family: inherit;"><br /></span><span style="font-family: inherit;">The striations are much less&nbsp;</span>noticeable<span style="font-family: inherit;">&nbsp;(both these images use the same grey scale). The down-side to this is that since more bolometer samples are rejected, there are fewer samples to form the map. For the first map, only 1.6 % of the available samples were rejected due to common-mode mis-match, leaving a total of 65.4 % available for the map. For the second map, 11.1 % were rejected due to common-mode mis-match, leaving 55.9 % available for the map.&nbsp;</span><br /><span style="font-family: inherit;"><br /></span><span style="font-family: inherit;">Going even further, the following map was created with </span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">com.corr_abstol </span><span style="font-family: inherit;">set to&nbsp;0.97 and &nbsp;</span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="http://www.starlink.ac.uk/docs/sun258.htx/node162.html">com.gain_fgood</a></span><span style="font-family: inherit;"> <span style="font-family: inherit;">set to 0.5:</span></span><br /><span style="font-family: inherit;"><span style="font-family: inherit;"><br /></span></span><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-S-wgl9J8dOY/UeQeTWEk6VI/AAAAAAAAERE/QxacHNFQLLk/s1600/striations3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="315" src="http://1.bp.blogspot.com/-S-wgl9J8dOY/UeQeTWEk6VI/AAAAAAAAERE/QxacHNFQLLk/s320/striations3.png" width="320" /></a></div><span style="font-family: inherit;"><span style="font-family: inherit;"><br /></span></span><span style="font-family: inherit;"><span style="font-family: inherit;">Here, 38.5% of samples were rejected due to common-mode mismatch, leaving only 28.6 % available for the map. You can see that the striations have been almost completely removed, but at the expense of higher noise.</span></span></div>http://pipelinesandarchives.blogspot.com/2013/07/removing-linear-striations-in-maps.htmlnoreply@blogger.com (David Berry)0tag:blogger.com,1999:blog-4994845427978586964.post-555735441889284255Tue, 09 Jul 2013 17:07:00 +00002013-07-09T07:07:37.533-10:00Automatic FLT masking<div dir="ltr" style="text-align: left;" trbidi="on">The estimation and subtraction of low frequency noise in each bolometer (the FLT model) very often causes ringing around bright sources. This ringing can be very bad for the first few iterations but often improves if sufficient iterations are performed. Masking the FLT model consists of excluding known bright sources from the estimation of the low frequency noise, and thus reducing the ringing. The ability to do this using the "<i><b>flt.xxx</b></i>" set of configuration parameters has been around for while, but sadly the ability to do <i>automatic</i>&nbsp; &nbsp;FLT masking has not been available - an external mask image needs to be supplied. By comparison, the AST model <i>can</i> be masked automatically using the <b><i>ast.zero_snr</i></b> and <b><i>ast.zero_snrlo</i></b> parameters. The corresponding parameters <b><i>flt.zero_snr</i></b>&nbsp;and&nbsp;<i style="font-weight: bold;">flt.zero_snrlo</i>&nbsp;do exist, but do not really work since the SNR values are only available once a map has been made. By then it is to late to do the FLT masking - the FLT model has already been applied and the ringing is therefore already present in the residuals.<br /><br />But a new parameter called <b><i>ast.skip</i></b>&nbsp;can get round this. This new parameter defaults to zero, but if set to a positive integer it gives the number of initial iterations for which the AST model (i.e. astronomical signal) is to be skipped. For example, if <b style="font-style: italic;">ast.skip</b> is set to 5 then the first five iterations will not include an AST model. The map will still be created at the end of each iteration, but it willnot be used to find the expected astronomical signal, and the residuals will be left unchanged. This means that all these first five iterations will start from essentially the same time series data - the initial cleaned raw data. However, what this means is that we can now create an FLT mask automatically from the SNR values in the map, and refine that mask five times. So when we get to the sixth iteration, we have a usable FLT mask and we have <i>not</i>&nbsp; introduced any ringing into the residuals.<br /><br />It's a bit like doing a separate run of five iterations to determine the FLT mask before starting again to do the main run.<br /><br />In summary, to use automatic FLT masking, add the following to your config:<br /><br /><i><b>ast.skip=5</b></i><br /><i><b>flt.zero_snr=5</b></i><br /><i><b>flt.zero_snrlo=3</b></i><br /><br /><div><br /></div><div>If you use a fixed number of iterations, you may want to increase your <i><b>numiter</b></i> value by the <i><b>ast.skip</b></i> value, so that you are doing the same number of <i>real</i> iterations. As usual, you can play around with the precise values of these parameters.</div><div><br /></div><div>FLT masking speeds up convergence as shown in the following plot of normalised change in the map against iteration number - red uses the new scheme described above, green uses a fixed external FLT mask, and blue is without FLT masking.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-IlZYmFJZ_9I/UdxApmTgj9I/AAAAAAAAEQE/b7wLtD6Rrv8/s1600/norm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="287" src="http://2.bp.blogspot.com/-IlZYmFJZ_9I/UdxApmTgj9I/AAAAAAAAEQE/b7wLtD6Rrv8/s320/norm.png" width="320" /></a></div><div><br /></div><div><br /></div></div>http://pipelinesandarchives.blogspot.com/2013/07/automatic-flt-masking.htmlnoreply@blogger.com (David Berry)0tag:blogger.com,1999:blog-4994845427978586964.post-5975147804676507019Thu, 04 Jul 2013 14:53:00 +00002013-07-04T04:53:22.367-10:00Changes to way makemap handles control-C interupts<div dir="ltr" style="text-align: left;" trbidi="on">There have been some changes to the way that makemap behaves when control-C is pressed (or a SIGINT signal is received by the process by any other means). When the first control-C is detected, makemap will continue to complete the current iteration (but a&nbsp;<i>second</i>&nbsp;interupt will cause makemap to terminate immediately). It will then pause and ask the user what to do next. The options are:<br /><br /><ol style="text-align: left;"><li>abort immediately with an error status</li><li>close the application returning the current output map</li><li>do one more iteration to finialise the&nbsp;map and then close</li></ol><div>Option 3 is important if you are using AST masking, since one final iteration should always be done without masking to ensure&nbsp;correct fluxes in the background regions (see the AST.ZERO_NOTLAST config parameter). If you are not using AST masking, or if you will be re-starting makemap to perform further iterations, then you should select option 2.</div><div><br /></div><div>At some point I will add a fourth option to ignore the interupt and continue iterating to convergence.</div><div><br /></div><div>Note if you are running makemap from within a script, then the handling of control-C interupts will probably be quite different, because the shell (or perl, python or whatever) will catch the interupt before it gets to makemap. You may get some sort of useful behaviour by finding the process ID for the makemap process itself, and sending a SIGINT to that process&nbsp;explicitly:</div><div><br /></div><div>% ps aux | grep makemap</div><div>dsb &nbsp; &nbsp; &nbsp;25407 &nbsp;0.0 &nbsp;0.0 &nbsp;43716 &nbsp;1952 pts/5 &nbsp; &nbsp;S &nbsp; &nbsp;Jul03 &nbsp; 0:00 makemap</div><div><br /></div><div>% kill&nbsp;-s INT 25407</div><div><br /></div><div><br /></div><div><br /></div></div>http://pipelinesandarchives.blogspot.com/2013/07/changes-to-way-makemap-handles-control.htmlnoreply@blogger.com (David Berry)0tag:blogger.com,1999:blog-4994845427978586964.post-8741803612565592441Thu, 04 Jul 2013 14:35:00 +00002013-07-04T04:35:20.830-10:00Monitoring itermaps whilst makemap is running<div dir="ltr" style="text-align: left;" trbidi="on">The main output NDF created by makemap is not closed until the last iteration has been completed. Since itermaps are usually placed in the SMURF extension of the main output NDF, this means that itermaps cannot be examined &nbsp;until makemap has completed. How annoying...<br /><br />But now you can get round this by using the new ITERMAPS parameter (an ADAM parameter, not a config parameter) to specify that the itermaps should be placed somewhere else. If you do this, each itermap NDF will be closed as soon as it has been written, allowing you to examine it immediately.<br /><br />So in one window, you start a big makemap job running (with "itermaps=1" in the config):<br /><br /><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">% makemap in=^infiles out=out config=^conf itermaps=fred</span><br /><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">and then in another window you can examine the itermaps as soon as they are created:</span><br /><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">% display fred.ch00i003</span><br /><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">If you want a list of all the available itermaps, do</span><br /><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">% ndfecho fred</span><br /><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">(this is a kappa command).</span></div>http://pipelinesandarchives.blogspot.com/2013/07/monitoring-itermaps-whilst-makemap-is.htmlnoreply@blogger.com (David Berry)0tag:blogger.com,1999:blog-4994845427978586964.post-7181886893146587689Thu, 04 Jul 2013 14:27:00 +00002013-07-04T04:27:41.367-10:00Re-starting makemap after an interupt or crash<div dir="ltr" style="text-align: left;" trbidi="on">makemap can take a very long time to run, and it is very annoying if a power failure, or some system glitch causes it to crash after several hours. The good news is that it is now possible to re-start it from (more or less) where it left off, so long as you follow a couple of requirements when running makemap:<br /><br /><ol style="text-align: left;"><li>You must include "<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">itermap=1</span>" or "<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">itermap=2</span>" in yourt config.</li><li>You must assign a value to the "ITERMAPS" ADAM parameter when running makemap. This new parameter specifies the file in which the itermaps are to be placed. Previously, there was no choice about where the itermaps went - they went into the SMURF extension of the main output NDF. Now, you can&nbsp;specify&nbsp;an alternative place using ITERMAPS - note, this is an ADAM parameter and so goes on the makemap command line, NOT a config parameter. The benefit of specifying an alternative &nbsp;location for itermaps is that, unlike the main output NDF, the specified file will be closed properly after writing each itermap. This means the itermaps will be usable even if makemap crashes, so long as you are not really unlucky and the crash occurs whilst it is actually writing an itermap. &nbsp;</li></ol><div>If you follow the above requirements when running makemap, you will be able to re-start makemap in the event of a crash by running makemap again in exactly the same way as you did before, but with the following changes:</div><div><ol style="text-align: left;"><li>Add "initialsky=xyz" to your config, replacing "xyz" by one of the ADAM parameter names - "REF", "MASK1" or "MASK2".</li><li>On the command line, set the chosen ADAM parameter to the path of the last itermap created by the previous run of makemap. &nbsp;</li><li>In the config, add "noi.calcfirst=1"</li></ol><div><br /></div><div><br /></div><div>For instance:</div></div><div><br /></div><div>% cat conf</div><div>...</div><div>itermap=1</div><div>...</div><div><br /></div><div><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">% makemap in=^infiles out=out config=^conf itermaps=fred</span></div><div><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">..</span></div><div><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">..</span></div><div><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div><div><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">oops - the computer died... OK, not to worry, I've got 67 itermaps in fred.sdf so I can re-start at iteration 68...</span></div><div><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div><div><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">% makemap in=^infiles out=out config="'^conf,initialsky=ref,noi.calcfirst=1'" \</span></div><div><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ref=fred.ch00i067</span></div><div><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div><div><span style="font-family: inherit;">Note, the re-started makemap will still need to go through the labourious task of cleaning the input time series data again, before it starts iterating.&nbsp;</span></div></div>http://pipelinesandarchives.blogspot.com/2013/07/re-starting-makemap-after-interupt-or.htmlnoreply@blogger.com (David Berry)0tag:blogger.com,1999:blog-4994845427978586964.post-2259098072871275182Wed, 15 May 2013 21:37:00 +00002013-05-15T11:37:14.175-10:00smurfGreat results from Skyloop<div style="text-align: justify;">This is just a quick post to show the clear benefits of using skyloop&nbsp;when reducing your own data&nbsp;(see our earlier blog or read: <a href="http://www.starlink.ac.uk/docs/sc21.htx/node51.html">http://www.starlink.ac.uk/docs/sc21.htx/node51.html</a>). Below is an image composed of several observations reduced individually using the iterative map maker. "Blooms" (bright false emission) are clearly evident at the edges of maps where high noise has made its way into the ast model.</div><br /><div class="separator" style="clear: both; text-align: center;">&nbsp;<a href="http://4.bp.blogspot.com/-bPmK1Iq41D4/UZP9h6u82oI/AAAAAAAAAFY/kLsLPmPHC7g/s1600/ImageMakeMap.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="375" src="http://4.bp.blogspot.com/-bPmK1Iq41D4/UZP9h6u82oI/AAAAAAAAAFY/kLsLPmPHC7g/s400/ImageMakeMap.jpeg" width="400" /></a></div><div class="separator" style="clear: both; text-align: justify;">Reducing the same data using skyloop helps to identify what is noise and what is true emission-particularly in the overlap regions. The benefit in this situation is clearly seen. The cost to the user is in computing power and time but the result is clearly worth it.</div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-4Mcd4pVuU_4/UZP9nnQ25hI/AAAAAAAAAFg/jySEjJizyvw/s1600/ImageSkyloop.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="370" src="http://3.bp.blogspot.com/-4Mcd4pVuU_4/UZP9nnQ25hI/AAAAAAAAAFg/jySEjJizyvw/s400/ImageSkyloop.jpeg" width="400" /></a></div><br />http://pipelinesandarchives.blogspot.com/2013/05/great-results-from-skyloop.htmlnoreply@blogger.com (Harriet Parsons)0tag:blogger.com,1999:blog-4994845427978586964.post-6359585117543722289Fri, 03 May 2013 15:45:00 +00002013-05-03T05:45:43.093-10:00Makemap now gives slightly different numbers<div dir="ltr" style="text-align: left;" trbidi="on">From today, people using a cutting edge starlink rsynced from /stardev at JAC may notice small changes in the maps created by MAKEMAP compared to previous versions.This is caused by a change to the way in which the (RA,Dec) of each bolometer sample is calculated. The change was needed to support FTS-2.<div><br /></div><div>The new scheme is mathematically equivalent to the old scheme, but groups the required transformations a little differently, resulting in <i>different</i> (not <i>worse</i>) numerical rounding errors. This has no&nbsp;effect&nbsp;on the majority of samples, but some samples that are very close to the edge of a pixel will be moved just that little tiny amount needed to push them over into the adjacent pixel. This results in&nbsp;noticeably&nbsp;<i>different</i> (not <i>worse</i>) noise in the final map.<div><br /></div><div>The black curve in the following plot is a histogram of the differences between the two maps produced by the two versions of makemap, for a small Uranus observation (S8A only). The red curve is a histogram of Error component (i.e. sqrt(Variance) ) for the old map. Note the logarithmic Y axis scale.&nbsp;</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-9d-91rUS3N4/UYPZVlEPIjI/AAAAAAAAEMU/cCnBI7g_TkY/s1600/error.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="297" src="http://4.bp.blogspot.com/-9d-91rUS3N4/UYPZVlEPIjI/AAAAAAAAEMU/cCnBI7g_TkY/s400/error.png" width="400" /></a></div><div><br /></div><div>Numerically, the RMS difference between the data values in the two maps is&nbsp;8.57E-5, the square root of the mean Variance value in the old image 53.0E-5, and the&nbsp;&nbsp;square root of the mean Variance value in the new image is 52.9E-5.</div></div><div><br /></div><div>So these changes are much smaller than the noise level, and so should not be a concern.</div></div>http://pipelinesandarchives.blogspot.com/2013/05/makemap-now-gives-slightly-different.htmlnoreply@blogger.com (David Berry)0tag:blogger.com,1999:blog-4994845427978586964.post-7064609346820639703Mon, 22 Apr 2013 11:45:00 +00002013-04-22T01:45:38.814-10:00Displaying images with the same scaling<div dir="ltr" style="text-align: left;" trbidi="on"><span style="font-family: inherit;">If like me you often want to produce a plot containing several images all scaled to the same limits, you may be interested in a new feature in KAPPA:DISPLAY that simplifies this. Setting the MODE parameter to "Current" will now re-use the scaling limits that were used on the previous invocation of display.&nbsp;&nbsp;No need to copy and paste the limits any more...&nbsp;As an example:</span><br /><br /><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">% gdclear</span><br /><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">% picdef mode=array xpic=3 ypic=1 prefix=pic</span><br /><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">% display map1 mode=per percentiles=\[2,98\]</span><br /><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">% picsel pic2</span><br /><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">% display map2 mode=cur</span><br /><br /><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">% picsel pic3</span><br /><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">% display map3 mode=cur</span><br /><div><br /></div><div>will display <span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">map1.sdf </span>on the left with 2% of pixels set to pure black and 2% set to pure white. It will then display&nbsp;<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">map2.sdf</span> in the center and <span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">map3.sdf</span> on the right, using the same scaling that was used with <span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">map1.sdf.</span></div></div>http://pipelinesandarchives.blogspot.com/2013/04/displaying-images-with-same-scaling.htmlnoreply@blogger.com (David Berry)0