An overly long summary page of information for the MD GR0 (grand reprocessing version 0) including the new reference stacks in the V3 tessellation with the start of new MD10 observations in OC142 (August 2011). This is version 0, the first start of full reprocessing of the MD fields and must be balanced along with other reprocessing, 3PI stacks, nightly science, generation of new reference stacks etc. Processing has also extended over a long period of time with many starts and stops for various reasons, thus, has become a very inhomogeneous set with code and labeling improvements as it progressed. A best attempt at documenting this has been done field by field.

The new ubercal reference catalog were available near the end of field reprocessing with MD08 (with PS1.REF.20120325) and MD09 (with a revised REF.20120524), but all others (MD10, 01, 02, 03, 04, 05, 06, 07) were originally processed using the original synthetic catalog. MD01,02,03,05 were not fully reprocessed originally due to various problems, so those fields will have a mix of exposures calibrated with different catalogs. There has continued to be limited reprocessing of MD exposures in order to create new refstacks for the season so there can be exposures with multiple reprocessed versions.

Creation of the new reference stacks need to be, at the very least if possible, made a week prior to the return to the particular MD field given in the current schedule
http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/Medium_Deep_Schedule. After the July, August, September time crunch of a new field a month, there should be time to continue reprocessing and creation of new reference stacks in advance.

12/20/11 MD06 processing recoverable, refstacks proceeding. MD08 a mess and will need to be re-run.

12/31/11 MD06 finally finished, distributed.

1/09/12 MD08, 09 on hold while finishing up full reprocessing and release of V3 products now (i.e., night stacks, SMFs, SSdiffs).

1/11/12 MD06,04 full camera SMFs on datastore, MD07 later today. MD04 set has some duplicates with both unmagicked SMFs (data_group=MD04.GR0, sub-sample of ~150/filter best seeing) and magicked SMFs (data_group=condor_MD04.V3_01_dg, full sample of ~400/filter loaded into PSPS Nov. 2011)

1/11 full V3 night stack reprocessing with MD06 started. -- important note: these are now being run allowing 2 or more inputs to make a stack (down from 4 during the magic era) so outer skycells may have strange behavior that we need feedback on.

1/23 MD04.V3 full reprocessing of night stacks finished and pushed out to datastore.

4/17 MD08.V3 restarted from scratch after raid crash in November/December with the new ipp-20120404 tag that will include new ubercal refcat (PS1.REF.20120325) and detrend improvements (dark modes, noise map, but not boost factors).

4/25 MD08.GR0 reprocessed and distributed, refstacks made and distributed, staticsky run and distributed.

stack photometry -- originally the same date 20120805, but revised runs now have dates for each in the form adding for staticsky .<#>x20YYMMMDD and skycal .cal20YYMMDD.

(U) updated -- new 2012/2013 versions exist for these refstack now (see MediumDeepFields) but not necessarily used for stack photometry (staticsky/skycal) for loading into PSPS (original MD.GR0 may be deeper and better to use). The refstack used is indicated in the Stack Photometry column. All updated refstacks of course use the ubercal catalog for calibration.

out-of-season/anti-yearly refstacks, yearly refstack: on hold for MD.GR1 (would need to update/reprocess large numbers of exposures to do).

iq/psf stack on hold until MD.GR1 (like out-of-season, would required many updates) -- should do for MD04,MD09 and requires development/code time.

deep stack for MD04,09 for MD.GR0 only, all others on hold until MD.GR1 (many many updates would be necessary..).

reminder that ubercal of pixels start with MD08 in the date sequence, finished processing of MD01,02,03,05 exposures into V3 will also be ubercal'd (including reprocessing of any needed to make the complete night stacks)

Questions:

background restored warps/stacks?

SMF loaded into PSPS limited to just the ones used in the refstack (or deepstack if available) or all available exposures Camera/SMFs?

MD01,10 have new refstack for this season, need to decide if using for phot+PSPS (limited set of exposures reprocessed recently to make)? Or use as update test for PSPS?

MD09.refstack.20120603 suffers from the weight image problem making an arc of lower sensitivity, but warps are still available for doing the deep stacks. 4-6 hrs/skycell to run, ~300 skycells, deepstack pantasks typically has 30 nodes, so ~2-3 days processing time, then redo? -- yes, since MD09 is a new test/comparison set with SAS -- done and date updated under refstack as MD09.refstack.20120831

Feedback is important

We would not want to continue full reprocessing of the MD fields if there has been a gross oversight or problem for a particular science interest. Please let us know of any issues you discover with example details. For example

tessellation and/or offsets issues in any of the fields

corner of skycell.055 seems to be regularly masked (and in neighbors with the overlap)

fields with offsets need to include sufficient images in old and new pointings (i.e., MD10), for the magic reference stack at least.

any issues with masking, mask values

been report of possible ones in early MD10 but still need details

depth/quality of the stacks, reprocessed images in the new format

any missing skycells, SMFs, other data products -- in particular MD10 which has been the only MD field full reprocessed and magicked for release of SMFs, warps.

MD10.refstack.20110722, MD10.refstack.20110821 r-band has two edge skycells (041,051) with poor quality and failed

with magic turned off now (01/2012), MD04+MD06 have 2+ warp inputs for night stack creation to test how well boundaries come out and has fault/quality issue with 2-3 input cases showing some inputs have incorrect PSF size values measured/passed. rare (~<1%) but should be looked into (ie stack_id=609561)

adding a quality column to ippMonitor for evaluating missing stack skycells would be useful to have on stack processed skyfile

MD10.GR0: from (351.979 -0.433) to (352.312, -0.433) however an additional offset looks like it should have been applied to cover more of the earlier observation dithers.

MD07.GR0: from (213.146, +53.417) to (213.454, +53.243) to best cover as much of the earlier observation dithers as possible.

MD08.GR0: very slight offset in RA to the W to just align with the MDS field pointing.

MD09.GR0: from (333.688, +0.283) closer to the current pointing of (334.188, +0.283) however an addition offset was added to move the center about half a skycell to the E to (333.980, +0.283) in order to cover as much of the earlier observation dithers as possible.

All fields should be looked at to see if there are any problems with the new tessellation and dither pattern.

MD07, MD09 will have a preliminary night stack made for testing the revised center as well as MD06 for NGC4258.

Reference versus Deep Stack

A requirements description for the needs in the different types of stacks would be useful to summarize.

Magic destreaking: a fairly clean reference stack (uses the convolved stack) with as much spatial area coverage as possible to enable release of non-stack data products. Particularly important is inclusion of older data on different pointings and of course a fairly balanced dither set.

Transient detection: a balance slanted towards best PSF but with some depth (1-1.5 magnitudes over night stacks) and also a fairly balanced dither set. Better with more recent data to limit AGN/variable star variations.

Deep stacks -- for deep static sky science.

As deep as possible. A maximum FWHM allowance? A minimum exposure time?

A specific request for at least ~10ks has been made, with best FWHM possible.

A fairly balanced dither set is almost by default with a large number of inputs. Issue really comes up in fields with shifted positions from older observations.

See summary for each MD field reference stack for details on specifics for that field. Due to processing+time constraints/early mistakes, an attempt has been made to merge the similar criteria under the refstack label -- full field coverage, best FWHM, 1-1.5 mag and >10ks depth over central FPA as possible

based on the steep edge on the FWHM distribution of input warps, a large number of inputs could be used without degrading the final FWHM by much (~0.2 pixels, ~0.05") even upwards of 150 input warps

of order 100 inputs found necessary to keep a minimal depth per pixel in many central regions with the dither and masking (see MD04 deep stack tests).

Additional considerations:

since magic destreaking uses the convolved stack, a quickstack version of the unconvolved stacks for the transients reference and deep stack could be faster to make with a larger number of inputs but requires additional development for regular processing.

fields with shifted centers with offsets to include early exposures, the inclusion of enough exposures to provide a reference stack for magic is necessary in order for data from those associated chips/skycells to be distributed (being older data, may not have the optimum quality). Mixing older and newer data may also have other problems.

for cases when a stack does not have a good sample of exposures with a reasonably small PSF FWHM, it should be possible to generate a reference stack with the fewer 'best' exposures for just SSdiffs and transient detection. MD02-g may be such a case, but could leave large regions of order a night stack number of inputs.

how much of an IQ variation can go into a stack before ppSub has difficulty modeling the variations within a skycell?

Deep stacks on hold until a solution is found to the overly large PSF made in the convolved stacks that is dependent on the number of input warps -- fixed, but other issues found and development necessary before full deepstack set is run on all fields.

post-Magic January 2012

All pixels can be released, refstacks for diffims only need to focus on detections for science.

some (de?)processing had been done to revert the magicked SMF catalogs back to their full form, status and completeness unknown.

without need for magic for releasing older data, can focus on more recent (and often better) data going into the refstacks.

Reprocessed Data Products

Each MD field has a variety of issues and details are summarized in each MD section. Label/data_group names have also evolved and the appropriate MD section should be referred to.

Chip+Warp

Currently processed but not distributed due to disk space load. Can be obtained via the Postage Stamp Server.

can be a mix of synthetic/ubercal catalog calibrations except MD08, 09. Generally if the chip_id, warp_id >= 433987, 393977 respectively it was processed using the 20120524v0 ubercal catalog.

Camera/SMFs

Fully processed and distributed.

can be a mix of synthetic/ubercal catalog calibrations except MD08, 09

a possible guideline to follow, including nightly science data processing, if MD exposures with a cam_id >=412069 (actually >=411444 of some V2 destined reprocessing) is used, those will be with the ipp-20120404,r33642 tag that includes the faint-end bias fix and the ubercal catalog 20120524v0 version also used for LAP.

Stacks

Generally, if a stack_id is >= 848641 then it was likely made from processed exposures using the 20120524v0 ubercal catalog (see night stacks where some were made using old warps if a complete set existed for MD01,02,03,05).

Night Stacks: All nights reprocessed and stacked into a night stack using the current night stack configuration used for nightly science night stacks when processed, including reference catalog

should attempt to make stack with 2 or more input warps now

some nights have >8 exposures, often in marginal conditions. Since marginal conditions can vary greatly, some selection should probably be made for inclusion in the night stack.

the sample of stacks can be a mix of synthetic/ubercal catalog calibrations for the pixels but the warps going into the stack should be of the same type of calibration.

Reference Stacks (refstack): To be made prior to the start of the MD field observations for the season to support the difference image products (magic and science). The focus of MD.GR0 was on the 2011/2012 season fields starting with MD10, then 01, 02 ... through MD09. The 2012/2013 season refstacks will to replace the 2011/2012 set and have their own page linked from MediumDeepFields.

out-of-season/anti-yearly refstacks, yearly refstack: on hold for MD.GR1 (would need to update/reprocess large numbers of exposures to do)

Deep Stacks (deeptest): Not a full production product yet. Test version exists for MD02.iqtest.20120927, MD10.psfrefstack.20110814.

IQ/PSF Stack: on hold until MD.GR1 (like out-of-season, would required many updates). A sample for MD09 done but requires development/code time.

Difference Images

Originally a possible reprocessing product, but was decided not useful at this time (for future reprocessing, need to outline interests for these if any). Will need to finish the convolution choice (input stack or reference stack) before started.

Stack Photometry (Static Sky + Skycal)

Stack photometry (staticsky) and recalibration (skycal) is run on refstacks, deepstacks and night stacks. In trying to run as a uniform group, will keep a summary here, but details like specific faults, poor quality etc will be in the respective MD section when time. For example some skycells can have a clear offset fix to the zeropoint, not actually set to 25.0 in stack pixels, for some the majority of the field is off from zeropoint of 25. There appears to be upwards of 3-5% error likely in the photometry across the FPA (see MD.GR0#MD09.GR0) even in the recalibration of the stack photometry (skycal stage). A much worse example is seen in MD01.refstack.20120803.

recalibration of the stack photometry catalogs has no method for distribution -- email for access

Re-revised planned loading into PSPS (March 2013): do the newer refstacks, so the older MD03--MD07 noted before below been have done as follows

Current run planned for loading into PSPS (January 2013): stack photometry (staticsky) re-run from Summer 2012 set to incorporate improvements/fixes in the Kron code. Photometry zeropoint recalibration (skycal) re-run to incorporate various bug fixes and improved tuning (psastro running on stack catalogs). Several previous re-runs for skycal and settled on .cal20130128 version for now.

ref stack label, data_groups: includes 5-1 filter sets now under same label, staticksky data_group will indicate <N> for # filters attempted, skycal data_group will include all under one data_group

deep stack label, data_groups: includes 5-1 filter sets now under same label, staticsky data_group will indicate N-filter, skycal data_group all under one name

label staticsky: MD<##>.deeptest.20120727.Nx20130114 -- <##> = 04, 09 only made so far
data_group staticsky: MD<##>.deeptest.20120727.<N>x20130114 -- <N> = 1,2,3,4,5 and number of filters available for the set, so 5 different data_group per MD
label=data_group skycal: MD09.deeptest.20120727.Nx20130114.cal20130128 -- all N filter set is run under one data_group now per MD
label=data_group skycal: MD04.deeptest.20120727.Nx20130114.cal20130128 -- all N filter set is run under one data_group now per MD

night stack label, data_group: only skycal needs to be rerun on problem skycells

New .cal20130128 has some bug fixes in the rerun has made some clear improvements in stack photometry uniformity (MD10.refstack.20120804, z-band, difference with matched refcat magnitudes with scaling changed to account for the mag ranges, 1/28/13) :

Clean-up Cycle

IPP datadisks -- can be remade/distributed via Postage Stamp Server if cleaned, faster if not cleaned

stage

time-frame

chip

once all warps finished

camera

not cleaned

warps

once all stack finished

stacks (ref,deep,night)

not cleaned

diffims (WS, SS)

not made

stack photometry (staticsky)

not cleaned

MD04, MD09 warps available still to make different stacks for development

Datastore -- standard distribution method of large sets of data bundles, not an archive. Specified ones can be remade/distributed via Postage Stamp Server.

stage

time-frame

chips

not distributed

camera

pixel/masks not distributed, catalogs/SMFs in ps1-md-GR-cat not cleaned

warps

not distributed

stacks (ref,deep,night)

ref not cleaned, deep not cleaned, night similar to nightly processing ~30 days for pixels

diffim (WS,SS)

not made/distributed

stack photometry (staticsky)

not cleaned for now

discovered that no clean-up of Datastore distribution dataproducts being cleaned up (12/2012) -- starting 3/2012 may need to start cleaning up pixel bundles from the GR datastore >1 year old

Stack Result Summary Posters at PS1SC 2012 Durham Meeting

Posters include summary map plots of psf_major(minor), 5-sig photometric depth and magnitude @50% recovery of 500 added fakes as well as full field, i-band images of the MD refstacks (reorg on machine serving these changed, not available at this time..)

MD10.GR0

Status:

reference stack (revised) processed and distributed. All exposures since April 2009 reprocessed and distributed except for SSdiffs. SSdiffs being held back until ppSub auto convolution choice is tested on.

8/2012: to be replaced with MD10.refstack.20120705, then replaced with MD10.refstack.20120804. See new refstack wiki for old/new refstack comparison maps.

Started for OC142 (July 15, 2011). Start with current statuswith a new reference stack and full sample of reprocessed exposures as an initial reprocessing version.

stack-stack diffims still using forced convolution of reference stack to input images

MD10.V3 center shifted to the current observation pointing even though there are earlier data that has offset dithered observations that populate the 0X8-0X9 skycells. Not as bad as MD07, but the revised reference stack may not provide for the outer skycells very well. Afterwards it was discovered the full extent of the previous observations wasn't quite accounted for, shown by these exposure plots with the MD10.V3 overlay and compared to a recent night stack set of exposures. This may need to be shifted in the next iteration.

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter

N total

N (FWHM<upper limit GPC1 pixels)

y

136

all

z

246

136 (<5.0 pixels), then finished reprocessing all

i

256

149 (<7.0 pixels), then finished reprocessing all

r

253

159 (<7.0 pixels), then finished reprocessing all

g

242

118 (<7.0 pixels), then finished reprocessing all

also initially attempted to include some rough cuts by zeropoint and background (in 20110722) but after reprocessing all exposures wouldn't be included in the stacktool selection based solely on basically FWHM major. (improvements to selection and rejection for a future test)

MD10.V3 reference stack

label: MD01.refstack.20110820

data_group: MD01.refstack.20110820

Initial goal was to maximize depth while minimizing the PSF FWHM with enough inputs (4 or more) to produce a stack for all skycells covered, in particular

Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20110722 (top blue), psfrefstack.20110814 (middle blue) and refstack.20100721 V2 (bottom blue). The unmatched blue/red overlap for refstack.20110722 FWHM major<6.5 pixels for exposures in the refstack is from using also low zeropoint and bright background cuts. Psfrefstack used <5.1 pixels.

The same refstacks with the selected exposures over time

Sample skycell coverage map by good_frac with N input warps, N rejected warps, dtime_script for making stack, and if quality error that rejects entire skycell. The ~130 input warps provided enough of a sample for the outer edge skycells from earlier exposures.

In tracking down an odd skycell (048, y-band), an odd rejection case was highlighted and prompted a modification to the rejection criteria based on the RMS of the stamps (see r32170). This may help reduce large number of rejections in future refstacks but likely needs some tuning (currently still rejecting too many?). Using the PSF FWHM cuts in the DEEP_STACK reduction also helps reject strange input warps as well.

Initially the stack processing was run without a reduction type and thus with just defaults that compresses the images (version 20110722gri, 20110715y, 20110718z) along with a test sample to produce the best PSF focused set (MD10.psfrefstack.20110814) and a test best deep stack (just skycell.055, i-band, not distributed). This is not optimal and a re-run was done with the DEEP_STACK reduction type (version 20110821) and should be the one used. The different reduction types include:

NULL (none/default values used from recipes/ppStack.config unless defined in gpc1/ppStack.config):

PSF.INPUT.MAX F32 NAN

PSF.INPUT.CLIP.NSIGMA F32 NAN

OUTPUT.NOCOMP BOOL FALSE

OUTPUT.LOGFLUX BOOL FALSE

OUTPUT.REPLICATE BOOL TRUE

DEEP_STACK:

OUTPUT.NOCOMP BOOL TRUE

OUTPUT.LOGFLUX BOOL FALSE

PSF.INPUT.MAX F32 7.5 (should add to PS1_REFERENCE as the reduction to use for refstacks)

PSF.INPUT.CLIP.NSIGMA F32 5.0 (should add to PS1_REFERENCE as the reduction to use for refstacks)

PS1_REFERENCE:

OUTPUT.NOCOMP BOOL TRUE

OUTPUT.LOGFLUX BOOL FALSE

OUTPUT.REPLICATE BOOL TRUE

NIGHTLY_STACK:

OUTPUT.NOCOMP BOOL FALSE

OUTPUT.LOGFLUX BOOL FALSE

THREEPI_STACK: (for comparison)

PSF.INPUT.MAX F32 12.0

OUTPUT.NOCOMP BOOL FALSE

OUTPUT.LOGFLUX BOOL TRUE

QUICKSTACK_DEEP METADATA (testing unconvolved stacks version only, similar to QUICKSTACK)

A summary table of the reprocessed reference stacks and the former V2 where center skycell for V3=skycell.055, V2=skycell.078.
(need to add FWHM upper limit used)

version label

filter

center skycell input warps (rejected)

center skycell FWHM (major)

MD10.refstack.20110821 V3

y

63(11)

0.96"

MD10.refstack.20110715 V3 error

y

93(6)

0.99"

MD10.psfrefstack.20110814 V3 error

y

41(3)

0.9"

MD10.refstack.20100721 V2

y

28(1)

1.06"

MD10.refstack.20110821 V3

z

72(0)

1.02"

MD10.refstack.20110718 V3 error

z

174(6)

1.09"

MD10.psfrefstack.20110814 V3 error

z

44(4)

0.98"

MD10.refstack.20100721 V2

z

23(0)

1.09"

MD10.refstack.20110821 V3

i

75(1)

0.99"

MD10.refstack.20110722 V3 error

i

132(6)

1.07"

MD10.psfrefstack.20110814 V3 error

i

46(2)

1.00"

MD10.deeprefstack.20110814 V3 error

i

217(11)

1.08"

MD10.refstack.20100721 V2

i

18(1)

1.32"

MD10.refstack.20110821 V3

r

76(0)

1.28"

MD10.refstack.20110722 V3 error

r

134(2)

1.33"

MD10.psfrefstack.20110814 V3 error

r

47(3)

1.28"

MD10.refstack.20100721 V2

r

19(1)

1.46"

MD10.refstack.20110821 V3

g

79(3)

1.4"

MD10.refstack.20110722 V3 error

g

123(0)

1.41"

MD10.psfrefstack.20110814 V3 error

g

45(0)

1.31"

MD10.refstack.20100721 V2

g

22(0)

1.57"

Using the MD10.refstack.20110722 and related set, histograms and mag-magerr plots for the reference stack, deeprefstack test, psfrefstack, typical night stack and V2 refstack are compared. The re-run MD10.refstack.20110821 will go a little deeper than the psfrefstack version.

A representative skycell mapping of the new tessellation and good_fraction coverage. Night stacks will have similar, if not more, trouble with covering the corner edge skycells: 012, 017, 021, 028, 071, 078, 082, 087.

Other plots that may be good to see

Mask fractions vs V2

mapping of all exposures WRT new tessellation to check maximum coverage given some fields offset and others have had extra dither patterns (i.e., like MD10)

MD01.GR0

Started for OC143 (August 13, 2011) with a new reference stack and small sample of reprocessed exposures released as night stacks. Due to a need of rebalancing the reprocessing load, the reprocessed sample includes only those exposures that may be used in the reference stack.

MD03 does have some offset dither exposures illustrated in the sample g-band V3 plot.

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter

N total

N (FWHM<upper limit GPC1 pixels)'"

y

178

106 (<5.0 pixels)

z

377

214 (<5.0 pixels)

i

351

172 (<5.0 pixels)

r

330

120 (<5.0 pixels)

g

340

124 (<5.5 pixels)

MD01.V3 reference stack

label: MD01.refstack.20110820

data_group: MD01.refstack.20110820

Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20110820 (top blue) and refstack.2010812 V2 (bottom blue). The unmatched blue/red overlap for V2 are due to ones included/rejected in the selected reprocessing.

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, V2=078) after running psphot on the stacks.

version label

filter

center skycell input warps (rejected)

center skycell FWHM (major)

MD01.refstack.20110820 V3

y

52(1)

0.84"

MD01.refstack.20110813 V3 error

y

106(1)

MD01.refstack.20100812 V2

y

21(3)

1.04"

MD01.refstack.20110820 V3

z

58(3)

0.87"

MD01.refstack.20110813 V3 error

z

115(2)

MD01.refstack.20100812 V2

z

60(2)

1.12"

MD01.refstack.20110820 V3

i

74(3)

1.00"

MD01.refstack.20100812 V2

i

50(0)

1.13"

MD01.refstack.20110820 V3

r

54(7)

1.10"

MD01.refstack.20100812 V2

r

56(1)

1.24"

MD01.refstack.20110820 V3

g

65(7)

1.20"

MD01.refstack.20100812 V2

g

54(5)

1.24"

with the approximate FWHM (major) as the PSF FWHM from a center field skycell after running psphot on the stacks. The MD01.refstack.20110813 V3 error was a run without the DEEP_STACK reduction

MD02.GR0

Started for OC144 (September 12, 2011) with a new reference stack and a small sample of reprocessed exposures to release night stacks as well. Due to a need of rebalancing the reprocessing load, only a minimal amount of processing required to make the reference stack was done.

MD02 does have some offset dither exposures illustrated in the sample g-band V3 plot, but many are single exposure coverage and won't typically be enough for a reference stack skycell. While warps will extend to some of the outer skycells, the night stack distribution will define what is likely for the skycells released due to magic. 012-017, 21-28, 31-38, 41-48, 51-58, 61-68, 71-78, 82-87

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter

N total

N (FWHM<upper limit GPC1 pixels)

y

147

85 (<6.0 pixels)

z

305

101 (<4.5 pixels)

i

266

117 (<5.5 pixels)

r

215

118 (<6.5 pixels)

g

218

121 (<7.0 pixels)

MD02.V3 reference stack

label: MD02.refstack.20110907

data_group: MD02.refstack.20110907

Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20110907 (top blue) and refstack.20101126 V2 (bottom blue). The unmatched blue/red overlap for V2 are due to ones included/rejected in the selected reprocessing.

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, V2=078) after running psphot on the stacks.

version label

filter

center skycell input warps(rejected)

center skycell FWHM (major)

MD02.refstack.20110907 V3

y

58(3)

1.01"

MD02.refstack.20101126 V2

y

47(3)

1.02"

MD02.refstack.20110907 V3

z

67(5)

0.90"

MD02.refstack.20101126 V2

z

84(10)

1.03"

MD02.refstack.20110907 V3

i

61(2)

1.00"

MD02.refstack.20101126 V2

i

102(15)

1.16"

MD02.refstack.20110907 V3

r

63(6)

1.15"

MD02.refstack.20101126 V2

r

67(15)

1.10"

MD02.refstack.20110907 V3

g

64(6)

1.32"

MD02.refstack.20101126 V2

g

71(24)

1.26"

with the approximate FWHM (major) as the PSF FWHM from a center field skycell after running psphot on the stacks.

Troubled skycells: none on initial inspection. 040 in g,r oddly seems to have enough coverage to make a stack.

Distributed under ps1-md-refstack -- stack images only.

Data volume on disk:

Processing time:

MD02.GR0 reprocessed sample

label: MD02.GR0

data_group: MD02.GR0.20110902 = y

data_group: MD02.GR0.20110903 = z

data_group: MD02.GR0.20110906 = g

data_group: MD02.GR0.20110905 = r

data_group: MD02.GR0.20110904 = i

dist_group: MD02.GR

Distributed under ps1-md-GR -- sample on night stack images only. SSdiff sample still to be run.

A typo in the loading script set the workdir to be MD01 rather than MD02 for all filter data groups (see string below) but will not cause any trouble other than the directory name in mirroring data

/gpc1/MD02.GR0/MD01.2011090*/MD02.V3/skycell.*/*unconv.fits

Data volume on disk:

Processing time:

chip-warp:

refstack: 8-9000s/skycell for the ~60 input warps, on averages looks longer than the previous refstack MD02.refstack.20101126 (to be investigated)

night stack: order 800-1200s/skycell

MD02.GR0 reprocessed remaining set

Chip->warp including camera SMF for all filters grizy:

label: MD02.GR0.20121210

data_group: MD02.GR0.20121210

Night stack for all filters grizy::

label: MD02.GR0

data_group: MD02.GR0.20121210.nightstack.<obsdate>

data_group if warps processed earlier but stack made with new tag: MD02.GR0.201212xx.nightstack.<obsdate>

MD03.V3 refstack needed before new observations in OC146 (November 10).

MD03 does have some offset dither exposures illustrated in the sample g-band V3 plot, but many of the outer skycells are only covered by single to a few exposure without the 4 or more for a stack.

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter

N total

N (FWHM<upper limit GPC1 pixels)'"

y

224

97 (<5.0 pixels)

z

526

150 (<4.2 pixels)

i

488

163 (<4.8 pixels)

r

377

160 (<4.9 pixels)

g

388

147 (5.5 pixels)

the number of exposures being reprocessed have been increased from that in MD01,02 in order to at least attempt to match the input warp numbers in the previous MD03.refstack (~100).

MD03.V3 reference stack

label: MD03.refstack.20111010

data_group: MD03.refstack.20111010

Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20111010 (top blue) and refstack.20101202 V2 (bottom blue). The unmatched blue/red overlap for V2 are due to ones included/rejected in the selected reprocessing.

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, V2=078) after running psphot on the stacks.

version label

filter

center skycell input warps(rejected)

center skycell FWHM (major)

MD03.refstack.20111010 V3

y

108(7)

0.91"

- MD03.reftest.20111011 V3

y

26(2)

0.83"

- MD03.reftest.20111011 V3

y

57(2)

0.87"

old MD03.refstack.20101202 V2

y

75(12)

1.13"

MD03.refstack.20111010 V3

z

125(18)

0.85"

- MD03.reftest.20111011 V3

z

19(1)

0.84"

- MD03.reftest.20111011 V3

z

60(2)

0.84"

old MD03.refstack.20101202 V2

z

101(27)

1.04"

MD03.refstack.20111010 V3

i

116(7)

0.97"

- MD03.reftest.20111011 V3

i

24(0)

0.98"

- MD03.reftest.20111011 V3

i

67(5)

0.97"

old MD03.refstack.20101202 V2

i

108(17)

1.12"

MD03.refstack.20111010 V3

r

123(4)

1.05"

- MD03.reftest.20111011 V3

r

24(1)

1.02"

- MD03.reftest.20111011 V3

r

64(3)

0.97"

old MD03.refstack.20101202 V2

r

115(44)

1.10"

MD03.refstack.20111010 V3

g

114(5)

~1.08" (looked at 054,053 as well)

- MD03.reftest.20111011 V3

g

17(2)

1.03"

- MD03.reftest.20111011 V3

g

67(2)

1.08"

old MD03.refstack.20101202 V2

g

90(29)

1.13"

overshot the number of input warps compared to V2. The average PSF FWHM_major seems much better than expected from the histograms (above) of input exposures.

Troubled skycells: none on initial inspection. Far outer edges with few dithers may/may not have successful skycells. Sample skycell coverage maps for V2 and V3 with good_frac with N input warps, N rejected warps, dtime_script for making stack, and flag of any quality error that rejects an entire skycell (ERR). The ~110-125 input warps provided a sample for some of the far outer skycells from earlier dithers to be covered still.

Distributed under ps1-md-refstack

Data volume on disk: ~330GB

Processing time summary: 12-18000 sec for 100-125 input warps. ~6 days to reprocess a ~50% of exposures through warp, create reference stack, and distribute (along with LAP and normal nightly science).

MD03.GR0 reprocessed sample

label: MD03.GR0

data_group: MD03.GR0.20111001 = y

data_group: MD03.GR0.20111002 = z

data_group: MD03.GR0.20111003 = i

data_group: MD03.GR0.20111004 = r

data_group: MD03.GR0.20111005 = g

dist_group: MD03.GR

Distributed under ps1-md-GR -- sample of night stack images only. SSdiff sample still to be run. Magicked camera SMFs to be done.

Data volume on disk:

Processing times:

MD03.GR0 reprocessed remaining set

Chip->warp including camera SMF for all filters grizy:

label: MD03.GR0.20121212

data_group: MD03.GR0.20121212

Night stack for all filters grizy::

label: MD03.GR0

data_group: MD03.GR0.20121212.nightstack.<obsdate>

data_group if warps processed earlier but stack made with new tag: MD02.GR0.201212xx.nightstack.<obsdate>

Stack photometry (staticsky, skycal):

odd fault to be fixed

sky_id=330056
MD03.refstack.20111010.Nx20130115
skycell.028

MD04.GR0

Status: Refstacks and sample night stacks finished and distributed. Very deep stack tests in progress. Full reprocessing of all possible night stacks finished and to datastore.

10/2 Starting with i-band rather than y for the sub-sample reprocessing exposures through warp.

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter

N total

N (FWHM<8 pixels)

N (FWHM<upper limit GPC1 pixels)'"

y

209

177

146 (<5.5 pixels)

z

343

321

196 (<5.2 pixels)

i

428

392

208 (<5.0 pixels)

r

357

327

203 (<5.6 pixels)

g

364

313

206 (<6 pixels)

the number of exposures being reprocessed have been increased from that in MD01,02 in order to at least attempt to match the input warp numbers in the previous MD03.refstack (~100). This field has also been the focus of deep stack tests, so the initial reprocessed set will attempt to be double the number from past versions (~200).

MD04.V3 reference stack

label: MD04.refstack.20111012 (~100)

data_group: MD04.refstack.20111012 (~100)

Histograms of the FWHM major and minor from camera stage for reprocessed grizy exposures (red) and refstack.20111012 (top blue) and refstack.20101102 V2 (bottom blue). The unmatched blue/red overlap for V2 are due to ones included/rejected in the selected reprocessing.

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, V2=078) after running psphot on the stacks.

version label

filter

center skycell input warps

center skycell FWHM (major)

MD04.refstack.20111012 V3

y

107(0)

0.90"

- MD04.reftest.20111011 V3

y

20(0)

0.88"?

- MD04.reftest.20111011 V3

y

58(5)

0.87"

old MD04.refstack.20101102 V2

y

100(14)

1.13"

MD04.refstack.20111012 V3

z

109(2)

0.90"

- MD04.reftest.20111011 V3

z

21(1)

0.92"?

- MD04.reftest.20111011 V3

z

52(1)

0.8"

old MD04.refstack.20101102 V2

z

105(8)

1.14"

MD04.refstack.20111012 V3

i

92(11)

0.88"

- MD04.reftest.20111011 V3

i

26(2)

0.93"?

- MD04.reftest.20111011 V3

i

55(5)

0.98"

- MD04.refstack.20111011 V3 (deep stack test, bad label)

i

206(16)

1.08"

- MD04.deeptest.20111012 V3

i

308(17)

1.16"

old MD04.refstack.20101102 V2

i

108(15)

1.22"

MD04.refstack.20111012 V3

r

128(9)

1.12"

- MD04.reftest.20111011 V3

r

26(0)

1.13"?

- MD04.reftest.20111011 V3

r

57(7)

1.06"

old MD04.refstack.20101102 V2

r

92(8)

1.24"

MD04.refstack.20111012 V3

g

113(8)

1.21"

- MD04.reftest.20111011 V3

g

22(0)

1.29"?

- MD04.reftest.20111011 V3

g

60(0)

1.34"?

old MD04.refstack.20101102 V2

g

101(13)

1.21"

was an minor error in the cutoff for i,r: a fewer number in i-band and few more in r-band compared to others. Should be okay for basic refstack, and planned deeper stack should offset any deepstack concerns.

Troubled skycells: none on initial inspection. Far outer edges with few dithers may/may not have successful skycells. Sample skycell coverage maps for V2 and V3 with good_frac with N input warps, N rejected warps, dtime_script for making stack, and flag of any quality error that rejects an entire skycell (ERR). The ~110 input warps provided a sample for some of the far outer skycells from earlier dithers to be covered still.

MD04.V3 deep stack

label/data_group:

MD04.refstack.20111011 (~200) for skycell.055, i-band

MD04.deeptest.20111012 (~300) for skycell.055, i-band

MD04.deeptest.20111015 (~300) full FPA, i-band

MD04.deeptest.20111017 (~300) full FPA, g-band

Histograms of the FWHM major and minor from camera stage for reprocessed exposures (red) and deeptest.20111012 (top blue, ~300 inputs) and refstack.20101102 V2 (bottom blue, ~100 inputs). Any unmatched blue/red overlap for V2 are due to ones included/rejected in the selected reprocessing.

platescale for the camera exposure fwhm_major is ~0.258 "/pixel and processing is limited to FWHM<12 pixels. For skycells is now 0.25 "/pixel and for ref/deep stack is limited to 7.5 pixels currently. May want to raise to 8-10.

ps1-md-GR: sample of night stack images only. SSdiff sample still to be run.

ps1-md-GR-cat: unmagicked camera SMFs with data_group MD04.GR0 released (dupe to the full magicked SMF release from condor processing that was magicked for PSPS loading Nov. 2011)

Heather/Serge successful reprocessing of chip-warp under condor: (more to be run with similar label and condor)

label: condor_MD04.V3_01

data_group: condor_MD04.V3_01_dg

dist_group: MD04.GR

Remaining nightly stack data_group: data_group of remaining sample of nightly stacks processed with ipp-20111222 tag and now allowing 2 or more input warps necessary to make stack (down from 4 during the magic era and typically the far outer skycells, but as a new case, will need to be looked at in more detail). Also added nightlystack.<obsdate> to make a group by night the observation made to help in tracking/monitoring (there will be a mix of data_groups with above MD04.GR0.201110xx earlier sample version)

data_group: MD04.GR0.20120117.nightlystack.<obsdate> = y

data_group: MD04.GR0.20120118.nightlystack.<obsdate> = z

data_group: MD04.GR0.20120119.nightlystack.<obsdate> = i

data_group: MD04.GR0.20120120.nightlystack.<obsdate> = r

data_group: MD04.GR0.20120121.nightlystack.<obsdate> = g

workdir: will now be under a similar directory gpc1/MD04.GR0/MD04.GR0.201201xx.nightlystack.<obsdate>

Distributed on datastore under

ps1-md-GR: remaining nightly stacks to be processed from the condor_MD04.V3_01_dg data_group. SSdiff sample still to be run.

Staticsky run for ingestion into PSPS. Eric notes the RA_PSF, DEC_PSF are NULL in the CMFs. When loaded into DVO coordinates are set and will be okay in PSPS. -- should be fixed in later re-run of stack photometry (Aug. 2012)

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter

N total

N (FWHM<8 pixels)

N (FWHM<upper limit GPC1 pixels)'"

y

338

224

174 (<5.0 pixels)

z

383

325

164 (<4.5 pixels)

i

406

343

185 (<5.0 pixels)

r

416

386

168 (<5.2 pixels)

g

426

379

163 (<5.7 pixels)

MD05.V3 reference stack

label: MD05.refstack.20111014 -- try to get out by 10/14

data_group: MD05.refstack.20111014

Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20111014 (top blue) and refstack.20110121 V2 (bottom blue). The unmatched blue/red overlap for V2 are due to ones included/rejected in the selected reprocessing.

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, V2=078) after running psphot on the stacks.

version label

filter

center skycell input warps

center skycell FWHM (major)

MD05.refstack.20111014 V3

y

133(4)

0.90"

- MD05.reftest.20111011 V3

y

22(0)

0.86"?

- MD05.reftest.20111011 V3

y

69(3)

0.85"

old MD05.refstack.20110121 V2

y

100(25)

1.16"

MD05.refstack.20111014 V3

z

132(4)

0.84"

- MD05.reftest.20111011 V3

z

25(0)

0.88"?

- MD05.reftest.20111011 V3

z

70(9)

0.83"

old MD05.refstack.20110121 V2

z

100(28)

1.01"

MD05.refstack.20111014 V3

i

147(17)

1.03"?

- MD05.reftest.20111011 V3

i

19(0)

0.99"?

- MD05.reftest.20111011 V3

i

63(8)

1.11"?

old MD05.refstack.20110121 V2

i

100(14)

1.24"

MD05.refstack.20111014 V3

r

156(15)

1.15"

- MD05.reftest.20111011 V3

r

23(1)

1.23"?

- MD05.reftest.20111011 V3

r

61(1)

1.12"

old MD05.refstack.20110121 V2

r

95(37)

1.31"

MD05.refstack.20111014 V3

g

154(8)

1.24"

- MD05.reftest.20111011 V3

g

21(5)

?1.28"?

- MD05.reftest.20111011 V3

g

70(5)

1.21"

old MD05.refstack.20110121 V2

g

95(10)

1.38"

*

Error: Macro Image(MD05.201111xx_g_test.png,300px) failed

Attachment 'wiki:MD.GR0: MD05.201111xx_g_test.png' does not exist.

Error: Macro Image(MD05.201111xx_i_test.png,300px) failed

Attachment 'wiki:MD.GR0: MD05.201111xx_i_test.png' does not exist.

Troubled skycells?

MD05.GR0 reprocessed set

Initial sample set of chip-warp and sub-sample night stack reprocessing:

Need to investigate tessellation center and dither for science with NGC4258

Exposures available and with preliminary FWHM cut of ones <2". All exposures were reprocessed for this field to be available in creating the refstacks and deepstacks.

Filter

N total

N (FWHM<8 pixels)

y

298

268

z

471

446

i

564

513

r

436

391

g

407

341

MD06.V3 reference stack

label: MD06.refstack.20111122

data_group: MD06.refstack.20111122

Test stack (reftest) sample also made again for skycell.055 with different number of input warps.

label: MD06.reftest.20111121

data_group: MD06.reftest.20111121

Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20111122 (top blue) and refstack.20110323 (bottom blue). Any unmatched blue/red overlap for (V0) are due to ones included/rejected in the selected reprocessing. The previous (V0) refstack was only 8 inputs so should be much improvement!

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, (V0)=078) after running psphot on the stacks.

skycell.087 i-band had quality failure, rerun excluding some earlier warps and okay. Needs more investigation into why it had a problem, others with quality issue may be similar.

MD06.GR0 reprocessed sample and finished set

label: condor_MD06.V3_01

data_group: condor_MD06.V3_01_dg for chip->warp

dist_group: NULL, changed camera to MD06.GR

Reprocessed night stack sample under the following label and data_group.

label: MD06.GR0

dist_group: MD06.GR

data_group of initial sample of night stacks processed with ipp-20111110 tag and using the default 4 or more input warps necessary to make stack

data_group: MD06.GR0.20111122 = y

data_group: MD06.GR0.20111123 = z

data_group: MD06.GR0.20111124 = i

data_group: MD06.GR0.20111125 = r

data_group: MD06.GR0.20111126 = g

data_group of remaining sample of night stacks processed with ipp-20111222 tag and allowing 2 or more input warps necessary to make stack (typically the far outer skycells, but as a new case, will need to be looked at in more detail)

data_group: MD06.GR0.20120109 = y

data_group: MD06.GR0.20120110 = z

data_group: MD06.GR0.20120112 = i

data_group: MD06.GR0.20120114 = r

data_group: MD06.GR0.20120116 = g

Distributed on datastore under

ps1-md-GR: sample of night stack images only (setting up for full set). SSdiff sample still to be run.

All MD07 g-band images over the V3 tessellation with an offset from the current pointing center (shown by the MD07 z-band night stack exposures from 9/28/11) in order to include as much of the earlier exposures as possible.

Exposures available and with preliminary FWHM cut of ones <2". All exposures were reprocessed for this field to be available in creating the refstacks and deepstacks (more important for this field given the two pointings since 4/2009).

Filter

N total

N (FWHM<8 pixels)

y

391

302

z

481

422

i

561

458

r

423

370

g

423

343

MD07.V3 reference stack

label: MD07.refstack.20111106

data_group: MD07.refstack.20111106

Test stack (reftest) sample also made again for skycell.055 with different number of input warps.

label: MD07.reftest.20111107

data_group: MD07.reftest.20111107

Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20111106 (top blue) and refstack.20100330 (V0) (bottom blue). Any unmatched blue/red overlap for (V0) are due to ones included/rejected in the selected reprocessing. The previous (V0) refstack was only 8 inputs so should be much improvement!

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, (V0)=078) after running psphot on the stacks.

version label

filter

center skycell input warps

center skycell FWHM (major)

MD07.refstack.20111106 V3

y

116(8)

0.81"

- MD07.reftest.201111107 V3

y

31(1)

0.74"

- MD07.reftest.201111107 V3

y

66(9)

0.77"

- MD07.reftest.201111107 V3

y

143(10)

0.85"

old MD07.refstack.20100330 (V0)

y

8(0)

0.91"

MD07.refstack.20111106 V3

z

126(10)

0.96"

- MD07.reftest.201111107 V3

z

29(2)

0.89"

- MD07.reftest.201111107 V3

z

59(4)

0.91"

- MD07.reftest.201111107 V3

z

126(10)

0.96"

- MD07.reftest.201111107 V3

z

154(14)

0.98"

old MD07.refstack.20100330 (V0)

z

8(0)

1.27"

MD07.refstack.20111106 V3

i

126(19)

1.07"

- MD07.reftest.201111107 V3

i

29(0)

1.00"

- MD07.reftest.201111107 V3

i

68(9)

1.03"

- MD07.reftest.201111107 V3

i

98(5)

1.05"

- MD07.reftest.201111107 V3

i

126(19)

1.06"

- MD07.reftest.201111107 V3

i

146(16)

1.07"

old MD07.refstack.20100330 (V0)

i

8(0)

1.28"

MD07.refstack.20111106 V3

r

138(6)

1.11"

- MD07.reftest.201111107 V3

r

25(2)

1.02"

- MD07.reftest.201111107 V3

r

60(7)

1.08"

- MD07.reftest.201111107 V3

r

115(18)

1.10"

- MD07.reftest.201111107 V3

r

138(7)

1.10"

old MD07.refstack.20110121 (V0)

r

8(0)

1.46"

MD07.refstack.20111106 V3

g

152(11)

1.15"

- MD07.reftest.20111107 V3

g

29(2)

1.08"

- MD07.reftest.20111107 V3

g

59(4)

1.11"

- MD07.reftest.20111107 V3

g

126(10)

1.13"

- MD07.reftest.20111107 V3

g

154(14)

1.14"

old MD07.refstack.20100330 (V0)

g

8(0)

1.67"

seems to be some strange rejection behavior in at least the i-band.

How well does the "best" selected inputs cover the two pointings?

note (V0) y-band refstack was at newer pointing, how many y-bands in old pointing?

MD07.GR0 reprocessed sample and finished set

Reprocessing of chip-->warp done mostly with condor. Hiccup with loss of tmp files stalled things in the middle of warp, but pantasks picked up the processing fine using the label.

label: condor_MD07.V3_01

data_group: condor_MD07.V3_01_dg

dist_group: NULL, changed camera to MD07.GR

Reprocessed night stack sample under the following label and data_group.

label: MD07.GR0

dist_group: MD07.GR

data_group of initial sample of night stacks processed with both ipp-20110622 and ipp-20111110 tag (see date) and using the default 4 or more input warps necessary to make stack

data_group: MD07.GR0.20111107 = y

data_group: MD07.GR0.20111108 = z

data_group: MD07.GR0.20111109 = i

data_group: MD07.GR0.20111110 = r

data_group: MD07.GR0.20111111 = g

data_group of remaining sample of night stacks processed with ipp-20111222 tag and allowing 2 or more input warps necessary to make stack

data_group: MD07.GR0.201201xx = y

data_group: MD07.GR0.201201xx = z

data_group: MD07.GR0.201201xx = i

data_group: MD07.GR0.201201xx = r

data_group: MD07.GR0.201201xx = g

Staticsky: only run on refstack skycells with all 5-filters available. This run is on stacks prior to ubercal catalogs so the zeropoint is fundamentally based on the synthcat values. This run provides a sample for testing the stack calibration stage. However, many faults (25/70 runs, i.e. skycells) occurred in the processing and need to still be investigated.

MD08.GR0

Status: finished

11/25: all filters all exposures running chip->warp

11/28: disk/data corruption extensive. will need to be restarted from scratch once diskspace available.

4/18/12: restarted from scratch after raid crash in November/December with the new ipp-20120404 tag to include new ubercal refcat (PS1.REF.20120325) and detrend improvements (dark modes, noise map but not the boost factors).

4/20: gri reprocessing finished and refstacks started. zy prepped but delayed until 4/22 for other tests.

Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20120422 (top blue) and refstack.20100713 (bottom blue). Any unmatched blue/red overlap for (V2) are due to exposures rejected in the selection cuts even at smaller FWHM.

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, V2=078) after running psphot on the stacks.

version data_group (stack_id)

filter

center skycell input warps (rej)

center skycell FWHM (major)

new MD08.refstack.20120422 V3

g

96 (2)

1.27"

MD08.reftest.20120417.c1f49 V3 (873695)

g

34 (1)

1.20"

MD08.reftest.20120417.c1f51 V3 (873696)

g

66 (2)

1.25"

MD08.reftest.20120417.c1f53 V3 (873697)

g

96 (2)

1.25"

MD08.reftest.20120417.c1f57 V3 (873698)

g

119 (7)

1.28"

MD08.reftest.20120417.all.50 V3 (861913)

g

70 (8)

1.21"

MD08.reftest.20120417.2011.50 V3 (861914)

g

50 (5)

1.21"

old MD08.refstack.20100713 V2

g

78 (4)

1.45"

new MD08.refstack.20120422 V3

r

90 (1)

1.19"

MD08.reftest.20120418.c1f42 V3 (873699)

r

33 (0)

1.16"

MD08.reftest.20120418.c1f45 V3 (873700)

r

72 (3)

1.19"

MD08.reftest.20120418.c1f46 V3 (873701)

r

90 (2)

1.19"

MD08.reftest.20120418.c1f50 V3 (873702)

r

119 (1)

1.21"

MD08.reftest.20120418.all.45 V3 (861915)

r

86 (4)

1.18"

MD08.reftest.20120418.2011.45 V3 (861916)

r

73 (6)

1.18"

old MD08.refstack.20110713 V2

r

89 (2)

corrupted wt file

new MD08.refstack.20120422 V3

i

91 (3)

1.09"

MD08.reftest.20120419.c1f41 V3 (873703)

i

33 (0)

0.92"

MD08.reftest.20120419.c1f43 V3 (873704)

i

55 (0)

0.94"

MD08.reftest.20120419.c1f45 V3 (873705)

i

91 (3)

1.09"

MD08.reftest.20120419.c1f46 V3 (873706)

i

117 (4)

1.10"

MD08.reftest.20120419.all.43 V3 (869922)

i

76 (8)

0.95"

MD08.reftest.20120419.2011.43 V3 (869923)

i

54 (3)

1.08"

old MD08.refstack.20100713 V2

i

81 (2)

1.34"

new MD08.refstack.20120422 V3

z

94 (3)

0.95"

MD08.reftest.20120420.c1f38 V3 (877041)

z

30 (0)

0.90"

MD08.reftest.20120420.c1f40 V3 (877042)

z

70 (2)

0.94"

MD08.reftest.20120420.c1f42 V3 (877043)

z

94 (3)

0.95"

MD08.reftest.20120420.c1f44 V3 (877044)

z

126 (4)

0.97"

MD08.reftest.20120420.all.39 V3 (877037)

z

73 (8)

0.93"

MD08.reftest.20120420.2011.39 V3 (877038)

z

43 (4)

0.92"

old MD08.refstack.20100713 V2

z

119 (8)

1.39"

new MD08.refstack.20120422 V3

y

93 (7)

0.84"

MD08.reftest.20120421.c1f35 V3 (880936)

y

31 (1)

0.79"

MD08.reftest.20120421.c1f38 V3 (880937)

y

66 (5)

0.82"

MD08.reftest.20120421.c1f40 V3 (880938)

y

93 (7)

0.84"

MD08.reftest.20120421.c1f42 V3 (880939)

y

117 (4)

0.85"

MD08.reftest.20120421.all.38 V3 (877039)

y

87 (11)

0.83"

MD08.reftest.20120421.2011.38 V3 (877040)

y

56 (3)

0.83"

old MD08.refstack.20100713 V2

y

74 (17)

1.68"

* i-band is an odd situation, 2010 had smaller FWHM with many exposures <1" versus 2011 with only a few <1" so 2010+2011 FWHM~1.09" with just ~15 more inputs.

Test psphot run on refstack

Troubled skycells

Previously had a few missing skycells in center and around edges (g-band example left). New has only a couple quality failures in i-band for skycell.087,012 (right) that at some point should be manually recovered.

Staticsky: only run on refstack skycells with all 5-filters available and distributed under ps1-md-staticsky. Convolved matching to target PSF much improved, generally the 4-6 pixels now similar to upper input PSF cutoff except for oddities that drive edge skycells >10,20,70 pixels. Work continues to better constrain selection of the target PSF.

label: MD08.refstack.20120422

data_group: MD08.refstack.20120422.5x20120425

dist_group: MD.staticsky

Eric Morganson has noted that the RA_PSF,DEC_PSF is NULLhttp://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/ticket/1505, need to track down if fixed and broken again. Should be able to use skycell WCS to get coordinates from xy -- should be fixed in later runs of stack photometry (Aug. 2012)

-- either tracking jump or mirror solution issue - check if used/rejected in z night stack, yes but rejected at end before final stack made by PSF chi2 clip.
neb://ipp048.0/gpc1/MD08.GR0.20120420/o5103g0041o.101597/o5103g0041o.101597.ch.442508

All MD09 g-band images over the V3 tessellation with an offset from the current pointing center (shown by the MD09 z-band night stack exposures from 9/28/11) in order to include as much of the earlier exposures as possible.

Exposures available and with preliminary FWHM cut of ones <2". All exposures were reprocessed for this field to be available in creating the refstacks and deepstacks.

Filter

N total

N (FWHM<8 pixels)

g

471

415

r

459

427

i

478

432

z

491

469

y

303

283

MD09.V3 reference stack

Reference stack using data from 2010+2011, a balance of N input warps versus best seeing, excluding exposures with large background, low zeropoints, and large elongation.

label: MD09.refstack.20120603

data_group: MD09.refstack.20120603

Test stack (reftest) sample also made again for skycell.055 with different number of input warps.

Histograms of the FWHM major and minor from camera stage for reprocessed all exposures (red) and refstack.20120603 (top blue) and refstack.20100721 (bottom blue).

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, V2=078) after running psphot on the stacks. Because the reftests were all run on the same date, the stack_id is important to note to separate by filter as well.

version data_group (stack_id)

filter

center skycell input warps (rej)

center skycell FWHM (major)

new MD09.refstack.20120603/20120831 V3

g

97 (0)

1.22"

MD09.reftest.20120602.c1f48 V3 (963600)

g

30(0)

1.17"

MD09.reftest.20120602.c1f50 V3 (963601)

g

56(1)

1.19"

MD09.reftest.20120602.c1f54 V3 (963602)

g

96(1)

1.22"

MD09.reftest.20120602.c1f56 V3 (963603)

g

117(2)

1.24"

MD09.reftest.20120602.all.49 V3 (963598)

g

95(8)

1.17"

MD09.reftest.20120602.2011.49 V3 (963599)

g

81(8)

1.16"

old MD09.refstack.20100721 V2

g

17 (2)

1.52"

new MD09.refstack.20120603/20120831 V3

r

89 (2)

1.09"

MD09.reftest.20120602.c1f41 V3 (975093)

r

31 (0)

1.02"

MD09.reftest.20120602.c1f43 V3 (975094)

r

59 (2)

1.05"

MD09.reftest.20120602.c1f45 V3 (975095)

r

89 (2)

1.09"

MD09.reftest.20120602.c1f47 V3 (975096)

r

126 (4)

1.13"

MD09.reftest.20120602.all.43 V3 (975092)

r

69 (7)

1.05"

MD09.reftest.20120602.2011.43 V3 (975091)

r

65 (8)

1.04"

old MD09.refstack.20100721 V2

r

17 (3)

1.79"

new MD09.refstack.20120603/20120831 V3

i

85 (2)

0.96"

MD09.reftest.20120602.c1f38 V3 (975158)

i

26 (1)

0.88"

MD09.reftest.20120602.c1f41 V3 (975159)

i

53 (3)

0.91"

MD09.reftest.20120602.c1f43 V3 (975160)

i

85 (5)

0.96"

MD09.reftest.20120602.c1f45 V3 (975161)

i

120 (5)

0.99"

MD09.reftest.20120602.all.43 V3 ()

i

97 (4)

0.97"

MD09.reftest.20120602.2011.43 V3 (975157)

i

57 (4)

0.94"

old MD09.refstack.20100721 V2

i

38(1)

1.42"

new MD09.refstack.20120603/20120831 V3

z

95 (3)

0.89"

MD09.reftest.20120602.c1f37 V3 (966289)

z

36 (1)

0.82"

MD09.reftest.20120602.c1f38 V3 (966290)

z

56 (2)

0.85"

MD09.reftest.20120602.c1f40 V3 (966291)

z

95 (2)

0.89"

MD09.reftest.20120602.c1f42 V3 (966292)

z

119 (2)

0.90"

MD09.reftest.20120602.all.38 V3 (966288)

z

87 (4)

0.85"

MD09.reftest.20120602.2011.38 V3 (966287)

z

64 (5)

0.83"

old MD09.refstack.20100721 V2

z

29(0)

1.17"

new MD09.refstack.20120603/20120831 V3

y

87 (8)

0.85"

MD09.reftest.20120602.c1f37 V3 (963607)

y

31(3)

0.79"

MD09.reftest.20120602.c1f41 V3 (963608)

y

59(2)

0.82"

MD09.reftest.20120602.c1f45 V3 (963609)

y

86(5)

0.86"

MD09.reftest.20120602.c1f52 V3 (963610)

y

119(0)

0.89"

MD09.reftest.20120602.all.38 V3 (963606)

y

81 (8)

0.78"

MD09.reftest.20120602.2011.38 V3 (963605)

y

64 (7)

0.78"

old MD09.refstack.20100721 V2

y

20(3)

1.35"

Test psphot run on refstack

Troubled skycells

Previously had a few missing/faulted skycells around edges (i-band example left). New refstacks do not have any quality failures so far and note the current nominal position of the populated skycells is shifted leftward (middle) to accomodate the reprocessing of the night stacks with the early pointing (right).

MD09 revised refstack.20120831

The refstack should be nearly identical (using the same warp processing) except for the removal of an arc of lower sensitivity/depth from the stacking process

label: MD09.refstack.20120831

data_group: MD09.refstack.20120831

Map of the magnitude of sources @5-sigma:

20120603

20120831

MD09 deep stack test 20120727

label: MD09.deeptest.20120727

data_group: MD09.deeptest.20120727

Some pre-input cuts were made in an attempt to clean out less useful exposures, focus is still on stack issues not how to deal with the worst of the input images --

the revised DEEP_STACK reduction configuration then imposes an addition FWHM cut based on the FWHM of the input warp skycells (can be wildly different from the equivalent camera exposure FWHM) of FWHM_mean+1.5*SIGMA

(left) Sample histogram of the available exposures by camera FWHM_MAJOR in i-band (red) and selected to attempt a deepstack with (blue) in top row, refstack in bottom row.

(right) Sample histograms for each skycell over the FPA by FWHM of the input warps in the i-band (blue) and rejected for poor PSF or other issues (red). The black dotted line is the input cut @7.5 pixels and the red line is the FWHM_mean+1.5*FWHM_SIGMA additional cut on the input FWHM that is often ~6 pixels given the available input warps -- initially set in an attempt to restrict the target PSF of the convolved version of the stack to ~1.5" but other problem(s) still exist that drive the target PSF much larger (not shown).

Newer deep stack tests being documented at MD09.PV1_deepstack_tests. A sample listing of similar skycell Ninput/rejected warps for this 20120727 run for comparison

Originally, only run on refstack skycells with all 5-filters available and distributed under ps1-md-staticsky. Convolved matching to target PSF much improved, generally the 4-6 pixels now similar to upper input PSF cutoff except for oddities that drive edge skycells >10 pixels (extended source measurements should be on the RAW/unconv image in this tag however). Work continues to better constrain selection of the target PSF.