Introduction

This page compiles the plans and needs to prepare and serve Highly Processed Data Products (HPDPs) to the Scientific Communities. There are two main ways considered to fulfil this: a web page, similar to that used for User Provided Data Products (UPDP, see also http://www.cosmos.esa.int/web/herschel/user-provided-data-products), and the HSA User Interface (HUI). The following focusses mostly on the latter.

Meetings and action items

The main action item resulting from the meeting is that each instrument representative shall come up with a list of use cases and requirement wish list applying to the HPDP they are supervising or acting as contact point for. These requirements will be compiled in the corresponding section below. Deadline: second half of September.

HPDP release notes

The following document is a template for release notes to be associated with HSC-curated HPDP deliveries.

HPDP plans per instruments

HPDPs have been identified by each instrument team, although it is likely that not all of them will be possible to prepared in certain cases. We indicate hereafter the HPDPs currently contemplated by each instruments, with some common delivery categories that will be considered or not.

Use Unimap to combine hundreds of obsids from four GOODS cosmological fields (custodian: B. Altieri)

Yes

No

Yes

Mini-Scan level 2.5

A series of observations taken as cross-scan maps, albeit with different phot. filters, resulting in the absence of level 2.5 products. The red observations are, however, available in both scanning directions hence level 2.5-type products can be generated.

Provide the point-source corrected spectrum for pointed observations; possibly focussing only on those with point-like sources not located in the central spaxel in them; a SpectrumTable

High

The results of the POC script can be better than those provided in the SPG, and this is especially so for off-centred point sources

Using the "POC" interactive pipeline script

This script has been validated sufficiently to be used for almost any flux level, but we would need to run sanity checks on the results to be sure this applies to the fainter sources. If we limit the provision to point-like sources, off-centred or not, then we need to identify those

Most of the effort is on identifying the candidates (if not done for all pointed observations, willy-nilly). Some effort on sanity checking

Blue spectra (<55mum)

Provide the blue end of the spectral range, which is not part of SPG as it is uncalibratable.

High

A slightly modified version of the SPG script can be used to create these ranges, where we do not the the flux calibration

Using a slightly modified ipipe/SPG script

need to identify the obsids with this range

not much effort, mostly on making sure that this "blue bulk" runs probably and creates correct products

On and Off cubes provided separately

The on-source and off-source Level 2 cubes provided for the unchopped line and all chop-nod observations

Mid

These provide the most simply way to identify off-source contamination

using the SplitOnOff ipipe script for chop-nod, and using the SPG script for unchopped line but saving the products slightly earlier in the Level 2 part

little - scripts ready, just need to modify it to run on all observations and produce the desired FITS files with the desired naming convention. the postcard can be created using the same task as used currently by PACS

all unchopped line and chop-nod

Unchopped obsids with continuum removed

Subtract the continuum from all unchopped obsids so as to provide a cleaner spectrum; Level 2/2.5 context

Mid

The continuum is very uncertain and it had been agreed last year at an ICC meeting that removing this was a "good thing to do”

Range scans: use the RV methods to smooth and subtract the continuum; Line scans: use an optimised version of the line-finding task in the flatfielding task of the SPG. Fit/smooth and remove continuum and then produce all final results

Need to test methods and thereafter processing should be automatic

Producing the scripts: a few weeks (?), checking the results: continuous (but semi-automatic)

all ~1200 on-sources unchopped obsid

We also need to produce a report on the unchopped mode: based on DF paper but more examples and a proper measure of uncertainty based on e.g. comparing repeat observations, scatter in super-off, etc.

Ad-hoc HPDPs

For obsids found to be badly handled by the SPG, an expert will make the necessary adjustments to the pipeline so it can run; Level 2/2.5 context

Mid

These obsids are either not provided, or the science products are not of the same standard as all other obsids

These are obsids identified in an ad-hoc manner as needing help. Jira tickets, Kayako, unchopped with bad transients identified while the unchopped study is done, etc

Will only be done if it is feasible in terms of time and manpower

so far <10 candidates suggested

Line list catalogue

Spectral Feature catalogue

Nice to have but ICC have no interest here and consider it Low

SPIRE and HIFI have one, so why not PACS? Catalogues always produce unexpected science and are a very good Legacy product

Need scripts to find lines, fit lines and maybe identify lines

Need to create the scripts: KE has a student at Leuven testing out fitting lines, and may have a student in 2016-17 to test out finding lines

If the student projects go well, some time in 2017 the resulting scripts could be run on the PACS spectroscopy archive (or subsets of that)

Is being done by KME as a science project

New transients script results for unchopped obsids DE-SCOPED

Producing the output of the unchopped obsids reduced with the new transients script

Low

This script is supposed to produce better results and users may want to use it (esp. if they read the DF paper) %RED; now realised that that this is not the case

Ancillary Data Products

HUI requirements

We provide here the HUI requirements needed to support the storage, queries and access to the various HPDPs and Ancillary Data Products: HPDP_ADP_HUI_Requirement_v1.3.pdf

Highly Processed Data Products Cosmos page

The Cosmos webpages hosting the HPDPs can be found at this link (currently orphan page)

Automatic README file generation

A Python script has been created by Ivan in order to automatically generate the xml README file required in order to ingest HPDPs into the HSA. The script is currently at version 3, with the following instructions.