To create Photon-HDF5 files, users can convert existing files or
save directly from suitably modified acquisition software. The
conversion option is generally the simplest approach and, when
using closed-source acquisition software, also the only one available
(until vendors start supporting Photon-HDF5).

To simplify saving (and converting) Photon-HDF5 files we developed and maintain,
phconvert, an open-source
python library serving as reference implementation for the
Photon-HDF5 format. While Photon-HDF5
can be created without phconvert,
using only a HDF5 library, we recommend taking advantage of phconvert
to simplify the writing step and to make sure that the saved file
conforms to the specifications. Phconvert, in fact, checks that all mandatory
fields are present and have correct names and types, and adds a description
to each field. Phconvert can be directly used in programs written in Python
or other languages that allow calling Python code (see next sections).
phconvert permissive license (MIT) allows integration with both open and
closed source software.

phconvert includes a browser-based interface using
Jupyter Notebooks to convert vendor-specific file
formats into Photon-HDF5 without requiring any python knowledge.
The formats currently supported are HT3 (from PicoQuant
TCSPC hardware), SPC/SET (from Becker & Hickl TCSPC hardware) as well as SM
(a legacy file format developed by the WeissLab, UCLA).

We provide a demo service
to run these notebooks online and convert one of these formats to Photon-HDF5
without software installation on the user’s computer.

Beyond the currently supported ones, other formats can be converted by
writing a Python function to load the data and by using phconvert to save
the data to Photon-HDF5. Taking the
existing phconvert loader functions
as examples, this task is relatively easy even for inexperienced Python programmers.
See also the notebook
Writing Photon-HDF5 files
(view online).

We encourage interested users to contribute to load functions to phconvert so that
out-of-the-box support for conversion of the largest number of formats can
be provided. If you have an input file format not supported by phconvert
please open a new issue
on GitHub.

To directly save Photon-HDF5 files from within an acquisition software,
there are several options. For programs written in Python, the obvious option
is using phconvert which makes simple creating Photon-HDF5 files while
assuring the validity of the output file. See for example the notebook
Writing Photon-HDF5 files
(view online).

For acquisition software written in other languages(e.g. C, MATLAB or LabVIEW),
it is in principle possible to call python using the Python C API
(see Embedding Python in Another Application).
However understanding the Python C API requires a fairly good proficiency in C
(and probably python).

In order to make it easy to create valid Photon-HDF5 in any language
(without duplicating the effort of creating a library like phconvert
in every language) we devised an alternative approach. The user can
save the photon-data arrays (timestamps, detectors, nanotimes, etc…)
in a plain HDF5 file. The remaining metadata is written in a simple
text file (YAML). Next, a script called
phforge reads the metadata and
the photon-data arrays and creates a valid Photon-HDF5 file using phconvert.
In this way, at the cost of a small inefficiency (writing some
temporary files), a user can easily and reliably generate Photon-HDF5
files from any language.
The metadata file is a text-based representation of the full Photon-HDF5
structure, excluding the photon-data arrays and some other field
automatically filled by phconvert. To store this metadata we use YAML markup
(a superset of JSON) for its simplicity and ability to describe hierarchical
structures. For example, a minimal metadata file describing only mandatory
fields is the following:

description:ThisisadummydatasetwhichmimicssmFRETdata.setup:num_pixels:2# using 2 detectorsnum_spots:1# a single confocal excitationnum_spectral_ch:2# donor and acceptor detectionnum_polarization_ch:1# no polarization selectionnum_split_ch:1# no beam splittermodulated_excitation:False# CW excitation, no modulationlifetime:False# no TCSPC in detectionphoton_data:timestamps_specs:timestamps_unit:10e-9# 10 ns

To save the photon-data arrays the user needs to call the HDF5 library
for the language of choice. For example, in MATLAB timestamps and detectors
arrays can be saved with the following commands:

In principle, it should be possible using a recent release of MATLAB (R2014b or later) to
directly call python functions.
Therefore it should be possible to directly call phconvert.
However, in our recent attempt, we weren’t able to configure MATLAB in order
to load the correct dynamic libraries (i.e. the HDF5 C library) required by phconvert.

If for some reason you cannot use phforge or phconvert, you have to implement
routines to write Photon-HDF5 files using the HDF5 library for your platform,
taking care of following the Photon-HDF5 specification.
In the following paragraph we provide a few suggestions on how to proceed
in this case.

To facilitate writing valid Photon-HDF5, we provide
a JSON file
containing all the official field names, a short description and a generic
type definition (array, scalar, string or group).
This JSON file can be used both to validate names and types of the data fields
and to retrieve the standard short description (this is, in fact, what
phconvert does). The developer needs to verify that all the mandatory fields
are present.
The description string should be saved for all the official fields in
an attribute named “TITLE”. For compatibility with h5labview, we recommend to
use a single-space string (” ”) for all the user fields that lack a description
(phconvert uses this workaround too).

Furthermore, the /identity group should include
the fields software_name and software_version to specify the name
and the version of the software that created the file.

Finally, you can verify that generate files are compliant with the
Photon-HDF5 specifications by using the phconvert function
phconvert.hdf5.assert_valid_photon_hdf5_tables().
This function will raise errors or warnings if the input file does not follows the specs.