DINEOF

From GHER

DINEOF is an EOF-based method to fill in missing data from geophysical fields, such as clouds in sea surface temperature. This page contains basic instructions to download and compile DINEOF. The source code and binaries are freely available for download right here!:

For more information on how DINEOF works, please refer to Alvera-Azcarate et al (2005) and Beckers and Rixen (2003) The multivariate application of DINEOF is explained in Alvera-Azcarate et al (2007), and in Beckers et al (2006) the error calculation using an optimal interpolation approach is explained. If you need a copy of any of these papers, don't hesitate to contact us! For more information about the Lanczos solver, see Toumazou and Cretaux (2001).

If you want to try DINEOF without installation on standard satellite data sets:

Daily reconstruction in the western Mediterranean Sea

You can see here a daily reconstruction of the Sea Surface Temperature of the Western Mediterranean Sea.

Daily filled fields, error maps and outliers for the past ten days are shown in the mentioned web page. The reconstruction is done on 6 months of data, and the whole procedure takes about two hours. The steps taken are the following:

Download cloudy data

Run DINEOF for outlier detection

Calculate and remove outliers from initial data set

Re-run DINEOF on the cleaned data set

Calculate error maps

Make plots, and display on web page

This product is updated daily with the latest SST data from the Mediterranean Sea.

Daily reconstruction in the Black Sea

You can see here a daily reconstruction of the Sea Surface Temperature of the Black Sea.

The same procedure as in the western Mediterranean Sea is applied to Black Sea SST (download cloudy data, run DINEOF for outlier detection, calculate and remove outliers from initial data set, re-run DINEOF on the cleaned data set, make plots, and display on the web page).

Getting started

List of DINEOF References

A list of publications made by our group about DINEOF and papers related to its development. There is also a list of papers by other people is using DINEOF for their research

Problems/Feedback

If you have problems to install or use DINEOF, we would be glad to help you. We have created a mailing list where you can ask your questions, and check the questions already posted by others. You can join as a member by going to: http://groups.google.com/group/dineof. Once you are a member, you can ask your question by sending an email to: dineof at googlegroups dot com (with obvious substitutions). Please join our discussion group! See also Troubleshooting

In order for us to help you, please provide the following information when reporting your problem through the mailing list:

Which version of DINEOF is used. Did you compile it yourself or did you use the binaries we provide?

Which operating system and which version are you working on? (for example: Windows 8.1, Ubuntu 14.04 ...)

Are you working on a 32-bit or 64 bit CPU?

Error message (the exact screen output, please)

If you are compiling DINEOF from source, please make sure to give us also the following information:

Which Fortran compiler (and which version) are you using (for example, gfortran 4.8.0)

Compiler flags used to compile ARPACK (If you did not compile ARPACK by yourself, then you can just specify from where you got the compiled library, e.g. Ubuntu software repository or MacPorts).

Your Makefile and the corresponding file in Compilers/ (e.g. Linux-gfortran.mk if you use gfortran on Linux).

Compile log (make.log) generated by compiling DINEOF:

make clean all > make.log

If you think there is a bug, please provide a stripped-down version of your problem which still exhibits the problem.

Please do not omit any of those points. The more information you provide, the more likely it will be that we can help you.

FAQ

In construction!!

Are there other ways to use DINEOF than with the compiled Fortran code?

If you are not needing to exploit all features of the compiled Fortran code, some wrappers or implementations in popular analysis softwares can be exploited:

Does the reconstruction take into account the time proximity of the images, i.e., does yesterday's image influence more than an image 6 months ago?

All the images have the same weight in the reconstruction. But in the recent versions, a filter is applied on the time-covariance matrix. This filtering provides a better consistence between successive images. Also note that even if there is no explicit information on time closeness, if the process is more correlated to recent events, the EOF decomposition will detect it and provide naturally a reconstruction which uses recent information rather than older ones.

How many days/months/years of data are necessary for a reconstruction?

As a general rule, the longest time series of images will provide the best results, as the principal modes of variability will be well reproduced. However, if the memory usage is limited, then the time series can be shortened.

Is it possible to reconstruct images with almost 100% of missing data?

Yes, it is possible, but the result may not be reliable. In general, images containing less than 5% of data do not provide useful information
and might aﬀect the final result.

How to deal with clouds that are not well detected by the algorithms?

There is now a quality control implemented in DINEOF: it provides a list of suspect pixel values according to:

the proximity with detected clouds

the distance to the local median value

the spatial coherence with the reconstruction

Why the reconstruction does partially remove gradients or variability?

Because the reconstruction with DINEOF is obtained as a truncated EOF decomposition. There are higher-order modes that are not considered for the final reconstruction, as the reconstruction with this number of modes does not provide the minimal cross-validation error.

It is also possible to keep the original pixels and only fill in the clouds. Doing so, the gradients and the variability is conserved but there might be artefacts at the edges of the filled regions.

Why isn't there any graphical user interface yet ?

DINEOF is intended for users who need to repeat calculation a number of times (be it because they are scientists who will need to test different parameter values on reconstuctions, or operational users aiming a recalculating regularly new analysis). In this case, rather then click every time across a series of menues, a command line approach with batch files is much more efficient. For the occasional user, a GUI is under development.

Can I use DINEOF with 1D or 3D fields instead of 2D images?

Yes, as DINEOF does not know anything about spatial position and even in the 2D case simply stacks all values of a given moment into a column vector. The same can be done for 1D and 3D very easily.