The NCO toolkit manipulates and analyzes data stored in
netCDF-accessible
formats, including
DAP,
HDF4, and
HDF5.
It exploits the geophysical expressivity of many
CF
(Climate & Forecast) metadata conventions,
the flexible description of physical dimensions translated by
UDUnits,
the network transparency of
OPeNDAP,
the storage features (e.g., compression, chunking, groups) of
HDF (the Hierarchical Data Format),
and many powerful mathematical and statistical algorithms of
GSL (the GNU Scientific Library).
NCO is
fast,
powerful,
and
free.

Traditional geoscience data analysis requires users to work with
numerous flat (data in one level or namespace) files.
In that paradigm instruments or models produce, and then repositories
archive and distribute, and then researchers request and analyze,
collections of flat files.
NCO works well with that paradigm, yet it also embodies the necessary
algorithms to transition geoscience data analysis from relying solely
on traditional (or “flat”) datasets to allowing newer
hierarchical (or “nested”) datasets.

The next logical step is to support and enable combining all
datastreams that meet user-specified criteria into a
single or small number of files that hold all the
science-relevant data organized in hierarchical structures.
NCO (and no other software to our knowledge) can do this now.
We call the resulting data storage, distribution, and analysis
paradigm Group-Oriented Data Analysis and Distribution
(GODAD).
GODAD lets the scientific question organize the data, not the
ad hoc granularity of all relevant datasets.
The User Guide illustrates
GODAD
techniques for climate data analysis:

Note that the “averagers”
(ncra and ncwa) are misnamed because they perform
many non-linear statistics as well, e.g., total, minimum, RMS.
Moreover, ncap2 implements a powerful domain language which
handles arbitrarily complex algebra, calculus, and statistics (using GSL).
The operators are as general as
netCDF
itself: there are no restrictions on the contents of input file(s).
NCO's internal routines are completely dynamic and impose no limit on
the number or sizes of dimensions, variables, and files.
NCO is designed to be used both interactively and with large batch jobs.
The default operator behavior is often sufficient for everyday needs,
and there are numerous command line (i.e., run-time) options, for
special cases.

How to Contribute: Volunteer, Endorse, or Donate

In March, 2000, NCO became an Open Source project hosted by
SourceForge.net.
This facilitates collaboration, code contributions, and support.
There is a TODO list hundreds of items long!
No matter what your programming background there is a task you can help with.
From re-organizing the TODO list itself, to improving this cheesy
webpage, to documentation, to designing and implementing new features
and interfaces, we need your help!
The project homepage
contains mailing lists, discussion forums, and instructions to make
contributing to NCO easy.

Many users feel unable to volunteer their time.
An equally effective contribution in the long-run would be your
endorsement, which may make the difference
between a declined and an accepted proposal.
An endorsement can be a few sentences that describes how NCO benefits your work or research.
E-mail your endorsement to
Charlie “my surname is zender” Zender
with Subject: “NCO Proposal Endorsement”.
This information is useful advocating for more NCO support.
“What future proposals?” you ask, “Aren't you already funded?”
Yes, in 2012 NASA funded us to implement netCDF4 groups and HDF support,
and in 2014 NASA funded us to improve SLD handling.
These funds are/were primarily for development and maintainance of
specific features.
To realize our grander ambition, i.e., to shift geoscience data
analysis from a flat- to a hierarchical-paradigm
(GODAD),
will require a sustained effort and software ecosystem that
understands and implement hierarchical dataset concepts.
And it's hard to sell a federal agency on the wisdom of investing in a paradigm shift!
Other more prosaic tasks that need work are, for example,
I/O parallelization (!!!), user-defined and non-atomic types, more CF conventions,
cloud-services, JSON back-end, and user-defined ncap2 functions.
If you send an endorsement, please include (at least) your Name, Title, and Institutional affiliation.

Lastly, as of June, 2003, if you're more strapped for time than money
and want to contribute something back, consider a monetary donation.
This may incentivize us to tackle your favorite TODO items.
Inspired by President Obama's plan to bring more transparency to
government investment, these homepage donation counters track the
influence of your monetary donations on NCO development:

Donations received between 20030624 and 20141126: US$144.55. Thank you, donors!

NCO features “incentivized” by these donations: More emoticons in the documentation :)

DOE ACME Project

“Lightweight Climate Analysis Tools for ACME”
is a US Department of Energy Cooperative Agreement (CA)
DE-SC0012998
from 20141215–20171214 that funds our
contribution
to the Accelerated Climate Modeling for Energy (ACME)
project,
a part of DOE's Earth System Modeling (ESM)
program.
The ACME project provides the resources to implement support for
parallel regridding and workflows in NCO accessible through UV-CDAT.
Spatially intelligent, parallelized analysis tools are a key component
of the Group-Oriented Data Analysis and Distribution
(GODAD) paradigm we
are developing for geoscience data analysis.

NASA ACCESS 2013 Project

The National Aeronautics and Space Administration (NASA) Cooperative
Agreement NNX14AH55A
funds our
project,
“Easy Access to and Analysis of NASA and Model Swath-like Data”
from 20140701–20160630 as part of the
Advancing Collaborative Connections for Earth System Science
(ACCESS)
program.
This URL,
http://nco.sf.net#prp_axs,
points to the most up-to-date information on the ACCESS 2013 project.

This ACCESS project provides resources to implement support in NCO
for Swath-like data (SLD), i.e., dataset with non-rectangular and/or
time-varying spatial grids in which one or more coordinates are multi-dimensional.
It is often challenging and time-consuming to work with SLD,
including all NASA Level 2 satellite-retrieved data, non-rectangular
subsets of Level 3 data, and model data (e.g., CMIP5) that are
increasingly stored on non-rectangular grids.
Spatially intelligent software tools is a key component of the
Group-Oriented Data Analysis and Distribution
(GODAD) paradigm we
are developing for geoscience data analysis.

We are currently recruiting a programmer (aka software engineer)
or postdoc based at UCI for at least two years, to accomplish our
ACCESS objectives.
As described in the proposal, this person will be responsible for
incorporating geospatial features and parallelism into NCO.
See the ads for more details.
(PDF,
TXT).

NASA ACCESS 2011 Project

The National Aeronautics and Space Administration (NASA) Cooperative
Agreement NNX12AF48A
funded our
project,
“Simplifying and accelerating model evaluation by NASA satellite data”
from 20120208–20140207 as part of the
Advancing Collaborative Connections for Earth System Science
(ACCESS)
program.
We appreciate the proposal reviewers for and the staff of the
Earth Science Division
(ESD)
in the Science Mission Directorate
(SMD)
Earth Science Data Systems (ESDS) Office.
NCO development was completely voluntary and without
institutional support from August, 2008–February, 2012.
This NASA support dramatically changed the scale and pace of NCO development.
This URL,
http://nco.sf.net#prp_access,
points to the most up-to-date information on the ACCESS 2011 project.

This ACCESS project provided human resources to implement support
in NCO for hierarchical group features in netCDF4 files and for NCO to
process HDF files.
Groups are a powerful HDF feature that many NASA satellite datasets
exploit, and that sources of netCDF data (e.g., the CMIP5 archive of
climate simulations for IPCC AR5) could better exploit.
By supporting groups in a generic fashion, NCO helps make possible
more widespread adoption of hierarchical data analysis, aka
Group-Oriented Data Analysis and Distribution
(GODAD).

We recruited two personnel, a programmer (aka software engineer)
and a graduate student (scientific specialist) both based at UCI for
at least two years, to accomplish our ACCESS objectives.
As described in the proposal, the responsibilities of positions are
roughly segregated as follows:
The
programmer
helped re-factor the code-base to support groups and improved the
build system so that now we have native Windows builds.
The
graduate student researcher
is analyzing and intercomparing snow cover and snow albedo datasets
from NASA MODIS and MISR datasets (in HDF format) and CMIP5
simulations (in netCDF format).
Her analysis informs the development of GODAD so that NCO can solve
the most important problems real-world researchers encounter in
evaluating GCMs against satellite data.

NSF SEI Project

The National Science Foundation Grant
NSF IIS-0431203
funded our
SEI Project,
“SEI(GEO): Scientific Data Operators Optimized for Efficient Distributed Interactive and Batch Analysis of Tera-Scale Geophysical Data”
from 20040901–20080831 as part of the
Science and Engineering Informatics
(SEI)
program.
We appreciate the proposal reviewers for and staff of the Divisions of
Information and Intelligent Systems
(IIS)
and
Shared Cyberinfrastructure
(SC) in the
Directorate for Computer and Information Science and Engineering
(CISE).
Until September 2004, NCO development was completely voluntary and without institutional support.
This NSF support dramatically changed the scale and pace of NCO development.
This URL,
http://nco.sf.net#prp_sei,
points to the most up-to-date information on the NCO SEI proposal.

The NSF project provided the human and technical resources to analyze,
design, and implement novel advanced methods for distributed
data analysis and reduction (DDRA) of geophysical data.
The efficacy of these powerful methods were demonstrated by analyses
of distributed climate prediction and observation datasets.
On the human side, we recruited full-time help for specific areas of
NCO, alongside our current and future volunteers.
We hired a
programmer
to help design, implement, and release major code changes.
We hired a
graduate student researcher
to pursue innovative Ph.D. research using
“grid-aware”distributed memory,
shared memory, and
client-server software
technologies to improve DDRA of climate datasets.
Another graduate student helped us design, implement, and release major
code changes.

On the institutional and hardware side, this project connected the
Earth System Modeling Facility
(ESMF) to the
California Institute for Telecommunications and Information Technology
(Cal-IT2)
OptIPuter at the
San Diego Supercomputer Center
(SDSC).
These supercomputers dedicated a TB of storage each to
OPeNDAP-served climate simulation
datasets for DDRA.
After the proof-of-concept NCO DDRA was complete, we attempted a DDRA
intercomparison of the internationally distributed
CMIP3 global climate change simulations that reside on the
Earth System Grid
(ESG).
Our techniques tremendously accelerated the analysis of different
climate prediction scenarios for the same model, and among different
climate models (see publications).

Get NCO Binary Executables

NCO developers are too short-handed to provide pre-built binary
executables for all platforms.
Our source tarballs are always up-to-date, and work on our
development systems (Fedora, Ubuntu, and Mac OS X).
We also attempt to provide (theoretically) platform-independent sources
in the most common Linux package formats (Debian and RPM).
Below are links to these and to packages for other platforms created
by volunteers.
Anyone willing to perform regular regression testing and porting
of NCO to other platforms is welcome.
Previous versions of these binaries are still available by searching
the directory index here.

If not, try our own most recent (we stopped building RPMs many years ago and are looking for a volunteer to do this instead) self-built NCO RPMs (install with, e.g., ‘yum install nco-3.9.5-1.fc7.i386.rpm’):

Volunteers have updated and maintained fairly up-to-date NCO packages in Fedora since it was added by Ed Hill in about 2004.
Thanks to Patrice Dumas, Ed Hill, and Orion Poplawski for packaging NCO RPMs over the years.
Thanks to Gavin Burris and Kyle Wilcox for documenting build procedures for RHEL and CentOS.

The most up-to-date binaries are probably those in the tarball below. Those unfamiliar with installing executables from tarballs may try the (older) DMG files (you may need to add /opt/local/bin to your executable path to access those operators).

These native Windows executables should be stand-alone, i.e., not
require users to have any additional software.
This is a new feature as of 20120615, please send us feedback.
To build NCO from source yourself using MSVC or Qt, please see the NCO Qt/MSVC build page.

Before using, first install curl (in the "Web" category of Cygwin Setup), and set export UDUNITS2_XML_PATH='/usr/local/share/udunits/udunits2.xml' (or wherever udunits2.xml is installed).
Thanks to Mark Hadfield and Pedro Vicente for creating Cygwin tarballs.
Thanks to Cygnus Solutions and RedHat Inc. for developing and supporting Cygwin over the years.

nco.texi is the most up-to-date.
Files nco.dvi, nco.ps, and nco.pdf are
contain all the mathematical formulae (typeset with TeX) missing from
the screen-oriented formats.
The screen-oriented formats—nco.html,
nco.info, nco.txt, and nco.xml—contain
all the documentation except the highly mathematical sections.

Other documentation:

This abbreviation key unlocks the mysteries of the source code abbreviations and acronyms.

FAQ: Frequently Asked Questions

These questions show up almost as frequently as my mother.
But they are more predictable:

I still have questions, how do I contact the NCO project?
The NCO project has various Q&A and discussion forums described
below.

Where can I find prebuilt NCO executables?
Pre-built executables of some versions of NCO for the operating
systems described above (Debian-compatible
GNU/Linux, Fedora/RedHat GNU/Linux, Gentoo GNU/Linux, and
Mac OS X).
Otherwise, you may be on your own.

Does NCAR support NCO?
The NCAR CISL Consulting Service Group (CSG) supports NCO like other
software packages.
The NCAR CISL-suported executables are made available through
“modules” so try module load nco.
If you notice problems with the NCO installation on CISL machines, or
if you would benefit from a more recent release or patch, then ask
cislhelp.
If you have a comment, suggestion, or bug report, then contact the
developers as described below.

Is there an easy way to keep up with new NCO releases?
Subscribe to the
nco-announce
mailing list.
This list is for NCO-related announcements, not for questions.
nco-announce is very low volume—one message every few months.

Help/Support/Contacts:

If you have support questions or comments please contact us via the
Project Forums (rather than personal e-mail) so other users can
benefit from and contribute to our exchange.
Let us know how NCO is working for you—we'd like to hear.
Have you read the documentation and browsed the
forums to see if your question/comment has been reported before?
Please read the Guide's suggestions for productive
Help Requests and Bug Reports.

NCO's browsable
Git Repository
contains up-to-the-minute sources and is the easiest way to stay
synchronized with NCO features.
Retrieving and using NCO source requires some familiarity with
development tools,
especially Git and
Make.
GitHub provides
generic instructions
for accessing their repositories.
You may retrieve any NCO distribution you wish.
Usually you wish to retrieve a recent tagged (i.e., released) version.
This command retrieves and places NCO version 4.4.8 (which is
tagged as nco-4.4.8) into local directory
nco-4.4.8:

This command retrieves the current (“bleeding edge”)
development version of NCO into a local directory named nco:

git clone https://github.com/czender/nco.git ~/nco

Track changes to the development version using

cd nco;git pull

One difference between running a "tagged" release
(e.g., nco-4.4.8) and the development version is that the
tagged release operators will print a valid version number (e.g.,
4.4.8) when asked to do so with the -r flag
(e.g., ncks -r).
The development version simply places today's date in place of the
version.
Once the autotools builds are working more robustly, the confusion
over versions should largely disappear.

Developer NCO Source Documentation

Automated source documentation, created by the
Doxygen tool is available.
Some developers find this documentation helpful, as it can clarify code and data
relationships in the code.

If pre-built executables do not satisfy you (e.g., are out-of-date)
and you want the latest, greatest features, then the first steps to
build (i.e., compile, for the most part) NCO from source code are to
install the prerequisites:
ANTLR version 2.7.7 (like this onenot version 3.x or 4.x!) (required for ncap2),
GSL (desirable for ncap2),
netCDF (absolutely required),
OPeNDAP (enables network transparency), and
UDUnits (allows dimensional unit transformations).
If possible, install this software stack from pre-built binaries
(commands to do so on Debian and RPM systems are given just below).
Failing that (e.g., lack of root access, or systems without packages
such as AIX), build these all with the same compiler and switches.
Recent versions of netCDF automatically build OPeNDAP and UDUnits.
NCO is mostly written in C99, and although you may mix and
match compilers, this is often difficult in practice and is not recommended.
The exception is ncap2 which is written in C++.
ANTLR, OPeNDAP, and NCO must be built with the same C++ compiler
to properly resolve the C++ name-mangling.
NCO does not yet support newer ANTLR versions because the
ANTLR 3.x and 4.x C++ interfaces are incomplete.

For the reasons explained above (compiler compatibility) install as
much pre-requisite and optional software as possible from pre-compiled
packages.
This is easy on modern package-oriented OSs.
For Debian-based systems:

For Windows with Cygwin, select and install the following packages with Cygwin Setup:

curl # DAP and wget pre-req

expat # expat XML parser for UDUnits

gsl # GSL

hdf5 # HDF

netcdf # netCDF

udunits2 # UDUnits

As of 20131101 there is no Cygwin package for Antlr, and the netcdf package does not yet support DAP.
Users should instead first download and install the Antlr found here.

Once you have installed the pre-requisites as shown above, you may
then build the latest stable NCO and install it in,
e.g., /usr/local with:

wget http://dust.ess.uci.edu/nco/nco.tar.gz

tar xvzf nco.tar.gz

cd nco-4.4.8

./configure --prefix=/usr/local

make

sudo make install

export PATH=/usr/local/bin\:${PATH}

export LD_LIBRARY_PATH=/usr/local/lib\:${LD_LIBRARY_PATH}

Please post questions about building or installing NCO to the
list
only after reading and attempting to follow these instructions.
To indicate you have done this, include the word “bonobo”
in the first sentence of your post.
Yes, “bonobo”.
Otherwise we will likely redirect you here.
For more sophisticated build/install options, see the next section.

Using NCO at UCI, NCAR, and other High Performance Computing Centers (HPCCs)

UC Irvine and NCAR users may find pre-built,
almost up-to-date NCO executables in the following locations.
Users at NCAR/NWSC should first try the CISL-supported
executables with module load nco.
The unsatisfied or adventurous may try my personal executables which
are built from the “main trunk” of NCO, not a tagged
version, and therefore may behave slightly differently.

NCAR CISL yellowstone.ucar.edu (Linux 2.6.x): ~zender/bin/LINUXAMD64

NCAR CISL mirage0.ucar.edu (Linux 2.6.x): ~zender/bin/LINUXAMD64

ORNL Rhea (Linux 2.6.x): ~zender/bin/LINUXAMD64

UCI ESS greenplanet.ps.uci.edu (Linux 2.6.x): ~zender/bin/LINUX

UCI ESS dust.ess.uci.edu (Linux 3.2.x): ~zender/bin/LINUXAMD64

Known Problems from 2013 (version 4.2.4) Onwards

Recent Generic Run-time Problems:

netCDF4 Strided Hyperslab bug:
Multiple users complain that access to strided hyperslabs of
netCDF4 datasets is orders of magnitude slower than expected.
This occurs with NCO and also with related software like NCL.
The cause appears to be that all recent versions of netCDF up
to 4.3.3 access strided hyperslabs via an algorithm
(in nc_get_vars()) that becomes unwieldy and error-prone
for large datasets.
We developed and implemented a transparent workaround (that avoids
the troublesome algorithm) for the most common case which is where
strides are taken in only one dimension, e.g., -d time,0,,12.
With the workaround introduced in NCO 4.4.6, strided access to
netCDF4 datasets now completes in nearly the same amount of time as
non-strided.
This workaround works transparently with any version of netCDF.
We are not yet sure that we have fully and correctly diagnosed the
cause nor that our workaround is always effective.
Comments welcome. Updates will be posted right here.

netCDF4 Renaming bugs:
Unforunately from 2007–present (February, 2015) the netCDF
library (versions 4.0.0–4.3.3-rc3) contained bugs or limitations
that prevent ncrename (and other netCDF4-based software)
from correctly renaming coordinate variables, dimensions, groups,
and attributes in netCDF4 files.
(To our knowledge the netCDF library calls for renaming always work
well on netCDF3 files so one workaround to netCDF4 bugs is convert to
netCDF3, rename, then convert back).
A summary of renaming limitations associated with particular versions
of netCDF4 is maintained in the online manual
here.
Important updates will also be posted here on the homepage.

Recent Operator-specific Run-time Problems:

Minimization/maximization of packed variables bug:
Versions 4.3.y—4.4.5 of ncwa
incorrectly handled packed variables during these operations.
The two workarounds are to unpack first or to perform the
statistics in single precision with the --flt option.
The solution is to upgrade to NCO 4.4.6.

Promoting packed records with missing values bug:
Versions 4.3.X—4.4.5 of ncra could produce (wildly)
inaccurate statistics when promoting (e.g., to double- from
single-precision) variables that are packed and that have missing
values.
The two workarounds are to unpack first or to perform the
statistics in single precision with the --flt option.
The solution is to upgrade to NCO 4.4.6.

Chunking while hyperslabbing bug:
Versions 4.3.X—4.4.4 of most operators could send incorrect
chunking requests to the netCDF library, resulting in failures.
This occurred only while simultaneously hyperslabbing.
The solution is to upgrade to NCO 4.4.5.

ncwa mask condition bug:
All versions through 4.4.3 of ncwa could return incorrect
mask values for negative numbers.
Thanks to Keith Lindsay for report, and to Henry Butowsky for fix.
Prior to this fix, the ncwa lexer would drop the negative
sign, if any, from the comparators appearing in the mask condition,
e.g., ncwa --mask_condition "lat < -20" was parsed as
"lat < 20" not "lat < -20".
Hence, users of ncwa --mask_condition (or of -B)
should upgrade. NB: The -m -M -T form of ncwa
masking is/was not buggy.
Thus the workaround is to use the -m -M -T form
of ncwa masking, while the long-term solution is to upgrade
to NCO 4.4.4+.

ncra, ncea, and ncrcat file close bug:
Versions 4.3.9—4.4.0 of ncra, ncea,
and ncrcat failed to correctly close and optionally remove
input files.
This could cause NCO to exceed system limits on the maximum number of
open files when hundreds-to-thousands of input files were specified
per NCO invocation.
The exact failure point is OS-dependent (NCO commands on Mac OS X 10.9
would fail when processing more than 256 files at a time).
This is embarassing because NCO has always been designed to work with
arbitrary numbers of input files and we want power users to be
comfortable running it on hundreds of thousands of input files.
The workaround is to avoid versions 4.3.9—4.4.0, while the
long-term solution is to upgrade to NCO 4.4.1+.

ncra MRO missing value bug:
Versions 4.3.6—4.3.9 of ncra could treat missing values
incorrectly during double-precision arithmetic.
A symptom was that missing values could be replaced by strange numbers
like, well, infinity or zero.
This mainly affects ncra in MRO (multi-record output) mode,
and the symptoms should be noticeable.
The workaround is to run the affected versions of ncra using the
--flt switch, so that single-precision floating point numbers
are not promoted prior to arithmetic.
The solution is to upgrade to NCO 4.4.0+.

ncwa hyperslabbing while averaging bug:
Versions 4.3.3—4.3.5 of ncwa could return incorrect
answers when user-specified hyperslabs were simultaneously extracted.
In such cases, hyperslab limits were not consistently applied.
This could produce incorrect answers that look correct.
This bug only affected hyperslabbed statistics (those produced
by simultaneously invoking -a and -d switches);
“global averages” were unaffected.
We urge all ncwa users to upgrade to NCO 4.3.6+.

ncpdq unpacking bug with auxiliary coordinates:
Versions 4.3.2–4.3.3 of ncpdq did not correctly
store unpacked variables.
These versions unpacked (when specified with -U or -P
upk) the values, but inadvertently stored their original packing
attributes with the unpacked values.
This would lead further operators to assume that the values were still
packed.
Hence consecutive operations could lead to incorrect values.
Fixed in version 4.3.4.
All ncpdq users are encouraged to upgrade.
NB: this bug did not affect the other arithmetic operators which
unpack data prior to arithmetic.