1. Introduction

Audience: This document is targeted toward
those who wish to use or share information based on the Darwin Core terms
using text files. It provides technical details on how to construct these
files and complementary metadata files that describe their content.

This document provides guidelines for formatting and sharing Darwin Core
terms [TERMS] in fielded text formats,
such as one or more comma separated value (CSV) files. Data conforming to the
Simple Darwin Core [SIMPLEDWC] (CSV
format and having the first row include Darwin Core standard term names) can
be shared in a single file, while a non-standard text file can be understood
using an [XML] metafile to describe its
contents and formatting.
More complex structure can be shared in multiple related files. The
description of content and relationships between files can be achieved using
the metafile. This guideline makes recommendations for the simple case of a
core file, upon which Darwin Core records are based, and
extensions that are linked to records in that core file.
Specifically, extension records have a many-to-one relationship with
records in the core file. For example, a core file might contain specimen
records, with one specimen per row in the file, while an extension file
contains one or more identifications for those specimens, with one
identification per row in the extension file, and with an identifier to the
specimen for each identification row. This example would allow many
identifications to be associated with each specimen.

1.1 Simple Example Metafile Content

A simple comma separated values
(CSV) data file with the following content:

1.2 XML versus Fielded Text

Many resources exist on the web describing the advantages of Extensible
Markup Language [XML] over less
structured content such as fielded text. The Darwin Core Text Guide
(this document) is not meant to promote the use of fielded text over
XML for data exchange, but rather to provide recommendations for how to
handle such data files when necessary.
Two scenarios that might benefit from the use of fielded text
are:

The transfer of large numbers of Darwin Core records and related data
from one database to another. Typically databases are very efficient at
exporting and importing comma separated text files.

The description of legacy data existing in a fielded text
format, such that it might be automatically understood and loaded into
another system. It could be that this system would then serve the data in
another format such as XML.

2. Metafile Content

The text metafile schema [TEXTSCHEMA] provides technical details
for the structure of a metafile by defining the elements and attributes
required to describe the contents and relationships between text files. These
elements and attributes, with descriptions and specifications for their use
in a metafile, are described in the following table.

2.1 The <archive> element

The <archive> element is the
container for the list of related files (one core and zero or more
extensions). The <archive> element has just one attribute,
metadata.

Attributes

Attribute

Description

Required

Default

metadata

Contains a qualified Uniform Resource Locator (URL) defining the
location of a metadata description of the entire archive. The format of
the metadata is not prescribed, but a standardized format such as
Ecological Metadata Language (EML), Federal Geographic Data Committee
(FGDC), or ISO 19115 family is recommended.

An <archive> must contain exactly one <core> element, representing the data entity
(the actual file and its column header mappings to Darwin Core terms)
upon which records are based.
If extensions are being used, each record in the core data must have
a unique identifier. The field for this identifier must be specified
in an explicit <id> field in order to associate extension
records with the core record.

An <archive> may define zero or more <extension> elements, each representing an
individual extension entity directly related to the core. In addition
to the general file attributes described below, every extension
entity must have an explicit <coreId> field to relate the
extension record to a row in the core entity. The extension itself
does not have to have a unique ID field and many rows can point to
the same core record.

A Unified Resource Identifier (URI) for the term identifying the
class of data represented by each row, for example,
http://rs.tdwg.org/dwc/terms/Occurrence for Occurrence records or
http://rs.tdwg.org/dwc/terms/Taxon for Taxon records. Additional
classes may be referenced by URI and defined outside the Darwin Core
specification. The row type defaults to the ambiguous
SimpleDarwinRecord. For convenience the URIs for classes defined by
the Darwin Core are listed below:

Simple Darwin Record

http://rs.tdwg.org/dwc/xsd/simpledarwincore/SimpleDarwinRecord

Occurrence

http://rs.tdwg.org/dwc/terms/Occurrence

Event

http://rs.tdwg.org/dwc/terms/Event

Location

http://purl.org/dc/terms/Location

GeologicalContext

http://purl.org/dc/terms/GeologicalContext

Identification

http://rs.tdwg.org/dwc/terms/Identification

Taxon

http://rs.tdwg.org/dwc/terms/Taxon

ResourceRelationship

http://rs.tdwg.org/dwc/terms/ResourceRelationship

MeasurementOrFact

http://rs.tdwg.org/dwc/terms/MeasurementOrFact

✓

http://rs.tdwg.org/dwc/xsd/simpledarwincore/SimpleDarwinRecord

fieldsTerminatedBy

Specifies the delimiter between fields. Typical values might be ","
or "\t" for CSV or Tab files respectively.

,

linesTerminatedBy

Specifies the row separator character.

\n

fieldsEnclosedBy

Specifies the character used to enclose (mark the start and end of)
each field. CSV files frequently use the double quote character
("), but the default is no enclosing character. Note that a comma
separated value file that has commas within the content of any field
must have an enclosing character.

"

encoding

Specifies the character
encoding for the data file. The encoding is extremely important,
but often ignored. The most frequently used encodings are:

UTF-8

8-bit Unicode Transformation Format.

UTF-16

16-bit Unicode Transformation Format.

ISO-8859-1

Commonly known as Latin-1 and a common default on systems
configured for a single western European language.

Windows-1252

Commonly known as WinLatin and a common default of legacy
versions of Microsoft Windows based operating systems.

UTF-8

ignoreHeaderLines

Specifies the number lines to ignore from the beginning of the
file. This can be used to ignore files with column headings or preamble
comments for example.

0

dateFormat

When verbatim dates are consistent in format, this field can be used
to indicate the format represented. It is recommended to use the
date, dateTime and time for field formats wherever possible, but
where verbatim dates are required, a format may be specified here.
This should be considered a 'hint' for consumers. It is recommended
that consumers support the minimum combinations of DD MM and YYYY
with the separators / and -. Examples:

If extensions are being used, the <core>
must contain an <id> element that indicates the identifier for
a record.

<coreId>

If extensions are being used, the <extension> element must contain a
<coreId> element that indicates the column in the extension
file that contains the core record identifier (the matching
<id> in the core file).

The files element must contain one or more <location> elements, each
defining where a file resides. Each core or extension entity can be composed
from one or more files. If an entity has data in more than one file, use the
<location> element multiple times, once for each file that makes up the
entity.

Elements

Element

Description

location

Specifies the location of the file being described, which may take
either of the following forms:

A web accessible URL such as
"http://www.gbif.org/data/specimen.csv" or
"ftp://ftp.gbif.org/tim/specimen.txt".

A filepath relative to the location of the metafile such as
"specimen.txt","./specimen.txt", "data/specimen.txt".

The field element is used to specify the location and content
of data within a file. There must be one field element for every term being
shared for the entity, whether explicitly or through the use of a default
value for all rows in the file.

Attributes

Attribute

Description

Required

Default

index

Specifies the position of the column in the row. The first column
has an index of 0, the second column 1, etc. If no column index is
specified, then the term and the default may be used to define a
constant value for all rows.

term

A Unified Resource Identifier (URI) for the term represented by
this field. For example, a field containing the scientific name would
have term="http://rs.tdwg.org/dwc/terms/scientificName".
Terms outside of the Darwin Core specification may be used, such as
those from the Dublin Core Metadata Initative, for example,
dcterms:modified would be
term="http://purl.org/dc/terms/modified".

✓

default

Specifies value to use if one is not supplied for the field in a
given row. If no index is supplied, the default can be used to define a
constant for all rows for a field that is not in the data file.

vocabulary

A Unified Resource Identifier (URI) for a vocabulary that the
source values for this field are based on. The URI ideally should
resolve to some machine readable definition like SKOS, RDF or at least
some simple text or html file often found for ISO or RFC standards. For
example http://rs.gbif.org/vocabulary/gbif/nomenclatural_code.xml,
http://www.ietf.org/rfc/rfc3066.txt or
http://www.iso.org/iso/list-en1-semic-3.txt .

3. Implementation Guide

3.1 Extension example

The following example illustrates the use of
extensions. In this example there are three files in the archive, all of
which are located in the same directory as the metafile. The whales.txt file
acts as a core file of Taxon records. The whales.txt file is extended by two
other files, types.txt and distribution.txt. The types.txt file contains
records of a type specified in an external definition at
http://http://rs.gbif.org/terms/1.0/Types and consists of Dublin Core and
Darwin Core terms, while the distribution.txt file contains records of a type
specified at http://http://rs.gbif.org/terms/1.0/Distribution and consists of
Darwin Core terms plus an additional term for threatStatus. Both extension
files are related to the core file by the taxonNameID fields. Presumably,
this archive contains information about whale species, type specimen records
for those species, and lists of countries and the threat status for those
species.

4. Database Example

4.1 MySQL

It is very easy to produce fielded text using the
SELECT INTO outfile command from MySQL. The encoding of the
resulting file will depend on the server variables and collations used, and
might need to be modified before the operation is done. Note that MySQL will
export NULL values as \N by default. Use the IFNULL() function as shown in
the following example to avoid this.