As data set size and complexity grows, the problem of managing and
analyzing scientific data sets grows. The ASC SDM project develops
tools that assist scientists in understanding and organizing their data
and information. The project has a tool development effort centered on
metadata techniques for managing and simplifying data access, and a
research effort. The research project, known as Sapphire, focuses on
applying feature extraction, data mining, and pattern recognition
techniques to large scale data.

The SDM project aims to deliver services for creating, storing,
retrieving, and searching data through an integrated environment that
supports end users' work as much within the context of that work as
possible. For example, a calculation summary tool ties together
simulation inputs and results and reduces the use of arcane commands,
lengthy and obscure path names, and time focused on transferring data.
A mining tool creates metadata behind the scenes that later can be used
in searches or presentation of the metadata. Graphical directory
browsers can launch end users' applications and can be configured to
recall lengthy command sequences.

Software

Metadata Tools

Eminer—A utility that creates metadata by mining through
local and remote file systems.

Hopper—A general
purpose directory browsing and file transfer tool, with the ability to
communicate via FTP, SSH, and other protocols.