Using the DataFabric

From BeSTGRID

The BeSTGRID DataFabric can be accessed in several different ways - each may be suitable for a different type of users or for a different scenario. This page provides a guide to the most common ways for accessing the DataFabric.

The primary means of accessing the DataFabric are:

Browsing the DataFabric via a web browser

Suitable for casual users and for browsing existing collections

Mounting the DataFabric as a filesystem via webDAV

Suitable for more involved users, for uploading larger collections of files, and for accessing the files on the DataFabric directly from applications.

Accessing the DataFabric directly via the iRODS protocol with the iDrop GUI client

Suitable for users requiring a convenient GUI interface, requiring high performance transfers but not needing the convenience of connecting the DataFabric as a remote drive

Accessing the DataFabric directly via the iRODS protocol with iCommands

For the most involved users, who need the most transfer performance they can get - or who need direct access to the iRODS advanced features (access control, metadata, ...)

For each of these scenarios, the exact use may still differ depending on the authentication mechanism used. The sections below describe how to start using the DataFabric for each of these scenarios.

The Data Fabric uses a system based on Access Control Lists (ACLs) to control what each user (or group of users) can do with each file or directory. These ACLs are a property of the storage system that is used to store the files in the Data Fabric (irods), and therefore they are independent of the method that is used to access the files (web interface, WebDAV, iDrop, icommands). Modification of ACLs is possible with some of these access methods, but is unavailable with others. If the method you are using to access your files in the Data Fabric does not support ACL modification, use web access if you need to change permissions.

When you store a file or create a directory in the data fabric, you are the owner and have full control - you can read, write or delete it and, by default, no one else can. You can give access to other users or groups, either just read access, or read and write, or full ownership. You can only give (or remove) access to others if you own the file yourself. But if you give 'own' to someone else, they can also give (and remove) access to others.

If you are changing the access rights to a directory, then the access rights to the files in this directory will be changed. For example, you can give write access to a user or group so they can store files into one of your directories. As normally configured, all users can read all directories (i.e. see the names of files within these directories).
Directories also have an 'inherit permissions' attribute that can be unset or set. When directories have this attribute set, new files and subdirectories added to the directory inherit the access permissions (ACLs) of the parent directory.

If you accidentally remove your own access from a file or directory you will not be able to read it, but you can restore access by changing the permissions to give yourself access again, because for the data fabric you are still the owner.

For users who do have a login at an institution that is part of Tuakiri, the New Zealand Access Federation, their Tuakiri login is the easiest way to access the DataFabric with a browser.

All New Zealand Universities and CRIs are members of Tuakiri. Please see the Tuakiri Subscriber List to see whether your instituion has completed the work of linking into Tuakiri. Users not affiliated with a Tuakiri member institutions but collaborating on projects where the PI is affiliated with Tuakiri member institution can request an account at the Tuakiri VirtualHome IdP. Please contact the Tuakiri support at support at tuakiri.ac.nz to request an account for your collaborators.

For users who do have a grid certificate, it may be the easiest to use their grid certificate as their identity on the DataFabric. They would delegate their credential into the MyProxy server, protecting the copy of the certificate with a username and password, and then login to the DataFabric with the MyProxy username and password.

The DataFabric will automatically create an account for each user on the first access. The account will be linked either with the Distinguished Name from the certificate or with the SharedToken received in the Shibboleth login.

For users who have both a Grid certificate and a Shibboleth login, it may be useful to link their two identities together - instead of having two separate DataFabric (iRODS) accounts. Please send a request to help@bestgrid.org to link your DataFabric account with a DN or a SharedToken from your other identity.

In the request, please include the following information:

The full DN included in your certificate - it is displayed on first page in Grix, and it can be also obtained with

The DataFabric comes with a webDAV interface, available at the same URL as the DataFabric web interface. The webDAV interface can be mounted into most current operating systems and desktop environments (Windows, Mac, Linux, POSIX). In most cases, the desktop environment already comes with built-in support for mounting a webDAV URL, but it may pay off to install additional tools - which can provide a more efficient and more reliable way of accessing the DataFabric.

Note that in the past, users were asked to create a MyProxy login for use with webDAV. This has been replaced by using the DataFabric account directly. For more (historic) information on the MyProxy usernames, please see the Creating a MyProxy login section below.

The actual instructions to mount the DataFabric vary across operating systems and desktop environments. The most common cases are below.

When accessing a project that has been made *anonymously* accessible, and the intention is to only access the project with the permissions of the anonymous user (typically read-only), it is *not necessary* to get a username and password. In that case, use the URL to the project home directory (example: https://df.bestgrid.org/BeSTGRID/home/GeoFabric) and when prompted for a username and password, either leave it blank, or enter "irods\anonymous" as the username and anything as the password. Otherwise proceed as documented below.

Please note that while this is the easiest solution to get going, the Windows XP *built-in webDAV client has severe limitations*, in particular with files larger than 2GB (it cannot read a directory if it contains a file larger than 2GB). We strongly recommend using one of the alternative solutions, in particular BitKinex

On Windows Vista, it is necessary to use external tools for mounting a webDAV URL - and Windows XP users may also get additional performance.

The tools available are NetDrive, WebDrive, and BitKinex. Read a brief comparison and review of these clients here, note that only BitKinex has shown to reliably transfer files greater than 2GB in size.

Inspector application that displays file information when requested from the Finder has a bug that prevents it displaying correct on-disk size for WebDAV directories. Even though the total directory size is reported correctly, Size on Disk returns very high values, unrelated to the actual folder size. This bug is known to the Apple developers. If more exact information is required, a du command can be used from the command line:

Open Temrinal

type the command:

du -k /Volumes/home/

The result would be the list of directories in the BeSTGRID home directory with their sizes and the total amount taken by the directories.

iDrop is a GUI tool for accessing the DataFabric directly via the iRods interface. iDrop provides better performance then the webDAV interface and is suitable for users who need to get better performance on transfering large files but can forgo the convience of connecting the DataFabric as a remote drive.

iDrop is a Java application that launches via a JavaWebStart link. You need to have Java 1.6 or higher installed - after that, you can just click on the webstart link to start iDrop for the first time. This should install an icon on your desktop - next time, you can use either this icon or the original link to start iDrop (and still get any updates released to the web application).

Because webDAV cannot handle a Shibboleth login, users who use a Shibboleth login on the web interface need to get a MyProxy username and password to access the webDAV interface.

To create a MyProxy login from a Shibboleth login, one needs to get a SLCS certificate based on the Shibboleth login, upload the certificate into MyProxy (choosing a username and password) and then login with the MyProxy username and password.

Unfortunately, this has to be repeated at least every 10 days - the lifetime of the SLCS certificates.

It is possible to set a password directly associated with a DataFabric (iRODS) account. Doing so is necessary for using clients like iDrop that talk directly to the iRODS server. And having an iRODS password is also an alternative to setting up a MyProxy login to use the DataFabric with a username and password (which has been necessary to access it via webDAV).

The Data Fabric can be accessed from the Globus (formerly Globus Online) online file transfer service. Globus offers Single Sign On to the Data Fabric for holders of an account in the Tuakiri Access Federation.

To make use of the Tuakiri Single Sign On, click 'Log In' on the Globus page, then 'alternate login', choose 'Tuakiri', then log in with your home institution. After a successful login, go to the 'Transfer Data' page. There, enter the name of the Data Fabric node you wish to connect to into one of the 'Endpoint' boxes. The available nodes are: