Tests added in package org.apache.hadoop.fs , for the FileSystem API, in 2.0:

Of those test types, we have "HCFS" tests, which are generic, and other tests, which either Mock a FileSystem implementation (MOCK), are abstract in their implementation (ABSTRACT), test the LocalFileSystem (LOCAL), or test HDFS specifically (HDFS).

AbstractFileSystem Tests

These are new to Hadoop 2.X. Many of these tests define a protected parameter which for the FileContext which can be overriden by extending classes, making them well suited for HCFS testing in 2.0 AbstractFileSystem implementations.

Sub packages, specific implementation tests

Many of these tests are decoupled from swift semantics by inheriting from the base class "SwiftFileSystemBaseTest", and thus, they may be adoptable as HCFS tests readily. In addition to generic tests, the swift-file-system test suite also includes Swift specific functional tests (for example, reconciliation of swift object paths with files ). These are taken from Steve Loughran's github Fork of the swift-file-system repository.

package with extended version of SwiftFileSystem specific for testing large uploads.

Conclusions

Hadoop 2.X adds several new tests and the FileContext related tests provide extension points for other HCFS tests.

This is an initial analysis. Comparison of missing tests would be much appreciated.

We have used the JDiff tool to diff between 1.X (1.1) and the 2.X branches (2.1-beta) and summarized classes by inspection in order to identify the ability to generalize these tests, but this may be prone to error. Comments and/or edits are welcome.

Meanwhile, other frameworks (i.e. swift) have implemented their own tests which in many cases can be abstracted into HCFS tests which are directly applicable to any FileSystem implementation.

More work will need to be done to normalize and integrate the corpus of existing tests into a single, robust, implementation independent testing suite that the community can share.