Details

The default blocksize property 'dfs.blocksize' now accepts unit symbols to be used instead of byte length. Values such as "10k", "128m", "1g" are now OK to provide instead of just no. of bytes as was before.

Description

Using "dfs.block.size=8388608" works
but "dfs.block.size=8mb" does not.

Allen Wittenauer
added a comment - 22/Jul/10 21:12 I thought all the units handling was built to support single letter units. (so m, not mb)
Also, pastebin.corp.yahoo.com doesn't appear to be available outside of Yahoo!. Please use an external service.

Harsh J
added a comment - 24/Dec/11 06:04 Some comments:
./hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c needs an update as well – both to dfs.blocksize rename and the method signature switch.
Found multiple issues with HttpFs and opened HDFS-2722 – as it could get more involved than this one's scope, need a separate patch for this.
Please update dfs.blocksize documentations (and also hdfs-default.xml) to indicate that users may supply k/m/g characters to indicate a block size.

FileSystem.getDefaultBlockSize() can't use DFSConfigKeys class because FileSystem is a class of hadoop-common project.
Also getDefaultBlockSize() hasn't been maintained for 2 years.
Should we open a new ticket to refactor getDefaultBlockSize() ?

Sho Shimauchi
added a comment - 24/Dec/11 13:43 FileSystem.getDefaultBlockSize() can't use DFSConfigKeys class because FileSystem is a class of hadoop-common project.
Also getDefaultBlockSize() hasn't been maintained for 2 years.
Should we open a new ticket to refactor getDefaultBlockSize() ?

Harsh J
added a comment - 24/Dec/11 16:40 Sho,
FileSystem instance underneath for HDFS operations is DistributedFileSystem, so it should be OK to use getDefaultBlockSize.
You even made the change in hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java that pertains to this. We should use it wherever is possible, and not repeat ourselves.

Thanks for the revised patch Sho - this looks good. Only a few further, mostly docs related, comments that pertain to HDFS:

For hdfs.c, would it be possible for you to also switch the blockSize and replication loading to use jFS's getDefaultBlockSize and getDefaultReplication calls instead of loading from Configuration? Would be a nice cleanup but if you want to pursue this on another JIRA please open a new one and we can do it there overall.

You can add more documentation notes to cluster_setup, explaining the most common prefixes (k, m, g) can be provided for block sizes instead of long values. Also worth it if you add the prefix list to the hdfs-default.xml docs.

Harsh J
added a comment - 27/Dec/11 08:53 Thanks for the revised patch Sho - this looks good. Only a few further, mostly docs related, comments that pertain to HDFS:
For hdfs.c, would it be possible for you to also switch the blockSize and replication loading to use jFS's getDefaultBlockSize and getDefaultReplication calls instead of loading from Configuration? Would be a nice cleanup but if you want to pursue this on another JIRA please open a new one and we can do it there overall.
You can add more documentation notes to cluster_setup, explaining the most common prefixes (k, m, g) can be provided for block sizes instead of long values. Also worth it if you add the prefix list to the hdfs-default.xml docs.

Hadoop QA
added a comment - 29/Dec/11 10:44 -1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12508830/hdfs-1314.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1744//console
This message is automatically generated.

Not sure why the patch failed. Perhaps its cause of the docs change in hadoop-common instead? Could you submit same patch without that change alone? I'll add back in later when committing (and will upload cumulative when am doing that).

Harsh J
added a comment - 29/Dec/11 11:00 Not sure why the patch failed. Perhaps its cause of the docs change in hadoop-common instead? Could you submit same patch without that change alone? I'll add back in later when committing (and will upload cumulative when am doing that).

Hadoop QA
added a comment - 04/Jan/12 09:14 -1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12509387/hdfs-1314.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1754//console
This message is automatically generated.