Suppose block size set in configuration file at client side is 64MB,block size set in configuration file at name node side is 128MB and block size set in configuration file at datanode side is something else.Please advice, If the client is writing a file to hdfs,which property would be executed.

> Hi,>>> Suppose block size set in configuration file at client side is 64MB,> block size set in configuration file at name node side is 128MB and block> size set in configuration file at datanode side is something else.> Please advice, If the client is writing a file to hdfs,which property> would be executed.>> Thanks,> Shalish.>

> I Think the client side configuration will take effect.>> Shumin> On Jul 12, 2013 11:50 AM, "Shalish VJ" <[EMAIL PROTECTED]> wrote:>>> Hi,>>>>>> Suppose block size set in configuration file at client side is 64MB,>> block size set in configuration file at name node side is 128MB and block>> size set in configuration file at datanode side is something else.>> Please advice, If the client is writing a file to hdfs,which property>> would be executed.>>>> Thanks,>> Shalish.>>>-- Regards,Varun Kumar.P

Replication, block size, etc. are all per-file and pure clientsupplied properties. They either take their default from the clientconfig, or directly from an API argument override.

On Sun, Jul 14, 2013 at 4:14 PM, varun kumar <[EMAIL PROTECTED]> wrote:> What Shumin told is correct,hadoop configurations has been over written> through client application.>> We have faced similar type of issue,Where default replication factor was> mentioned 2 in hadoop configuration.But when when ever the client> application writes a files,it was having 3 copies in hadoop cluster.Later on> checking client application it's default replica factor has 3.>>> On Sun, Jul 14, 2013 at 4:51 AM, Shumin Guo <[EMAIL PROTECTED]> wrote:>>>> I Think the client side configuration will take effect.>>>> Shumin>>>> On Jul 12, 2013 11:50 AM, "Shalish VJ" <[EMAIL PROTECTED]> wrote:>>>>>> Hi,>>>>>>>>> Suppose block size set in configuration file at client side is 64MB,>>> block size set in configuration file at name node side is 128MB and block>>> size set in configuration file at datanode side is something else.>>> Please advice, If the client is writing a file to hdfs,which property>>> would be executed.>>>>>> Thanks,>>> Shalish.>>>>> --> Regards,> Varun Kumar.P

When client issues the command "hadoop fs -put local_sourcehdfs_destination", client contact the namenode that he wants to write andthen NameNode creates a blockID and ask three datanodes ( if replication onthe client side is set to 3) to host the replica. THis information is sentto client and then client starts writing the data on to the data nodes byforming pipe line..And then client write that much amount of data onto ablock which he has set.Regards,Som Shekhar Sharma+91-8197243810On Sun, Jul 14, 2013 at 4:24 PM, Harsh J <[EMAIL PROTECTED]> wrote:

The default block size is 64MB which is good at the client end. Make surethe same at your end also in conf. You can increase the size of each blockto 128MB or greater than that only thing you can see the processing will befast but at end there may be chances of losing data.

> Hi,>>> Suppose block size set in configuration file at client side is 64MB,> block size set in configuration file at name node side is 128MB and block> size set in configuration file at datanode side is something else.> Please advice, If the client is writing a file to hdfs,which property> would be executed.>> Thanks,> Shalish.>

> Shalish,>> The default block size is 64MB which is good at the client end. Make sure> the same at your end also in conf. You can increase the size of each block> to 128MB or greater than that only thing you can see the processing will be> fast but at end there may be chances of losing data.>> Thanks,> Kiran>>> On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <[EMAIL PROTECTED]> wrote:>>> Hi,>>>>>> Suppose block size set in configuration file at client side is 64MB,>> block size set in configuration file at name node side is 128MB and block>> size set in configuration file at datanode side is something else.>> Please advice, If the client is writing a file to hdfs,which property>> would be executed.>>>> Thanks,>> Shalish.>>>>

the conf that client running on will take effect. On Jul 13, 2013 4:42 PM, "Kiran Dangeti" <[EMAIL PROTECTED]> wrote:

Shalish,>>The default block size is 64MB which is good at the client end. Make sure the same at your end also in conf. You can increase the size of each block to 128MB or greater than that only thing you can see the processing will be fast but at end there may be chances of losing data. >>Thanks,>Kiran>>>>On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <[EMAIL PROTECTED]> wrote:>>Hi,>>>>>> Suppose block size set in configuration file at client side is 64MB,>>block size set in configuration file at name node side is 128MB and block size set in configuration file at datanode side is something else. >>Please advice, If the client is writing a file to hdfs,which property would be executed.>>>>Thanks,>>Shalish.>

The client side conf will take precedence. Further you can use the FileSystem API which can set the block size:create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize)

Cheers,Subroto Sanyal

On Jul 13, 2013, at 9:10 PM, Shalish VJ wrote:

> Hi,> > Ru sure?> or have u ever tried it out.> Pls advice.> > From: Azuryy Yu <[EMAIL PROTECTED]>> To: [EMAIL PROTECTED] > Sent: Saturday, July 13, 2013 3:19 PM> Subject: Re: Hadoop property precedence> > the conf that client running on will take effect.> On Jul 13, 2013 4:42 PM, "Kiran Dangeti" <[EMAIL PROTECTED]> wrote:> Shalish,> > The default block size is 64MB which is good at the client end. Make sure the same at your end also in conf. You can increase the size of each block to 128MB or greater than that only thing you can see the processing will be fast but at end there may be chances of losing data.> > Thanks,> Kiran> > > On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <[EMAIL PROTECTED]> wrote:> Hi,> > > Suppose block size set in configuration file at client side is 64MB,> block size set in configuration file at name node side is 128MB and block size set in configuration file at datanode side is something else.> Please advice, If the client is writing a file to hdfs,which property would be executed.> > Thanks,> Shalish.> > >

Hi, Suppose block size set in configuration file at client side is 64MB,block size set in configuration file at name node side is 128MB and block size set in configuration file at datanode side is something else.Please advice, If the client is writing a file to hdfs,which property would be executed.

> Hi,>>> Suppose block size set in configuration file at client side is 64MB,> block size set in configuration file at name node side is 128MB and block> size set in configuration file at datanode side is something else.> Please advice, If the client is writing a file to hdfs,which property> would be executed.>> Thanks,> Shalish.>

+

Raj K Singh 2013-07-12, 16:54

NEW: Monitor These Apps!

All projects made searchable here are trademarks of the Apache Software Foundation.
Service operated by Sematext