Hi , I have some sample date like this 1,prasad,Newyork 2,Tarak,Mexico I want to load this data in to hive table using partition using current date and when I load this data tomorrow again it should be partitioned by using tomorrow's date. is this possible to achieve this in hive.????
... View more

Hi I have the data in the data file as shown below 7373743343333444. 7373743343333432. This data should be converted to decimal values and should be in a position of 8.7 where 8 are the digits before decimal and 7 are the digits after decimal. I am trying to read the data file like below. val readDataFile = Initialize.spark.read.format("com.databricks.spark.csv").option("header", "true").option("delimiter", "|").schema(***SCHEMA*****).load(****DATA FILE PATH******) I have tried this val changed = dataFileWithSchema.withColumn("COLUMN NAME ", dataFileWithSchema.col("COLUMN NAME ").cast(new DecimalType(38,3))) println(changed.show(5)) but it gives me only the zeros like this 7373743343333444.0000 But I want digits How can I achieve this ? can you please help me?
... View more

Hi guys, I have a table in hbase like shown below: hbase(main):005:0> scan 'glookups' ROW COLUMN+CELL UDS.WT_PRD_MD.MR column=lk:lval, timestamp=1526974749860, value=Denver UDS.WT_PRD_MD.MR column=lk:val, timestamp=1526974739603, value=DN 1 row(s) in 0.0230 seconds now I wanted it to map to hive table and the script is shown below: create
EXTERNAL TABLE global_lookups(key String,val String,lvl String) STORED
BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH
SERDEPROPERTIES ("hbase.columns.mapping"=":key,lk:lvl,lk:val")TBLPROPERTIES("hbase.table.name"="EOS:globallookups"); which works fine .But my scenario is bit different I want the row key to be splitted and stored in different columns in hive for example if this my row key UDS.WT_PRD_MD.MR I want UDS should be in one column WT_PRD_MD in one column and the value of lk:lval in one column and the value of lk:val in another column I wrote the following script but it's not working create EXTERNAL TABLE global_lookups(sorname String,fieldname String,fieldCode String,fieldDescription String) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping"=":key.subString(0,3),:key.subString(5,14),lk:lvl,lk:val")TBLPROPERTIES("hbase.table.name"="EOS:globallookups"); How can I achieve this ???
... View more

Hi guys, I have a dataframe as show below. +------+----+ | col1|col2|col3 +------+----+ |abc| c|h |abc| c|h |abc| c|h +------+----+ for the col2 I have a value c as shown in the above picture. Now for all the c values should be replaced with cat and h which is the value of col3 should be replaced with hat. I can achieve this easily with the below statement dataframe.withColumn("col2", when(col("col2").equalTo("c"), "cat").otherwise(col("col2") ) but I want to have the original values also . Like say my final dataframe should be looked as shown below +------+----+----+-----+----+ | col1|col2|col3|col4|col5 | +------+----+----+-----+----+ |abc| c|cat|h|hat | |abc| c|cat|h|hat | |abc| c|cat|h|hat | +------+----+---+-----------+ How can I achieve this ??? Any help....:)
... View more

Hi guys, I have table employer like below in hbase 1 column=personal_data:age, timestamp=1472494356226, value=28 1 column=personal_data:name, timestamp=1472494329458, value=Kishore 2 column=personal_data:age, timestamp=1472494969330, value=35 2 column=personal_data:name, timestamp=1472494969330, value=Gopi I have another table Department like below in hbase 1 column=personal_data:age, timestamp=1472494356333, value=28 1 column=personal_data:department, timestamp=1472494329433, value=Admin 2 column=personal_data:age, timestamp=1472494969345, value=35 2 column=personal_data:deparment, timestamp=1472494962340, value=IT Can we join these two tables in hbase without using spark-scala and without loading this in to hive (since we can join these loading in to hive) I want to join these two tables in hbase only. Can we achieve this ?????? Any suggestions would be appreciated.
... View more

Hi guys, In sql I have a case statement like below: select col1, (case when col2=28 then true else false) from employer which returns as follows col1 value, True col2 Value,False Do we have the same in hbase for example if the value is found then it should return true.If the value is not found it should return false. Can we achieve this HBASE query? Regards
... View more

hi, I am trying to insert my dataframe using orc and bzip2 compression but it is throwing me the error java.lang.IllegalArgumentException: Codec [bzip2] is not available. Available codecs are uncompressed, lzo, snappy, zlib, none.
at org.apache.spark.sql.hive.orc.OrcOptions.<init>(OrcOptions.scala:49)
at org.apache.spark.sql.hive.orc.OrcOptions.<init>(OrcOptions.scala:32)
at org My code is fields.write.format("orc").option("compression","bzip2").saveAsTable("prasadtest.descargatest")
I am using spark 2 for this.
... View more