I can see two different flavors of class:org.apache.hadoop.hive.serde2.ColumnProjectionUtils. One is available in hive-serde-0.11.0.1.3.0.0-107.jar and the other in hive-exec-0.11.0.1.3.0.0-107.jar. I have both the jars in my client clasppath and end up with the problem of: java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.ColumnProjectionUtils.setReadColumnIDs(Lorg/apache/hadoop/conf/Configuration;Ljava/util/ArrayList;)V

Though I couldn’t find this problem in Apache Hive-11 release(Though both the jars have duplicate copy of class).

It will helpful to know whether this change is intentional or a bug in hdp release?
I can find a bug in open-source related to this:

Hi Zeke,
The org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumnIDs(Lorg/apache/hadoop/conf/Configuration;Ljava/util/List;) is in hive 0.10 and later’s hive-serde jar. Do you have it in your class path? or you have an earlier version of hive-serde?
Thanks,
Yi
...

The org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumnIDs(Lorg/apache/hadoop/conf/Configuration;Ljava/util/List;) is in hive 0.10 and later’s hive-serde jar. Do you have it in your class path? or you have an earlier version of hive-serde?

Hey guys,
I’m having similar issue which seems happening during mapreduce. Pasted call stack below.
How can I change the order of jars in hadoop classpath?
2013-09-24 20:45:03,205 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumnIDs(Lorg/apache/hadoop/conf/Configuration;Ljava/util/List;)V
at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:427)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java...

I’m having similar issue which seems happening during mapreduce. Pasted call stack below.
How can I change the order of jars in hadoop classpath?

2013-09-24 20:45:03,205 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumnIDs(Lorg/apache/hadoop/conf/Configuration;Ljava/util/List;)V
at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:427)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:190)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:411)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

Subroto,
Between Hive 9 and Hive 10 the signature of this method changed to use a List instead of an ArrayList. Based on the traceback you have something is trying to use an ArrayList rather than a List. Do you maybe have some old UDFs or custom JARs that might be trying to use an old interface? I don’t think there was any intentional change by HWX that would cause this.
carter$ find ./ -name ColumnProjectionUtils.java | xargs grep setReadColumnIDs
…
.//branch-0.9/serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java: public static void setReadColumnIDs(Configura...

Between Hive 9 and Hive 10 the signature of this method changed to use a List instead of an ArrayList. Based on the traceback you have something is trying to use an ArrayList rather than a List. Do you maybe have some old UDFs or custom JARs that might be trying to use an old interface? I don’t think there was any intentional change by HWX that would cause this.

Hi Ted,
The content of the classes are not same but, they have same name and are in same package. Swapping the order might be a workaround.
Can you confirm whether this is bug from Distribution or not ?? If it is when can we expect it to be resolved?
Cheers,
Subroto Sanyal
...

The content of the classes are not same but, they have same name and are in same package. Swapping the order might be a workaround.
Can you confirm whether this is bug from Distribution or not ?? If it is when can we expect it to be resolved?

Hi Subroto,
If you look at the code for these two copies are they actually two different classes? If so, is there a way you could try swapping the order of the jars on the classpath? Since java will use the version that it finds first.
Thanks,
Ted.
...

If you look at the code for these two copies are they actually two different classes? If so, is there a way you could try swapping the order of the jars on the classpath? Since java will use the version that it finds first.