Description

LazyBinarySerDe will serialize the data in binary format while supporting LazyDeserialization.
This will be used as the SerDe for value between map and reduce, and also between different map-reduce jobs.

1/ Remove duplicated functions in TestLazyBinarySerDe which exist in TestBinarySortableSerDe already. By make those function public static in TestBinarySortableSerDe, TestLazyBinarySerDe shares the same functions.
2/ Fixed the copy constructors in all LazyBinaryPrimitive classes so that the copy constructor copies the value of the data as well.
3/ Cached ((ListObjectInspector)oi).getListElementObjectInspector() in LazyBinaryArray to a local variable.
4/ Added a private function in LazyBinarySerDe so that serializing a row and serializing a struct call the same function. Before, they were using different but quite similar code pieces.
5/ Simplified parsing code in LazyBinaryArray/LazyBinaryStruct/LazyBinaryMap. In particular, if valueIsNull[i] in LazyBinaryArray is true, then we don't need to update valueStart[i], valueLength[i] and lastElementEnd. Similar changes are done in LazyBinaryStruct and LazyBinaryMap.

Yuntao Jia
added a comment - 31/Jul/09 18:17 A newer patch, which incorporates most of Zheng's comments:
1/ Remove duplicated functions in TestLazyBinarySerDe which exist in TestBinarySortableSerDe already. By make those function public static in TestBinarySortableSerDe, TestLazyBinarySerDe shares the same functions.
2/ Fixed the copy constructors in all LazyBinaryPrimitive classes so that the copy constructor copies the value of the data as well.
3/ Cached ((ListObjectInspector)oi).getListElementObjectInspector() in LazyBinaryArray to a local variable.
4/ Added a private function in LazyBinarySerDe so that serializing a row and serializing a struct call the same function. Before, they were using different but quite similar code pieces.
5/ Simplified parsing code in LazyBinaryArray/LazyBinaryStruct/LazyBinaryMap. In particular, if valueIsNull [i] in LazyBinaryArray is true, then we don't need to update valueStart [i] , valueLength [i] and lastElementEnd. Similar changes are done in LazyBinaryStruct and LazyBinaryMap.
Thanks Zheng for the reviews.

Zheng Shao
added a comment - 31/Jul/09 00:52 @ HIVE-640 .2.patch:
LazyBinaryArray/LazyBinaryStruct/LazyBinaryMap: If valueIsNull [i] is true, valueStart [i] and valueLength [i] do not need to be set, lastElementByteEnd does not need to be updated.

@HIVE-640.2.patch:
LazyBinaryByte etc: The copy constructors should copy the value of the data as well. LazyBinaryString/LazyBinaryBoolean is already doing that but not others.
It's also OK to remove these copy constructors at all, since they are never used.

LazyBinaryArray.java:150: We should get ((ListObjectInspector)oi).getListElementObjectInspector() and save it to a local variable, so we don't need to call the function again and again.

LazyBinaryUtils.java:295: "Returns the lazy binary object inspector that can be used to translate an object of that typeInfo to a standard object type. " -> "Returns the lazy binary object inspector that can be used to inspect an lazy binary object of that typeInfo"

LazyBinaryString.java: The length of the string can be stored in a VInt instead of fixed size 4 bytes. This will help us save some space (especially for short strings).

LazyBinarySerDe.java: Remove the commented code (Line 464). It seems the only difference between serializing the row and serializing a struct is that we don't need the total number of bytes at the beginning. Can we add a parameter to serialize(Output byteStream,Object obj, ObjectInspector objInspector), and let serialize(Object o, OI oi) to call that directly?

Zheng Shao
added a comment - 31/Jul/09 00:42 @ HIVE-640 .2.patch:
LazyBinaryByte etc: The copy constructors should copy the value of the data as well. LazyBinaryString/LazyBinaryBoolean is already doing that but not others.
It's also OK to remove these copy constructors at all, since they are never used.
LazyBinaryArray.java:150: We should get ((ListObjectInspector)oi).getListElementObjectInspector() and save it to a local variable, so we don't need to call the function again and again.
LazyBinaryUtils.java:295: "Returns the lazy binary object inspector that can be used to translate an object of that typeInfo to a standard object type. " -> "Returns the lazy binary object inspector that can be used to inspect an lazy binary object of that typeInfo"
LazyBinaryString.java: The length of the string can be stored in a VInt instead of fixed size 4 bytes. This will help us save some space (especially for short strings).
LazyBinarySerDe.java: Remove the commented code (Line 464). It seems the only difference between serializing the row and serializing a struct is that we don't need the total number of bytes at the beginning. Can we add a parameter to serialize(Output byteStream,Object obj, ObjectInspector objInspector), and let serialize(Object o, OI oi) to call that directly?

TestLazyBinarySerDe:hexString: Can you make TestBinarySortableSerDe:hexString static and remove TestLazyBinarySerDe:hexString? In the future, we can move that function to a utility class, but for now, let's at least use the same function to avoid code replication.
There are several other functions that we can make the same change.

TestLazyBinarySerDe:testShorterSchemaDeserialization etc: Can you extract some common code and put into a function?

Zheng Shao
added a comment - 30/Jul/09 22:33 Haven't finished the review yet. Here are some of the comments:
@ HIVE-640 .2.patch:
TestLazyBinarySerDe:hexString: Can you make TestBinarySortableSerDe:hexString static and remove TestLazyBinarySerDe:hexString? In the future, we can move that function to a utility class, but for now, let's at least use the same function to avoid code replication.
There are several other functions that we can make the same change.
TestLazyBinarySerDe:testShorterSchemaDeserialization etc: Can you extract some common code and put into a function?

Second version patch, there are a few more changes.
1/ Fixed two bugs in ObjectInspectorUtils.java at Line 465 and 474. At both places, the second "loi1" should be "loi2".
2/ Removed the "isNull" tag in all LazyBinaryPrimitive classes. Because currently, primitive types are only initialized when they are not null, so the tag is not necessary.
3/ Changed LazyBinaryStruct so that it supports metadata changes. In particular, serialization data of an old table schema can be deserialized with a new table schema that has more fields added at the end. For example, if the data is serialized with "Table (key int, field int)", it can be deserialized with a new table schema "Table (key int, field int, value double)". The new added last field will be defaulted to null.
4/ Changed the default value of null bits from 1 to 0 to support the above feature 3.
5/ Improved the unit test to cover two more cases: tests LazyBinaryMap and test serialization and deserialization with different schemas.
6/ Fixed a few comments

Yuntao Jia
added a comment - 30/Jul/09 20:33 Second version patch, there are a few more changes.
1/ Fixed two bugs in ObjectInspectorUtils.java at Line 465 and 474. At both places, the second "loi1" should be "loi2".
2/ Removed the "isNull" tag in all LazyBinaryPrimitive classes. Because currently, primitive types are only initialized when they are not null, so the tag is not necessary.
3/ Changed LazyBinaryStruct so that it supports metadata changes. In particular, serialization data of an old table schema can be deserialized with a new table schema that has more fields added at the end. For example, if the data is serialized with "Table (key int, field int)", it can be deserialized with a new table schema "Table (key int, field int, value double)". The new added last field will be defaulted to null.
4/ Changed the default value of null bits from 1 to 0 to support the above feature 3.
5/ Improved the unit test to cover two more cases: tests LazyBinaryMap and test serialization and deserialization with different schemas.
6/ Fixed a few comments

We will also need a performance test.
In order to test the performance, just replace LazySimpleSerDe with LazyBinarySerDe in PlanUtils.getIntermediateFileTableDesc() and PlanUtils.getReduceValueTableDesc().

Zheng Shao
added a comment - 30/Jul/09 00:17 We will also need a performance test.
In order to test the performance, just replace LazySimpleSerDe with LazyBinarySerDe in PlanUtils.getIntermediateFileTableDesc() and PlanUtils.getReduceValueTableDesc().

Talked with Yuntao offline.
We want to make sure that LazyBinarySerDe can allow the simplest schema evolution - adding/deleting columns at the end of the table.

In order to do that, we need to modify LazyBinaryStruct a little bit. We need to write a single null bytes, then 8 fields, and then the next null byte. This makes sure if data with 8 fields are read by LazyBinarySerDe intitialized with 9 fields, we can still successfully deserialize the data (the missing fields will be null).

Zheng Shao
added a comment - 29/Jul/09 21:01 Talked with Yuntao offline.
We want to make sure that LazyBinarySerDe can allow the simplest schema evolution - adding/deleting columns at the end of the table.
In order to do that, we need to modify LazyBinaryStruct a little bit. We need to write a single null bytes, then 8 fields, and then the next null byte. This makes sure if data with 8 fields are read by LazyBinarySerDe intitialized with 9 fields, we can still successfully deserialize the data (the missing fields will be null).

2/ Added package "org.apache.hadoop.hive.serde2.lazybinary" which include the LazyBinary classes for all Hive supported data types, such as Boolean, Byte, Short, Integer, Long, Float, Double, String, Array, Struct and Map. It also include a LazyBinaryFactory class and a utility function class.

3/ Added package "org.apache.hadoop.hive.serde2.lazybinary.objectinspector" which include the object inspector classes for non-primitive data types, such as List, Struct and Map. It also includes an object inspector factory class. For primitive data types, such as Boolean, Byte, Short and so on, I used the corresponding writable object inspector classes. For instance, I used WritableBooleanObjectInspector for LazyBinaryBoolean.

4/ Added a unit test for LazyBinarySerDe. It tests serializatoin and deserialization of all above data types except LazyBinaryMap. I will include a test for that in the future.

Yuntao Jia
added a comment - 29/Jul/09 05:26 The first version patch. The major changes are:
1/ Added the LazyBinarySerDe class.
2/ Added package "org.apache.hadoop.hive.serde2.lazybinary" which include the LazyBinary classes for all Hive supported data types, such as Boolean, Byte, Short, Integer, Long, Float, Double, String, Array, Struct and Map. It also include a LazyBinaryFactory class and a utility function class.
3/ Added package "org.apache.hadoop.hive.serde2.lazybinary.objectinspector" which include the object inspector classes for non-primitive data types, such as List, Struct and Map. It also includes an object inspector factory class. For primitive data types, such as Boolean, Byte, Short and so on, I used the corresponding writable object inspector classes. For instance, I used WritableBooleanObjectInspector for LazyBinaryBoolean.
4/ Added a unit test for LazyBinarySerDe. It tests serializatoin and deserialization of all above data types except LazyBinaryMap. I will include a test for that in the future.

1/ Create "LazyBinary" classes for all data types. It will be a hierarchy of classes which is similar to the hierarchy of the "Lazy" classes.
The most top level class is LazyBinaryObject which has two child classes, LazyBinaryPrimitive and LazyBinaryNonPrimitive.
LazyBinaryPrimitive has several child classes defined for all primitive types, such as LazyBinaryBoolean for Boolean, LazyBinaryInteger
for Integer and so on. LazyBinaryNonPrimitive has 4 child classes for 4 complex data types in Hive, including LazyBinaryStruct,
LazyBinaryMap, LazyBinaryList and LazyBinaryString. The reason of having LazyBinaryPrimitive and its child classes is that we
can easily define a LazyBinaryList as a list of LazyPrimitive objects.
2/ Create LazyBinary object inspectors for some of the LazyBinary classes that handle complex data types. They are
LazyBinaryStructObjectInspector, LazyBinaryMapObjectInspector, LazyBinaryListObjectInspector and LazyBinaryStringObjectInspector.
3/ Create the LazyBinarySerDe class which serializes a datastream to LazyBinary format and later deserialize them back to LazyBinary types.

Yuntao Jia
added a comment - 21/Jul/09 22:35 Changes need to be made to the source code:
1/ Create "LazyBinary" classes for all data types. It will be a hierarchy of classes which is similar to the hierarchy of the "Lazy" classes.
The most top level class is LazyBinaryObject which has two child classes, LazyBinaryPrimitive and LazyBinaryNonPrimitive.
LazyBinaryPrimitive has several child classes defined for all primitive types, such as LazyBinaryBoolean for Boolean, LazyBinaryInteger
for Integer and so on. LazyBinaryNonPrimitive has 4 child classes for 4 complex data types in Hive, including LazyBinaryStruct,
LazyBinaryMap, LazyBinaryList and LazyBinaryString. The reason of having LazyBinaryPrimitive and its child classes is that we
can easily define a LazyBinaryList as a list of LazyPrimitive objects.
2/ Create LazyBinary object inspectors for some of the LazyBinary classes that handle complex data types. They are
LazyBinaryStructObjectInspector, LazyBinaryMapObjectInspector, LazyBinaryListObjectInspector and LazyBinaryStringObjectInspector.
3/ Create the LazyBinarySerDe class which serializes a datastream to LazyBinary format and later deserialize them back to LazyBinary types.

Here is my initial proposal for the Lazy Binary SerDe. It should have the following properties:
1/ Lazy, which means the real fields are not deserialized until accessed, just like SimpleLazySerDe.
2/ Binary, which means the data are stored in the compact binary format. However it is different from BinarySortable that the stored data does not preserve the orders of the original data. More specifications on how different data types are stored are described below.
2.1/ Null fields in a row. To represent that, we use a single bit to represent whether each filed is null or not. 0b means null and 1b means not. Eight bits forms a byte, if there are less than eight bytes at the end, we use one more byte (8 bits). They are stored at the beginning of each row. Take a 10 columns table for an example, we begin each row with two bytes(16 bits). If in one row, the first column and the 10th column are null, then we will store 01111111b and 10111111b, which are 127 and 191 decimal numbers.
2.2/ Null fields in container types and complex types, such as list, map and struct. Similarly, we use a single bit to represent whether each element is null or not. For recursive data, such as a list of list, we store those bytes at each level. We store some bytes at the beginning of the list to indicate whether each list element is null. At the beginning of each list element, which is another list, we store some bytes too to indicate whether its elements are null or not.
2.3/ For elements that are null, we do not store them.
2.4/ For int and long primary types, we store them with the varied sized int and varied size long, such as vint and vlong in the WritableUtils in hadoop.
2.5/ For other primary types, including double, float, Boolean, byte and short, store them in binary format. For example, Boolean takes one byte and double takes eight bytes.
2.6/ For String, we first store its size as an vint, then followed by all the string bytes. For an empty string, we just store it size.
2.7/ For List, we first store its size as an vint, then followed by the bytes representing whether the fields are null or not. Then the real elements are stored. For an empty list, we just store its size.
2.8/ For Map, we first store its size as an vint, then followed by the bytes representing whether the keys and values are null or not. Each pair of key and value requires two consecutive bits. So there are twice as many bits as the size of the map. The key-value pairs are stored afterwards. For an empty map, we just store its size.
2.9/ For Struct, we first store the bytes representing whether each filed is null or not. The we will store the real data fields.
3/ We will use the standard writable object inspector.
4/ We will use the BytesWritable class as the serialization class.

Yuntao Jia
added a comment - 17/Jul/09 17:30 Here is my initial proposal for the Lazy Binary SerDe. It should have the following properties:
1/ Lazy, which means the real fields are not deserialized until accessed, just like SimpleLazySerDe.
2/ Binary, which means the data are stored in the compact binary format. However it is different from BinarySortable that the stored data does not preserve the orders of the original data. More specifications on how different data types are stored are described below.
2.1/ Null fields in a row. To represent that, we use a single bit to represent whether each filed is null or not. 0b means null and 1b means not. Eight bits forms a byte, if there are less than eight bytes at the end, we use one more byte (8 bits). They are stored at the beginning of each row. Take a 10 columns table for an example, we begin each row with two bytes(16 bits). If in one row, the first column and the 10th column are null, then we will store 01111111b and 10111111b, which are 127 and 191 decimal numbers.
2.2/ Null fields in container types and complex types, such as list, map and struct. Similarly, we use a single bit to represent whether each element is null or not. For recursive data, such as a list of list, we store those bytes at each level. We store some bytes at the beginning of the list to indicate whether each list element is null. At the beginning of each list element, which is another list, we store some bytes too to indicate whether its elements are null or not.
2.3/ For elements that are null, we do not store them.
2.4/ For int and long primary types, we store them with the varied sized int and varied size long, such as vint and vlong in the WritableUtils in hadoop.
2.5/ For other primary types, including double, float, Boolean, byte and short, store them in binary format. For example, Boolean takes one byte and double takes eight bytes.
2.6/ For String, we first store its size as an vint, then followed by all the string bytes. For an empty string, we just store it size.
2.7/ For List, we first store its size as an vint, then followed by the bytes representing whether the fields are null or not. Then the real elements are stored. For an empty list, we just store its size.
2.8/ For Map, we first store its size as an vint, then followed by the bytes representing whether the keys and values are null or not. Each pair of key and value requires two consecutive bits. So there are twice as many bits as the size of the map. The key-value pairs are stored afterwards. For an empty map, we just store its size.
2.9/ For Struct, we first store the bytes representing whether each filed is null or not. The we will store the real data fields.
3/ We will use the standard writable object inspector.
4/ We will use the BytesWritable class as the serialization class.