Have been working with John on his cluster on this issue. This patch seems to fix the issue (more testing to do).

Splits are triggered if the compaction run returns true. The return up out of compaction was coming up from the depths of store file and on the way could be mangled if multiple families in a region; one might compact but the subsequent one might not. Because of the latter, we'd not run split check.

stack
added a comment - 25/Jun/08 19:49 Have been working with John on his cluster on this issue. This patch seems to fix the issue (more testing to do).
Splits are triggered if the compaction run returns true. The return up out of compaction was coming up from the depths of store file and on the way could be mangled if multiple families in a region; one might compact but the subsequent one might not. Because of the latter, we'd not run split check.

Recreated entire use case scenario and the issue is gone. We are now seeing normal region splits.

However, we have experienced a new behavior during those splits. We are writing and the client receives an IllegalStateException:

Trying to commit: Exception in thread "main" java.lang.RuntimeException: java.lang.IllegalStateException: region offline: items,823ce1e3-d414-474f-ac70-c4081cecef0f,1214434560891
at org.apache.hadoop.hbase.HTable.getRegionServerWithRetries(HTable.java:1062)
at org.apache.hadoop.hbase.HTable.commit(HTable.java:763)
at org.apache.hadoop.hbase.HTable.commit(HTable.java:744)
at HBase.AddAttributes(HBase.java:220)
at PoJaMigratorItems.Add(PoJaMigratorItems.java:143)
at PoJaMigrator.AddItems(PoJaMigrator.java:123)
at PoJaMigrator.AddAllData(PoJaMigrator.java:57)
at PoJaMigrator.<init>(PoJaMigrator.java:27)
at PoJaMigrator.main(PoJaMigrator.java:35)
Caused by: java.lang.IllegalStateException: region offline: items,823ce1e3-d414-474f-ac70-c4081cecef0f,1214434560891
at org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:438)
at org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:350)
at org.apache.hadoop.hbase.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:318)
at org.apache.hadoop.hbase.HTable.getRegionLocation(HTable.java:114)
at org.apache.hadoop.hbase.HTable$ServerCallable.instantiateServer(HTable.java:1021)
at org.apache.hadoop.hbase.HTable.getRegionServerWithRetries(HTable.java:1036)
... 8 more

Jonathan Gray
added a comment - 25/Jun/08 23:28 Recreated entire use case scenario and the issue is gone. We are now seeing normal region splits.
However, we have experienced a new behavior during those splits. We are writing and the client receives an IllegalStateException:
Trying to commit: Exception in thread "main" java.lang.RuntimeException: java.lang.IllegalStateException: region offline: items,823ce1e3-d414-474f-ac70-c4081cecef0f,1214434560891
at org.apache.hadoop.hbase.HTable.getRegionServerWithRetries(HTable.java:1062)
at org.apache.hadoop.hbase.HTable.commit(HTable.java:763)
at org.apache.hadoop.hbase.HTable.commit(HTable.java:744)
at HBase.AddAttributes(HBase.java:220)
at PoJaMigratorItems.Add(PoJaMigratorItems.java:143)
at PoJaMigrator.AddItems(PoJaMigrator.java:123)
at PoJaMigrator.AddAllData(PoJaMigrator.java:57)
at PoJaMigrator.<init>(PoJaMigrator.java:27)
at PoJaMigrator.main(PoJaMigrator.java:35)
Caused by: java.lang.IllegalStateException: region offline: items,823ce1e3-d414-474f-ac70-c4081cecef0f,1214434560891
at org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:438)
at org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:350)
at org.apache.hadoop.hbase.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:318)
at org.apache.hadoop.hbase.HTable.getRegionLocation(HTable.java:114)
at org.apache.hadoop.hbase.HTable$ServerCallable.instantiateServer(HTable.java:1021)
at org.apache.hadoop.hbase.HTable.getRegionServerWithRetries(HTable.java:1036)
... 8 more