Details

Description

Keith Turner re-wrote the accumulo continuous ingest test using gora, which has both hbase and accumulo back-ends.

I put a billion entries into HBase, and ran the Verify map/reduce job. The verification failed because about 21K entries were missing. The goraci README explains the test, and how it detects missing data.

I re-ran the test with 100 million entries, and it verified successfully.
Both of the times I tested using a billion entries, the verification failed.
If I run the verification step twice, the results are consistent, so the problem is
probably not on the verify step.

Here's the versions of the various packages:

package

version

hadoop

0.20.205.0

hbase

0.92.1

gora

http://svn.apache.org/repos/asf/gora/trunk r1311277

goraci

https://github.com/ericnewton/goraci tagged 2012-04-08

The change I made to goraci was to configure it for hbase and to allow it to build properly.

Issue Links

relates to

HBASE-5986Clients can see holes in the META table when regions are being split

Save yourself a mountain of time during the Verify step by taking a look at GORA-117.

When I pre-split the data, my 100M runs were clean. When I leaned on the split button during ingest, I had problems even at 100M. Could be I just got (un)lucky, but I hope it will reduce your debugging time.

Eric Newton
added a comment - 10/Apr/12 23:29 @stack
Save yourself a mountain of time during the Verify step by taking a look at GORA-117 .
When I pre-split the data, my 100M runs were clean. When I leaned on the split button during ingest, I had problems even at 100M. Could be I just got (un)lucky, but I hope it will reduce your debugging time.

I managed to get things to work after a few detours and figuring the tool (I'm a little slow). Here is output of a verify run after uploading 1B rows using the Generator tool (I have a five node cluster):

However, I just did a run of 100M. I suspect split/balancing is causing the data loss. So, during the 100M run, I pushed the split button on the master web page, every 5 seconds until I had several hundred tablets. Then I left it alone.

Eric Newton
added a comment - 12/Apr/12 14:25 Yes, that looks like a clean run.
My run of 1B completed correctly yesterday.
However, I just did a run of 100M. I suspect split/balancing is causing the data loss. So, during the 100M run, I pushed the split button on the master web page, every 5 seconds until I had several hundred tablets. Then I left it alone.
12/04/12 14:17:01 INFO mapred.JobClient: goraci.Verify$Counts
12/04/12 14:17:01 INFO mapred.JobClient: UNDEFINED=37099
12/04/12 14:17:01 INFO mapred.JobClient: REFERENCED=89961385
12/04/12 14:17:01 INFO mapred.JobClient: UNREFERENCED=10001516
Perhaps if you do this you can replicate the problem.

The counts for the 1B run seem odd to me , but maybe thats just an artifact of how many map task you ran for the generator and how much data each task generated. If a map task does not does not generate a multiple of 25,000,000 then it will leave some unreferenced. It generates a circular linked list every 25M.

If you were to run 10 map task each generating 100M, then this should generate 1B with all nodes referenced. Minimizing the number of unreferenced is ideal, because the test can not detect the loss of unreferenced nodes. I should probably add this info to the readme.

Keith Turner
added a comment - 12/Apr/12 15:31 The counts for the 1B run seem odd to me , but maybe thats just an artifact of how many map task you ran for the generator and how much data each task generated. If a map task does not does not generate a multiple of 25,000,000 then it will leave some unreferenced. It generates a circular linked list every 25M.
12/04/12 03:54:11 INFO mapred.JobClient: REFERENCED=564459547
12/04/12 03:54:11 INFO mapred.JobClient: UNREFERENCED=1040000000
If you were to run 10 map task each generating 100M, then this should generate 1B with all nodes referenced. Minimizing the number of unreferenced is ideal, because the test can not detect the loss of unreferenced nodes. I should probably add this info to the readme.

Notice that map input records is 1M less that 250M, which indicates that the inputformat did not provide all records in the table. The missing rows all belong to the single region. I rerun the test again after a couple of hours, and it passed. But the failed test created 244 maps, instead of 246, which is the current region count, so I am suspecting there is something wrong in the split calculation or in the supposed transactional behavior for split/balance operations in the meta table. I am still inspecting the code and the logs, but any pointers are welcome.

Enis Soztutar
added a comment - 09/May/12 03:05 In one of my 0.92.x tests on a 10 node cluster, 250M inserts, I did manage to get the verify to fail:
12/05/08 11:11:18 INFO mapred.JobClient: goraci.Verify$Counts
12/05/08 11:11:18 INFO mapred.JobClient: UNDEFINED=972506
12/05/08 11:11:18 INFO mapred.JobClient: REFERENCED=248051318
12/05/08 11:11:18 INFO mapred.JobClient: UNREFERENCED=972506
12/05/08 11:11:18 INFO mapred.JobClient: Map-Reduce Framework
12/05/08 11:11:18 INFO mapred.JobClient: Map input records=249023824
Notice that map input records is 1M less that 250M, which indicates that the inputformat did not provide all records in the table. The missing rows all belong to the single region. I rerun the test again after a couple of hours, and it passed. But the failed test created 244 maps, instead of 246, which is the current region count, so I am suspecting there is something wrong in the split calculation or in the supposed transactional behavior for split/balance operations in the meta table. I am still inspecting the code and the logs, but any pointers are welcome.

Lars Hofhansl
added a comment - 17/Jul/12 04:28 Trying to figure out whether this is something to worry about or not.
Is this a permanent data loss issue, or is it "just" .META. being out of whack temporarily (or at least fixable with hbck)?

@Lars,
We have been running this for a while as nightlies, and apart from the reported HBASE-5986, HBASE-6060, and HBASE-6160, we did not run into more issues. All of them can be considered META issues w/o actual data loss. Let's see what Eric would say.

Enis Soztutar
added a comment - 17/Jul/12 08:15 @Lars,
We have been running this for a while as nightlies, and apart from the reported HBASE-5986 , HBASE-6060 , and HBASE-6160 , we did not run into more issues. All of them can be considered META issues w/o actual data loss. Let's see what Eric would say.

Eric Newton
added a comment - 17/Jul/12 13:38 I will try to reproduce the bug with 0.94 and hadoop 1.0.3 sometime in the next week.
We normally run this test against Accumulo for 24 hours, both with and without agitation (random killing of servers). I would be concerned if there was apparent data loss, even if it was transient.

In the past we just killed Accumulo processes, the tablet servers, loggers, and master process(es). We were not looking to test HDFS. For the next release, Accumulo has moved its write ahead logging to HDFS. So for the next release we plan to start killing HDFS processes as part of our testing. See ACCUMULO-578, ACCUMULO-623, and ACCUMULO-636.

Keith Turner
added a comment - 17/Jul/12 14:00 In the past we just killed Accumulo processes, the tablet servers, loggers, and master process(es). We were not looking to test HDFS. For the next release, Accumulo has moved its write ahead logging to HDFS. So for the next release we plan to start killing HDFS processes as part of our testing. See ACCUMULO-578 , ACCUMULO-623 , and ACCUMULO-636 .

Just FYI in case, I've been working on adding "long running ingestion tests while randomly killing of servers", and other types of integration tests over at HBASE-6241, HBASE-6201. Feel free to chime in.

Enis Soztutar
added a comment - 17/Jul/12 14:30 Just FYI in case, I've been working on adding "long running ingestion tests while randomly killing of servers", and other types of integration tests over at HBASE-6241 , HBASE-6201 . Feel free to chime in.

Eric Newton, this issue has been dormant for 11 months. Do you think all these issues were resolved based on the other lined JIRAs above (e.g. HBASE-5987, HBASE-6060, etc)? Or is this still reproducible?

Ian Varley
added a comment - 06/Jun/13 03:58 Eric Newton , this issue has been dormant for 11 months. Do you think all these issues were resolved based on the other lined JIRAs above (e.g. HBASE-5987 , HBASE-6060 , etc)? Or is this still reproducible?

Ian Varley I don't know if it has been fixed. I have not re-tested more recent versions of HBase. Since the test has now been incorporated into HBase without any additional findings, perhaps it has been fixed.

Eric Newton
added a comment - 06/Jun/13 12:57 Ian Varley I don't know if it has been fixed. I have not re-tested more recent versions of HBase. Since the test has now been incorporated into HBase without any additional findings, perhaps it has been fixed.

Ian Varley
added a comment - 06/Jun/13 14:27 OK, thanks Eric Newton . I would move to close unless anyone has reported something similar in the meantime or is actively working on it. Reopen if you disagree.