hbase-dev mailing list archives

[jira] Updated: (HBASE-505) Region assignments should never time out so long as the region server reports that it is processing the open request

Date

Wed, 26 Mar 2008 22:31:24 GMT

[ https://issues.apache.org/jira/browse/HBASE-505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-505:
------------------------
Fix Version/s: 0.2.0
Affects Version/s: 0.2.0
Marking this a blocker for 0.2.
> Region assignments should never time out so long as the region server reports that it
is processing the open request
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-505
> URL: https://issues.apache.org/jira/browse/HBASE-505
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Reporter: Jim Kellerman
> Assignee: Jim Kellerman
> Fix For: 0.2.0
>
>
> Currently, when the master assigns a region to a region server, it extends the reassignment
timeout when the region server reports that it is processing the open. This only happens once,
and so if the region takes a long time to come on line due to a large set of transactions
in the redo log or because the initial compaction takes a long time, the master will assign
the region to another server when the reassignment timeout occurs.
> Assigning a region to multiple region servers can easily corrupt the region. For example:
> region server 1 is processing the redo log creating a new mapfile. It takes more than
one interval to do so so the master assigns the region to region server 2. region server 2
starts processing the redo log creating essentially the same mapFile as region server 1, but
with a different name.
> region server 2 can fail to open the region if region server 1 deletes the old log file
or if it tries to open the new mapFile that region server 1 is creating.
> region server 1 can fail to open the region if it tries to open the mapFile that region
server 2 is creating.
> Often region server 1 eventually succeeds and reports to the master that it has finished
opening the region, but the master tells it to close that region because it has assigned it
to another server. Region server 2 often fails to open the region, because the old log file
has been deleted, or it fails to process the new map file created by region server 1.
> Proposed solution:
> During the open process the region server should send a MSG_PROCESS_OPEN with each heartbeat
until the region is opened (when it sends MSG_REGION_OPEN). The master will extend the reassignment
timeout with each MSG_PROCESS_OPEN it receives and will not assign the region to another server
so long as it continues to receive heart beat messages from the region server processing the
open.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.