I got the a master to come up after setting hbase.master.namespace.init.timeout to some absurd value

I see the master registering dead region servers (though I cannot find where it pick them up, not in the WAL, Archive or data)

and I see the master registering the following

Starting namespace manager (since 1hrs, 20mins, 5sec ago)

even though cloudera manager shows healthy

list the catalog in hbase shell gives me the following error

hbase(main):004:0> list
TABLE
ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2373)
at org.apache.hadoop.hbase.master.MasterRpcServices.getTableNames(MasterRpcServices.java:907)
at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55650)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2182)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)

fsck /hbase -files -blocks shows healthy

hbck shows zero inconsistencies

I am on version

Hadoop 2.6.0+cdh5.11.1+2400HBase 1.2.0+cdh5.11.1+319

I did have a master colocated with a region server and was wondering if I ran into this

Re: hbase Master failed to become active master

Not sure which version of CDH you met with this issue. Note that the hbase namespace table is a system table required for HBase to function properly. This table is not handled with a higher priority than other tables as noted in HBASE-14190. If the HBase Master tries to split a WALs on dead or ghost Region Servers then the HBase Master might get stuck trying to split these WALs on invalid Region Servers. The HBase Master can also get stuck trying to split corrupt 83-byte or smaller WAL files on startup in which case just sidelining those blocking WAL files would help. So increasing the hbase.master.namespace.init.timeout may or may not help depending on whether the Master is stuck with any other tasks like WAL splitting.

Another workaround is to recreate the hbase:namespace table in which case it will get loaded quickly.