In the world of NoSQL

Got the following exception when starting the datanode after it had terminated due to a disk failure (without rebooting the server):

2013-10-1111:24:02,122 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode:Exception in secureMain
java.net.BindException: Problem binding to [0.0.0.0:50010] java.net.BindException: Address already in use;For more details see: http://wiki.apache.org/hadoop/BindException
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:718)
at org.apache.hadoop.ipc.Server.bind(Server.java:403)
at org.apache.hadoop.ipc.Server.bind(Server.java:375)
at org.apache.hadoop.hdfs.net.TcpPeerServer.<init>(TcpPeerServer.java:106)
at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:555)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:741)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:344)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1795)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1728)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1751)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1904)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1925)2013-10-1111:24:02,126 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 12013-10-1111:24:02,128 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at hbase10.network.com/1.2.3.4
************************************************************/

After an application crashes it might leave a lingering socket, so to reuse that socket early you need to set the socket flag SO_REUSEADDR when attempting to bind to it to be allowed to reuse it. The HDFS datanode doesn’t do that, and I didn’t want to restart the HBase regionserver (which was locking the socket with a connection it hadn’t realized was dead).
The solution was to bind to the port with an application that sets SO_REUSEADDR and then stop that application, I used netcat for that: