Very long hang during OTP app startup

Very long hang during OTP app startup

Using Erlang R18 and rebar3. This is an application, or release, built with rebar3 and using generated management script to start and stop the application. I see very strange, and random behavior at startup in that when I start the application with "../<app name> console" I will see erlang “PROGRESS REPORT” messages as included applications start, like amps_client. It appears my OTP supervisor and children start normally. In fact one of the children opens a socket and handshakes with an external system. Once the handshake completes I expect to start receiving data from the external system, but nothing else happens until I see the following in the terminal where I started the application:

At this point the app receives messages on the tcp socket and everything appears completely normal. However, sometimes it is hours before I see this last “PROGRESS REPORT” and normal operation starts, other times maybe a minute or less. I have this feeling that a supervisor or child in another tree, not mine, is hung in an init() callback or something. I just am at a loss on how to debug this problem. I have never seen this type of behavior before.

My release includes my application of course, but also lager, amqp_client and rabbit_common, and xmerl.