In the Netherlands, it would only refer to the afternoon. However, I think that in Belgium, "middag" is the period aroud noon. In the Netherlands a common phrase for that period would be "tussen de middag".