I have had some time to check what is happening. Your GMT5 classic and GMT 6 modern mode examples are not exactly the same. In modern mode, if you dont give -D it will automatically determine the best resolution. This turns out to be -Dc. In classic it defaults to -Dl instead. If you try your pscoast command in 5.4.2 with -Dc it will fail too. I will see if we are able to fix that bug but you can avoid it by using -Dl.