On Windows, it can't rely on the bang path or even really a .sh extension. Before HADOOP-12850, this worked by having exec-maven-plugin invoke bash directly in the executable attribute. I did a quick hack change locally to confirm that switching back to direct invocation of bash would work.

Allen Wittenauer, I assume you want to retain the behavior of invoking through /usr/bin/env so that it can look up bash from the user environment, right? Can you confirm? If we want to keep that, then I think it puts me on the path of setting up a separate native-win profile for the special case in hadoop-dist/pom.xml.

Chris Nauroth
added a comment - 07/Mar/16 19:22 On Windows, it can't rely on the bang path or even really a .sh extension. Before HADOOP-12850 , this worked by having exec-maven-plugin invoke bash directly in the executable attribute. I did a quick hack change locally to confirm that switching back to direct invocation of bash would work.
Allen Wittenauer , I assume you want to retain the behavior of invoking through /usr/bin/env so that it can look up bash from the user environment, right? Can you confirm? If we want to keep that, then I think it puts me on the path of setting up a separate native-win profile for the special case in hadoop-dist/pom.xml.

It would be good to lookup bash from the environment, but honestly it's an extreme edge case. So if it needs to be hard-coded, then so be it. My big goal was mainly to pull large chunks of shell out of the pom files so that they could get some love from shellcheck and to make them easier to debug. I'm in the process of doing the same thing with HADOOP-12892.

Allen Wittenauer
added a comment - 07/Mar/16 19:34 It would be good to lookup bash from the environment, but honestly it's an extreme edge case. So if it needs to be hard-coded, then so be it. My big goal was mainly to pull large chunks of shell out of the pom files so that they could get some love from shellcheck and to make them easier to debug. I'm in the process of doing the same thing with HADOOP-12892 .

I'm attaching patch v001. This goes back to the strategy of invoking the interpreter directly. I tested this successfully on Windows and Linux.

I also changed the bang lines within the scripts. Technically, that's not really necessary for the fix, but I didn't want people reading it to mistakenly think that it's routing through env during the build.

Chris Nauroth
added a comment - 08/Mar/16 00:22 - edited I'm attaching patch v001. This goes back to the strategy of invoking the interpreter directly. I tested this successfully on Windows and Linux.
I also changed the bang lines within the scripts. Technically, that's not really necessary for the fix, but I didn't want people reading it to mistakenly think that it's routing through env during the build.

Chris Nauroth
added a comment - 08/Mar/16 05:49 There are no tests, because this is a change in the build process only. As I mentioned earlier, I did manual testing on both Windows and Linux to confirm that bundling Snappy works correctly.

On CentOS, it's /usr/bin/bash. This sounds like good enough reason to keep the old bang line using /usr/bin/env. Here is patch v002 with just the pom.xml change. It's not necessary to change the bang lines to fix this issue.

Chris Nauroth
added a comment - 08/Mar/16 22:12 On CentOS, it's /usr/bin/bash. This sounds like good enough reason to keep the old bang line using /usr/bin/env. Here is patch v002 with just the pom.xml change. It's not necessary to change the bang lines to fix this issue.

I looked into shelldocs more. First, there is the same fix required for invoking the shell directly through maven-exec-plugin. However, that alone isn't sufficient to get shelldocs working on Windows. It looks like there are also going to be some requirements around translating paths using cygpath -w for the GPG commands. Even getting past that, I have some as-yet-undiagnosed error preventing it from working fully.

At this point, I think I want to proceed with committing the existing patch v002 based on Andrew's +1. I'd like to defer shelldocs and releasedocmaker to a separate issue. I don't have time to dig into those further right now, and getting the distro build working (without the site) is still a helpful step for anyone who wants to do a distro build from trunk on Windows.

Chris Nauroth
added a comment - 09/Mar/16 05:04 I looked into shelldocs more. First, there is the same fix required for invoking the shell directly through maven-exec-plugin. However, that alone isn't sufficient to get shelldocs working on Windows. It looks like there are also going to be some requirements around translating paths using cygpath -w for the GPG commands. Even getting past that, I have some as-yet-undiagnosed error preventing it from working fully.
At this point, I think I want to proceed with committing the existing patch v002 based on Andrew's +1. I'd like to defer shelldocs and releasedocmaker to a separate issue. I don't have time to dig into those further right now, and getting the distro build working (without the site) is still a helpful step for anyone who wants to do a distro build from trunk on Windows.
I'll wait 24 hours before committing just in case anyone disagrees.