Rob Langley
added a comment - 2014-08-27 16:32 Looks like https://github.com/jenkinsci/git-plugin/commit/897426d8004155c25f272189052b695c3aaadceb may have removed part of the original fix https://github.com/jenkinsci/git-plugin/commit/df304f7fe1d85798410ad7c3165a7d3cadcb4d05

Mark Waite
added a comment - 2014-08-27 23:29 - edited I don't understand why you think the polling change may have removed part of the earlier fix .
If you are using Windows, then isn't it more likely that the new guessing logic trying to find the ssh.exe program is somehow wrong in your environment?

As yet another guess, based on comparing the source code of the git-client-plugin and your log, it seems that the command "git ls-remote" is attempting to fork the "ssh" command from inside "git ls-remote" but is unable to find the "ssh" command on the host where it is attempting to execute. The call to "ssh" seems to hint that this is executing on your Linux machine. Also, since the call to ssh is from inside "git ls-remote", it seems to be beyond the control of the git plugin or the git client plugin to make the ssh program available on the PATH when executing "git ls-remote".

Are you confident that ssh is in the PATH on that machine in the context where the "git ls-remote" command is being executed?

Mark Waite
added a comment - 2014-08-27 23:43 As yet another guess, based on comparing the source code of the git-client-plugin and your log, it seems that the command "git ls-remote" is attempting to fork the "ssh" command from inside "git ls-remote" but is unable to find the "ssh" command on the host where it is attempting to execute. The call to "ssh" seems to hint that this is executing on your Linux machine. Also, since the call to ssh is from inside "git ls-remote", it seems to be beyond the control of the git plugin or the git client plugin to make the ssh program available on the PATH when executing "git ls-remote".
Are you confident that ssh is in the PATH on that machine in the context where the "git ls-remote" command is being executed?

So I've done a little more investigation, it would appear the git ls-remote command is being run on the Debian master rather than the windows slave. To ascertain this I renamed every git.exe on the windows machine which made no difference, after reverting them I did the same to /usr/bin/git which resulted in the following error:

Started on 28-Aug-2014 10:17:24
Polling SCM changes on XXX
Using strategy: Default
[poll] Last Built Revision: Revision 131f327074161ac81f7dc523363ec08841739669 (origin/master)
> git ls-remote -h git@XXX:XXX/XXX.git master # timeout=10
FATAL: hudson.plugins.git.GitException: Error performing command: git ls-remote -h git@XXX:XXX/XXX.git master
hudson.util.IOException2: hudson.plugins.git.GitException: Error performing command: git ls-remote -h git@github.bromium.net:rob-langley/glowing-octo-dangerzone.git master
at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:462)
at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:357)
at hudson.scm.SCM.poll(SCM.java:374)
at org.jenkinsci.plugins.multiplescms.MultiSCM.compareRemoteRevisionWith(MultiSCM.java:92)
at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:357)
at hudson.scm.SCM.poll(SCM.java:374)
at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1449)
at hudson.model.AbstractProject._poll(AbstractProject.java:1420)
at hudson.model.AbstractProject.poll(AbstractProject.java:1331)
at com.cloudbees.jenkins.GitHubPushTrigger$1.runPolling(GitHubPushTrigger.java:73)
at com.cloudbees.jenkins.GitHubPushTrigger$1.run(GitHubPushTrigger.java:98)
at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: hudson.plugins.git.GitException: Error performing command: git ls-remote -h git@XXX:XXX/XXX.git master
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1414)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1195)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1119)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1110)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getHeadRev(CliGitAPIImpl.java:2004)
at hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl(GitSCM.java:495)
at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:460)
... 18 more
Caused by: java.io.IOException: Cannot run program "git": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
at hudson.Proc$LocalProc.<init>(Proc.java:244)
at hudson.Proc$LocalProc.<init>(Proc.java:216)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:780)
at hudson.Launcher$ProcStarter.start(Launcher.java:360)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1403)
... 24 more
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:135)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
... 29 more
Done. Took 16 ms
No changes

Following up from this I swapped /usr/bin/git for a shell script that echo'ed the path and got:

Which is completely wrong for a Debian box, I think the polling is picking up the path from the last build job then either incorrectly trying to use it on the Debian box or incorrectly running the polling on the Debian box. I imagine it's probably the former.

Rob Langley
added a comment - 2014-08-28 09:30 So I've done a little more investigation, it would appear the git ls-remote command is being run on the Debian master rather than the windows slave. To ascertain this I renamed every git.exe on the windows machine which made no difference, after reverting them I did the same to /usr/bin/git which resulted in the following error:
Started on 28-Aug-2014 10:17:24
Polling SCM changes on XXX
Using strategy: Default
[poll] Last Built Revision: Revision 131f327074161ac81f7dc523363ec08841739669 (origin/master)
> git ls-remote -h git@XXX:XXX/XXX.git master # timeout=10
FATAL: hudson.plugins.git.GitException: Error performing command: git ls-remote -h git@XXX:XXX/XXX.git master
hudson.util.IOException2: hudson.plugins.git.GitException: Error performing command: git ls-remote -h git@github.bromium.net:rob-langley/glowing-octo-dangerzone.git master
at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:462)
at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:357)
at hudson.scm.SCM.poll(SCM.java:374)
at org.jenkinsci.plugins.multiplescms.MultiSCM.compareRemoteRevisionWith(MultiSCM.java:92)
at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:357)
at hudson.scm.SCM.poll(SCM.java:374)
at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1449)
at hudson.model.AbstractProject._poll(AbstractProject.java:1420)
at hudson.model.AbstractProject.poll(AbstractProject.java:1331)
at com.cloudbees.jenkins.GitHubPushTrigger$1.runPolling(GitHubPushTrigger.java:73)
at com.cloudbees.jenkins.GitHubPushTrigger$1.run(GitHubPushTrigger.java:98)
at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: hudson.plugins.git.GitException: Error performing command: git ls-remote -h git@XXX:XXX/XXX.git master
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1414)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1195)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1119)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1110)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getHeadRev(CliGitAPIImpl.java:2004)
at hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl(GitSCM.java:495)
at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:460)
... 18 more
Caused by: java.io.IOException: Cannot run program "git": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
at hudson.Proc$LocalProc.<init>(Proc.java:244)
at hudson.Proc$LocalProc.<init>(Proc.java:216)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:780)
at hudson.Launcher$ProcStarter.start(Launcher.java:360)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1403)
... 24 more
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:135)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
... 29 more
Done. Took 16 ms
No changes
Following up from this I swapped /usr/bin/git for a shell script that echo'ed the path and got:
C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell1.0\;C:\Program Files (x86)\Gitin;C:\Program Files (x86)\Java\jre7in
Which is completely wrong for a Debian box, I think the polling is picking up the path from the last build job then either incorrectly trying to use it on the Debian box or incorrectly running the polling on the Debian box. I imagine it's probably the former.

If it were me, I'd first go looking for other places which may have introduced that incorrect setting of the PATH environment variable. For example, check that the Jenkins system settings did not somehow get a PATH setting with those flawed values. A "grep" for that path string in the Jenkins home directory may quickly show you where that value has been incorrectly placed.

I propose to close this as "not a bug", since there is no indication that the root of the problem is due to a git plugin or git client plugin bug.

Mark Waite
added a comment - 2014-08-28 14:54 If it were me, I'd first go looking for other places which may have introduced that incorrect setting of the PATH environment variable. For example, check that the Jenkins system settings did not somehow get a PATH setting with those flawed values. A "grep" for that path string in the Jenkins home directory may quickly show you where that value has been incorrectly placed.
I propose to close this as "not a bug", since there is no indication that the root of the problem is due to a git plugin or git client plugin bug.

Before we close this I'd like to add some more info and understand what the correct resolution is.

Removing envinject does solve the issue, as does rolling back the git plugin to 2.2.4 rather then 2.2.5. This testing was done on two separate machines from the original report, a OSX master and win7 slave. Can we reopen this as reverting the git plugin also fixes this issue which suggests it may be at least partially to blame.

Rob Langley
added a comment - 2014-08-28 16:02 Before we close this I'd like to add some more info and understand what the correct resolution is.
Removing envinject does solve the issue, as does rolling back the git plugin to 2.2.4 rather then 2.2.5. This testing was done on two separate machines from the original report, a OSX master and win7 slave. Can we reopen this as reverting the git plugin also fixes this issue which suggests it may be at least partially to blame.
Thanks,

I'm not persuaded that the change of the plugin from 2.2.5 back to 2.2.4 actually had anything to do with the "fix". I'm not aware of anything in the git plugin changes from 2.2.4 to 2.2.5 which would alter the value of a PATH environment variable and its impact on the PATH passed to a command line git invocation.

I don't intend to investigate this any further, and I think at the moment I'm the primary person tracking git and git client plugin bug reports.

Whether you choose to reopen it or not, my behavior will be the same, I'll ignore this bug report. I assume that eventually I'll return to it and close it completely as either "not a bug" or "cannot reproduce".

Mark Waite
added a comment - 2014-08-28 16:11 I'm not persuaded that the change of the plugin from 2.2.5 back to 2.2.4 actually had anything to do with the "fix". I'm not aware of anything in the git plugin changes from 2.2.4 to 2.2.5 which would alter the value of a PATH environment variable and its impact on the PATH passed to a command line git invocation.
I don't intend to investigate this any further, and I think at the moment I'm the primary person tracking git and git client plugin bug reports.
Whether you choose to reopen it or not, my behavior will be the same, I'll ignore this bug report. I assume that eventually I'll return to it and close it completely as either "not a bug" or "cannot reproduce".

Rob Langley
added a comment - 2014-08-28 17:04 I have rebuilt the plugin on the 2.2.5 base with 897426d8004155c25f272189052b695c3aaadceb reverted and that looks to fix it for me.
As to why I'm not sure yet but there is something in that commit that is either changing the behaviour or breaking an assumption elsewhere in the code.

Mark Waite
added a comment - 2014-08-28 17:12 I'd guess then that the portion which exposes the misconfiguration in your Jenkins job or Jenkins environment to the git plugin is at line 255 and 256 where the comment says:
// add env contributing actions & values from last build to environment - fixes JENKINS-22009
That's the key change which was in that commit. As far as I can tell, it is doing "the right thing" by making the environment variables available inside the plugin.
I think that its discovery of your EnvInject related configuration issue is a happy side effect, not a bug which the git plugin can fix.

Following the code, it looks like ln:486 in GitSCM.java deicides that it's possible to use fast remote poll.

// fast remote polling needs a single branch and an existing last build
if (!requiresWorkspaceForPolling() && buildData.lastBuild != null && buildData.lastBuild.getMarked() != null) {
// FIXME this should not be a specific case, but have BuildChooser tell us if it can poll without workspace.
final EnvVars environment = GitUtils.getPollEnvironment(project, workspace, launcher, listener, false);
GitClient git = createClient(listener, environment, project, Jenkins.getInstance(), null);

This then uses getPollEnvironment (and in turn addEnvironmentContributingActionsValues) to get the last build env. This is working correctly as we saw in my previous comments, but it then uses this env on the Jenkins master to do the poll which is why I'm hitting the issue.

This as least gives a workaround of forcing the polling to use a workspace. I guess EnvInject could be messing with this code but I don't know much about it.

Rob Langley
added a comment - 2014-08-28 17:56 Following the code, it looks like ln:486 in GitSCM.java deicides that it's possible to use fast remote poll.
// fast remote polling needs a single branch and an existing last build
if (!requiresWorkspaceForPolling() && buildData.lastBuild != null && buildData.lastBuild.getMarked() != null) {
// FIXME this should not be a specific case, but have BuildChooser tell us if it can poll without workspace.
final EnvVars environment = GitUtils.getPollEnvironment(project, workspace, launcher, listener, false);
GitClient git = createClient(listener, environment, project, Jenkins.getInstance(), null);
This then uses getPollEnvironment (and in turn addEnvironmentContributingActionsValues) to get the last build env. This is working correctly as we saw in my previous comments, but it then uses this env on the Jenkins master to do the poll which is why I'm hitting the issue.
This as least gives a workaround of forcing the polling to use a workspace. I guess EnvInject could be messing with this code but I don't know much about it.

Mark Waite
added a comment - 2014-08-30 22:38 - edited I'm reopening this because Rob's question seems very reasonable, and there is a hint that JENKINS-24516 may be the same issue, manifest in a different manner.
If it is the same issue, then the challenge becomes writing an automated test which will show the failure. Pull requests of an automated test to show the failure are certainly welcomed...

EnvironmentContributingAction is too wider net where are I believe ParametersAction will be much better. At least with my testing it looks to do the right thing with or without EnvInject installe. @danielbeck - it should only add parameters rather then environment variables which should avoid any instance of copying of node specific environments. It also passes the test case added for JENKINS-22009.

As for a test case, I'm not sure. EnvInject hit this as EnvInjectPluginAction implements EnvironmentContributingAction, is it possible to set something in a test to create some thing similar?

Rob Langley
added a comment - 2014-09-01 13:43 Having spent a little more time looking at this, the above will cause JENKINS-22009 to reoccur, I think a better solution would be the following:
diff --git a/src/main/java/hudson/plugins/git/util/GitUtils.java b/src/main/java/hudson/plugins/git/util/GitUtils.java
index 3d41ad3..d770209 100644
--- a/src/main/java/hudson/plugins/git/util/GitUtils.java
+++ b/src/main/java/hudson/plugins/git/util/GitUtils.java
@@ -266,8 +266,8 @@ public class GitUtils implements Serializable {
if (buildActions != null) {
for (Action action : buildActions) {
// most importantly, ParametersAction will be processed here (for parameterized builds)
- if (action instanceof EnvironmentContributingAction) {
- EnvironmentContributingAction envAction = (EnvironmentContributingAction) action;
+ if (action instanceof ParametersAction) {
+ ParametersAction envAction = (ParametersAction) action;
envAction.buildEnvVars(b, env);
}
}
EnvironmentContributingAction is too wider net where are I believe ParametersAction will be much better. At least with my testing it looks to do the right thing with or without EnvInject installe. @danielbeck - it should only add parameters rather then environment variables which should avoid any instance of copying of node specific environments. It also passes the test case added for JENKINS-22009 .
As for a test case, I'm not sure. EnvInject hit this as EnvInjectPluginAction implements EnvironmentContributingAction, is it possible to set something in a test to create some thing similar?

Rob Langley
added a comment - 2014-09-02 11:00 Mark, I just created a PR for this complete with a test case that should catch future regressions: https://github.com/jenkinsci/git-plugin/pull/253
Thanks,