Jenkins git submodule update fails

Question

I have a git repo which has one submodule. Both belong to a team on BitBucket. My jenkins machine is a AWS windows server with the git plugin. I am using SSH keys for authentication. I have three jenkins jobs. One clones the main repo. This is successful. One clones the second repo on its own (the repo which will be used as a submodule). This is also successful. In my third build job I tell jenkins to recursively update the submodules. This fails and says public-key error. How can this be the case if I can clone the repo on its own?

Console output below:

Started by user anonymous
Building on master in workspace C:\Program Files (x86)\Jenkins\jobs\MainRepo\workspace
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository [email protected]:team/mainrepo.git
 > git.exe init C:\Program Files (x86)\Jenkins\jobs\mainrepo\workspace # timeout=10
Fetching upstream changes from [email protected]:team/mainrepo.git
 > git.exe --version # timeout=10
using GIT_SSH to set credentials 
 > git.exe -c core.askpass=true fetch --tags --progress [email protected]:team/mainrepo.git +refs/heads/*:refs/remotes/origin/*
 > git.exe config remote.origin.url [email protected]:team/mainrepo.git # timeout=10
 > git.exe config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git.exe config remote.origin.url [email protected]:team/mainrepo.git # timeout=10
Fetching upstream changes from [email protected]:team/mainrepo.git
using GIT_SSH to set credentials 
 > git.exe -c core.askpass=true fetch --tags --progress [email protected]:team/mainrepo.git +refs/heads/*:refs/remotes/origin/*
 > git.exe rev-parse "refs/remotes/origin/master^{commit}" # timeout=10
 > git.exe rev-parse "refs/remotes/origin/origin/master^{commit}" # timeout=10
Checking out Revision 6b3f6535c45e79ee88f4918d464edead48d83369 (refs/remotes/origin/master)
 > git.exe config core.sparsecheckout # timeout=10
 > git.exe checkout -f 6b3f6535c45e79ee88f4918d464edead48d83369
 > git.exe rev-list 6b3f6535c45e79ee88f4918d464edead48d83369 # timeout=10
 > git.exe remote # timeout=10
 > git.exe submodule init # timeout=10
 > git.exe submodule sync # timeout=10
 > git.exe config --get remote.origin.url # timeout=10
 > git.exe submodule update --init --recursive
FATAL: Command "git.exe submodule update --init --recursive" returned status code 128:
stdout: 
stderr: Cloning into 'my-submodule'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:team/my-submodule.git' into submodule path 'my-submodule' failed

hudson.plugins.git.GitException: Command "git.exe submodule update --init --recursive" returned status code 128:
stdout: 
stderr: Cloning into 'my-submodule'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:team/my-submodule.git' into submodule path 'my-submodule' failed

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1693)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:62)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.execute(CliGitAPIImpl.java:953)
    at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:90)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1098)
    at hudson.scm.SCM.checkout(SCM.java:485)
    at hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
    at hudson.model.Run.execute(Run.java:1738)
    at hudson.matrix.MatrixBuild.run(MatrixBuild.java:301)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:410)
Finished: FAILURE

Answer 1

Based on the earlier answers here, I re-prioritized upgrading my client's Jenkins. Now they are on Jenkins 2.41 with Git plugin 3.0.1 and before additional configuration this did not fix the issue. I found the configuration to be a little tricky:

1. Add top-level repository to Source Code Management -> Git
2. Select the "Additional Behaviours" Add button
   • Select "Advanced sub-modules behaviours"
3. I tested with only "Recursively update submodules" and got the "Permission denied" error (see bottom*)
4. However, I now select both "Recursively update submodules" and "Use credentials from default remote of parent repository" in Jenkins 2.41

Once I select both options, it uses the credentials I had configured for the top-level repository and everything works for me. Here is what the dialog looks like in 2.41 with Git plugin 3.0.1:

* Here is the essence of my "Permission denied" error:

Cloning into 'thirdparty'...

stderr: Permission denied (publickey). fatal: The remote end hung up unexpectedly Clone of 'ssh://[email protected]//thirdparty' into submodule path 'thirdparty' failed

PS Just before posting, I did my usual double check to make sure I'm not duplicating an answer. In this case, I see that @danielfn's comment points to something that is nearly identical to my answer, but 1. this didn't help me and I ended up figuring it out by trial and error and 2. it is StackOverflow policy to post answers here rather than referencing external links.

Answer 2

As an alternative, you may use 'Source Code Management' - 'Multiple SCMs' to manually configure all the sub-modules and add 'Additional Behaviours' - 'Check out to a sub-directory' for each one.

Answer 3

It seems that they have fixed it with the versions git client plugin 1.20.0-beta1 and git plugin 2.5.0-beta1. However, they can only be added to Jenkins by specifying to pull updates from the experimental update center.