Gitlab 7.10 install doesn't create proper ssh git user config?

Question

I am setting up a Gitlab instance on my home VM lab and am baffled with the way SSH is configured in Gitlab so I would appreciate some insight into whether I am doing something wrong or is Gitlab using some convention I am not aware of.

I've just installed Gitlab 7.10 CE on CentOS 7 under ESXi 5.5 U2 using yum install gitlab-ce. Prior to reconfiguring gitlab for the first time, I modified /etc/gitlab/gitlab.rb to use external postgresql database, defined external SMTP server, external port and external git repository directory (NFS-backed directory from my NAS).

Everything runs smoothly (gitlab-ctl reconfigure throws no exceptions, gitlab-rake gitlab:check resolves no errors, I can log on to the instance, I can create projects and I see the changes being propagated to NFS-backed directory) - until the time I try to push to git repository for the first time using generated instructions from project web-page (I am using msysgit to push to git repo, if that is even remotely relevant).

There is no other instance/installation of git on server machine other than the one embedded by gitlab - the official instructions didn't mention that as needed and that sounds reasonable; why would I need two git instances anyway? It sounds like trouble.

If needed, I'll post gitlab-rake check output here as an update but I would like to reduce the size of the post initially.

client user ssh key -> added via Gitlab web UI
server git bin path -> /opt/gitlab/embedded/bin/git
server git home dir -> /var/opt/gitlab/
server git shell -> sh
gitlab project group -> algorithms

Here's the chronology of my efforts:

Action:

$ git push -u origin master
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Response: add the authorized key to git user's .ssh directory.

Action:

$ git push -u origin master
sh: git-receive-pack: command not found
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

$ ssh [email protected] echo \$PATH
/usr/local/bin:/usr/bin

Response: add the ~git/.profile file to git user with embedded git path exported. Add the ~git/.ssh/environment file with embedded git path exported (for non-interactive shell mode). Amended /etc/ssh/sshd_config to accept the new configuration. Restarted sshd.

Action:

$ ssh [email protected] echo \$PATH
/usr/local/bin:/usr/bin:/opt/gitlab/embedded/bin/

$ git push -u origin master
fatal: 'algorithms/sedgewick-book.git' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Action (motivated by SO post): Added explicit AllowUsers <...> git directive to sshd even though I am not using a non-standard port.

Response: none. Same error message as before.

Action: I am starting to suspect something is missing from the SSH url, so I started using full path to my remote repository, even though I shouldn't have to and Gitlab doesn't generate it during the initial project setup.

$ git push origin master
Counting objects: 6, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (5/5), 520 bytes | 0 bytes/s, done.
Total 5 (delta 0), reused 0 (delta 0)
remote: GitLab: No such user or key
To [email protected]:/mnt/git/repositories/algorithms/sedgewick-book.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to '[email protected]:/mnt/git/repositories/algorithms/sedgewick-book.git'

Response: None. See error above.

---

At this point, I must admit defeat. I've tried using different gitlab namespace (namely, my own username instead of project group), I've tried aligning exactly my gitlab username with the username used on SSH key, I've tried creating a commit through Gitlab UI on origin/master and then unprotecting the branch - the end result is always the same, error above.

It is highly suspect that I should be specifying the absolute path to the repository (the client should be completely oblivious to where the repository is located!) and the Gitlab groups do not have any metadata I can see that could interfere and inject some additional data under the table (I see them as trivial convenience containers of projects).

Finally, this SO post had a very similar problem to me but the solution proposed doesn't make much sense - it's as if I should be creating a separate Gitlab user for each group, which would be completely bonkers, not to mention I've already tried to align the repository path name with Gitlab username, to no avail (nota bene, that is also really questionable but I was desperate so I gave it a shot).

The error indicates Gitlab doesn't know how to find the (git) user I am pushing with but I don't know what else to try - I've added the SSH key to Gitlab user and also tried to align the Gitlab username with git client name (used on the SSH public key itself).

I do not know what else to try. Any hints, please?

Answer 1

I was able to make this work - I will add the explanation in hopes someone else might be spared the hair pulling I went through.

The solution was trivial but the error reported is misleading and the underlying cause unexpected. It was only by accident that I stumbled upon the link which precisely described the cause and the solution.

Essentially, the root of the problem is I was adding the client's authorized SSH key manually to ~git/.ssh/authorized_keys. Gitlab really dislikes this as it inserts some additional metadata information about its repositories as a part of the key itself - you're supposed to add the SSH key to user's profile solely via web UI.

The reason I went with manual adding of the SSH key in the first place is that I was getting Permission denied (publickey,gssapi-keyex,gssapi-with-mic) errors despite the SSH key being defined so I went with the above.

Arguably, that may be a questionable design choice and it seems like a hack - I am not at all convinced placing anything but the key into the authorized_keys is wise: it doesn't seem it was designed to hold metadata information (one is essentially lying about the purpose of the file itself, as it holds "something else" in addition to the keys) and it contradicts the existing best practices regarding the placement of keys in the file (this is the first time I saw anything like this). On the other hand, one could argue the Gitlab's git user is really an internal component of Gitlab and you're not required nor expected to tamper with it manually. That could be fixed with improved documentation.