0

I am using different git repositories for source code versioning cloned from different upstream origins (e.g. GitHub.com, gitlab.com, self-hosted gitlab etc.), all with SSH key access. git pull always works, but after I create a new commit using 

touch test && git add . && git commit -a -m "Test"

git push will always time out, on all repositories with all upstream origins. After that, all other git operations involving the remote (git clone, git pull, ...) will time out as well, even on other repositories with different usptreams.

However, I am using gpg signed commits by default. If I disable that by setting

git config --global commit.gpgsign false

delete my local repositories with new commits, clone them again and create new unsigned commits, pushing works.

So it seems to be some kind related to the commit signing. Any ideas? On my other machine, which is configured the same way (same software versions, same configs), pushing signed commits works.

EDIT: As supposed in the commits, I've tested this issue with commits that are unencrypted but have large commit messages, and interestingly got failures for commit messages around 4KB. So maybe this issue is related to the size of the commit messages?

EDIT 2: It is strangely connected to the MTU of my network interface, if I lower it from 1500 to 500, everything works fine. A Wireshark dump of a failing git push session with MTU 1500 looks like this:

Wireshark dump of failing git push

Any ideas?

Improve this question
4
  • Does it occur if you create an unsigned commit with a large commit message (let's say 4kB of random garbage)
    – grawity_u1686
    Commented Feb 21, 2019 at 14:32
  • I have tested it with commit messages up to 27KB, which worked flawlessly. But then I thought about compression and generated commit messages with pseudo-random data, and surprisingly it fails already for messages ~4KB in size. So maybe it is size-related?
    – LukeLR
    Commented Feb 21, 2019 at 22:48
  • I somewhat suspect it's IP MTU related. Is the cutoff somewhere near 1400 bytes? Are you using IPv6? Are you using any form of tunnel or VPN? Can you ssh into your GitLab server (into an interactive shell) and just paste the same large amount of text into it?
    – grawity_u1686
    Commented Feb 22, 2019 at 4:57
  • Yes, I am Using IPv6, but no tunnel or VPN, just plain Ethernet. No, unfortunately I can't ssh into the GitLab server, since I am not the Administrator, but I tried lowering the MTU - and it worked! So it definitely is MTU-related. Any ideas how to fix it, besides lowering the MTU?
    – LukeLR
    Commented Feb 22, 2019 at 19:37

1 Answer 1

Reset to default
0

The path from you to the GitLab server has a link with MTU lower than the standard 1500. (This could be any router or the final server.) Additionally,

  • either the system in question doesn't send you ICMP "Packet Too Big" error messages,
  • or your computer doesn't receive them (because they're blocked by another router in the middle),
  • or your computer itself blocks/ignores them (overzealous iptables-ing).

This leads to your computer getting stuck whenever it sends large IP packets (TCP segments): it doesn't receive a TCP ACK, but it doesn't receive an ICMP error indication either, so all it can do is keep retrying.

(If it had received the "Packet Too Big" ICMP packet, it would automatically adjust the MTU for that particular destination and resend the packet in smaller fragments – this is called "Path MTU Discovery".)

Investigating PMTUD

Reset your MTU to the standard 1500, then run Wireshark or tcpdump capturing "icmp or icmp6" packets. After you try pushing again and the connection hangs, the packet capture should show either some incoming ICMP packets, or just outgoing TCP retransmissions.

If you see incoming ICMP "Packet too big" responses, usually the problem is your own firewall which blocks ICMP unnecessarily. If you don't see this, the problem is somewhere further upstream.

(You can also test tracepath to the GitLab server, although this tool itself relies on ICMP so it will only show anything useful if PMTUD is already working.)

Fixing local ICMP blocking

Open up your iptables/nft ruleset and remove overly broad "block all ICMP" rules if you have any. (See http://shouldiblockicmp.com/ for a list of required-to-allow types/subtypes.)

Workarounds for upstream MTU problems

On Linux, in addition to ICMP-based PMTUD you can also use TCP-level MTU discovery:

sudo sysctl net.ipv4.tcp_mtu_probing=1

This doesn't rely on any error responses, it just tells TCP that whenever it detects packet loss after sending large segments, it should retransmit with smaller ones (instead of just re-sending exactly the same as before). The TCP connection will still appear to hang for a second or two before adjusting its MSS and resuming.

You can also use iptables "TCP MSS clamping" to make the TCP connections use a lower maximum segment size from the very beginning.

Improve this answer
9
  • Thanks for your answer! :) I'll investigate the MTU problem. But that wouldn't explain why it does work on my other machine, on the same source network, to the same destination servers, would it?
    – LukeLR
    Commented Feb 24, 2019 at 18:52
  • 1
    The path isn't necessarily the same from both sources, and your firewall settings aren't necessarily the same between both computers.
    – grawity_u1686
    Commented Feb 24, 2019 at 20:59
  • But let's say there's two paths A and B, where A is working and B has MTU problems. It would be quite a coincidence if machine A reliably always uses path A and therefore has a working git push whereas machine B reliably always uses path B and therefore git push does not work... wouldn't it? Still, I'll do some wiresharking as soon as I find the time. Would be an interesting case!
    – LukeLR
    Commented Feb 26, 2019 at 15:08
  • 1
    If both machines are behind the same public address, that's unlikely. But if the problem occurs in a place where they can be distinguished (e.g. if they get different public addresses, or if the problem/cause is still inside the company LAN), then it's quite possible. When ISPs use ECMP load balancing, it is usually deterministic and based on hashing the IP addresses, so it's not unusual for (e.g.) odd public addresses to use path A, but even addresses to use path B, and things like that.
    – grawity_u1686
    Commented Feb 26, 2019 at 15:37
  • 1
    Finally, one thing that wasn't checked is whether both your machines use the same IP version -- it could be that machine A persistently uses IPv4, while the other always uses IPv6. Those have independent configurations, independent routing, and independent problems.
    – grawity_u1686
    Commented Feb 26, 2019 at 15:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .