apt-get throws 400 URI failure when trying to download docker-ce

Question

I'm having a strange issue with trying to install docker-ce instead of the outdated distro packages on Debian Stretch. I followed their simple guide verbatim:

https://docs.docker.com/install/linux/docker-ce/debian/

The key is present and verified and the sources.list line generated looks fine as expected:

deb [arch=amd64] https://download.docker.com/linux/debian stretch stable

However, a complete re-download of the package lists (stretch, stretch-updates and debian-security otherwise) results in this:

W: The repository 'https://download.docker.com/linux/debian stretch Release' does not have a Release file.
E: Failed to fetch https://download.docker.com/linux/debian/dists/stretch/stable/binary-amd64/Packages  

Complete apt-get update output:

Get:1 http://security.debian.org/debian-security stretch/updates InRelease [94.3 kB]
Ign:2 https://download.docker.com/linux/debian stretch InRelease                      
Ign:3 https://download.docker.com/linux/debian stretch Release                  
Ign:4 https://download.docker.com/linux/debian stretch/stable amd64 Packages    
Ign:5 https://download.docker.com/linux/debian stretch/stable all Packages      
Ign:6 https://download.docker.com/linux/debian stretch/stable Translation-en    
Ign:7 https://download.docker.com/linux/debian stretch/stable Translation-en_US 
Ign:8 https://download.docker.com/linux/debian stretch/stable all Contents (deb)
Ign:9 https://download.docker.com/linux/debian stretch/stable amd64 Contents (deb)
Ign:10 https://download.docker.com/linux/debian stretch all Contents (deb)      
Ign:11 https://download.docker.com/linux/debian stretch amd64 Contents (deb)    
Ign:4 https://download.docker.com/linux/debian stretch/stable amd64 Packages    
Ign:5 https://download.docker.com/linux/debian stretch/stable all Packages      
Ign:6 https://download.docker.com/linux/debian stretch/stable Translation-en    
Ign:7 https://download.docker.com/linux/debian stretch/stable Translation-en_US 
Ign:8 https://download.docker.com/linux/debian stretch/stable all Contents (deb)
Ign:9 https://download.docker.com/linux/debian stretch/stable amd64 Contents (deb)
Ign:10 https://download.docker.com/linux/debian stretch all Contents (deb)      
Ign:11 https://download.docker.com/linux/debian stretch amd64 Contents (deb)    
Ign:4 https://download.docker.com/linux/debian stretch/stable amd64 Packages    
Ign:5 https://download.docker.com/linux/debian stretch/stable all Packages      
Ign:6 https://download.docker.com/linux/debian stretch/stable Translation-en    
Ign:7 https://download.docker.com/linux/debian stretch/stable Translation-en_US 
Ign:8 https://download.docker.com/linux/debian stretch/stable all Contents (deb)
Ign:9 https://download.docker.com/linux/debian stretch/stable amd64 Contents (deb)
Ign:10 https://download.docker.com/linux/debian stretch all Contents (deb)      
Ign:11 https://download.docker.com/linux/debian stretch amd64 Contents (deb)    
Ign:4 https://download.docker.com/linux/debian stretch/stable amd64 Packages
Ign:5 https://download.docker.com/linux/debian stretch/stable all Packages
Ign:6 https://download.docker.com/linux/debian stretch/stable Translation-en
Ign:7 https://download.docker.com/linux/debian stretch/stable Translation-en_US
Ign:8 https://download.docker.com/linux/debian stretch/stable all Contents (deb)
Ign:9 https://download.docker.com/linux/debian stretch/stable amd64 Contents (deb)
Ign:10 https://download.docker.com/linux/debian stretch all Contents (deb)
Ign:11 https://download.docker.com/linux/debian stretch amd64 Contents (deb)
Ign:4 https://download.docker.com/linux/debian stretch/stable amd64 Packages
Ign:5 https://download.docker.com/linux/debian stretch/stable all Packages
Ign:6 https://download.docker.com/linux/debian stretch/stable Translation-en
Ign:7 https://download.docker.com/linux/debian stretch/stable Translation-en_US
Get:12 http://security.debian.org/debian-security stretch/updates/main Sources [195 kB]
Ign:8 https://download.docker.com/linux/debian stretch/stable all Contents (deb)
Ign:9 https://download.docker.com/linux/debian stretch/stable amd64 Contents (deb)
Ign:10 https://download.docker.com/linux/debian stretch all Contents (deb)
Ign:11 https://download.docker.com/linux/debian stretch amd64 Contents (deb)
Err:4 https://download.docker.com/linux/debian stretch/stable amd64 Packages

Ign:5 https://download.docker.com/linux/debian stretch/stable all Packages
Ign:6 https://download.docker.com/linux/debian stretch/stable Translation-en
Ign:7 https://download.docker.com/linux/debian stretch/stable Translation-en_US
Get:13 http://security.debian.org/debian-security stretch/updates/main amd64 Packages [476 kB]
Ign:8 https://download.docker.com/linux/debian stretch/stable all Contents (deb)
Ign:9 https://download.docker.com/linux/debian stretch/stable amd64 Contents (deb)
Ign:10 https://download.docker.com/linux/debian stretch all Contents (deb)      
Ign:11 https://download.docker.com/linux/debian stretch amd64 Contents (deb)   
Get:14 http://security.debian.org/debian-security stretch/updates/main Translation-en [211 kB]
Ign:15 http://ftp.us.debian.org/debian stretch InRelease                         
Get:16 http://ftp.us.debian.org/debian stretch-updates InRelease [91.0 kB]            
Get:17 http://ftp.us.debian.org/debian stretch Release [118 kB]    
Get:18 http://ftp.us.debian.org/debian stretch-updates/main Sources [14.3 kB]
Get:19 http://ftp.us.debian.org/debian stretch-updates/main amd64 Packages [11.1 kB]
Get:20 http://ftp.us.debian.org/debian stretch-updates/main Translation-en [11.1 kB]
Get:21 http://ftp.us.debian.org/debian stretch-updates/main amd64 Contents (deb) [352 kB]
Get:22 http://ftp.us.debian.org/debian stretch Release.gpg [2,434 B]
Get:23 http://ftp.us.debian.org/debian stretch/main Sources [6,746 kB]
Get:24 http://ftp.us.debian.org/debian stretch/main amd64 Packages [7,084 kB]
Get:25 http://ftp.us.debian.org/debian stretch/main Translation-en [5,384 kB]
Get:26 http://ftp.us.debian.org/debian stretch/main amd64 Contents (deb) [31.4 MB]
Fetched 52.2 MB in 13s (3,784 kB/s)                                                                                                                       
Reading package lists... Done
W: The repository 'https://download.docker.com/linux/debian stretch Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Failed to fetch https://download.docker.com/linux/debian/dists/stretch/stable/binary-amd64/Packages  
E: Some index files failed to download. They have been ignored, or old ones used instead.

Both curl and wget pull the exact URLs apt is complaining about without issues and show nothing of note in their verbose logs.

Looking at strace, I see this:

read(5, "400 URI Failure\nURI: https://download.docker.com/linux/debian/dists/stretch/stable/binary-all/Packages\nMessage: \n\n400 URI Failure\nURI: https://download.docker.com/linux/debian/dists/stretch/stable/binary-amd64/Packages\nMessage: \n\n400 URI Failure\nURI: https://download.docker.com/linux/debian/dists/stretch/stable/i18n/Translation-en\nMessage: \n\n400 URI Failure\nURI: https://download.docker.com/linux/debian/dists/stretch/stable/i18n/Translation-en_US\nMessage: \n\n", 64000) = 461
stat("/var/lib/apt/lists/partial/download.docker.com_linux_debian_dists_stretch_stable_binary-all_Packages", 0x7ffded218d60) = -1 ENOENT (No such file or directory)
unlink("/var/lib/apt/lists/partial/download.docker.com_linux_debian_dists_stretch_stable_binary-all_Packages") = -1 ENOENT (No such file or directory)
stat("/var/lib/apt/lists/download.docker.com_linux_debian_dists_stretch_Contents-all.lz4", 0x7ffded218b80) = -1 ENOENT (No such file or directory)
stat("/var/lib/apt/lists/partial/download.docker.com_linux_debian_dists_stretch_Contents-all", 0x7ffded218bd0) = -1 ENOENT (No such file or directory)
write(1, "Ign:4 https://download.docker.com/linux/debian stretch/stable all Packages\n", 75Ign:4 https://download.docker.com/linux/debian stretch/stable all Packages
) = 75
stat("/var/lib/apt/lists/partial/download.docker.com_linux_debian_dists_stretch_stable_binary-amd64_Packages", 0x7ffded218d60) = -1 ENOENT (No such file or directory)
unlink("/var/lib/apt/lists/partial/download.docker.com_linux_debian_dists_stretch_stable_binary-amd64_Packages") = -1 ENOENT (No such file or directory)
write(1, "Err:5 https://download.docker.com/linux/debian stretch/stable amd64 Packages\n", 77Err:5 https://download.docker.com/linux/debian stretch/stable amd64 Packages

That 400 URI failure has been discussed before, e. g. here: https://askubuntu.com/questions/493180/method-gave-invalid-400-uri-failure-message-when-running-apt-get-update#497377

However, it doesn't appear to be the same locale issue - running the same test with wget I see it uses en_US.UTF-8 and this is my locale output:

LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

So, nothing too special. This is pretty much a bog-standard Debian Stretch install with not much else done than getting apache2, sendmail, dovecot and other standard applications like them.

I do not have a proxy configured in $http_proxy, in any of the profile / bashrc files or for apt specifically. I can't find a good open HTTPS proxy and seem to be unable to throw one together but one that let a few bytes trickle through showed the 400 errors as well so it's unlikely that my VPS provider is pulling some shady trick on me. (If they even could with HTTPS - I'm with RamNode who recommend one of their KVMs for Docker use.) My Sid desktop with a much more hacky configuration has no issues downloading the files, as did someone on the Docker IRC using Stretch, which is at least a bit strange.

I'm out of ideas and will have to install manually for now, which works but is of course not a good idea. Any guesses on what the problem could be?

Edit: OK, I'm at my wits' end here. I set up mitmproxy in order to capture the traffic to download.docker.com. I configured it as a reverse proxy, put the proxy IP in the hosts table as download.docker.com and executed apt-get update again. It goes through flawlessly, the requests look like completely normal HTTP/1.1 in Wireshark and go through in openssl as well. The encoding is simply ASCII(-compatible).

If someone knows how to explain this, please do. I could MITM mitmproxy (straight to port 80 not possible because of forced HTTPS) and compare the incoming and outgoing requests on the proxy system but I doubt it will yield anything and I already have a headache...

Edit2: Testes with a second Debian Stable machine (which updates correctly) as a comparison. The requests sent by them are the same, byte for byte according to Wireshark.

Answer 1

Solution (sort of)

It works through mitmproxy because I disabled certificate validation for it - with that configuration, I don't need the proxy. I have no idea how a failed certificate validation can lead to a 400 error because the HTTP communication shouldn't happen in the first place but there's clearly something happening here that's beyond my level of expertise. At least there's a workaround to try if someone has a similar problem with an HTTPS repo...