Forcing HTTPS through proxy by DNS / How works "Smart DNS" with HTTPS?

Question

I'd like to force HTTP(S) traffic to certain websites from my home network through a proxy running on a VPS.

I installed tinyproxy on the VPS, running on port 8080. On my computer I'm running

sudo ssh -i /home/user/.ssh/id_rsa -nNT -L 80:localhost:8080 -L 443:localhost:8080 remoteuser@vps

to access the proxy locally. When I configure localhost:80 as a proxy in Firefox' settings everything works fine (HTTP and HTTPS).

But as I don't want all traffic to go through that proxy, I added this to my PC's /etc/hosts:

127.0.0.1     server.example 

server.example is the name of a webserver configured for both HTTP and HTTPS.

Accessing http://server.example works, while opening https://server.example in Firefox fails with the error

SSL_ERROR_RX_RECORD_TOO_LONG

Also curling the page over HTTPS fails:

curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

As far as I understand this issue is caused by tinyproxy responing to the HTTPS request with plain HTTP.

First, is it theoretically possible to implement what I want? Those so-called "Smart DNS" services seem exactly to do that.

Is this problem caused by my setup or is tinyproxy rather not able not do that? Is there a different proxy server which is capable of that?

EDIT: At the moment I'm running the SSH tunnel on my PC and the DNS modification is also locally, but I'd like to deploy this on my router later, so that the traffic from any device of my network goes through this proxy.

EDIT2: As @SteffenUllrich wrote in the comments, when HTTPS is routed over a HTTP proxy, the client first sends an unencrypted CONNECT repuest before the handshake happens. See here (SSH tunnel to VPS is running on 10.0.5.4):

This is necessary, because else the proxy wouldn't be able to determine to which server the request should be forwarded to.

But how exactly do those "Smart DNS" proxy servers work? (For general information see here)

Because they do not need CONNECT requests, they seem to be working without. But how can they tetermine the server the request should be forwarded to? Here's a dump of a HTTPS connection using such a "Smart DNS" proxy (IP is 37.x.x.x). (The DNS is manipulated to point to that proxy):

So, how does this work? And furthermore, is there a way to archieve that on my VPS (maybe not with Tinyproxy, but with other software)?

Answer 1

Looks like tinyproxy won't really transparently proxy HTTPS connections, the client has to use the CONNECT method which means it has to be setup to use the proxy.

You cannot transparently proxy HTTPS without installing a MITM certificate anyway, otherwise each access will give you certificate errors or fail due to that reason.

You probably need something more sophisticated like squid.

Answer 2

But how exactly do those "Smart DNS" proxy servers work? They do not need CONNECT requests, they seem to be working without.

I think these Smart DNS proxies work by extracting the target domain from the HTTP request for plain HTTP. For HTTPS they probably extract the domain from the SNI TLS extension in the ClientHello message at the start of the TLS handshake.

A similar setup could be build for example with the squid proxy. This proxy can be configured to be used as transparent proxy at the central router. Since all clients use this router as gateway to the internet the traffic on the relevant ports (80,443 for http and https) can be redirected to the proxy with packet filter rules, i.e. no special DNS setup is needed.

The proxy then can extract the real target from the HTTP header (plain HTTP) or the ClientHello (HTTPS) without breaking the SSL encryption - see squid:SSL Peek And Splice for details. The squid proxy can also be configured to have different upstream proxies (like your VPS) or direct connection depending on the target.