Webserver establishing new outgoing connections from port 443

Question

I've been trying to configure the firewall on a webserver running Apache using iptables commands. I took the approach of blocking all outgoing connections, except for those that are required. Everything seems to be working fine, although in the firewall logs I see some blocked outgoing connections with the source port of 443:

IPTables-Dropped: IN= OUT=eth0 SRC={SERVERIP} DST={DESTIP} LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=36863 DF PROTO=TCP SPT=443 DPT=37096 WINDOW=0 RES=0x00 RST URGP=0

In my firewall configuration file, I use the following rule:

iptables -A OUTPUT -p tcp --sport 443 -m state --state ESTABLISHED,RELATED -j ACCEPT

The fact that the packets are being dropped, suggests that the connection from port 443 has to be a NEW connection, which I want to block since I don't see the reason why my server should connect to any other machine this way.

The destination IPs change and look like IPs of ordinary clients, based on their location and whois information.

What am I missing here? Is this expected behavior that is somehow dictated by the HTTPS connection or are those connections suspect? Should I ACCEPT NEW connections as well?

TLDR; The webserver seems to be trying to establish NEW connections from port 443 to various clients, as indicated by the firewall log. Should this be happening? Should I allow this traffic? What could be its source?

EDIT: We're running Ubuntu Server 12.04

apache2 -v
Server version: Apache/2.2.22 (Ubuntu)
Server built:   Jul 12 2013 13:37:10

iptables -V
iptables v1.4.12

iptables -L -v
Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  814 86082 ACCEPT     all  --  lo     any     anywhere             anywhere
2382K   99M ACCEPT     all  --  any    any     anywhere             anywhere             ctstate RELATED,ESTABLISHED
    2   108 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:ssh
    0     0 ACCEPT     udp  --  any    any     anywhere             anywhere             udp dpt:ssh
 1404 77906 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:https
    0     0 ACCEPT     udp  --  any    any     anywhere             anywhere             udp dpt:https
  286 15356 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:http
    0     0 ACCEPT     udp  --  any    any     anywhere             anywhere             udp dpt:http
    0     0 ACCEPT     udp  --  any    any     anywhere             anywhere             udp spt:domain dpts:1024:65535 state ESTABLISHED
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp spt:domain dpts:1024:65535 state ESTABLISHED
    1    83 ACCEPT     udp  --  any    any     anywhere             anywhere             udp spts:1024:65535 dpt:domain state NEW,ESTABLISHED
    0     0 ACCEPT     udp  --  any    any     anywhere             anywhere             udp spt:domain dpt:domain state NEW,ESTABLISHED
    0     0 ACCEPT     icmp --  any    any     anywhere             anywhere             icmp echo-reply state RELATED,ESTABLISHED
   47  3892 ACCEPT     icmp --  any    any     anywhere             anywhere             icmp echo-request state NEW,RELATED,ESTABLISHED
   94 28900 LOGGING    all  --  any    any     anywhere             anywhere

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  any    lo      anywhere             anywhere
    0     0 ACCEPT     all  --  any    any     anywhere             anywhere             ctstate RELATED,ESTABLISHED

Chain OUTPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  527 34635 ACCEPT     udp  --  any    any     anywhere             anywhere             udp spts:1024:65535 dpt:domain state NEW,ESTABLISHED
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp spts:1024:65535 dpt:domain state NEW,ESTABLISHED
  407 60108 ACCEPT     udp  --  any    any     anywhere             anywhere             udp spt:domain dpts:1024:65535 state ESTABLISHED
    0     0 ACCEPT     udp  --  any    any     anywhere             anywhere             udp spt:domain dpt:domain state ESTABLISHED
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:ssh state NEW,ESTABLISHED
 1634  206K ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:https state NEW,ESTABLISHED
    0     0 ACCEPT     udp  --  any    any     anywhere             anywhere             udp dpt:https state NEW,ESTABLISHED
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:http state NEW,ESTABLISHED
    0     0 ACCEPT     udp  --  any    any     anywhere             anywhere             udp dpt:http state NEW,ESTABLISHED
25185   42M ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp spt:https state NEW,RELATED,ESTABLISHED
    0     0 ACCEPT     udp  --  any    any     anywhere             anywhere             udp spt:https state NEW,RELATED,ESTABLISHED
 1198  260K ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp spt:http state NEW,RELATED,ESTABLISHED
    0     0 ACCEPT     udp  --  any    any     anywhere             anywhere             udp spt:http state NEW,RELATED,ESTABLISHED
4891K  358M ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp spt:ssh state ESTABLISHED
    0     0 ACCEPT     udp  --  any    any     anywhere             anywhere             udp spt:ssh state ESTABLISHED
    0     0 ACCEPT     icmp --  any    any     anywhere             anywhere             icmp echo-request state NEW,RELATED,ESTABLISHED
   47  3892 ACCEPT     icmp --  any    any     anywhere             anywhere             icmp echo-reply state RELATED,ESTABLISHED
   69  2831 LOGGING    all  --  any    any     anywhere             anywhere

Chain LOGGING (2 references)
 pkts bytes target     prot opt in     out     source               destination
   89 16387 LOG        all  --  any    any     anywhere             anywhere             limit: avg 15/min burst 5 LOG level warning prefix "IPTables-Dropped: "
  163 31731 DROP       all  --  any    any     anywhere             anywhere

Answer 1

Is this expected behavior that is somehow dictated by the HTTPS connection or are those connections suspect? Should I ACCEPT NEW connections as well?

No it's not, yes they are, and no you shouldn't. Your server should not initiate new connections; it should accept incoming ones and respond on the session that is already open. If you were to allow your system to initiate new connections as long as they originate from a known port, that would be an attack vector for anyone trying to hack your system or use it e.g. in a bot net.

A probable reason for this traffic is that your apache server still thinks that the connection is open and is trying to reply to it, while your firewall thinks that the connection is closed and thus that the session is no longer active. This may happen if a connection is idle for a longer time than the conntrack module thinks reasonable.

To fix it, you need to make sure that the timeout values in Apache and the conntrack values are in sync. For apache, check the KeepAliveTimeout value. For the conntrack module, check your sysconfig for all conntrack items. Also check any firewall or routers between your webserver and the internet, to make sure it's not one of those that is dropping the connections.