HTTPS @Scale

HTTPS @Scale
Arvind Mani, LinkedIn

Bio
● Head, Data & Infrastructure Security @
LinkedIn
● Prior - Yahoo, PayPal, Symantec, McAfee

What is HTTPS?
● HTTPS is HTTP over a channel
secured by TLS or SSL.
● TLS (predecessor SSL) are crypto
protocols designed to authenticate peer
in client to server communication and
ensure confidentiality and integrity of
data.

LinkedIn HTTPS Timeline
Mar 2012 Opt-in HTTPS
June 2013 team assembled (netops, security,
CDNOps, traffic) to make site default HTTPS.
Dec 2013 Launch default HTTPS in NL
Jan - Jun 2014 Rollout default HTTPS to US,
EU, most of APAC
Dec 2014 - Default HTTPS except CN and ZA

Challenges
Mixed Content
Site Speed
Scaling TLS
Session Upgrade
Security

What is it?
On a https page, resource origin is insecure.
What are the consequences?
MITM
Mixed Active Content Blocking - script, frames
Mixed Content

Measuring Mixed Content
Content-Security-Policy-Report-Only Header
default-src https:; img-src https: data:; report-uri
https://www.linkedin.com/lite/contentsecurity
Sample
{"csp-report":{"blocked-
uri":"http://www.lixanalytics.com/foo","document-
uri":"https://www.linkedin.com/profile?trk=ww","original-
policy":"default-src https:; img-src https: data:; report-uri
https://www.linkedin.com/csp","referrer":"","violated-
directive":"default-src https:"}}

CSP Gotchas
● Set on just HTTPS response
● Sample
● Exclude report requests from any page-view
rate-limiting
● unsafe-eval and unsafe-inline
● May not report on NPAPI Plugins
● Must Scale
● No cookies even if report-uri is same domain

Fixing Mixed Content
Ads 3rd, 4th, 5th party pixels must all be https,
even in iframe!
Source Code Scanner fix hardcoded external
links in dust templates
Dynamic Scanner fix logical bugs, links
constructed by backend applications

HTTP vs HTTPS
It all boils down to TLS Handshake latency!

TLS False Start
● Full handshake is 2
roundtrip
● Client validates
“Finished” from
server before
sending Application
Data
● With Client side
False Start, client
sends “Application
Data” before
validating Finished
=> just 1 roundtrip
● Is this secure?

TLS Session Ticket
● Pre-Master
Secret
● What is in a
ticket?
● Fallback is full
handshake
● Is this secure?
● Tickets and
PFS

Network Roundtrip
● Terminate TLS closer to user
● Where to build Points-of-Presence (PoP)?
See LinkedIn presentation at Velocity 2015
● Static content - CDN map

Online Certificate Status Protocol
● Claims of 30% slow down:
o DNS
o TCP
o OCSP request/response for chain
However:
● OCSP response cached by browser (even days), ocsp
response served off CDN
● Chrome disables OCSP checks by default

OCSP Stapling
Server caches OCSP response, sends in Certificate Status
Request
Pros
● Captive Portal
● Preserves browsing privacy
● Supported by nginx, traffic server, Apache, etc
Cons
● OCSP stapling (solution) not supported by mobile
● Stapling increases TLS handshake message size.

Cert Chain Size, initCwnd, SAN

Scaling TLS
TLS Termination hardware vs software
CDN TLS capacity multi CDN strategy
TLS Tuneup TLS record size, byte overhead,
AES-NI

“No login” HTTP -> HTTPS Upgrade
http? ajax? bot?
ramp
segment
yes no no yes
● Set “migration underway” hint cookie with TTL 10 minutes (avoid repeatedly upgrading bot that
refuses to be upgraded)
● 301 to HTTPS
● Kill old authentication cookie+session/mint new authentication cookie+session
● Set new authentication cookie secure => new cookie was never on wire in plaintext!
● Set TLS hint cookie that is NOT secure
● Implement reverse for rollback

SSL Stripping
How it Works?
Attacker MITM HTTP requests. 301 or rejecting
HTTP requests does not help.
Mitigation
Host Strict Transport Security (HSTS) and
preload

HSTS
Strict-Transport-Security: max-age=31536000; includeSubDomains;
preload
Window of vulnerability! Rollback with max-age=0
Preload
● Submit site https://hstspreload.appspot.com/ to enter
into hardcoded list
● Domain should not serve HTTP traffic
● includeSubDomains (www), preload flag, max age >=
18 weeks
Handle preload with care - hard to rollback

Perfect Forward Secrecy
● ClientKeyExchange - with RSA “pre-master”
secret encrypted with server public key
● DH vs DHE vs ECDHE
● Forward Secrecy with Session Resumption

TLS Configuration
● Disable weak/export ciphers
● Disable SSL 3.0, TLS 1.0??
● Enable PFS ciphersuites
● Attacks on SSL/TLS - insecure
renegotiation, BEAST, CRIME, FREAK,
POODLE, logjam, heartbleed (OpenSSL
implementation)

Pinning*
What problem does Pinning solve?
Rogue CA, some MITM, reduces trust requirements
What do you Pin?
server certificate, server public key, public key of cert in
chain
Public-Key-Pins-Report-Only: max-age=2592000;
pin-sha256="E9CZ9INDbd+2eRQozYqqbQ2yXLVKB9+xcprMF+44U1g=";
pin-sha256="LPJNul+wow4m6DsqxbninhsWHlwfp0JecwQzYpOLmCQ=";
report-uri="https://other.example.net/pkp-report"

Pinning Made Easier
● Fiddler, Corporate MITM supported*
● Reports on Pin failures
● Backup Pin required
● Testing
chrome://net-internals/#hsts

EdgeNetwork Security*
● HSM
● Secure Edge <-> Datacenter communication

References
● https://www.imperialviolet.org/2010/06/25/ov
erclocking-ssl.html
● https://github.com/linkedin/adfullssl
● http://www.w3.org/TR/CSP/ (note Level 2)
● http://velocityconf.com/devops-web-
performance-
2015/public/schedule/detail/42026
● https://blog.mozilla.org/tanvi/2013/04/10/mix

References
● https://www.igvita.com/2013/10/24/optimizin
g-tls-record-size-and-buffering-latency/
● https://hstspreload.appspot.com/
● http://vincent.bernat.im/en/blog/2011-ssl-
perfect-forward-secrecy.html

We’re Hiring
Contact
amani@linkedin.com
https://www.linkedin.com/in/arvindmani
Projects
● Inter-service Authentication & Access Control
● Web Application Security Frameworks & Tools
● Anti-scraping/DoS infrastructure
● Crypto libraries & services

HTTPS @Scale

More Related Content

What's hot

What's hot (20)

Similar to HTTPS @Scale

Similar to HTTPS @Scale (20)

HTTPS @Scale

Editor's Notes