Postgres connections at scale
•
0 likes•466 views
PostgreSQL connections at scale was the presentation by our external speaker at our 8th opensource database meetup. The presentation helps you comprehend on database connections with its cost, gauge the need for a connection pooler, Pgbouncer overview with its features, monitoring, and deployment best practices.
Report
Share
Postgres connections at scale
- 2. No copyright, Nothing reserved >_ thedataguy.in About me Bhuvanesh (aka The DataGuy) Data Architect + DBA https://linkedin.com/in/rbhuvanesh/ BhuviTheDataGuy BhuviTheDataGuy BhuviTheDataGuy (Tech Blog) https://thedataguy.in (Tech Blog) https://bhuvane.sh/
- 3. No copyright, Nothing reserved >_ thedataguy.in This presentation is based on my personal experience & research about PostgreSQL and pgbouncer. No copyright !! Nothing reserved !!! Icons used - https://undraw.co/ GIFs used - https://giphy.com/ Disclaimer
- 4. No copyright, Nothing reserved >_ thedataguy.in Agenda 1. A database connection and its Cost 2. Need for a connection pooler 3. Pgbouncer - Introduction 4. How pgbouncer works and its features 5. Monitoring the pgbouncer 6. Pgbouncer deployment best practices
- 5. No copyright, Nothing reserved >_ thedataguy.in Why PostgreSQL doesn’t scale to a large number of connections?
- 6. No copyright, Nothing reserved >_ thedataguy.in Life cycle of a connection in MySQL Client MySQL Server Handshake & Authentication Thread cache User thread THD Query execution Results Close Conn Reference: https://mysqlserverteam.com/mysql-connection-handling-and-scaling/ Create conn
- 7. No copyright, Nothing reserved >_ thedataguy.in Cost of a single PostgreSQL connection ● Each connection is a process fork with roughly consume 10 MB memory (a research from heroku) ● Each connection will simultaneously open up to 1000 files (default configuration) Let’s assume your DB is consuming 400 connections, then ● 10MB * 400 connections = 4GB Memory ● 1000 Files * 400 connections = 4,00,000 Files Example:
- 8. No copyright, Nothing reserved >_ thedataguy.in What about Idle connections? States of PostgreSQL connection: Active currently running Idle Its not doing anything Idle in transaction currently not doing anything and could be waiting for an input Idle in transaction(aborted) the connections that were idle in the transaction that have since been aborted “Idle connections are not just idle, they eat the resources in all the ways.”
- 9. No copyright, Nothing reserved >_ thedataguy.in Cost of the idle connections A benchmark from AWS: ● Open 100 connections. ● Leave the connections idle for 10 minutes. ● Close the connections. Reference: https://aws.amazon.com/blogs/database/resources-consumed-by-idle-postgresql-connections/
- 10. No copyright, Nothing reserved >_ thedataguy.in More incoming connections? Then your PostgreSQL server is in Danger! We need a Superhero Not the Black Panther, but pgbouncer
- 11. No copyright, Nothing reserved >_ thedataguy.in pgbouncer ● Lightweight connection pooler ● Reduce the number of backend connections ● Connections economy (creating a connection is a fork and acquire a ProcArrayLock) ● No need for special authentication PGBouncer Applications Postgre SQL
- 12. No copyright, Nothing reserved >_ thedataguy.in What does a connection pooler do? Postgresql Server Applications PGBouncer Queue 1 2 3 1 2 3 Opening connections Pool size = 3 Server connections Client connections POOL
- 13. No copyright, Nothing reserved >_ thedataguy.in Lets deep dive into pgbouncer
- 14. No copyright, Nothing reserved >_ thedataguy.in Pgbouncer pooling mode Session ● Each client(app) connection will open backend connections and the connection will remain open until the client closes. ● Client and Server connections are mapped ● It is almost similar to using the database directly without any connection pooler. ● The connection is transparent. PGBouncer Applications Postgre SQL
- 15. No copyright, Nothing reserved >_ thedataguy.in Pgbouncer pooling mode Transaction ● Client connect to the server connection only during the transaction, after the transaction the same server connection can be used to run another session’s transaction. ● All the queries inside the {begin… end} will be executed in one server connection. ● Session variables and prepared statements will not work here. ● The connection is not transparent. PGBouncer Applications Postgre SQL T1 {Begin.. Insert ... Delete... end} T2 {Begin.. Update... Drop... end} T3 {Begin.. Create... Insert... end}
- 16. No copyright, Nothing reserved >_ thedataguy.in Pgbouncer pooling mode Statement ● Very aggressive mode. ● The server connection back to the pool when the statement completes. ● Multi statement is not possible here. ● Auto commit is always on in this mode. PGBouncer Applications Postgre SQL T1 Insert … Update… delete Insert Update Delete
- 17. No copyright, Nothing reserved >_ thedataguy.in Adding the PostgreSQL Server Like a connection string ;; Common patterns db1 = host=primary dbname=db1 db2 = host=standby dbname=db2 * = host=standby ;; Custom pool size db2 = host=standby dbname=db2 pool_size=50 reserve_pool=10 List of parameters: ● dbname ● host ● port ● user ● password ● auth_user ● client_encoding ● datestyle ● timezone ● pool_size ● reserve_pool ● max_db_connections ● pool_mode ● connect_query ● application_name Pool size value can be override based on users ● Pool_size =10 ● No of databases = 10 ● No of users = 2 ● Pool size can be override to (no of db * no of users) = 20
- 18. No copyright, Nothing reserved >_ thedataguy.in Authentication Auth File with auth type ● A txt file that contains username and the password ● Password can be plain text of MD5 hash(recommended) ● Supported auth types: ○ any ○ trust ○ plain ○ md5 ○ cert ○ hba ○ pam pg_hba.conf ● Its very similar to postgresql’s hba methold. ● Same postgresql’s syntax will work here. ● But LDAP, pam and a few other methods will not work. Auth_user with query ● Automatically loads the username and password from the target database. ● Just give an user and password(mention it in the auth_file) for the authentication(like a dedicated user), then it’ll fetch the user, password from the database. ● example: SELECT usename, passwd FROM pg_shadow WHERE usename=$1
- 19. No copyright, Nothing reserved >_ thedataguy.in Authentication cont... ;; auth file auth_type = trust auth_file = /etc/pgbouncer/userlist.txt ;; HBA-style ;auth_hba_file = ;; Auth user with query ;auth_query = SELECT usename, passwd FROM pg_shadow WHERE usename=$1
- 20. No copyright, Nothing reserved >_ thedataguy.in Connections and Pool Parameter Description max_client_conn Maximum number of connection allowed in pgbouncer default_pool_size How many server connections to allow per user/database pair. min_pool_size Minimum number of server connections reserve_pool_size reserve_pool_timeout If the pool is full, and a connection is waiting more than the reserve_pool_timeout then it’ll use extra connection from this reserve pool. It should be less in size.
- 21. No copyright, Nothing reserved >_ thedataguy.in Example 1: How the connection pool makes the connection Connection: db1 = host=localhost dbname=db1 Pool: default_pool_size = 20 pool_mode = session Auth file: user1:xxxxxx user2:yyyyy PGBouncer App1 Postgre SQL Connection scenario: ● App1 is connecting to the DB via user1 with 5 connections ● App2 is connecting to the DB via user2 with 5 connections App2 User1 5 conn User1 5 conn User2 5 conn User2 5 conn 10 conn
- 22. No copyright, Nothing reserved >_ thedataguy.in Example 2: where the pool values override Connection: db1 = host=localhost dbname=db1 Pool: default_pool_size = 20 pool_mode = session Auth file: user1:xxxxxx user2:yyyyy PGBouncer App1 Postgre SQL Connection scenario: ● App1 is connecting to the DB via user1 with 15 connections ● App2 is connecting to the DB via user2 with 15 connections App2 User1 15 conn User1 15 conn User2 15 conn User2 15 conn 30 conn
- 23. No copyright, Nothing reserved >_ thedataguy.in Set the Hard limit for the pool size Parameter Description max_db_connections Pgbouncer will not allow more than this value for a database max_user_connections Maximum pool for a user Option 1: Set the limit based on DB or user level Option 2: Define separate connections for each DB and set the pool size via connection db1 = host=34.72.164.39 dbname=db1 pool_size=10 db2 = host=34.72.164.39 dbname=db2 pool_size=10 [OR] db1 = host=34.72.164.39 dbname=db1 pool_size=10 max_db_connections=10 db2 = host=34.72.164.39 dbname=db2 pool_size=10 max_user_connections=10
- 24. No copyright, Nothing reserved >_ thedataguy.in Lets see the real world example ● PostgreSQL max connections = 300 ● No of databases = 1 ● No of users = 1 ● Pool size = 120 ● Pool mode = session ● Max client connections = 4k ● Reserve pool size = 10 ● Reserve pool timeout = 10s
- 25. No copyright, Nothing reserved >_ thedataguy.in Applications pgbouncer Active Pool pool_size 120 Waiting area 4000 PostgreSQL 200 connections First 120 connections Active server connections 120 Last 80 connections 10 sessions executed and returning the results 10 conn back to pool 10 conn move to pool Recently moved 10 connections Other sessions returning the results How pgbouncer works internally? Reserve pool Size = 10 reserve_pool _timeout = 5 [From reserve Pool] Active Server Connections 10
- 26. No copyright, Nothing reserved >_ thedataguy.in Zero downtime Maintenance
- 27. No copyright, Nothing reserved >_ thedataguy.in Seamless switchover to a standby App 1 App 2 App 3 Queue/ Waiting area PostgreSQL pool Standby 1 Pause 2 Reload 3 Resume Wait until all the running transaction will commit Pool will reconnect to the new Database Config file: db1 = host= primary dbname=db1 db1 = host= standby dbname=db1 PGBouncer Replication Max_client_connections Before switchover After switchover
- 28. No copyright, Nothing reserved >_ thedataguy.in Restart without an actual restart -R, --reboot Do an online restart. That means connecting to the running process, loading the open sockets from it, and then using them. If there is no active process, boot normally. Use this in your systemctl or systemd file: pgbouncer -R /etc/pgbouncer/pgbouncer.ini 1 2 3 4 5 Online Restart Flow 1. Create new pgbouncer instance, make server connections 2. Send SUSPEND command to the old pgbouncer instance 3. Transfer the pool from old instance to new instance 4. Shutdown the old instance 5. Resume the work on the new instance
- 29. No copyright, Nothing reserved >_ thedataguy.in Monitoring CLI and Prometheus exporter
- 30. No copyright, Nothing reserved >_ thedataguy.in Monitor via CLI Pgbouncer config: ● Max pool = 300 ● Max client = 800 Connect to pgbouncer db: psql -h 127.0.0.1 -p 6432 -U user1 pgbouncer Run a pgbench test: pgbench -h 127.0.0.1 -U user1 -C -c 800 -j 2 -t 10000 -p 6432 postgres
- 31. No copyright, Nothing reserved >_ thedataguy.in Monitoring with the prometheus exporter and Grafana Reference: https://medium.com/searce/grafana-dashboard-for-pgbouncer-and-monitor-with-percona-pmm-3170d3eb4d14 Prometheus exporter: pip3 install prometheus-pgbouncer-exporter Grafana Dashboard: The dashboard is open source(we developed this dashboard) Dashboard ID: 13353
- 32. No copyright, Nothing reserved >_ thedataguy.in Pgbouncer deployment pattern Pattern 1 Pattern 2 Pattern 3 Pattern 4 Pattern 5
- 33. No copyright, Nothing reserved >_ thedataguy.in What I didn’t cover? 1. Timeout parameters 2. High availability of pgbouncer 3. TLS 4. server_reset_query - an important parameter 5. And more
- 34. No copyright, Nothing reserved >_ thedataguy.in Thank you !!! And any Questions? Get this deck here: bit.ly/pgbouncer-deck