Redis Everywhere - Sunshine PHP
- 2. HELLO WORLD
• Ricard
Clau, born and grown up in Barcelona
• Software
engineer at Hailo in London
• Symfony2
lover and PHP believer
• Open-source
• Twitter
contributor, sometimes I give talks
@ricardclau / Gmail ricard.clau@gmail.com
- 3. WHAT IS REDIS?
• REmote
• Created
• Open
DIctionary Server
in 2009 by Salvatore Sanfilipo (@antirez)
source
• Advanced
in-memory key-value data-structure server
• Part
of the NoSQL movement
• Stop
thinking SQL, think back to structures!
- 5. REDIS EVERYWHERE?
Redis is not a silver bullet... so let´s rephrase that with...
The amazing story on how we gradually introduced Redis
in a social videogaming company... and it saved us!
- 6. AGENDA
• REDIS
introduction
• Getting
started with PHP / Symfony2: sessions, caching
• Recipes
and comparison with other technologies
• Sharding
data. And some criticism!
• War
Stories from a 7.5M DAU project
• Final
thoughts
- 8. DATA TYPES
• Strings
• Lists
up to 512 Mb
of elements sorted by insertion order (max 2^32 - 1)
• Sets
unordered collection of elements supporting unions,
intersections and differences (max 2^32 - 1)
• Hashes
key-value pairs (max 2^32 - 1)
• Sorted
sets automatically ordered by score (max 2^32 - 1)
- 9. ONLY IN-MEMORY?
• Your
data needs to fit in your memory
has configurable persistence: RDB snapshots, AOF
persistence logs, combine both or no persistence at all
• It
• Think
like memcached on steroids with persistence
• http://redis.io/topics/persistence
• “Memory
is the new disk, disk is the new tape”
- 10. INTERESTING FEATURES
• Master-slave
• Pipelining
replication
to improve performance
• Transactions
• Publisher
• Scripting
(kind of with MULTI ... EXEC)
/ Subscriber
with LUA (~stored procedures)
• Predictable
performance (check commands complexity)
- 11. HARDWARE TIPS
is single-threaded, so a 2 core machine is enough
• Redis
using EC2, maybe m1.large (7.5Gb) is the most suitable,
but if it is not enough you might consider any m2 memory
optimized type
• If
• Amazon
• CPU
• Read
ElastiCache finally supports Redis!
is rarely the bottleneck with Redis
carefully the doc to maximize performance!
- 12. NEW IN REDIS 2.8
• CONFIG
REWRITE and IPv6 support
• Improvements
in scripting and key expiring algorithm
• Keyspace
notifications via PUB / SUB. (~Triggered events)
• Partial
resynchronization with slaves
• Better
•A
consistency support (allow writes only when N slaves)
fully rewritten Redis Sentinel
- 13. ALSO IN 2.8 (H|Z|S)?SCAN
• Ability
to iterate the data structures
• SCAN
cursor [MATCH pattern] [COUNT count]
• (H|Z|S)SCAN
• Cursor
• Count
• When
key cursor [MATCH pattern] [COUNT count]
is kind of a pointer that we send back and forth
is NOT the number of elements but the complexity
using MATCH some steps in iteration may return nil
- 15. PHP CLIENTS
• Predis
(PHP) vs phpredis (PHP extension)
• Predis
is very mature, actively maintained, feature complete,
extendable, composer friendly and supports all Redis versions
• Phpredis
is faster, but not backwards compatible
• Network
latency is the biggest performance killer
• http://redis.io/clients
- 16. REDIS-CLI AND MONITOR
• Redis-cli
to connect to a Redis instance
• Experiment
with http://redis.io/commands
• With
Monitor we can watch live activity!
- 17. SESSIONS, CACHING
• Super-easy
to implement with commercial frameworks
used: GET <session-id>, SETEX <session-id>
<expires> <serialized-data>
• Commands
• Fast
performance and also persistent!
• Caching
• Be
and temp-data storage work exactly equal
careful with temp-data storage and memory usage!
- 18. SYMFONY2 INTEGRATION
• There
must be a bundle for that...
• https://github.com/snc/SncRedisBundle
of the box: Session management, Monolog handler,
Swiftmailer spooling, Doctrine caching
• Out
• Full
integration with Symfony2 profiler
- 21. REDIS AS A QUEUE
• Using
• We
Lists and LPUSH / RPOP instructions
can have different languages accessing data
used that to send OpenGraph requests to Facebook
(REALLY slow API) with Python daemons
• We
• ~80k
• If
messages/minute processed with 1 m1.large EC2
you need some advanced message queuing features check
RabbitMQ (and Álvaro Videla´s talk) and others
- 22. REAL-TIME ANALYTICS
• We
can use hashes to group many stats under one key
• Most
of the times we have counters:
HINCRBY “stats" <key> <increment>
• HMGET “stats” <key1>
<key2> ... to retrieve values and
HMSET “stats” “stat1” <value1> “stat2” <value2> to set them
• HGETALL
to get the full hash
- 23. LEADERBOARDS
• Super-easy
• Each
with Redis, heavy consuming with other systems
board with a single key, using sorted sets
• ZINCRBY “rankXXX” <increment>
<userId>
• Top
N ZREVRANGE “rankXXX” 0 N [WITHSCORES]
• We
stored several milions of members under 1 key
- 24. WHO IS ONLINE? (I)
can use SETS to achieve it!
• We
• One
set for every minute, SADD <minute> <userId>
Time
+1m
5
+2m
2
+3m
1
4
2
1
5
4
1
3
SUNION
1
7
3
1
2
3
7
2
- 25. WHO IS ONLINE? (II)
• Online
users == everyone active in the last N minutes
• SUNION
<now> <now-1> <now-2> ...
• SUNIONSTORE “online” <now>
• Number
• Obtain
<now-1> <now-2> ...
of users: SCARD “online”
online users with SMEMBERS “online”
- 26. FRIENDS ONLINE?
• If
we store user´s friends in a SET with key friends-<user-id>
online: SINTERSTORE “friends-<userid>-online”
“online” “friends-<userid>”
10
4
2
1
3
7
SINTER
ONLINE
5
1
9
3
7
15
FRIENDS-12-ONLINE
• Imagine
how you would do it without Redis!
FRIENDS-12
• Friends
- 27. DISTRIBUTED LOCKING (I)
• Problem
in heavy-write applications: 2 almost concurrent
requests trying to update same records
• SELECT
FOR UPDATE? Easy to create deadlocks!
R1read
R1 updates
are lost!!!!!
R2read
timeline
R1
R2
R1save
R2save
- 28. DISTRIBUTED LOCKING (II)
• Locking
• SET
• And
•8
can be solved with Redis with Exclusive SET
<lock> 1 NX EX <seconds>
keep retrying until a specific timeout
m1.xlarge instances to lock 7.5M Daily Active Users
Zookeeper, perhaps better for scaling
or if we need different locking patterns
• Alternatives: Apache
- 30. NEEDS TO FIT IN MEMORY
• Redis
• Proxy
Cluster will be available in 3.0 (delayed many times)
servers: Twemproxy (also for memcached)
• Client-side
• Sharding
partitioning (supported in many clients)
strategies (range vs hash partitioning)
• Presharding
• Memory
can help scaling horizontally
is cheaper every day but this is a real need!
- 31. 100000-199999
id % 4 == 0
id % 4 == 1
200000-299999 300000-399999
id % 4 == 2
id % 4 == 3
1-99999
Easy to predict scale
Load not properly distributed
Load distributed ok
Fairly easy to reshard
RANGE PARTITIONING
May work in some cases, not a silver bullet
- 32. HASH
hash(key) % 4 == 0
hash(key) % 4 == 1
CRC16
CRC32
MD5
SHA1
MURMUR3
DIST
hash(key) % 4 == 2
hash(key) % 4 == 3
MODULUS
KETAMA
Distribution also depending on hash algorithm
Complicated resharding
HASH PARTITIONING
Better distribution... but still issues
If few keys, it also becomes useless
- 33. Many Redis instances in one
initial server
When needed, just split it
into more servers and you
don´t need resharding!!!
Be careful configuring
memory limits
PRESHARDING
Maybe overengineered
- 35. VS OTHER DATABASES
• Cassandra
or RIAK reshard automatically (at a big cost)
• Complicated
to fully automate failovers (Sentinel problems)
• Severe
• Tricky
• All
issues when slaves get out of sync, getting better
to inspect big sets and hashes prior to 2.8
NoSQL have issues, check aphyr.com “Call me maybe”
- 36. ABOUT REDIS CLUSTER
• CAP Theorem: Consistency, Availability, Partition
• Purists
• CAP
tolerance
blame it to not properly respect the principles
theorem is... a theorem and it comes with a cost
• The
goal is to provide some consistency and some failure
resistance without sacrificing other practical qualities
• We
could debate for hours around that point!
- 38. PROGRESSIVE USAGE
• DragonCity
• First
had ~2M DAU when we introduced Redis
usages: Queues PHP-Python and Temporary storage
• Introduced
• Used
Locking with Redis -> End of deadlocks
for temporary contests involving rankings
• Session
• About
tracking, cheat detection and many others
7.5M DAU and similar amount of servers
- 39. REDIS IS SUPER-FAST
• Your
network gets killed much before Redis is down
• Always
increase your ulimit values in Redis servers
• We
had Gigabit network connectivity between machines but
it was not enough to handle the data sent and retrieved
• Redis
• Do
was just at 20% of CPU
some heavy-load tests before launching!
- 40. SURPRISES WITH MEMORY
documentation: “Sets of binary-safe strings”so,
we changed values in sets from being just numbers to number
+ :<char> to add some info to values
• Redis
• We
expected ~20% memory increase but in was 500%
• https://github.com/antirez/redis/blob/unstable/src/t_set.c#L55
• If
value can be represented as long, it is stored more efficiently
• http://alonso-vidales.blogspot.co.uk/2013/06/not-all-redis-
values-are-strings.html
- 42. REDIS IS PERFECT FOR...
• Intensive
read-write data applications
• Temporary
• Data
stored data
that fits in memory
• Problems
that fit Redis built-in data types
• Predictability
in performance needed (all commands
complexity documented)
- 43. BUT IS NOT SUITABLE WHEN..
• Big
data sets, archive data
• Relational
• We
data (RDBMS are absolutely fine to scale)
don´t know how we will access data
• Reporting
• It
applications (no where clauses)
gets tricky for distributed applications (replication, clustering)
• ALWAYS
choose the right tool for the job!
- 44. SPECIAL THANKS
• Ronny
López (@ronnylt) & Alonso Vidales (@alonsovidales)
• All
backend and systems engineers at @socialpoint
• Of
course, to all of you for being here today!
our open-source project Redtrine, PR welcome:
https://github.com/redtrine/redtrine
• Check