SCAN vs KEYS performance in Redis

Question

A number of sources, including the official Redis documentation, note that using the KEYS command is a bad idea in production environments due to possible blocking. If the approximate size of the dataset is known, does SCAN have any advantage over KEYS?

For example, consider a database with at most 100 keys of the form data:number:X where X is an integer. If I want to retrieve all of these, I might use the command KEYS data:number:*. Is this going to be significantly slower than using SCAN 0 MATCH data:number:* COUNT 100? Or are the two commands essentially equivalent in this circumstance? Would it be accurate to say that SCAN is preferable to KEYS because it protects against the scenario where an unexpectedly large set would be returned?

Answer 1

You shouldn't care about current command execution but about the impact to all other commands, since Redis processes commands using a single thread (i.e. while a command is being executed all others need to await until executing one ends).

While keys or scan might provide you similar or identical performance executed alone in your case, some milliseconds blocking Redis will significantly decrease overall I/O.

This the main reason to use keys for development purposes and scan on production environments.

OP said:

"While keys or scan might provide you similar or identical performance executed alone in your case, some milliseconds blocking Redis will significantly decrease overall I/O." - This sentence seems to indicate that one command blocks Redis, and the other doesn't, which can't be the case. If I am guaranteed 100 results from my call to KEYS, in what way is it worse than SCAN? Why do you feel that one command is more prone to blocking?

There should be a good difference when you can paginate the search. It's not the same being forced to get 100 keys in a single pass than being able to implement pagination and get 100 keys, 10 by 10 (or 50 and 50). This very small interruption can let other commands sent by the application layer be processed by Redis. See what Redis official documentation says about this:

Since these commands allow for incremental iteration, returning only a small number of elements per call, they can be used in production without the downside of commands like KEYS or SMEMBERS that may block the server for a long time (even several seconds) when called against big collections of keys or elements

.

Answer 2

The answer is in the SCAN documentation

These commands allow for incremental iteration, returning only a small number of elements per call, they can be used in production without the downside of commands like KEYS or SMEMBERS that may block the server for a long time (even several seconds) when called against big collections of keys or elements.

So ask for small chunks of data rather than getting whole of it

Also as Matías Fidemraizer pointed out, Redis is single threaded and KEYS is a blocking call thus blocking any incoming requests for operation until execution of KEYS is done.

Whether your data is small or not, it never hurts to apply best practices.

Answer 3

1. There is no performance difference between KEYS and SCAN other than pagination (count) where the amount bytes transferred (IO) from redis to client will be controlled in pagination.

2. The count option it self has its own specification where sometimes you will not get data, but still scan cursor is on, so will get data in the next iterations. So the count option should be reasonable amount say 200 to something max to avoid multiple round trip time. I think this value depends on total number of keys in your db.

3. There is no point/difference when we use SCAN within LUA compare to KEYS, though there is no IO involved, still both are blocking other calls till entire big collection get iterated. I haven't tried this, my guess it is.

Answer 4

The main difference between KEYS and SCAN is KEYS returns the matched keys in one request while SCAN uses the cursor. Both behind-the-scenes iterate all over the keys.

During my testing, I created a Redis database with 50k records, I tested against KEYS and SCAN.

Here is what I have:

• KEYS takes 3000 microseconds
• SCAN - No COUNT, takes 11000 microseconds
• SCAN - COUNT:100000, takes 14000 microseconds. Split into two commands
• SCAN - COUNT:10000, takes 14000 microseconds. Split into 6 commands

My conclusion is:

• As long as we have a fairly small number of keys, we can still use KEYS, a little bit faster then SCAN
• In case we use SCAN, we need to choose the good COUNT so that it will split into smaller chunks. Sometimes, the performance is not that good comparing to KEYS. Ex: when I choose 10k, each chunk takes 2300 microseconds

There is no point/difference when we use SCAN within LUA compare to KEYS, though there is no IO involved, still both are blocking other calls till entire big collection get iterated. I haven't tried this, my guess it is.

-> From the screenshot, I can see that using LUA with SCAN, the commands are still able to split into smaller chunks without blocking other calls

LUA script

EVAL "local cursor = '0' local count = 0 repeat local result = redis.call('SCAN', cursor, 'match', ARGV[1], 'count', 100000) cursor = result[1] local keys = result[2] until cursor == '0' if count == 0 then return 'no keys' else return count end" 0 USER:BB*