I think you're correct in your logic that stat curr_connections is the number of connections that are current.
curr_connections - Number of open connections to this Memcached server, should be the same value on all servers during normal operation. This is something like the count of mySQL's "SHOW PROCESSLIST" result rows.
Source: Memcached statistics (stats command)
When I setup memcached I noticed that it always maintained 10 as the least amount of curr_connections too.
$ echo stats | nc localhost 11211 | grep curr_conn
STAT curr_connections 10
But why 10?
If you run memcached in verbose mode you'll notice the following output:
$ memcached -vv
...
<26 server listening (auto-negotiate)
<27 server listening (auto-negotiate)
<28 send buffer was 212992, now 268435456
<28 server listening (udp)
<28 server listening (udp)
<28 server listening (udp)
<29 send buffer was 212992, now 268435456
<28 server listening (udp)
<29 server listening (udp)
<29 server listening (udp)
<29 server listening (udp)
<29 server listening (udp)
If you tally up the listening servers (8) + 2 servers (auto-negotiated) you'll discover why there are 10 base servers to start, at least that's what I thought at first. But you need to dig deeper to better understand what's going on.
It would appear that memcached is multi-threaded and so the output that you were taking notice of isn't really how one would tally up the connections.
Notice the threads
The output of ps -eLf shows the threads:
$ ps -eLf | grep memc
saml 20036 4393 20036 0 6 20:07 pts/7 00:00:00 memcached -vv
saml 20036 4393 20037 0 6 20:07 pts/7 00:00:00 memcached -vv
saml 20036 4393 20038 0 6 20:07 pts/7 00:00:00 memcached -vv
saml 20036 4393 20039 0 6 20:07 pts/7 00:00:00 memcached -vv
saml 20036 4393 20040 0 6 20:07 pts/7 00:00:00 memcached -vv
saml 20036 4393 20041 0 6 20:07 pts/7 00:00:00 memcached -vv
There is one master node and 5 worker threads.
Here's what the output of lsof looks like when I've made 3 connections to memcached using the same method as you, telnet localhost 11211.
So it would appear that each thread is maintaining a connection (or a reference to each connection) as they're made and kept open. As soon as the telnet connections are closed they go away from this list.
So how can we count the connections?
So if you wanted to tally up the connections you could either subtract 10 from the curr_connections result, or you could run lsof and count the number of connections that are associated with the main PID. Notice this output:
$ lsof |grep memcac | grep IPv
memcached 20036 saml 26u IPv4 7833065 0t0 TCP *:memcache (LISTEN)
memcached 20036 saml 27u IPv6 7833066 0t0 TCP *:memcache (LISTEN)
memcached 20036 saml 28u IPv4 7833069 0t0 UDP *:memcache
memcached 20036 saml 29u IPv6 7833070 0t0 UDP *:memcache
memcached 20036 saml 30u IPv6 7962078 0t0 TCP localhost:memcache->localhost:38728 (ESTABLISHED)
That last line is an active connection. So we could count them like this:
$ lsof -p $(pgrep memcached) | grep "memcache->" | wc -l
1
But your output shows IPv4 & IPv6, what's up with that?
To simplify the example even more let's start memcached up and force it to only listen on a single IPv4 address, 192.168.1.20. In the above examples we were starting memcached up on all interfaces and connecting to it using localhost, so let's take another look.
$ memcached -vv -l 192.168.1.20
...
<26 server listening (auto-negotiate)
<27 send buffer was 212992, now 268435456
<27 server listening (udp)
<27 server listening (udp)
<27 server listening (udp)
<27 server listening (udp)
Notice that we're only getting 1/2 as many? Previously we had 2 auto-negotiated servers and 8 UDP, this time we have 1 auto and 4 UDP. Why? Well we've told memcached to only listen on the IPv4 interface, previously it was listening on everything, the IPv4 and localhost. We can convince ourselves of this by attempting to connect to the server on localhost:
$ telnet localhost 11211
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
See we can't connect. But we can using the IPv4 address:
$ telnet 192.168.1.20 11211
Trying 192.168.1.20...
Connected to 192.168.1.20.
Escape character is '^]'.
What does stats say now?
STAT curr_connections 5
See? The curr_connections is showing the number 5 (1 auto + 4 UDP).
4 UDPs?
Why are we running 4 UDPs? This would appear to be a default setting in memcached. You can see the settings using the stats settings command when youtelnet` to the server:
stats settings
STAT maxbytes 67108864
STAT maxconns 1024
STAT tcpport 11211
STAT udpport 11211
STAT inter 192.168.1.20
STAT verbosity 2
...
STAT num_threads 4
STAT num_threads_per_udp 4
...
Can we change that value? Sure it's the -t # switch to memcached.
$ memcached -vv -l 192.168.1.20 -t 1
...
<11 server listening (auto-negotiate)
<12 send buffer was 212992, now 268435456
<12 server listening (udp)
So now we only have the main listener (auto) and 1 UDP thread. If we check stats now:
STAT curr_connections 2
Incidentally, we can't set the number of threads to 0.
$ memcached -vv -l 192.168.1.20 -t 0
Number of threads must be greater than 0
So the lowest we can get curr_connections to go is 2. If we open up 6 telnets (5 from ourself - greeneggs and 1 from another remote server named skinner) we'll see the following in our lsof output:
$ lsof -p $(pgrep memcached) | grep ":memcache"
memcached 24949 saml 11u IPv4 847365 0t0 TCP greeneggs.bubba.net:memcache (LISTEN)
memcached 24949 saml 12u IPv4 847366 0t0 UDP greeneggs.bubba.net:memcache
memcached 24949 saml 13u IPv4 855914 0t0 TCP greeneggs.bubba.net:memcache->greeneggs.bubba.net:48273 (ESTABLISHED)
memcached 24949 saml 14u IPv4 872538 0t0 TCP greeneggs.bubba.net:memcache->skinner.bubba.net:41352 (ESTABLISHED)
memcached 24949 saml 15u IPv4 855975 0t0 TCP greeneggs.bubba.net:memcache->greeneggs.bubba.net:48298 (ESTABLISHED)
memcached 24949 saml 16u IPv4 855992 0t0 TCP greeneggs.bubba.net:memcache->greeneggs.bubba.net:48305 (ESTABLISHED)
memcached 24949 saml 17u IPv4 859065 0t0 TCP greeneggs.bubba.net:memcache->greeneggs.bubba.net:48316 (ESTABLISHED)
memcached 24949 saml 18u IPv4 859077 0t0 TCP greeneggs.bubba.net:memcache->greeneggs.bubba.net:48326 (ESTABLISHED)
So we still have the auto + the 1 UDP and 6 other connections. Our stats command shows this:
STAT curr_connections 8
So no surprise there.
