Scaling to 100,000 cc

Configuring Linux for many concurrent connections.

Pretty much worked out of the box due to the nature of nginx connection management. Did not reach 1,000,000 cc in initial testing, hitting some TCP limit somewhere at 240,000cc.

Memory is an important limit.

Kernel TCP buffers read 4096 + write 4096
xtomp buffers      read 400  + write 0
xtomp connections  sizeof ngx_connection_t xtomp_session_t xtomp_subscriber_t 2 * ngx_event_t etc...

nofiles limit needs to be set, which is number of open files for the user xtomp uses to spawn a worker.
Linux has one file per socket.
The xtomp process user is never root which makes things more complicated.

Hard to configure ulimit above 65536 even as root.

PAM settings in /etc/security/limits.conf have no effect.

As root, set into /proc filesystem to tune kernel limits.

echo 4000000 > /proc/sys/fs/file-max
echo 4000000 > /proc/sys/fs/nr_open

echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
sysctl -w net.ipv4.tcp_rmem="1024 65536 8388608"
sysctl -w net.ipv4.tcp_wmem="1024 65536 8388608"

Magic is in xtomp.conf

worker_processes  1;
worker_rlimit_nofile 1000000;
events {
    worker_connections  1000000;
}

This configures one million concurrent connections using one worker process. xtomp does not support more than one process yet.

Check real limits of the worker process

pid=$(ps -efwww|grep "xtomp: worker"|grep -v grep|cut -c 10-15|tr -d ' ')
cat /proc/${pid}/limits

Max memory for TCP needs to be quite high, dependes on how much RAM you have.
Measured in pages.

getconf PAGE_SIZE

So this is 8Gig max TCP memory

sysctl -w net.ipv4.tcp_mem="50576 64768 1953125"

Default mem per connection

sysctl -w net.core.rmem_default=212992
sysctl -w net.core.wmem_default=212992

reduce if we expect lots of small messages (this wil be too low for normal operations)

sysctl -w net.core.rmem_default=4096
sysctl -w net.core.wmem_default=4096

sysctl -w net.core.rmem_default=65536
sysctl -w net.core.wmem_default=65536

Some TCP stack tuning /etc/sysctl.conf

# General gigabit tuning:
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_syncookies = 1
# this gives the kernel more memory for tcp
# which you need with many (100k+) open socket connections
net.ipv4.tcp_mem = 50576   64768   1953125
net.core.netdev_max_backlog = 2500
# I was also masquerading the port comet was on, you might not need this
net.ipv4.netfilter.ip_conntrack_max = 1048576

Check memory usage in Kb

ps -aux | egrep 'VSZ|xtomp'

Now we have to test so we need to setup clients, this hits socket limits since clients listen on a local port from 0 - 65536

First step is max the ephemeral ports available to clients

echo "1024 65535" > /proc/sys/net/ipv4/ip_local_port_range

This gives you 64511 ports per IP

The you need multiple IP addresses for a million concurrent connections you need 16 IP addresses.

Little known secret is that localhost listens on 127.0.0.1/24 so there are plenty of IP addresses.

I used a modified version of xtomp.js (https://github.com/teknopaul/stomp-js) that uses nodejs' sockets. I bind clients to a IP between 127.0.0.1 and 127.0.0.201 randomly.

xtomp.socket.connect({
    port : xtomp.port, 
    host : xtomp.host,
    localAddress : '127.0.0.' + (Math.floor(Math.random() * 200) + 1)
});