Benchmarking ZeroMQ Erlang bindings over Gbit network.

If you are planning a large, or not even that large systems, you need some values to estimate how much load can the system take, what hardware you need etc. If you plan on using a message bus in your system, you really need to know its performance in the setup you plan on using it in. These automated benchmarks may make your life a little bit easier.
More info on ZeroMQ
Check out the benchmarks section!


I have started with the erlang 0mq binding. Other bindings may have similar performance tests, and they are also included in the distribution. Check out the /perf folder of whatever you are using.


Erlang - Erlang

First, you will need to adapt the perfgraphs.py script to use SSH. Find definitions of localcmd and remotecmd in both thr_run and lat_run methods and make them use SSH instead of local commands. My version looked like this:

localcmd = "ssh -l username 192.168.2.120 ./zmqPerf.sh local_thr"+(active and "_active" or "")+".erl tcp://eth0:5557 "+str(msgsize)+" "+str(msgcount)
remotecmd = "escript perf/remote_thr.erl tcp://192.168.2.120:5557 "+str(msgsize)+" "+str(msgcount)

The other machine I was using had the IP 192.168.2.120, hence this value. Also, you would probably like to avoid entering your SSH password all the time (it didn't work for me at all, but even if..), so it is a good idea to use public keys.

You can do that the following way:
$ ssh-keygen -t dsa
when asked for the passphrase, leave it empty. That generates a key in ~/.ssh/id_dsa.pub, and you will need to upload it to the server. SSH to the other machine and create a directory ~/.ssh/authorized_keys
then upload the key to the server
$ cat ~/.ssh/id_dsa.pub | ssh user@host "cat - >> ~/.ssh/authorized_keys"

Then try logging to the server with
$ ssh user@host
If everything went well, you should be directly logged in.


Since we need to execute some commands on the server, such as cd in the erlzmq2 directory followed with some parameter-dependent commands starting the performance measurements, the easyest way to do this is y writing a miniature shell script.

#! /bin/bash
cd erlzmq2
./perf/$1 $2 $3 $4

This does the job, since it takes the required arguments from the python script, such as the file to run and its parameters. Save it as zmqPerf.sh in your home directory at the server you are SSH'ing to and make it executable.

I assumed you installed erlzmq2 in your home folder. Don't forget to run the make once you got it with GIT.
If you need to benchmark bindings to other languages, you will need to adapt the shell script to start whatever you need started.

Here are the results of my measurements. I have tested zmq between erlang bindings on two machines (AMD, 6 and 4 cores, more then enough RAM, gigabit ethernet on a consumer level Gbit switch, both running Ubuntu 10.10 and the current version of erlzmq2)

Erlang - C++

I have updated the script to work with the performance tests included in the zmq distribution. You can find the script here.  Note that you need to change hostnames and usernames for SSH to work, and that the .sh filename may be different (lat.sh in my case).
On the server side, you need to make a change in your zmqPerf.sh

#! /bin/bash

#cd erlzmq2
cd zeromq2-1
./perf/$1 $2 $3 $4 | tee outfile


Once I ran the tests, following results came back:

Only one mode (named passive, because of my laziness) is available, since there are no other options in C++ version, unlike the erlang binding which has 2 modes.


Interpretation of the results I leave to you, and I hope this article may encourage  you and help you run benchmarks in your specific environment.