On Benchmarking Databases: MySQL on Joyent versus AWS (part 1)
August 17, 2009 - by jason
Last week we announced the MySQL accelerators and scalable architecture offerings. Like the Zeus Accelerators, this is a continuation of our goal to offer individual servers and entire architectures of known performance. What one really wants from an entire infrastructure is for it to simply do what you need it to do. Not to be 50x servers with 4GB of RAM each with something something CPU, and blah blah drives.
In this first article, I wanted to talk about two things: 1) how one should do benchmarks and 2) what's exactly behind saying that we're "3x better".
Jason's Principles of Benchmarking
- Limit your variables
- Always use external clients with accesses of <1 ms latencies (unless that's part of what you're looking at but please see #1)
- Prime and normalize the systems. This is the same as burning them in. Run the tests, rest, run the tests, rests, run the tests, rest. And then get started. For example, I've heard some concern about how Amazon Web Services "first writes" are slow. This helps eliminate such things and hopefully allows for something closer to an "equilibrium state" than a first event BIG BANG type of thing.
- Do a sufficient number of experiments for the numbers to be statistically powerful.
- For your initial framing, look at the 100% state, the 50% state (equilibrium) and the 0% state. In the case of databases, 100% reads, 100% writes and 50% read:writes
I'll likely add to this list in the future.
Fundamentally, these are experiments like any others. You want to make sure you're measuring what you intend to measure and that you don't have an improper influence on the results.
In the case of Joyent Accelerators versus AWS EC2, we used Sysbench's OTLP benchmark and with both it's read-write and read-only tests. There is some existing, but rudimentary benchmark numbers using Sysbench's OTLP from Rightscale. So we could at least know that we were on the right track.
We used /, /mnt and a properly formatted ext3 filesystem on EBS. The EBS numbers trended slightly better so they were used. We ran test on all of the instance sizes and ended up having to go with the m1.large. These are 7.5GB of RAM Fedora Core 8 Basic 64bit instances running the MySQL 5.1.36 download from sun.com and the default my.cnf. We ended up having to go with the m1.large instances because it turns out that for 64-bit those are the minimum instances. Even 256 MB Joyent Accelerators are 64-bit, and I wanted us to compare 64-bit builds to 64-bit builds.
Systems: MySQL 5.1.36 64-bit on a 1GB Joyent Accelerator versus a m1.large 7.5 GB RAM Fedora Core 8.
Prime: 30 minutes, 10 minutes of rest, reboot.
Benchmark: Sysbench OTLP at 8 and 16 threads (the MySQL defaults). Ran for 10 minutes. Rinse. Repeat.
The average results from the 7.5GB m1.large instance were in line with the Rightscale numbers: 309 tps (transactions per second) for read-write and 417 tps for read-only (with standard deviations around 2%).
The 1GB Joyent Accelerators? The read-write result was 968 tps and the read-only average was 1247 tps. Both with pretty tight standard deviations as well.
There's a number of reasons we believe that from the start we can achieve greater OTLP database performance, and also why the systems themselves achieve an excellent level of performance compared to EC2. Next time I'll start with some fundamental system benchmarks and a bit about how we design our systems like you would with "super computers", and how the databases performance numbers are only going to get better with a few more refinements.