Performance Battle of NoSQL blob storages #3: Redis

We have already measured performance stats for Apache Cassandra and Apache Kafka as well. To include Redis within a comparison of persistent storages could see like some misunderstanding at first sight. On the other hand, there are certain use-cases allowing us to think about to store data in main memory, especially in private data centers. Primarily once your cluster includes a machine having almost equal size of hard drive and RAM :-)

Redis is enterprise, or advanced, key-value store with optional persistence. There are couple of reasons why everyone loves Redis. Why I do?

1. It's pretty simple.

Following command can install redis server on ubuntu. That's all.

apt-get install redis-server

2. It's incredible fast. Look at following tables. One million remote operations per second.

3. It supports large set of commands. More than some kind of database, it's rather enterprise remote-aware hash-map, hash-set, sorted-list or pub/sub channel solution supporting TTL or Lua script environment among others. See all commands.

4. It's optimized to use low computer resources, cpu and ram. Despite the redis server is single thread app it can achieve such great performance.

I've already started to talk about purposes at the beginning. We have primarily targeted two points. First, we can use Redis within super-fast deployment of our app when latency matters.

Secondly, I wanted to compare in-memory and persistent stores. Does it really worth to think about such in-memory solution?

Setup

I used following setup:

• Redis server 2.2.6: HP Proliant BL460c gen 8, 32core 2.6 GHZ, 192GB RAM, ubuntu 12 server
• Tests executor: xeon, 16 cores, 32gb ram, w8k server
• 10Gbps network
• jedis java client
• kryo binary serialization

As Redis cluster feature was in development in the time of measuring these numbers, I used only one machine. 32 cores were really overestimated as Redis used one plus one core indeed.

Performance Measurement of Redis

Batch Size

Appending using LPUSH to some eight different keys.

Blob size \ Batch Size [TPS]	128	256	1024	32768
100b	570k	570k	557k	600k
20kb	38k	40k	35k	33k

As main memory is touched only, it's all about network transmission. It's almost same for all sizes of batches.

Variable Connections

Utilizing LPUSH again to append to different number of keys. Every keys is accessed using different connection.

Blob size \ Connections [TPS]	1	2	4	8	32	128
100b	446	750k	646k	560k	960k	998k
20kb	9.2k	16.8k	20.8k	34k	35k	52k

Ohhh. One million inserted messages per second to one Redis instance. Incredible. Java client uses NIO so this is the answer why it somehow scales with much more tcp connections. Increasing number of network pipes where a client can push the data enables better throughput.

Occupied Memory

Blob within a list.

Blob size	Bytes per Message
100b	152
20kb	19kb

There is some build-in compression which appeared within large message test.

Long Running

The goal of this test is to fill the main memory (192GB) with one redis instance to find out if there are some scalability limitations.

Blob size	TPS
100b	842k
20kb	18.2k

Redis fill the main memory with blob messages till OOM. The shape of this progress within the time is almost flat.

Persistence

Even if Redis stores the data in the main memory, there is a way how to persist the data.

Blob size \ [TPS]	With EOF (every second)	With AOF (always)	Without
100b	800k	330k	960k

Redis forks a new thread for I/O. Numbers are almost same if the data are persisted within one second frames. TPS goes significantly down when Redis writes every updated key to the drive but this mode ensures best durability.

Large Messages

How the performance is affected when message size increases to tens of mbytes?

Blob size	TPS
500 kb	1418
5 mb	85
100 mb	1,2

Conclusion

The numbers are impressive. One-thread app successfully process almost one million of small messages in memory.

Redis performance is incredible. We could expect certain limitations because of one-threaded design which causes "serializable" behavior. Maybe this lockless implementation is the reason why redis server can handle such great throughput.

On the other hand, the right comparison against other competitors uses redis persistence feature. The performance is much more worse. Three times. Well, the persistence requirement can be the decision maker.

I've already mentioned great command set. You can probably model almost any behavior. Even if Redis primarily targets caches, there are commands allowing to calculate various stats, hold unique sets etc. The script made from these commands is powerful and very fast.

Everything is always about performance and features. Redis has both of them :-)

Read More

Performance Battle of NoSQL blob storages #2: Apache Kafka

The first article of this series brought scaling factors for blob-based content on Apache Cassandra. It's well know piece of software, requiring full installation on nodes, management application and so on. You also need to tune various configs to achieve best performance results. I've spend nice time playing with yamls on ubuntu :-)

The configuration is sometimes tricky. I was little bit confused once or twice so I planned to hire Cassandra guru to our team as issues I encountered seems really complicated :-)

Well, we do not need much functionality in HP Service Virtualization. The core is to replicate messages to achieve reliability. The next steps is to process them. Yes, the last part is to aggregate the results. Sounds exactly like map and reduce ala hadoop. Evaluation of Apache Storm or Apache Samza are different stories, but they allow me to find Apache Kafka.

Kafka is pretty nice software, much more simpler comparing to Cassandra how I've described it above. The only operation to use it is to depend on kafka jars in your pom files. That's it! Maven downloads couple of dependent jar files and your environment is (almost) ready.

As you can see below, Kafka is incredibly fast, much more faster than Cassandra. Last year I read some article. An author labels Redis as most incredible software he had met so far. I agreed. Now there are two candidates for this label :-)

It was also very beneficial to read their technical documentation for me as for technical engineer. The guys did performance research how to store data on hard drives and they also employed founded approaches in Kafka implementation. Their documentation contains interesting technical papers, like e.g. lmax pdf does with disruptor.

Setup

The setup was identical to the described in first article.

Performance Measurement of Kafka Cluster

Batch Size

• 8 topics
• two partitions per one topic

Blob size \ Batch Size [TPS]	128	256	512	1024	32768
100 b	392k	432k	433k	495k	396k
20 kb	9k	--	--	--	--

There is little difference between various batch sizes so you can tune the value according your needs. Note that overall throughput is incredible: raw 400 mbits/s.

Number of Connections

• batch size is 512

Blob size \ Connections [TPS per connection]	1	2	4	8	16	32
100 b	84k	74k	62k	58k	37k	17k
20 kb	3.2k	2.5k	1.7k	--	--	--

You can see that number of connections significantly increases the throughput of Kafka during writes. Two connections handles 150k but, 8 ones allows 464k messages per all connections.

Number of Partitions

• 8 connections

Blob size \ Partitions [TPS]	1	2	4	8	16	64
100 b		535k	457k	493k	284k	--

Two partitions brought little difference in the overall score. There is same approach or pattern like in hyper threading.

Long Running

The goal of this test is to verify the behavior once the cluster is under the continual heavy load for couple of minutes - as the underlying storage goes out of space (150GB). As Kafka is really fast, such space lasts couple of minutes.

Blob size \ [TPS]
100b	490k

The test generates 150GB of data successfully stored by Kafka. The throughput result is almost the same as in short test.

Replication

This is crucial verification how is Kafka affected by the replication.

Blob size \ Replicas [TPS]	1	2
100 b	577k	536k

There is difference less than 10% when two nodes are involved in the replication. Great! Both nodes handles 400mbits throughput.

Large Messages

Previous tests use relatively small messages, how does it behave with larger message?

Blob size \ [TPS]
100b	640k
20 kb	6k
500kb	314

The best throughput achieved last - large - message. Even for so large entity, it handles unbelievable 1.2Gbits/s. Note that all of this is a remote communication, we have 10gbps network.

Occupied Space

As the kafka stores byte array, the occupied space for this system depends on the serialization framework. I used famous kryo.

Blob size \ [Occupied space in bytes per message]
100b	183b

Here is the structure of serialized entity.
class Message {

    private UUID messageId;
    private UUID virtualServiceId;
    private String targetUrlSuffix;
    private int responseStatusCode;
    private long time;
    private byte[] data;
}

Conclusion

Kafka surprised me a lot. It's performance is incredible. The installation is just the dependency on a jar file, the configuration is very easy. The API is really simple, It's up to you what kind and form of the content you prefer.

The only drawback is that this piece of software is pretty young. In the time this article was being written, beta version 0.8 is out. For example, async API is now in a proposal only.

On the other hand, there is large set of various articles, videos and other materials, how one used it in his project, especially along with Apache Storm.

Well,  if you want to use messaging within your new solution you should definitely look at Apache Kafka.

Read More

Performance Battle of NoSQL blob storages #1: Cassandra

Preface

We spend last five years on HP Service Virtualization using MsSQL database. Non-clustered server. Our app utilizes this system for all kinds of persistence. No polyglot so far. As we tuned the performance of the response time - we started at 700ms/call and we achieved couple milliseconds per call at the end when DB involved - we had to learn a lot of stuff.

Transactions, lock escalation, isolation levels, clustered and non clustered indexes, buffered reading, index structure and it's persistence, GUID ids in clustered indexes, bulk importing, omit slow joins, sparse indexes, and so on. We also rewrite part of NHibernate to support multiple tables for one entity type which allows use scaling up without lock escalation. It was good time. The end also showed us that famous Oracle has half of our favorite features once we decided to support this database.

Well, as I'm thinking about all issues which we encountered during the development, unpredictive behavior is the feeling I have once I did all this with my DB specialist colleague. When we started to care about the performance more and more we must have understood those features and we usually took out default behavior and plug-in some special one - we could have done that as we understood essentials of the system. This is the way.

As I've described those features above, a lot of NoSQL storages allow to use them on your own. You can easily provide inverted index in key-value store even if the database itself does not support such functionality. What you need to do is to write error-less code and maintain indexes by your own code. Look at twitter example for famous key-value store Redis. It is not so simple as three/four tables in MySql with build-in indexing, isn't it?

There is nothing bad about SQL databases. I still believe that they fit the purpose of 80% of programs as they are best for difficult querying and/or less than massive scaling. But who need real massive scaling? Everyone?

Why we are going to use NoSQL?

Here in this article and the whole series, I'm going to evaluate certain attributes of storages for persistence of binary data of variable length. The goal is to persist network traffic in binary data format.

No relation decomposition, no querying of particular attributes of particular messages. No implicit transactions complying with ACID. No indexes and so on. You can find the reasons in the preface, we just hope that we know what we do :-)

Cassandra

The first evaluated storage is Cassandra. It's famous NoSQL database, commercial support provides virtual confidence for fast support and so on.

Cassandra is somewhere in the middle between schemaless and schemaware (SQL like) databases as it supports schema and CQL query language. I was pretty surprised when I have saw features like time series with dynamic columns, build-in lists and sets. Next good stuff is OpsCenter monitoring tool, I always show some screenshots from the application when I want to demonstrate easy-of-use monitor for some distributed app. Pretty nice.

Hardware

Hardware used within the tests:

• DB cluster:
• HP Proliant BL460c gen 8, 32core 2.6 GHZ, 32GB RAM, W12k server
• HP Proliant BL460c gen 8, 32core 2.6 GHZ, 192GB RAM, ubuntu 12 server
• Tests executor:
• xeon, 16 cores, 32gb ram, w8k server
• 10Gbps network
• Java 7

Performance Measurement of Cassandra Cluster

The setup used one seed and one replica, java tests use datastax driver. Replication factor two when not explicitly defined. After some issues:

#GC and #cassandra ...ufff :-( pic.twitter.com/ohTciftP1e
— Martin Podval (@MartinPodval) February 17, 2014

... my final CQL create script was:

CREATE COLUMNFAMILY recording (


PARTKEY INT,


ID uuid,


VS_ID uuid,


URL ascii,


MESSAGE_TIME BIGINT,


DATA BLOB,


PRIMARY KEY((PARTKEY, VS_ID), ID)


);

Batch Size

Datastax driver allows to use queries in batches, how does it scale?

Blob size \ Batch Size [TPS]	128	256	512	1024	8196	32768
100 bytes (8 clients)	67 100	63 900	55 400	55 800	23 500	Timeouted
1 kb (4 clients)	15 600	20 600	20 300	19 400	5 500	N/A
5 kb (1 client)	N/A	8 300	9 200	10 000	3 000

256 rows seems like best size of the batch.

Number of Connections

The tests utilizes variable count of connections to the Cassandra cluster. Note that the batch size was 256 and consistency was set to ONE.

Blob size \ Connections [TPS]	1	2	4	8	24
100 b	27 000	53 000	75 000	62 000	62 000
1 kb	17 900	31 000	42 000	40 000	--
5 kb	9 000	14 600	16 000	10 300	--
20 kb	3 300	4 500	3 800

You can see that 2 or 4 connections per one client are enough whilst one connection slows down the overall performance.

Number of Partitions

The table contains partition key, see the CQL code snippet. The test generates variable values for the partition key. Batch size was also 256 and consistency one.

Blob size \ Partitions [TPS]	1	8	64
100 b	58 000	71 000	70 000
10 kb	13 200	9100	7 300

Partition key says where the data will reside, it defines target machine. As two nodes are involved in my tests, its better to use more than one partition.

Variable Consistency

Tuning of consistency is the most beautiful attribute in Cassandra. You can define consistency level in every aspect of the communication with cassandra driver in any language. There are different levels, see the documentation.

The test used 8 partitions, 8 connections and 256 batch size.

Blob size \ Consisency [TPS]	Any	One	Two
100 b	67 000	66 000	66 000

Well, I'm surprised. There is almost no difference between consistency levels for such blob size despite totally different purpose or implementation. This could happen due to almost ideal environment, very fast network and machines as well.

Variable Replicas

I used two replicas by default but this tests tries to reveal the price for second copy of the data.

Blob size \ Replicas [TPS]	1	2
100 b	121 000	72 000
10 kb	19 200	8 200

There is apparent penalty for second replica.

Long Running

Previous tests used a few millions of inserts/rows but the current one uses hundred of millions. This is the way how to definitely omit the advantage of various caches and clever algorithms. This simulates heavy weight load test to the server.

8 partitions, 8 connections, 256 batch size.

Blob size \ Replicas+Consistency [TPS]	1+ONE	2+ONE	2+TWO
100 b (1 billion)	121 000	63 000	62 00
10 kb (30 m)	--	8 200	--

Large Messages

The largest message in these tests has 20 kb so far, but what is the behavior when our app would meet some large file as a attachment?

Consistency One, 2 partitions, 2 connections, 2 batch size.

Blob size  [TPS]
500 kb	90
5 mb	9.5

Occupied Space

Apache Cassandra automatically enables compression since 1.1 version.

Blob size \ [Occupied space in bytes per message]
100b	89b

Here is the structure of serialized entity.

CREATE COLUMNFAMILY recording (

    PARTKEY INT,

    ID uuid,

    VS_ID uuid,

    URL ascii,

    MESSAGE_TIME bigint,

    DATA blob,

    PRIMARY KEY((PARTKEY, VS_ID), ID)

);

Conclusion

This kind of testing is almost opposite in comparison to neflix tests. They analyzed scalability but we have done testing of simple cluster having two machines only. We need to figure out the performance of one node (obviously in cluster deployment).

I was impressed as the numbers above are good, the overall throughput on the network reached hundred mbps coming out from one client. 

What next. Same kinds of tests agains Kafka, Redis, Couchbase, Aerospike, Couchbase.

Read More