-
Notifications
You must be signed in to change notification settings - Fork 91
Redis Driver
This driver implements functionality required to execute the TPC-C benchmark against a Redis key-value store. With Redis, there are two approaches one might take in trying to model a relational database. First, records can be implemented using the Redis hash data-type where the Redis key represents the primary key, the Redis hash keys represent the column names, and the Redis has values represent the record's data. The other option is to store each record's value for each attribute at a separate Redis key.
In terms of the TPC-C benchmark, consider the WAREHOUSE table and a record that has a primary key of FOO. Using the Redis hash approach, you would use the Redis "hmset" command to store all of the column name/value pairs at the key WAREHOUSE:FOO. Using the individual key approach, you would need to store values at WAREHOUSE:FOO:W_ID, WAREHOUSE:FOO:W_NAME, WAREHOUSE:FOO:W_STREET_1, WAREHOUSE:FOO:W_STREET_2, WAREHOUSE:FOO:W_CITY, WAREHOUSE:FOO:W_STATE, WAREHOUSE:FOO:W_ZIP, WAREHOUSE:FOO:W_TAX, and WAREHOUSE:FOO:W_YTD.
In development of this driver, we actually implemented both methods of representing the relational structure and benchmarked them to determine any performance differences. Since both performed similarly on the benchmark, we chose to go with the Redis hash implementation, reducing the overall number of keys stored in the database and improving the ease of accessing and writing information to the key-value store.
In both methods, you will want to maintain a set of the primary keys for each table. For example, the WAREHOUSE:IDS key stores a set of all primary key values for the WAREHOUSE table. Ideally, this would be implemented using a Redis sorted set to model how a relational database stores data on disk, sorted by primary key. However, we had difficulty implementing a sorted set with the Python driver for Redis. Therefore, the primary key values were stored in a regular Redis set and sorting was done in the driver when necessary.
To facilitate execution on multiple nodes, we included data partitioning in the tuple loading phase, based on the Redis Pre-Sharding technique. With the exception of the ITEM table, which was stored in its entirety at node, the appropriate node at which to store a tuple was determined by taking the particular warehouse id modulo the number of nodes.
Many of the TPC-C transactions include queries specifying various predicates. Redis, however, is a key-value store and only supports basic operations. Therefore, we had to add additional structures to facilitate query simulation in order to complete the TPC-C transactions. The closest parallel to this approach would be an index in a relational model. For example, consider the ORDER STATUS transaction that requires finding all order lines for a particular order. As order line tuples are added, we create a key that models the search predicate and store all ids that satisfy that search predicate. Consider the following order line:
{'OL_O_ID' : 1, 'OL_D_ID' : 2, 'OL_W_ID' : 25, 'OL_NUMBER' : 17})
The primary key value "1-2-25-17" would be added to the set of ids matching this predicate at the Redis key "ORDER_LINE:INDEXES:SUMOLAMOUNT:1-2-25".
Redis seemed to outperform the other NoSQL systems tested in class. One possible explanation could be that Redis is more than just a simple key- value store. Redis allows storage of more complex data types (e.g. sets, hashes) and has special operators for working with each data type. The more robust functionality lends itself to modeling some relational functionality.
In is interesting to note that performance did not scale as more nodes were added to the system. More testing is needed with more granular differences in node numbers to get a better feel for the effect of adding nodes to the overall performance of the system. However, since the current Redis key-value store is seemingly meant to be used as a single node, or master-slave, configuration, it might be the internal construction of Redis that is the limiting factor. It would be interesting to see how the new Redis cluster would compare in a similar test.
This driver is dependent on the Redis.py Python client library for Redis.
The Redis README notes that using tcmalloc instead of the standard glibc malloc would improve performance and reduce memory fragmentation. Since our goal was to maximize Redis performance, we decided to use tcmalloc. Our Linux distribution, Ubuntu 10.04.02 LTS 64-bit, has a missing package dependency which prevents the installation of the google-perftools development package. As such, in order to compile Redis against the google-perftools library, we had to first remove the libgoogle-perftools0 Ubuntu package, and then compile and install google-perftools from source. We used google-perftools version 1.7. This package, in turn, had a dependency of libunwind 0.99, which we also installed from source. Compilation of libunwind required that we pass the -U_FORTIFY_SOURCE flag to the C compiler, which disables the stack smashing protector. Finally, we compiled Redis version 2.2.4 from source using the command line "make USE_TCMALLOC=yes". This work was performed on each node in our cluster, using a shell script to provide automation and guarantee standardization.
The limiting factor in performance seems to be in the communications between the benchmark executor and the worker nodes. Modeling queries in Redis requires more network communications than a relational database. A query typically requires 2 round-trips to the Redis data-store. The first finds the appropriate IDS and the second returns that values for an ID. If the query returns multiple records in the result set, the number of round trips to the Redis data store increases proportionally. This lag can be alleviated somewhat by using pipelining to group commands together, reducing network traffic. Despite implementing this approach, we found that network I/O was the limiting factor in all TPC-C transactions.
In the completion of this project, we did not experiment with denormalizing the TPC-C data. It might be possible to improve transaction throughput by modifying the way in which data is stored in Redis. In addition, the use of sorted sets to correctly model the storage of a relational table on disk, would probably improve performance by removing the need to sort that data in-memory. Finally, it would be interesting to test the yet-to-be-released Redis Cluster to see how it compares to the current Redis data store.