First of all, TPC-C is the de facto OLTP benchmark standard. It is a set of specifications, and any database can publish its test results under this standard, so there’s no issue of quarreling over the testing tools used.

Secondly, TPC-C is closer to real-world scenarios as it includes a transaction model within it. In the flow of this transaction model, there are both high-frequency simple transaction statements and low-frequency inventory query statements. Therefore, it tests the database more comprehensively and practically.

Testing TPC-C on CockroachDB

This year, CockroachDB released its TPC-C performance results. However, unfortunately, they did not use a tool recognized by the database industry that implements the TPC-C standard for testing. Instead, they used their own implementation of a TPC-C tool. The compliance level of this tool was not recognized. In the white paper officially released by them, it is also mentioned that this TPC-C cannot be compared with the TPC-C standard.

Therefore, I thought of using a highly recognized tool in the industry for testing. Here, I chose Benchmarksql version 5.0.

Benchmarksql 5.0 supports the PostgreSQL protocol, Oracle protocol, and MySQL protocol (the MySQL protocol is supported in the code, but the author hasn’t fully tested it, so the official documentation doesn’t mention MySQL). Among these, the PostgreSQL protocol is supported by CockroachDB.

Test Preparation

After preparing the Benchmarksql code, don’t rush into testing. There are three main pitfalls here that need to be addressed first.

1. CockroachDB does not support adding a primary key after table creation. Therefore, you need to include the primary key when creating the table. Specifically, in the run folder under the root directory of the Benchmarksql code, create a sql.cdb folder. Copy tableCreates.sql and indexCreates.sql from the sql.common folder at the same level into sql.cdb. Then move the primary keys in indexCreates.sql into the table creation statements in tableCreates.sql. For how to define indexes while creating tables, please refer to the database documentation syntax via Google.

2. CockroachDB is a “strongly typed” database. This is my own way of describing it. It has a rather peculiar behavior: when you add different data types (e.g., int + float), it will report an error saying, “InternalError: unsupported binary operator: <int> + <float>”. Generally, databases don’t behave like this; most would perform some implicit conversions, or in other words, they are very tolerant of SQL writers. But CockroachDB is unique in that if you don’t specify the type, it reports an error. This greatly reduces the burden of type inference in its internal implementation.

   This behavior causes Benchmarksql to fail to run the tests properly. The solution is to add the required type at the position where the error occurs. For example, change update t set i = i + ?; (the ? is generally filled in using prepare/execute) to update t set i = i + ?::DECIMAL;. Yes, CockroachDB specifies types explicitly by adding ::<type_name> at the end. But strangely, not all additions require type specification.

3. CockroachDB does not support SELECT FOR UPDATE. This is the easiest to solve: comment out all FOR UPDATE clauses in Benchmarksql. CockroachDB itself supports the serializable isolation level; lacking FOR UPDATE doesn’t affect consistency.

Starting the Test

After overcoming the pitfalls mentioned above, you can proceed with the normal testing process: creating the database, creating tables and indexes, importing data, and testing. You can refer to Benchmarksql’s HOW-TO-RUN.txt.

Test Results

On my test machine with 40 cores, 128 GB of memory, and SSD, under 100 warehouses, the tpmC is approximately 5,000. This is about one-tenth of PostgreSQL 10 on the same machine. PostgreSQL can reach around 500,000 tpmC.

CockroachDB uses the PostgreSQL protocol. If you want to use Sysbench for testing, you need to enable pg protocol support in Sysbench. Sysbench already supports the pg protocol, but it is not enabled by default during compilation. You can configure it with the following command:

./configure --with-pgsql

Of course, preliminary work involves downloading the Sysbench source code and installing the necessary PostgreSQL header files required for compilation (you can use yum or sudo to install them).

Testing

The testing method is no different from testing MySQL or PostgreSQL; you can test any of the create, read, update, delete (CRUD) operations you like. The only thing to note is to set auto_inc to off.

This is because CockroachDB’s auto-increment behavior is different from PostgreSQL’s. It generates a unique id, but it does not guarantee that the ids are sequential or incremental. This is fine when inserting data. However, during delete, update, or query operations, since all SQL statements use id as the condition for these operations, you may encounter situations where data cannot be found.

That is:

When auto_inc = on (which is the default value in Sysbench)

Table Structure

CREATE TABLE sbtest1 (
   id INT NOT NULL DEFAULT unique_rowid(),
   k INTEGER NOT NULL DEFAULT 0:::INT,
   c STRING(120) NOT NULL DEFAULT '':::STRING,
   pad STRING(60) NOT NULL DEFAULT '':::STRING,
   CONSTRAINT ""primary"" PRIMARY KEY (id ASC),
   INDEX k_1 (k ASC),
   FAMILY ""primary"" (id, k, c, pad)
)

Data

root@:26257/sbtest> SELECT id FROM sbtest1 ORDER BY id LIMIT 1;
+--------------------+
|         id         |
+--------------------+
| 354033003848892419 |
+--------------------+

As you can see, the data does not start from 1, nor is it sequential. Normally, the id in a Sysbench table should be within the range [1, table_size].

SQL

UPDATE sbtest%u SET k = k + 1 WHERE id = ?

Taking the UPDATE statement as an example, id is used as the query condition. Sysbench assumes that this id should be between [1, table_size], but in reality, it’s not.

Example of Correct Testing Command Line

sysbench --db-driver=pgsql --pgsql-host=127.0.0.1 --pgsql-port=26257 --pgsql-user=root --pgsql-db=sbtest \
        --time=180 --threads=50 --report-interval=10 --tables=32 --table-size=10000000 \
        oltp_update_index \
        --sum_ranges=50 --distinct_ranges=50 --range_size=100 --simple_ranges=100 --order_ranges=100 \
        --index_updates=100 --non_index_updates=10 --auto_inc=off prepare/run/cleanup

INSERT Testing

Let’s discuss the INSERT test separately. The INSERT test refers to Sysbench’s oltp_insert. The characteristic of this test is that when auto_inc is on, data is inserted during the prepare phase of the test; otherwise, only the table is created without inserting data. Because when auto_inc is on, after the prepare phase, during the run phase, the inserted data will not cause conflicts due to the guarantee of the auto-increment column. When auto_inc is off, the id of the data inserted during the run phase is randomly assigned, which aligns with some actual testing scenarios.

For CockroachDB, when testing INSERT operations with auto_inc set to off, after the prepare phase, during the run phase of data insertion, you can observe the monitoring metrics (by connecting to CockroachDB’s HTTP port) under the “Distribution” section in “KV Transactions”. You’ll notice a large number of “Fast-path Committed” transactions. This indicates that transactions are committed using one-phase commit (1PC). That is, the data involved in the transaction does not span across CockroachDB nodes, so there’s no need to ensure consistency through two-phase commit transactions. This is an optimization in CockroachDB, which is very effective in INSERT tests and can deliver excellent performance.

If auto_inc is on, although for other tests that require read-before-write operations, the results in CockroachDB might be inflated, it is still fair for the INSERT test. If time permits, you can supplement the tests to see the differences.