System Variables

The system variable names in TiDB are defined in tidb_vars.go. This file also includes some default values for variables, but the place where they are actually assembled is defaultSysVars.

This large struct array defines the scope, variable names, and default values for all variables in TiDB. Besides TiDB’s own system variables, it also includes compatibility with MySQL’s system variables.

Scope

In TiDB, there are three types of variable scopes literally:

They are ScopeNone, ScopeGlobal, and ScopeSession. They represent:

• ScopeNone: Read-only variables
• ScopeGlobal: Global variables
• ScopeSession: Session variables

The actual effect of these scopes is that when you use SQL to read or write them, you need to use the corresponding syntax. If the SQL fails, the SQL operation does not take effect. If the SQL executes successfully, it merely means the setting is complete, but it does not mean that it takes effect according to the corresponding scope.

Let’s use the method mentioned in the first article to start a single-node TiDB for demonstration:

ScopeNone

Take performance_schema_max_mutex_classes as an example,

MySQL  127.0.0.1:4000  SQL > select @@performance_schema_max_mutex_classes;
+----------------------------------------+
| @@performance_schema_max_mutex_classes |
+----------------------------------------+
| 200                                    |
+----------------------------------------+
1 row in set (0.0002 sec)
MySQL  127.0.0.1:4000  SQL > select @@global.performance_schema_max_mutex_classes;
+-----------------------------------------------+
| @@global.performance_schema_max_mutex_classes |
+-----------------------------------------------+
| 200                                           |
+-----------------------------------------------+
1 row in set (0.0004 sec)
MySQL  127.0.0.1:4000  SQL > select @@session.performance_schema_max_mutex_classes;
ERROR: 1238 (HY000): Variable 'performance_schema_max_mutex_classes' is a GLOBAL variable

As you can see, the scope of ScopeNone can be read as a global variable,

MySQL  127.0.0.1:4000  SQL > set global performance_schema_max_mutex_classes = 1;
ERROR: 1105 (HY000): Variable 'performance_schema_max_mutex_classes' is a read-only variable
MySQL  127.0.0.1:4000  SQL > set performance_schema_max_mutex_classes = 1;
ERROR: 1105 (HY000): Variable 'performance_schema_max_mutex_classes' is a read-only variable
MySQL  127.0.0.1:4000  SQL > set session performance_schema_max_mutex_classes = 1;
ERROR: 1105 (HY000): Variable 'performance_schema_max_mutex_classes' is a read-only variable

But it cannot be set in any way.

To trace the usage of ScopeNone, you will see

In setSysVariable, when this type of scope variable is encountered, an error is directly returned.

In ValidateGetSystemVar, it is handled as a global variable. From a theoretical standpoint, these ScopeNone variables are essentially a single copy in the code. Once TiDB is started, they exist in memory as read-only and are not actually stored in TiKV.

ScopeGlobal

Using gtid_mode as an example,

MySQL  127.0.0.1:4000  SQL > select @@gtid_mode;
+-------------+
| @@gtid_mode |
+-------------+
| OFF         |
+-------------+
1 row in set (0.0003 sec)
MySQL  127.0.0.1:4000  SQL > select @@global.gtid_mode;
+--------------------+
| @@global.gtid_mode |
+--------------------+
| OFF                |
+--------------------+
1 row in set (0.0006 sec)
MySQL  127.0.0.1:4000  SQL > select @@session.gtid_mode;
ERROR: 1238 (HY000): Variable 'gtid_mode' is a GLOBAL variable

It works the same way as MySQL global variable reading,

MySQL  127.0.0.1:4000  SQL > set gtid_mode=on;
ERROR: 1105 (HY000): Variable 'gtid_mode' is a GLOBAL variable and should be set with SET GLOBAL
MySQL  127.0.0.1:4000  SQL > set session gtid_mode=on;
ERROR: 1105 (HY000): Variable 'gtid_mode' is a GLOBAL variable and should be set with SET GLOBAL
MySQL  127.0.0.1:4000  SQL > set global gtid_mode=on;
Query OK, 0 rows affected (0.0029 sec)
MySQL  127.0.0.1:4000  SQL > select @@global.gtid_mode;
+--------------------+
| @@global.gtid_mode |
+--------------------+
| ON                 |
+--------------------+
1 row in set (0.0005 sec)
MySQL  127.0.0.1:4000  SQL > select @@gtid_mode;
+-------------+
| @@gtid_mode |
+-------------+
| ON          |
+-------------+
1 row in set (0.0006 sec)

The setting method is also compatible with MySQL. At this point, we can shut down the single-instance TiDB and restart it,

MySQL  127.0.0.1:4000  SQL > select @@gtid_mode;
+-------------+
| @@gtid_mode |
+-------------+
| ON          |
+-------------+
1 row in set (0.0003 sec)

And you can see that the result can still be read, meaning that this setting was persisted to the storage engine. Looking closely at the code, you can see:

The actual implementation involves executing an internal replace statement to update the original value. This constitutes a complete transaction involving acquiring two TSOs and committing the entire process, making it slower compared to setting session variables.

ScopeSession

Using rand_seed2 as an example,

MySQL  127.0.0.1:4000  SQL > select @@rand_seed2;
+--------------+
| @@rand_seed2 |
+--------------+
|              |
+--------------+
1 row in set (0.0005 sec)
MySQL  127.0.0.1:4000  SQL > select @@session.rand_seed2;
+----------------------+
| @@session.rand_seed2 |
+----------------------+
|                      |
+----------------------+
1 row in set (0.0003 sec)
MySQL  127.0.0.1:4000  SQL > select @@global.rand_seed2;
ERROR: 1238 (HY000): Variable 'rand_seed2' is a SESSION variable

Reading is compatible with MySQL.

MySQL  127.0.0.1:4000  SQL > set rand_seed2='abc';
Query OK, 0 rows affected (0.0006 sec)
MySQL  127.0.0.1:4000  SQL > set session rand_seed2='bcd';
Query OK, 0 rows affected (0.0004 sec)
MySQL  127.0.0.1:4000  SQL > set global rand_seed2='cde';
ERROR: 1105 (HY000): Variable 'rand_seed2' is a SESSION variable and can't be used with SET GLOBAL
MySQL  127.0.0.1:4000  SQL > select @@rand_seed2;
+--------------+
| @@rand_seed2 |
+--------------+
| bcd          |
+--------------+

The setting is also compatible with MySQL. It can be simply observed that this operation only changes the session’s memory. The actual place where it finally takes effect is SetSystemVar.

There are some tricks here.

Actual Scope of Variables

The previous section covered setting session variables. Based on MySQL’s variable rules, setting a global variable does not affect the current session. Only newly created sessions will load global variables for session variable assignment. Ultimately, the active session variable take effect. Global variables without session properties still have unique characteristics, and this chapter will cover:

1. Activation of session variables
2. Activation of pure global variables
3. Mechanism of global variable function

These three aspects.

Activation of Session Variables

Whether a session variable is also a global variable only affects whether it needs to load global variable data from the storage engine when the session starts. The default value in the code is the initial value for eternity if no loading is required.

The actual range where a variable operates can only be observed in SetSystemVar.

For example, in this part, s.MemQuotaNestedLoopApply = tidbOptInt64(val, DefTiDBMemQuotaNestedLoopApply) changes the s structure, effectively changing the current session,

Whereas atomic.StoreUint32(&ProcessGeneralLog, uint32(tidbOptPositiveInt32(val, DefTiDBGeneralLog))) changes the value of the global variable ProcessGeneralLog, thereby affecting the entire TiDB instance when set tidb_general_log = 1 is executed.

Activation of Pure Global Variables

Pure global variables in current TiDB are used for background threads like DDL, statistics, etc.

Because only one TiDB server requires them, session-level variables hold no meaning for these.

Mechanism of Global Variable Function

Global variables in TiDB don’t activate immediately after setting. A connection fetches the latest global system variables from TiKV to assign them to the current session the first time it’s established. Concurrent connection creation results in frequent access to the TiKV node holding a few global variables. Thus, TiDB caches global variables, updating them every two seconds, significantly reducing TiKV load. The problem arises that after setting a global variable, a brief wait is necessary before creating a new connection, ensuring new connections will read the latest global variable. This is one of the few eventual consistency locations within TiDB.

For specific details, see this commentary in loadCommonGlobalVariablesIfNeeded.

Metrics

Compared to system variables, Metrics in TiDB are simpler, or straightforward. The most common Metrics are Histogram and Counter, the former is used to record actual values for an operation and the latter records occurrences of fixed events. All Metrics in TiDB are uniformly located here, with AlertManager and Grafana scripts also available separately under alertmanager and grafana.

There are many Metrics, and from a beginner’s perspective, it’s best to focus on a specific monitoring example. Let’s take the TPS (transactions per second) panel as an example.

Click EDIT and you will see the monitoring formula is:

The tidb_session_transaction_duration_seconds is the name of this specific metric. Since it is a histogram, it can actually be expressed as three types of values: sum, count, and bucket, which represent the total sum of values, the count (which functions the same as a counter), and the distribution by bucket, respectively.

In this context, [1m] represents a time window of 1 minute, indicating the precision of the measurement. The rate function calculates the slope, essentially the rate of change, indicating how many times something occurs per second. The sum function is used for aggregation, and when combined with by (type, txn_mode), it represents aggregation by the dimensions of type and txn_mode.

The Legend below displays the dimensions above using {{type}}-{{txn_mode}}. When surrounded by {{}}, it can display the actual label names.

In this representation, the final states of transactions are commit, abort, and rollback. A commit indicates a successful user-initiated transaction, rollback indicates a user-initiated rollback (which cannot fail), and abort indicates a user-initiated commit that failed.

The second label, txn_mode, refers to two modes: optimistic and pessimistic transactions. There’s nothing further to explain about these modes.

Corresponding to the code:

This segment of code shows that tidb_session_transaction_duration_seconds is divided into several parts, including namespace and subsystem. Generally, to find a variable in a formula like tidb_session_transaction_duration_seconds_count within TiDB code, you need to remove the first two words and the last word.

From this code snippet, you can see it’s a histogram, specifically a HistogramVec, which is an array of histograms because it records data with several different labels. The labels LblTxnMode and LblType are these two labels.

Checking the references, there is a place for registration, which is in the main function we discussed in the first article, where metrics are registered.

Other references show how metrics are instantiated. Why do we do this? Mainly because as the number of labels increases, the performance of metrics becomes poorer, which is related to Prometheus’s implementation. We had no choice but to create many instantiated global variables.

Taking the implementation of Rollback as an example, its essence is to record the actual execution time of a transaction when Rollback is truly executed. Since it’s a histogram, it is also used as a counter in this instance.

How to Check the Support Level of a Syntax

There are usually two methods:

1. Check through the parser repo
2. Directly check within the TiDB repo

Both of these methods require the environment setup from the previous article. If you haven’t tried that yet, give it a go.

Preparation

1. Install GoYacc Support

   

   The GoYacc Support plugin is a creation by a colleague at our company, a third-party plugin officially accepted by JetBrains, a well-regarded product. It includes syntax highlighting and intelligence, which is great!

2. Download parser repo

   If you’re checking syntax directly from the parser, you need to download it manually. If you’re navigating from TiDB, IDEA will automatically download the code, so no extra steps are needed.

Check via parser repo

Open the parser using IDEA, switch to the branch you need, and locate the parser.y file. However, it is more recommended to check from within TiDB.

Check via TiDB repo

1. Open the TiDB project with IDEA and switch to the required branch

   

2. Find the parser.y file; make sure to select the broadest search scope

   

   Alternatively, you can find it in the file list,

   

   

Let’s take checking the SHOW ENGINES SQL statement as an example.

The entry point for the entire statement parsing is Start. Below it is the StatementList, followed by Statement. Under the large list of Statements, you can find ShowStmt.

However, ShowStmt is actually quite complex. Another way is to directly search for ShowEngines within parser.y, since naming follows Golang conventions, with camel case and capitalized letters for public exposure. Naturally, if familiar with the code, you’d know ShowEngines is under ShowTargetFilterable. Its first branch is ShowEngines.

What is the level of support for SHOW ENGINES?

You can look at how ast.ShowEngines is processed. Here, you can’t just jump to it; you need to copy and search.

You only need to see how it’s processed under TiDB, and you can skip test files.

One is the actual implementation,

The other is the build schema, which you can ignore for now,

Entering fetchShowEngines, you can see the specific implementation is simple, running an internal SQL to read a system table.

Checking SHOW ENGINES ends here. You can see that it’s fully supported.

Which statements only have syntax support?

Taking the temporary table creation syntax as an example, find its position in the parser.y file.

It’s an option.

You can see that if the temporary table option is specified, it simply returns true with an attached warning, stating that the table is still treated as a regular table. Previously, the parser had a lot of operations that only returned without doing anything, not even a warning, but these are now rare.

Advantages of Querying via TiDB repo

You can see that checking via the TiDB repo allows you to find the parser’s detailed hash using IDEA. If you check directly via the parser, you need to first look up the parser’s hash in TiDB’s go.mod, then check out to the corresponding hash in the parser. If you need to check specific implementations, you have to go back to TiDB, making back-and-forth checks less convenient compared to looking within a single project. The only advantage is the ease of blaming commit history.

Viewing and Modifying Default Configuration

The default configurations can be easily viewed in TiDB, specifically the variable defaultConf. The configurations listed here are the actual default settings.

Taking the first Host configuration as an example, it has a mapping to toml and json files.

This essentially shows how it’s written in a toml file. The DefHost following it is the specific default value.

Something important to note is that configurations have a hierarchical relationship. For example, the log-related configuration in the configuration file is:

In the code, it is represented as:

This denotes a configuration called “level” under the log configuration.

What if you want to add more levels? For instance, the most complex configuration for CopCache adds another layer under tikv-client called copr-cache.

Since toml files do not support multi-level nesting, this leads to the most complex configuration writing in TiDB.

To use non-default configurations with the TiDB started through IDEA as mentioned above, the simplest way is to modify this defaultConf.

Summary

From this, you can see that checking whether a statement is supported, and whether it’s just syntax support or has a specific implementation, can be achieved with the described methods. You also learned how to view and modify default configurations, allowing you to conduct some verifications yourself. In the next article, I plan to introduce TiDB’s system variables.

There are many articles on reading the source code of TiDB, often referred to as the “Twenty-Four Chapters Scriptures”. However, these introductions typically proceed from a macro to a micro perspective. This series attempts to introduce how to read TiDB’s source code from an easier angle. The goals we aim to achieve are:

1. Enable readers to start reading TiDB’s code themselves, rather than understanding it passively through pre-written articles.
2. Provide some common examples of looking into the details of the code, such as examining the scope of a variable.

After all, teaching people to fish is better than giving them fish. While the code changes often, the methods remain mostly unchanged.

Why choose TiDB to read?

1. I am not familiar with TiKV or PD.

2. TiDB is the entry point directly interacting with users and is also the most likely to be questioned.

3. TiDB can run independently and be debugged. If you want to run some SQL after reading the code to verify your understanding, it can be easily done.

Preparations

1. A development machine

   TiDB is a pure Golang project. It can be conveniently developed on Linux, MacOS, and even Windows. My environment is Windows 10.

2. A copy of the TiDB source code, available for download at the official repo.

3. Golang environment, following the official guide is straightforward.

4. Goland or IntelliJ IDEA + Golang plugin

   I personally feel there’s no difference between the two. Why not recommend VSCode + Golang plugin? Mainly because I’m used to the JetBrains suite, and indeed commercial software tends to be higher quality than community software. For long-term use, it’s recommended to pay for it. Students can use it for free, but need to renew the license every year.

Environment Setup

1. After installing the Golang environment, remember to set the GOPATH.

2. The TiDB code doesn’t need to be developed under the GOPATH, so you can place it anywhere. I usually create a directory called work and throw various codes in there.

3. Open Goland/IDEA. I use IDEA because I often look at code in other languages.

4. Open with IDEA, select the tidb directory.

   

5. At this point, IDEA typically prompts you to set up GOROOT and enable Go Modules. Follow the recommendations.

The environment setup is now complete.

Entry Points

At the beginning, someone advised me to start with the session package. However, after some experience, I personally feel there are two better entry points: the main function and the dispatch function.

main Function

The main function of TiDB can be seen at link. You can roughly go through what happens when starting a tidb-server from top to bottom.

From top to bottom:

• Parse flags

• Output version information and exit

• Register store and monitoring

• Configuration file check

• Initialize temporary folders, etc.

• Set global variables, CPU affinity, log, trace, print server information, set binlog, set monitoring

• Create store and domain

  The createStoreAndDomain method is important, as critical background threads are created here.

• Create server and register stop signal function

• Start the server

  Within runServer, the srv.Run() actually brings up the tidb-server. In the Run() function here, the server continuously listens to network requests, creating a new connection for each new request and using a new goroutine to serve it continually.

• After this, cleanup work is done when the server needs to stop, ultimately writing out the logs.

Thus, the entire main function process ends. Through the main function, you can see the complete lifecycle of a server from creation to destruction.

Additionally, with IDEA, you can easily start and debug TiDB. Click on this triangle symbol as shown in the image below:

A pop-up with options to run and debug the main function will appear. Essentially, this starts a TiDB with default configurations. TiDB defaults to using mocktikv as the storage engine, so it can be started on a single machine for various testing and validation.

As for how to modify the configuration for starting and debugging, this will be introduced in subsequent articles in the series.

dispatch Function

From here, we can proceed further to another suitable entry point function, dispatch.

The dispatch function has several characteristics:

1. Requests coming from clients only enter the dispatch function, meaning from this point onward, user requests are executed. If you set breakpoints here, you can conveniently filter out SQL executed by internal threads.

2. From here, various requests are dispatched into different processing logic, ensuring you don’t miss any user requests. It avoids situations like spending significant time reading text protocol code only to find out the user is actually using a binary protocol.

3. dispatch itself is located at a very early stage, meaning its parameters mostly come directly from the client’s initial information. If it’s a text protocol, directly reading parameters can parse out the SQL text.

At the start, dispatch primarily focuses on obtaining tokens corresponding to the token-limit parameter. Requests that can’t get a token won’t execute, which explains why you can create many connections but only 1000 SQL executions are allowed simultaneously by default.

Next, we enter the most crucial switch case:

These commands are MySQL protocol commands, so it’s apparent from here exactly what TiDB implements. For comparison, you can refer to this link (this link is only for the text protocol). For full details, see the figure below:

Within dispatch, the most important are mysql.ComQuery, as well as the trio mysql.ComStmtPrepare, mysql.ComStmtExecute, and mysql.ComStmtClose. The latter trio is more frequently used in actual production, hence even more important. In contrast, mysql.ComQuery is generally used only for some simple tests and validations.

Since dispatch is the entry point for interfacing with clients, it can conveniently tally how many requests the database has handled. The so-called QPS derived from monitoring statistics is essentially the number of times this function executes per second. Here arises an issue: in cases like multi-query requests, such as select 1; select 1; select 1;, multiple statements sent together are regarded as a single request by dispatch, but as multiple by clients. While using the binary protocol, some clients prepare a statement, then execute, and finally close it. Seemingly equivalent to executing a single SQL from the client’s perspective, the database actually completes three requests.

In summary, users’ perceived QPS may not necessarily align with the number of dispatch function calls. In later versions, the QPS panel in TiDB’s monitoring was changed to CPS, which stands for Commands Per Second, representing the number of commands executed per second.

Looking at the callers of dispatch can also reveal information that helps explain some frequently asked questions:

1. An EOF error in dispatch typically means the client has actively disconnected, so there’s no need to maintain the database connection, and it is severed.

2. In case of an undetermined error (indicating a transaction’s commit is uncertain—whether it has succeeded or failed needs manual intervention for verification), manual intervention is required immediately, and the connection will be closed.

3. If writing binlog fails and ignore-error = false, previously the tidb-server process wouldn’t exit but couldn’t provide services. Now, the tidb-server will exit directly.

4. For all other dispatch errors, the connection will not be closed, allowing service to continue, but the failure information will be logged as “command dispatched failed”, which is arguably one of the most critical logs for TiDB.

Conclusion

This concludes the introduction from setting up the environment to finding a reasonable entry point to start reading code. Subsequent posts in the series will cover aspects such as configuration (adjustments, default values), variables (default values, scope, actual range, activation), supported syntax, etc. Stay tuned.