Recently, I encountered a problem where a user wanted to synchronize data from PostgreSQL to TiDB (which uses the same protocol as MySQL) and wanted to know whether the data after synchronization is consistent. I hadn’t dealt with this kind of issue before, so I did a bit of research.

Typically, to verify data consistency, you compute a checksum on both sides and compare them.

TiDB (MySQL) Side

For the verification of a specific table, the following SQL is used:

SELECT bit_xor(
    CAST(crc32(
        concat_ws(',',
            col1, col2, col3, …, colN,
            concat(isnull(col1), isnull(col2), …, isnull(colN))
        )
    ) AS UNSIGNED)
)
FROM t;

Let’s look at a specific example:

DROP TABLE IF EXISTS t;
CREATE TABLE t (i INT, j INT);
INSERT INTO t VALUES (2, 3), (NULL, NULL);
SELECT bit_xor(
    CAST(crc32(
        concat_ws(',',
            i, j,
            concat(isnull(i), isnull(j))
        )
    ) AS UNSIGNED)
)
FROM t;

The result is:

+-------------------------------------------------------------------------------------------------------------------------------------------+
| bit_xor(
    CAST(crc32(
        concat_ws(',',
            i, j,
            concat(isnull(i), isnull(j))
        )
    ) AS UNSIGNED)
) |
+-------------------------------------------------------------------------------------------------------------------------------------------+
|                                                           5062371 |
+-------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

PostgreSQL Side

The goal is simply to write the same SQL as above, but PostgreSQL does not support bit_xor, crc32, isnull, nor does it have unsigned types. Therefore, the solution is relatively straightforward—relying on UDFs (User-Defined Functions).

After some research, the main missing functions can be addressed with a few custom implementations.

bit_xor:

CREATE OR REPLACE AGGREGATE bit_xor(IN v bigint) (SFUNC = int8xor, STYPE = bigint);

crc32:

CREATE OR REPLACE FUNCTION crc32(text_string text) RETURNS bigint AS $$
DECLARE
    tmp bigint;
    i int;
    j int;
    byte_length int;
    binary_string bytea;
BEGIN
    IF text_string = '' THEN
        RETURN 0;
    END IF;

    i = 0;
    tmp = 4294967295;
    byte_length = bit_length(text_string) / 8;
    binary_string = decode(replace(text_string, E'\\', E'\\\\'), 'escape');
    LOOP
        tmp = (tmp # get_byte(binary_string, i))::bigint;
        i = i + 1;
        j = 0;
        LOOP
            tmp = ((tmp >> 1) # (3988292384 * (tmp & 1)))::bigint;
            j = j + 1;
            IF j >= 8 THEN
                EXIT;
            END IF;
        END LOOP;
        IF i >= byte_length THEN
            EXIT;
        END IF;
    END LOOP;
    RETURN (tmp # 4294967295);
END
$$ IMMUTABLE LANGUAGE plpgsql;

isnull:

CREATE OR REPLACE FUNCTION isnull(anyelement) RETURNS int AS $$
BEGIN
    RETURN CAST(($1 IS NULL) AS INT);
END
$$ LANGUAGE plpgsql;

After creating the three UDFs above, let’s test the previous example. Note that UNSIGNED should be changed to BIGINT.

DROP TABLE IF EXISTS t;
CREATE TABLE t (i INT, j INT);
INSERT INTO t VALUES (2, 3), (NULL, NULL);
SELECT bit_xor(
    CAST(crc32(
        concat_ws(',',
            i, j,
            concat(isnull(i), isnull(j))
        )
    ) AS BIGINT)
)
FROM t;

The result:

 bit_xor
---------
 5062371
(1 row)

It’s exactly the same as on the TiDB (MySQL) side.

Postscript

1. I haven’t tested more extensively; this is just a simple test.
2. UDFs are indeed a great feature that greatly enhance flexibility.

Database Structure

The database consists of four tables:

1. STUDENT: Contains student number (SNO), name (SNAME), gender (SSEX), birthday (SBIRTHDAY), and class (CLASS).
2. COURSE: Includes course number (CNO), name (CNAME), and teacher number (TNO).
3. SCORE: Records student number (SNO), course number (CNO), and degree (DEGREE).
4. TEACHER: Holds teacher number (TNO), name (TNAME), gender (TSEX), birthday (TBIRTHDAY), professional title (PROF), and department (DEPART).

Sample Data

• Students such as Zeng Hua, Kang Ming, and Wang Fang are stored with specific details, including their class and gender.
• Courses like “Introduction to Computers” and “Operating Systems” are associated with teacher numbers.
• Scores are recorded for students across various courses.
• Teachers are described with their professional roles and departments.

Query Problems

Several SQL queries are suggested for practice, such as:

• Extracting specific columns like SNAME, SSEX, and CLASS from the STUDENT table.
• Listing distinct departments for teachers.
• Calculating and sorting grades within the SCORE table.
• Performing database operations to find student averages, count of students per class, and comparing scores.

Advanced Query Exercises

• Performing set operations and conditional joins to answer complex questions like finding students who scored more than others or comparing teachers’ scores.
• Use of SQL functions like DATE_PART, subqueries, and unions to gather specific data.

Additional Queries

• Techniques to refine queries for performance, like avoiding the NOT IN method.
• Handling conditions like age calculations using AGE(SBIRTHDAY) and filtering by name patterns.

Overall, these exercises provide a robust framework for practicing SQL skills on a structured set of sample data, focusing on various database manipulation and retrieval techniques.7. Query:

- `SELECT A.TNAME, B.CNAME FROM TEACHER A JOIN COURSE B ON A.TNO = B.TNO WHERE A.TSEX='男';`
- Explanation: Joins teacher and course tables to select male teachers and their course names.

1. Query:

   • SELECT A.* FROM SCORE A WHERE DEGREE=(SELECT MAX(DEGREE) FROM SCORE B);
   • Explanation: Selects all columns from the highest score in the score table.

2. Query:

   • SELECT SNAME FROM STUDENT A WHERE SSEX=(SELECT SSEX FROM STUDENT B WHERE B.SNAME='李军');
   • Explanation: Selects student names who have the same gender as the student named ‘Li Jun.’

3. Query:

   • SELECT SNAME FROM STUDENT A WHERE SSEX=(SELECT SSEX FROM STUDENT B WHERE B.SNAME='李军') AND CLASS=(SELECT CLASS FROM STUDENT C WHERE C.SNAME='李军');
   • Explanation: Selects student names who have the same gender and class as the student named ‘Li Jun.’

4. Two Answers:

   • SELECT A.* FROM SCORE A JOIN STUDENT B ON A.SNO = B.SNO JOIN COURSE C ON A.CNO = C.CNO WHERE B.SSEX='男' AND C.CNAME='计算机导论';
   • SELECT * FROM SCORE WHERE SNO IN(SELECT SNO FROM STUDENT WHERE SSEX='男') AND CNO=(SELECT CNO FROM COURSE WHERE CNAME='计算机导论');
   • Explanation: Both queries select scores of male students for the course ‘Introduction to Computer Science.’