We’ve all been there. You’re deep in a technical discussion with an AI assistant—analyzing code, exploring architecture, or debugging a complex issue. The conversation is rich with insights, and you think: “This would make a great blog post.”

But then reality hits: you need to switch to your blog repository, format the content, commit it, push it, and wait for the build. By the time you’re back, the original context is gone, and the momentum is lost.

What if you could publish directly from where you are?

Building on Existing Automation

In my previous post about automatically publishing a blog using GitHub Actions, I set up a workflow where pushing to the blog repository triggers an automatic build and deployment to GitHub Pages. This solved the build and deployment automation, but there was still one manual step remaining: creating the post file itself.

The workflow I described there handles:

1. Checking out the blog repository
2. Building the Hugo site with make
3. Deploying to jackysp.github.io

But you still needed to be in the blog repository to create the post. That’s where MCP changes everything.

Enter MCP: The Missing Link

The Model Context Protocol (MCP) is revolutionizing how AI agents interact with external systems. Instead of treating AI as a passive tool, MCP enables agents to act as autonomous agents with direct access to your tools and workflows.

In my setup, I’ve connected MCP-enabled agents (like Cursor) directly to my blog repository via GitHub MCP. This means:

• No context switching: Stay in your current working directory, whether it’s a random project folder or a deep codebase exploration
• Preserve conversation flow: The AI maintains the full context of your discussion
• Direct publishing: Create and publish posts without leaving your IDE

The Architecture: Seamless Integration

Here’s how the complete workflow operates:

┌─────────────────────────────────────────────────────────┐
│  AI Agent (Cursor/Claude) with MCP enabled              │
│  - Context: Any code repository or discussion            │
│  - Tool: GitHub MCP Server                               │
└──────────────────┬──────────────────────────────────────┘
                   │
                   │ Creates post via GitHub MCP
                   │
                   ▼
┌─────────────────────────────────────────────────────────┐
│  Blog Repository (jackysp/blog)                         │
│  - content/posts/[new-post].md                          │
│  - Commit: "Publish: [title]"                           │
└──────────────────┬──────────────────────────────────────┘
                   │
                   │ Push to master branch
                   │
                   ▼
┌─────────────────────────────────────────────────────────┐
│  GitHub Actions (from previous post)                     │
│  - Build: Hugo static site generation                   │
│  - Deploy: Push to jackysp.github.io                    │
└──────────────────┬──────────────────────────────────────┘
                   │
                   │ Published
                   │
                   ▼
┌─────────────────────────────────────────────────────────┐
│  Live Site (jackysp.github.io)                          │
│  - Post is live and accessible                           │
└─────────────────────────────────────────────────────────┘

The GitHub Actions part remains exactly as described in the previous post—no changes needed there. The MCP layer adds the ability to trigger it from anywhere.

The Workflow in Action

1. AI-Powered Content Creation

When you’re discussing a technical topic with an AI agent, you can simply ask:

“Turn this discussion into a blog post and publish it.”

The AI agent, with access to GitHub via MCP, can:

• Extract key insights from your conversation
• Format content according to Hugo front matter requirements
• Create properly structured markdown files
• Handle images and assets
• Commit and push to the repository

2. Automated Build & Deploy

The moment a post is pushed to the master branch, the same GitHub Actions workflow from the previous post kicks in:

on:
  push:
    branches: [ master ]

The workflow (as detailed in the previous post):

1. Checks out the blog repository with submodules
2. Builds the Hugo site using make
3. Deploys the built artifacts to jackysp.github.io

All without manual intervention.

3. Governance Through Contracts

To ensure quality and prevent accidents, I’ve implemented an AI Publishing Contract (PUBLISHING.md) that defines:

• Allowed paths: Only content/** and static/** can be modified
• Post format: Required front matter fields (title, date, tags, slug, summary)
• Image handling: Standardized location and reference format
• Commit conventions: Single commit per post with descriptive messages

This contract ensures that AI agents can publish content while respecting the repository structure and quality standards.

Why This Matters: The Developer Experience Revolution

Zero Context Switching

Traditional workflow:

1. Copy conversation → Switch to blog repo → Format → Commit → Push → Wait
2. Context lost, momentum broken

New workflow:

1. Ask AI to publish → Done
2. Context preserved, workflow continuous

Capturing Technical Insights

The best technical insights often emerge during active problem-solving. With this workflow, you can:

• Document discoveries in real-time
• Turn debugging sessions into tutorials
• Transform architecture discussions into deep-dives
• Share codebase explorations as learning resources

Scaling Knowledge Sharing

Previously, the friction of publishing meant many valuable insights were never written down. Now, the barrier to publishing is minimal, making it easier to:

• Share learnings with your team
• Build a personal knowledge base
• Contribute to the developer community
• Document your problem-solving journey

Technical Implementation Details

MCP Server Configuration

The GitHub MCP server provides the AI agent with:

• Repository read/write access
• File creation and modification
• Commit and push capabilities
• Branch management

GitHub Actions Workflow

The CI/CD pipeline (as described in the previous post) handles:

• Go environment setup (for Hugo builds)
• Repository checkout with submodules
• Site generation via make
• Deployment to GitHub Pages repository

No changes needed to the existing workflow—it just gets triggered from a new entry point.

Hugo Site Configuration

Posts follow Hugo’s standard structure:

• Location: content/posts/
• Format: YAML front matter + Markdown content
• Images: Stored in content/posts/images/
• Draft control: draft: true/false for preview/publish

The Future: AI-Augmented Documentation

This workflow represents a shift toward AI-augmented documentation. Instead of treating AI as a writing assistant, we’re treating it as a publishing agent that can:

• Understand context from code discussions
• Extract technical insights automatically
• Format and structure content appropriately
• Publish without breaking workflow

As MCP and similar protocols mature, we’ll see more sophisticated capabilities:

• Automatic code analysis and explanation
• Multi-post series generation from extended discussions
• Cross-referencing with existing content
• SEO and metadata optimization

Getting Started

If you want to set up a similar workflow:

1. Set up automated publishing (see my previous post)
2. Enable MCP in your AI agent (Cursor, Claude Desktop, etc.)
3. Configure GitHub MCP server with repository access
4. Define publishing contracts for governance
5. Start publishing from your conversations

The technical details are straightforward, but the impact on productivity and knowledge capture is profound.

Conclusion

The intersection of AI agents, MCP protocols, and automated CI/CD creates a new paradigm for technical publishing. By building on the existing GitHub Actions automation and adding MCP as the entry point, we eliminate context switching and reduce friction.

This isn’t just about automating blog posts—it’s about preserving the flow state of technical discovery and making knowledge sharing as natural as having a conversation.

The future of technical documentation is here, and it’s conversational.

This post was created and published using the exact workflow described above—from a discussion about workflow automation to a live blog post, all without leaving the conversation context.

Most Moka pot guides treat brewing as a recipe problem: grind size, water temperature, and heat level.

That framing breaks down completely when you introduce:

• A Brikka (pressure valve + burst extraction)
• An induction hob
• A steel induction adapter plate

At this point, you are no longer “brewing coffee”. You are operating a multi-stage thermal + pressure system with delayed feedback.

This article reframes Brikka-on-induction as a control problem, not a recipe.

Brikka vs Classic Moka: A Structural Difference

Classic Moka pots operate as continuous-flow systems.

Brikka is fundamentally different.

Brikka is a Triggered System

• No flow occurs until a pressure threshold is reached
• Once triggered, the valve opens abruptly
• Extraction happens in a very short, high-energy window

From a systems perspective:

Classic Moka = streaming pipeline
Brikka = edge-triggered event

Induction + Adapter Plate: Where the Model Changes

With an induction hob and the official Bialetti adapter plate, the heat path becomes:

Induction coil
→ Steel adapter plate
→ Aluminum boiler
→ Water

The adapter plate introduces significant thermal inertia.

Low or medium power levels often never leave the heat accumulation phase.

Engineering Goal: Reach the Trigger, Then Stop

Because Brikka extracts only after the valve opens, the primary goal is:

Drive the system to the trigger point as efficiently as possible — then stop.

Any energy added after triggering only increases bitterness.

Parameter Design

Water

• Cold water to the fill line

Cold water ensures a linear pressure ramp.

Beans

• Medium to medium-light roast

Brikka amplifies front-loaded flavors.

Grind

• Timemore C5 ESP: baseline 1.2.0
• Adjust only ±0.0.5

Brikka has a narrow operating window.

Heat Strategy

Preheating Phase (Before Grinding):

• Start heating the adapter plate at Level 9 while measuring beans and grinding
• The official stainless steel adapter plate is thick and conducts heat slowly
• Preheating ensures the plate reaches operating temperature before brewing begins

Brewing Phase:

• Phase 1: Level 9 until trigger
• Phase 2: Cut power immediately at first continuous output
• Rinse boiler bottom with cold water

Residual heat is sufficient.

Timing as Validation

• Trigger time: 3–5 minutes
• Much longer indicates system inefficiency

Time validates the system, not flavor.

Design Trade-offs

High initial power stresses equipment but exits the thermal dead zone.

Immediate cutoff sacrifices volume but preserves flavor boundaries.

Final Mental Model

Brikka is not a brewer.
It is a pressure-triggered extraction event.

Once the event fires, the system should coast.

Respecting that boundary makes the system predictable and repeatable.

OceanBase targets high availability and scalability in a shared-nothing cluster. The core engineering challenge is to make four critical subsystems work together with predictable latency and correctness:

• Write transactions must be durable, replicated, and efficiently committed.
• Tablet replay must recover state quickly and safely.
• SQL parse to execute must optimize well while respecting multi-tenant constraints.
• Unit placement must map tenants to physical resources without fragmentation.

This article focuses on motivation, design, implementation highlights, and tradeoffs, using concrete code entry points from the OceanBase codebase.

System Architecture

OceanBase adopts a shared-nothing architecture where each node is equal and runs its own SQL engine, storage engine, and transaction engine. Understanding the overall architecture is essential before diving into implementation details.

Cluster, Zone, and Node Organization

graph TB
    subgraph Cluster["OceanBase Cluster"]
        subgraph Z1["Zone 1"]
            N1["OBServer Node 1"]
        end
        subgraph Z2["Zone 2"]
            N2["OBServer Node 2"]
        end
        subgraph Z3["Zone 3"]
            N3["OBServer Node 3"]
        end
    end
    
    subgraph Proxy["obproxy Layer"]
        P1["obproxy 1"]
        P2["obproxy 2"]
    end
    
    Client["Client Applications"] --> Proxy
    Proxy --> N1
    Proxy --> N2
    Proxy --> N3

Key Concepts:

• Cluster: A collection of nodes working together
• Zone: Logical availability zones for high availability and disaster recovery
• OBServer: Service process on each node handling SQL, storage, and transactions
• obproxy: Stateless proxy layer routing SQL requests to appropriate OBServer nodes

Data Organization: Partition, Tablet, and Log Stream

graph TB
    subgraph Table["Table"]
        P1["Partition 1"]
        P2["Partition 2"]
        P3["Partition 3"]
    end
    
    subgraph LS1["Log Stream 1"]
        T1["Tablet 1"]
        T2["Tablet 2"]
    end
    
    subgraph LS2["Log Stream 2"]
        T3["Tablet 3"]
    end
    
    subgraph LS3["Log Stream 3"]
        T4["Tablet 4"]
    end
    
    P1 --> T1
    P1 --> T2
    P2 --> T3
    P3 --> T4
    
    T1 --> LS1
    T2 --> LS1
    T3 --> LS2
    T4 --> LS3

Key Concepts:

• Partition: Logical shard of a table (hash, range, list partitioning)
• Tablet: Physical storage object storing ordered data records for a partition
• Log Stream (LS): Replication unit using Multi-Paxos for data consistency
• Replication: Each tablet has multiple replicas across zones, with one leader accepting writes. Log streams replicate data via Multi-Paxos protocol across different zones.

Multi-Tenant Resource Model

graph TB
    subgraph Tenant["Tenant"]
        T1["Tenant 1 MySQL Mode"]
        T2["Tenant 2 Oracle Mode"]
        T3["System Tenant"]
    end
    
    subgraph Pool["Resource Pool"]
        RP1["Pool 1"]
        RP2["Pool 2"]
        RP3["Pool 3"]
    end
    
    subgraph Unit["Resource Unit"]
        U1["Unit 1 CPU Memory Disk"]
        U2["Unit 2 CPU Memory Disk"]
        U3["Unit 3 CPU Memory Disk"]
    end
    
    subgraph Server["Physical Server"]
        S1["Server 1"]
        S2["Server 2"]
        S3["Server 3"]
    end
    
    T1 --> RP1
    T2 --> RP2
    T3 --> RP3
    
    RP1 --> U1
    RP2 --> U2
    RP3 --> U3
    
    U1 --> S1
    U2 --> S2
    U3 --> S3

Key Concepts:

• Tenant: Isolated database instance (MySQL or Oracle compatibility)
• Resource Pool: Groups resource units for a tenant across zones
• Resource Unit: Virtual container with CPU, memory, and disk resources
• Unit Placement: Rootserver schedules units to physical servers based on resource constraints

Layered Architecture

graph TB
    subgraph App["Application Layer"]
        Client["Client Applications"]
    end
    
    subgraph Proxy["Proxy Layer"]
        ODP["obproxy Router"]
    end
    
    subgraph OBServer["OBServer Layer"]
        subgraph Node["OBServer Node"]
            SQL["SQL Engine"]
            TX["Transaction Engine"]
            ST["Storage Engine"]
        end
    end
    
    subgraph Storage["Storage Layer"]
        subgraph LS["Log Stream"]
            Tablet["Tablet"]
            Memtable["Memtable"]
            SSTable["SSTable"]
        end
        Palf["Paxos Log Service"]
    end
    
    subgraph Resource["Resource Layer"]
        Tenant["Tenant"]
        Unit["Resource Unit"]
        Pool["Resource Pool"]
        RS["Rootserver"]
    end
    
    Client --> ODP
    ODP --> SQL
    SQL --> TX
    TX --> ST
    ST --> LS
    LS --> Palf
    Tenant --> Unit
    Unit --> Pool
    Pool --> RS
    RS --> Node

Key Concepts:

• SQL Engine: Parses, optimizes, and executes SQL statements
• Transaction Engine: Manages transaction lifecycle, commit protocols, and consistency
• Storage Engine: Handles data organization, memtables, and SSTables
• Log Service: Provides Paxos-based replication and durability
• Rootserver: Manages cluster metadata, resource scheduling, and placement

Design Overview

At a high level, each node runs a full SQL engine, storage engine, and transaction engine. Data is organized into tablets, which belong to log streams. Log streams replicate changes using Paxos-based log service. Tenants slice resources via unit configurations and pools, while rootserver components place those units on servers.

The following sections walk through each path with the relevant implementation anchors.

Architecture Diagrams

Transaction Write Path

flowchart LR
  subgraph A["Transaction Write Path"]
    C["Client"] --> S["SQL Engine"]
    S --> T["Transaction Context"]
    T --> M["Memtable Write"]
    M --> R["Redo Buffer"]
    R --> L["Log Service"]
    L --> P["Replicated Log"]
    P --> K["Commit Result"]
  end

Tablet Replay Path

flowchart LR
  subgraph B["Tablet Replay Path"]
    L["Log Service"] --> RS["Replay Service"]
    RS --> E["Tablet Replay Executor"]
    E --> LS["Log Stream"]
    LS --> TB["Tablet"]
    TB --> CK["Replay Checks"]
    CK --> AP["Apply Operation"]
    AP --> ST["Updated Tablet State"]
  end

SQL Compile and Execute

flowchart LR
  subgraph C["SQL Compile and Execute"]
    Q["SQL Text"] --> P["Parser"]
    P --> R["Resolver"]
    R --> W["Rewriter"]
    W --> O["Optimizer"]
    O --> LP["Logical Plan"]
    LP --> CG["Code Generator"]
    CG --> PP["Physical Plan"]
    PP --> EX["Executor"]
  end

Unit Placement

flowchart LR
  subgraph D["Unit Placement"]
    UC["Unit Config"] --> RP["Resource Pool"]
    RP --> PS["Placement Strategy"]
    PS --> CS["Candidate Servers"]
    CS --> CH["Chosen Server"]
    CH --> UN["Unit Instance"]
  end

Write Transaction: From Memtable to Replicated Log

Motivation

A write transaction must be both fast and durable. OceanBase uses memtables for in-memory writes, and a log stream for redo replication. The design must allow low-latency commit while supporting parallel redo submission and multi-participant (2PC) transactions.

Design Sketch

• Each transaction is represented by a per-LS context (ObPartTransCtx).
• Redo is flushed based on pressure or explicit triggers.
• Commit chooses one-phase or two-phase based on participants.
• Logs are submitted via a log adapter backed by logservice.

Implementation Highlights

• Transaction context lifecycle and commit logic are in src/storage/tx/ob_trans_part_ctx.cpp.
• Redo submission is driven by submit_redo_after_write, which switches between serial and parallel logging based on thresholds.
• Commit decides between one-phase and two-phase commit depending on participant count.
• The log writer (ObTxLSLogWriter) submits serialized logs via ObITxLogAdapter, which is wired to logservice (ObLogHandler).

Tradeoffs

• Serial vs parallel redo: Serial logging is simpler and cheaper for small transactions, but parallel logging reduces tail latency for large transactions at the cost of more coordination.
• 1PC vs 2PC: 1PC is fast for single-participant transactions; 2PC is required for distributed consistency but increases coordination overhead.
• In-memory batching vs durability: Larger batching improves throughput but can delay durability and increase replay time.

Tablet Replay: Reconstructing State Safely

Motivation

Recovery needs to be deterministic and safe: the system must replay logs to reconstruct tablet state without violating invariants or applying obsolete data.

Design Sketch

• Logservice schedules replay tasks per log stream.
• Tablet replay executor fetches the LS, locates the tablet, validates replay conditions, and applies the log.
• Specialized replay executors handle different log types (e.g., schema updates, split operations).

Implementation Highlights

• Replay orchestration lives in src/logservice/replayservice/ob_log_replay_service.cpp.
• Tablet replay logic is in src/logservice/replayservice/ob_tablet_replay_executor.cpp.
• Specific tablet operations are applied in dedicated executors, such as ObTabletServiceClogReplayExecutor in src/storage/tablet/ob_tablet_service_clog_replay_executor.cpp.

Tradeoffs

• Strictness vs throughput: Replay barriers enforce ordering for correctness but can reduce parallelism.
• Tablet existence checks: Allowing missing tablets can speed recovery but requires careful validation to avoid partial state.
• MDS synchronization: Metadata state updates improve correctness but add contention via locks.

SQL Parse to Execute: Compile Pipeline for Performance

Motivation

OceanBase supports MySQL and Oracle compatibility with rich SQL features. The compile pipeline must be fast, cache-friendly, and yield efficient execution plans.

Design Sketch

• SQL text enters the engine via ObSql::stmt_query.
• Parsing produces a parse tree.
• Resolution turns the parse tree into a typed statement tree.
• Rewrite and optimization generate a logical plan.
• Code generation produces a physical plan and execution context.

Implementation Highlights

• Entry and query handling: src/sql/ob_sql.cpp (stmt_query, handle_text_query).
• Resolver: ObResolver in src/sql/resolver/ob_resolver.h.
• Transform and optimize: ObSql::transform_stmt and ObSql::optimize_stmt in src/sql/ob_sql.cpp.
• Code generation: ObSql::code_generate in src/sql/ob_sql.cpp.

Tradeoffs

• Plan cache vs compile accuracy: Plan caching reduces latency but may reuse suboptimal plans under changing data distributions.
• Rewrite aggressiveness: More transformations can yield better plans but increase compile cost.
• JIT and rich formats: Faster execution for some workloads, but added complexity and memory pressure.

Unit Placement: Scheduling Tenant Resources

Motivation

Multi-tenancy requires predictable isolation and efficient resource utilization. Unit placement must respect CPU, memory, and disk constraints while minimizing fragmentation.

Design Sketch

• Unit config defines resource demands.
• Resource pool groups units by tenant and zone.
• Placement strategy scores candidate servers to pick a host for each unit.

Implementation Highlights

• Resource types and pools: src/share/unit/ob_unit_config.h, src/share/unit/ob_resource_pool.h, src/share/unit/ob_unit_info.h.
• Placement policy: src/rootserver/ob_unit_placement_strategy.cpp uses a weighted dot-product of remaining resources to choose a server.
• Orchestration: src/rootserver/ob_unit_manager.cpp handles creation, alteration, and migration of units and pools.

Tradeoffs

• Greedy placement vs global optimality: Dot-product scoring is efficient and practical but may not be globally optimal.
• Capacity normalization: Assuming comparable server capacities simplifies scoring but may bias placement in heterogeneous clusters.
• Latency vs stability: Fast placement decisions can lead to more churn; conservative placement improves stability but can reduce utilization.

Closing Thoughts

These four paths demonstrate how OceanBase balances correctness, performance, and operability. The code structure follows clear separation of responsibilities: transaction logic is in storage/tx, replication and replay are in logservice, SQL compilation is in sql, and scheduling is in rootserver and share/unit. The tradeoffs are explicit and largely encoded in thresholds and policies, which makes tuning feasible without invasive rewrites.

If you are extending OceanBase, start with the entry points highlighted above and follow the call chains into the relevant subsystem. It is the fastest way to build a mental model grounded in the actual implementation.

WebVM is a virtual machine (VM) that executes entirely within a web browser. It’s an innovative project that brings the power of a Linux environment straight to your browser, eliminating the need for traditional virtual machine setups. WebVM operates within a sandboxed environment, ensuring secure execution of applications without affecting the host system.

Understanding WebVM

The source repository provides a frontend for the WebVM demo. By forking the repository and following the instructions outlined in the GitHub Actions, you can build an image using the Dockerfile located at:

• Debian Large Dockerfile

This image can then be hosted on GitHub Pages, as demonstrated in my demo.

WebVM’s primary functionality is to stream resources to the browser, minimizing client-side resource consumption. It enables the execution of various applications that are typically restricted to virtual machines, all within a browser. Additionally, WebVM allows for the embedding of these applications through custom front-ends.

Default Image and Capabilities

The default WebVM image is Debian-mini, which may have limited capabilities. To enhance its functionality, I have opted for the Debian-large image, which has been extended to a 2GB disk capacity. This provides a more robust environment with additional tools and packages.

Usage and Benefits

WebVM offers a range of applications and capabilities:

1. Custom Image Creation: Create custom images tailored to your specific requirements, allowing for a personalized virtual environment.

2. Web-Based Linux Terminal: Access a web-based Linux terminal to execute Linux commands directly within the browser. This includes:

   • SSH/SCP File Transfers: Securely transfer files using SSH and SCP protocols.
   • HTTP Server Initiation: Start an HTTP server using python3 -m http.server.

3. Sandboxed Security: Operates within a sandbox environment, ensuring secure execution of applications without affecting the host system.

4. Serverless Architecture: Embraces a serverless architecture by executing entirely on the client side. Running a Linux server within a browser presents a unique and innovative approach to virtualization.

Alternative Options

Yes, there are alternative options. JSLinux is a preferred and faster option. However, it does not allow modifications to the image, which can be a limitation if you require a customized environment.

Additional Tips

• Internet Connectivity via Tailscale:

  WebVM can connect to the internet via Tailscale. It utilizes the first available node as an exit node. If you execute:

  curl https://ifconfig.me
  

  You will obtain your current node’s IP address.

• DNS Functionality:

  The DNS functionality of Tailscale is currently experiencing issues. It’s recommended to use IP addresses to connect to other nodes within your Tailscale network instead of domain names.

• Default Credentials:

  You can obtain the default user:password and root:password credentials by checking the Dockerfile:

  • Default Credentials in Dockerfile

Conclusion

WebVM is a powerful tool that brings the versatility of a Linux environment to your browser. Whether you’re looking to experiment with Linux commands, develop applications, or require a portable and sandboxed environment, WebVM offers a serverless and secure solution. Its ability to create custom images and operate entirely on the client side sets it apart from other web-based virtual machines.

Feel free to explore WebVM and customize it to suit your needs. Happy coding!

What Modifications Did I Make?

1. Handlebar Grips

   The original grips were glued on and became loose over time, so I replaced them with new, more secure ones.

2. Bell

   Safety first! I added a bell to alert pedestrians and other cyclists.

3. CNC Mount Base for Action Cameras

   Installed at the center of the handlebar, it’s perfect for mounting my DJI OSMO Pocket 1 or a phone holder.

4. Front Bottle Bag from Decathlon Riverside

   A convenient spot for a water bottle and small items.

5. Front Light

   Mounted just above the front wheel to illuminate the path without blinding anyone.

6. Soft Saddle from Decathlon

   For a more comfortable ride.

7. Four Universal Easy Wheels

   These make the bike easier to push when folded. The set of four cost me 100 SGD.

8. Taillight

   For visibility and safety during night rides.

9. Front Bag

   Provides additional storage space for essentials.

10. Anti-Loose Accessory

    Helps keep components tight and secure.

    

11. Tool Set

    Essential for on-the-go repairs and adjustments.

    

Why Did I Choose These Modifications?

I experimented with mounting accessories on the handlebar, like the bottle holder and front light, but it made the handlebar too crowded and unstable. For instance, the bottle holder wasn’t secure when attached to the handlebar. I decided to utilize existing mounts, such as the taillight bracket, to streamline the setup.

The additional accessories are intended to make the bike even easier to push when folded. It’s already convenient, but I thought, why not enhance it further?

Areas for Improvement

• Anti-Loose Accessory: It’s not working as well as I’d hoped and might need to be replaced.

• Front Bag Straps: They’re a bit long, and I’m concerned they could get caught in the front wheel. I might need to shorten them.

Cost Breakdown

Most accessories cost between 5–50 SGD, except for the universal wheels, which were 100 SGD for all four. Initially, I wasn’t sure if I needed all four wheels—maybe two would have sufficed—but I ultimately decided to get all four for better stability.

Anything Else?

I can’t think of anything else at the moment. If you have any suggestions or feedback, I’d love to hear them!

Tips for Buying Accessories in Singapore

• Decathlon: A great place to find affordable accessories, and they have a 365-day return policy!

• Shopee: Good for finding unique or hard-to-find items.

• Taobao: Offers a wide range of accessories, but shipping can be expensive. You can ask a friend in China to help you buy and ship the items to Singapore.

I don’t recommend buying from bike shops in Singapore, as they tend to be more expensive—about twice the price.

Let me know if there’s anything else you’d like to know or discuss!

As a long-time podcaster, I’ve always enjoyed sharing my thoughts and ideas through audio. While the world of video content—and the role of a YouTuber—has its allure, the complexities of video editing have kept me anchored in the realm of podcasting. My journey has involved leveraging platforms like Spotify Creator (formerly Anchor) for hosting and distributing my recordings. This platform offers a wide array of features for free, including audio recording, editing capabilities, and automatic promotion to Spotify.

However, I sought a more comprehensive solution, one that would allow me to listen to my own podcast while driving, using the Podcast app on my CarPlay device. To achieve this, I ventured into publishing on Apple Podcasts (podcastsconnect.apple.com), which also offers free hosting. With a self-designed cover and episodes uploaded, I was set—or so I thought.

The Challenges of Traditional Podcasting

Despite having the technical setup, I faced significant challenges:

• Consistency: Maintaining a regular publishing schedule proved difficult.
• Voice Quality: My voice quality was inconsistent, affecting listener engagement.
• Content Preparation: Crafting well-structured episodes without improvisation was challenging.
• Enhancements: Incorporating background music and other audio elements to enrich the listening experience required additional effort.

These hurdles led to my podcast being suspended for approximately two years. I found myself in need of a solution that could simplify the process and revitalize my passion for podcasting.

Discovering NotebookLM: An AI-Powered Podcasting Tool

Recently, I stumbled upon NotebookLM (notebooklm.google.com), an innovative application developed by Google. NotebookLM harnesses the power of artificial intelligence to generate podcast content. Users can provide a topic and related documents, and the AI takes over, creating engaging podcast episodes.

My Experience with NotebookLM

Intrigued, I decided to give NotebookLM a try. The results were nothing short of astounding:

• Effortless Production: The AI effortlessly generated a half-hour episode featuring two speakers discussing the topics in English.
• Enhanced Content: It went beyond the provided information, utilizing search engines to gather additional relevant data from the internet.
• Quality Output: The quality of the generated content was exceptionally high, surpassing what I could produce on my own.
• Incorporated Music: Appropriate background music was added, enhancing the overall listening experience.
• Cost-Free: All these features were available entirely for free.

A Case Study: Deep Dive into TiDB

To put NotebookLM to the test, I created an episode about TiDB, a product developed by my current employer. The process was seamless, and the final product was impressive. You can listen to the episode here: Deep Dive into TiDB.

Conclusion

The integration of AI into podcast creation through tools like NotebookLM has the potential to revolutionize the way we produce content. It removes many of the barriers that podcasters face, such as time constraints, technical challenges, and the need for consistent quality.

For anyone looking to start or rejuvenate their podcast without the traditional hassles, I highly recommend giving NotebookLM a try. It’s remarkable to see how AI can not only match but enhance human capabilities in creative endeavors.

I hope this helps! Let me know if there’s anything you’d like to add or modify in your blog post.

Key Features

• Compact Size: The trifold design results in a very small folded size.
• Lightweight: Weighing between 10 to 11 kg, making them easy to carry.
• Easy Transportation: Suitable for air travel, and can be taken on public transport like MRT and buses.
• Thin Tires: Facilitates easy movement and efficient riding.
• Front Bracket: Allows for quick changes of front bags.
• Popularity: Highly popular in Singapore.
• Non-standard Accessories: May require specific accessories due to unique design.

My bike is a Brompton clone, and I’ll share some insights on the various brands and models available.

Brompton Bikes

Brompton is the original brand offering high-quality trifold bikes.

• Price Range: Approximately 10,000 RMB to 40,000 RMB, with common models around 15,000 RMB.
• Models:
  • C Line: Features 3-speed internal hub gears.
  • P Line: Equipped with 4-speed external shift gears.

Brompton Clone Brands

For those seeking more affordable options, several brands offer Brompton-like bikes ranging from 3,000 to 7,000 RMB.

Aceoffix

• Gears: Available in 3 or 5-speed external shift gears.
• Frame Material: Options of steel or aluminum frames.
• Compatibility: Standard size compatible with Brompton accessories.

Cran

• Gears: Offers 7 or 9-speed external shift gears.
• Sizes:
  • 7-Speed: Standard size.
  • 9-Speed: Non-standard size.
• Frame Material: Primarily steel; aluminum frames may have stability issues.

3Sixty

• One of the earliest Brompton clones.
• Features internal shift gears.

Yubu (鱼布)

• Custom assembled by a Brompton enthusiast in Hangzhou.
• Gears: 3-speed external shift gears.
• Reputation: Known for high quality and reliability.

Dagger

• A new entrant in the market.
• Price: Offers bikes at 3,700 RMB with features comparable to higher-priced models (>5,000 RMB).
• Value: High cost-performance ratio.

Choosing the Right Bike

• Avoid Internal Shift Gears: While reliable, they add weight to the bike.
• Optimal Gears: 3 or 5-speed external gears are stable and sufficient for most needs.
• Portability Concerns: 9-speed models might be less convenient for transportation due to size.
• Standard Size Preference: Ensures compatibility with standard transfer cases and accessories.
• Pedals: Consider non-foldable pedals with quick-release options for convenience.

Essential Accessories

• Lighting: Front and rear lights are mandatory by law in Singapore.
• Additional Gear:
  • Bottle Bracket
  • Bell
  • Phone Bracket
  • Helmet
  • Front and Rear Lights
• Bag: A Brompton two-strap bag for carrying essentials.

Riding Tips

• UV Protection: Use sun cream or wear sun-protective clothing.
• Saddle Upgrade:

  • Wide Saddle: For rides longer than 1 hour. 90 degrees is ideal for the saddle angle.

    

  • Thin Saddle: Suitable for shorter rides. 60 degrees is ideal for the saddle angle.

    

• Accessory Shopping: Most accessories can be found at stores like Decathlon, though some items may require sourcing elsewhere due to sizing.

Note: While many accessories are readily available at retailers like Decathlon, certain items like helmets may have limited sizing options.

Sanrenmu 9201

• Price: 198 Yuan
• Lock Type: Axis Lock
• Handle Material: Plastic
• Blade Material: D2 Steel

First Impressions

The 9201 features an axis lock mechanism, which I found to be very smooth and easy to open and close. The D2 steel blade offers good edge retention, and the plastic handle keeps the knife lightweight.

Pros

• Ease of Use: The axis lock allows for ambidextrous operation, making it convenient for both left and right-handed users.
• Lightweight: The plastic handle reduces overall weight, making it comfortable to carry.

Cons

• Poor Detailing: Some edges and surfaces were rough, causing minor cuts to my hand during use.
• Finish Quality: The attention to detail was lacking, which detracted from the overall experience.

Personal Modifications

I used sandpaper and a rotary tool to smooth out the rough edges. While this improved the handling, I accidentally scratched the blade slightly during the process.

Sanrenmu 9008

• Price: 168 Yuan
• Lock Type: Frame Lock with Safety Lock
• Handle Material: Steel
• Blade Material: Unknown
• Features: Quick-opening flipper

First Impressions

Out of the box, the 9008 showcased excellent workmanship. The quick-opening flipper worked flawlessly, and the safety lock adds an extra layer of security when the knife is not in use.

Pros

• Build Quality: The knife feels solid and well-constructed.
• Quick-opening Flipper: Provides rapid deployment of the blade.
• Safety Lock: Prevents accidental opening.

Cons

• Weight: The steel handle makes it quite heavy, which isn’t ideal for everyday carry.
• Button Quality: The button mechanism isn’t as smooth as I would like.
• Blade Material: Unspecified steel that doesn’t hold an edge as well as higher-end materials.

Usage

Due to its weight, I decided to keep the 9008 in my car. It features a window-breaking hammer, making it a practical tool for emergencies.

Sanrenmu 820

• Price: 298 Yuan
• Lock Type: Frame Lock
• Handle Material: Titanium
• Blade Material: VG10 Steel
• Features: Ball bearings (no safety lock or flipper)

First Impressions

The 820 stands out with its titanium handle and VG10 blade, offering a premium feel. It operates on ball bearings, ensuring a smooth action when opening and closing.

Pros

• Material Quality: The VG10 blade and titanium handle are high-quality materials that enhance durability and aesthetics.
• Workmanship: Excellent fit and finish.

Cons

1. Opening Mechanism Issues:

   • The stop pin is too small.
   • The lock interface is large and deep, making one-handed opening difficult.

2. Pocket Clip:

   • The clip is excessively tight, making it hard to attach to pockets easily.

Personal Modifications

I disassembled the 820 and polished the lock interface, significantly improving the ease of one-handed opening. Post-modification, the knife opens smoothly, though there’s a slight looseness when closed.

Reflections and Insights

• Axis Lock vs. Frame Lock:

  • Axis Lock (9201): Offers ambidextrous use and works well with a plastic handle but has a complex structure that’s challenging to disassemble.
  • Frame Lock (820 & 9008): Simpler design and easier to disassemble (especially the 820 after some practice). However, it’s more suited to one-handed use and relies heavily on precise craftsmanship for smooth operation.

• Handle Materials:

  • Titanium Handle (820): Initially underestimated, the titanium handle provides a superior feel compared to plastic or steel. Once accustomed to it, switching back to other materials feels like a downgrade.
  • Steel Handle (9008): Durable but adds unnecessary weight, making it less ideal for carrying around.
  • Plastic Handle (9201): Lightweight but doesn’t offer the same premium feel as metal handles.

• Practicality vs. Aesthetics:

  • The 9008, while heavy, serves a practical purpose in the car for emergencies.
  • The 820, after modifications, has become my preferred everyday carry despite the tight pocket clip.

• DIY Modifications:

  • Personalizing and improving the knives was a rewarding experience, even if there were minor mishaps like scratching the blade on the 9201.
  • Disassembling and reassembling the 820 has turned it into not just a tool but a hobbyist’s toy.

Conclusion

Each knife has its place:

• Sanrenmu 9008: Stays in the car for emergency situations, thanks to its sturdy build and window-breaking feature.
• Sanrenmu 820: Becomes my go-to carry knife after modifications. Its high-quality materials and improved functionality make it stand out.
• Sanrenmu 9201: Although I’m unsure of its future use, it might serve as an additional car knife or a backup.

If you’re considering a Sanrenmu folding knife, think about what features matter most to you—be it the locking mechanism, handle material, or ease of carry. Also, don’t shy away from making personal adjustments to tailor the knife to your preferences.

Note: This review is based on personal experiences and modifications. Results may vary depending on individual usage and skills.

Recommended Usage

• Compile all Go files in the current directory:

  go build
  

• Compile all Go files explicitly:

  go build *.go
  

Common Pitfalls

Compiling a Single File

Running:

go build main.go

will only build main.go. This can lead to errors if main.go depends on other Go files in the same package, as those files won’t be included in the build process.

Including Non-Go Files

Using:

go build *

will cause an error if the * wildcard includes non-Go files. The compiler will output an error message stating that only Go files can be compiled. This serves as a precise reminder to exclude non-Go files from the build command.

By using go build correctly, you can ensure that all necessary files in your package are compiled together, avoiding missing dependencies and other common compilation issues.

Table of Contents

• Introduction to Ollama
  • A Popular Choice
  • Ease of Use
  • Built with Golang
• My Practices with Ollama
  • Preferred Models
    • Llama 3.1
    • Mistral
    • Phi-3
    • Qwen-2
  • Hardware Constraints
• Exploring UIs for Ollama
  • OpenWebUI
  • Page Assist
  • Enchanted
  • AnythingLLM
  • Dify
• Diving into llamaindex
• Experimenting with Candle
• Conclusion

Introduction to Ollama

A Popular Choice

Ollama has rapidly become a favorite among developers interested in local LLMs. Within a year, it has garnered significant attention on GitHub, reflecting its growing user base and community support.

Ease of Use

One of Ollama’s standout features is its simplicity. It’s as easy to use as Docker, making it accessible even to those who may not be deeply familiar with machine learning frameworks. The straightforward command-line interface allows users to download and run models with minimal setup.

Built with Golang

Ollama is written in Golang, ensuring performance and efficiency. Golang’s concurrency features contribute to Ollama’s ability to handle tasks effectively, which is crucial when working with resource-intensive LLMs.

My Practices with Ollama

Preferred Models

Llama 3.1

I’ve found that Llama 3.1 works exceptionally well with Ollama. It’s my go-to choice due to its performance and compatibility.

Mistral

While Mistral also performs well, it hasn’t gained as much popularity as Llama. Nevertheless, it’s a solid option worth exploring.

Phi-3

Developed by Microsoft, Phi-3 is both fast and efficient. The 2B parameter model strikes a balance between size and performance, making it one of the best small-sized LLMs available.

Qwen-2

Despite its impressive benchmarks, Qwen-2 didn’t meet my expectations in practice. It might work well in certain contexts, but it didn’t suit my specific needs.

Hardware Constraints

Running large models on hardware with limited resources can be challenging. On my 16GB MacBook, models around 7B to 8B parameters are the upper limit. Attempting to run larger models results in performance issues.

Exploring UIs for Ollama

Enhancing the user experience with UIs can make interacting with local LLMs more intuitive. Here’s a look at some UIs I’ve tried:

OpenWebUI

OpenWebUI offers a smooth and user-friendly interface similar to Ollama’s default UI. It requires Docker to run efficiently, which might be a barrier for some users.

• Features:
  • Basic Retrieval-Augmented Generation (RAG) capabilities.
  • Connection to OpenAI APIs.

Page Assist

Page Assist is a Chrome extension that I’ve chosen for its simplicity and convenience.

• Advantages:
  • No requirement for Docker.
  • Accesses the current browser page as input, enabling context-aware interactions.

Enchanted

Enchanted is unique as it provides an iOS UI for local LLMs with support for Ollama.

• Usage:
  • By using Tailscale, I can connect it to Ollama running on my MacBook.
  • Serves as an alternative to Apple’s native intelligence features.

AnythingLLM

AnythingLLM offers enhanced RAG capabilities. However, in my experience, it hasn’t performed consistently well enough for regular use.

Dify

Dify is a powerful and feature-rich option.

• Pros:
  • Easy to set up with an extensive feature set.
• Cons:
  • Resource-intensive, requiring Docker and running multiple containers like Redis and PostgreSQL.

Diving into llamaindex

llamaindex is geared towards developers who are comfortable writing code. While it offers robust functionalities, it does have a learning curve.

• Observations:
  • Documentation is somewhat limited, often necessitating diving into the source code.
  • The llamaindex-cli tool aims to simplify getting started but isn’t entirely stable.
    • Works seamlessly with OpenAI.
    • Requires code modifications to function with Ollama.

Experimenting with Candle

Candle is an intriguing project written in Rust.

• Features:

  • Uses Hugging Face to download models.
  • Simple to run but exhibits slower performance compared to Ollama.

• Additional Tools:

  • Cake: A distributed solution based on Candle, Cake opens up possibilities for scaling and extending use cases.

Conclusion

Exploring local LLMs has been an exciting journey filled with learning and experimentation. Tools like Ollama, llamaindex, and Candle offer various pathways to harnessing the power of LLMs on personal hardware. While there are challenges, especially with hardware limitations and setup complexities, the control and privacy afforded by local models make the effort worthwhile.

Feel free to share your experiences or ask questions in the comments below!

Discovering Tailscale Through WebVM

My journey with Tailscale began when I came across WebVM, an impressive project that allows you to run a virtual machine directly in your browser. Intrigued by the possibilities, I delved deeper and discovered that Tailscale could help me create a seamless, private network across all my devices.

What is Tailscale?

Tailscale is a mesh VPN network built on top of WireGuard, specifically using the WireGuard-go implementation. It allows you to create a secure, encrypted network between your devices, no matter where they are located.

Key Features

• Free Plan Available: Tailscale offers a free plan that is sufficient for personal use, allowing up to 20 devices.
• Ease of Use: Setting up Tailscale is straightforward. With minimal configuration, you can have your own network up and running quickly.
• Cross-Platform Support: Tailscale works exceptionally well across the Apple ecosystem, including iOS, tvOS, and macOS.
• Magic DNS Service: It provides a built-in DNS service that makes it easy to address your devices by name.

Performance on Different Platforms

While Tailscale shines on Apple devices, in my experience, it hasn’t performed as well on Windows. I encountered some connectivity and stability issues on Windows machines, which may vary based on individual setups.

My Tailscale Setup

Here’s how I leveraged Tailscale to connect my devices and access my home network seamlessly.

Running Tailscale on Apple TV

I installed Tailscale on my Apple TV, which stays online 24/7. This makes it an excellent candidate for a consistently available node in my network.

• Enabling Subnet Routing: By enabling subnet routing on the Apple TV, I can access other devices on the same local network, such as my NAS and router, as if I were connected locally.
• Setting Up an Exit Node: I configured the Apple TV as an exit node, allowing me to route internet traffic through my home network. This is useful when I need to access geo-restricted content or ensure a secure connection.

Connecting Other Devices

I also installed Tailscale on my MacBook and iPhone, which allows all my personal devices to communicate over the secure network, no matter where I am.

Benefits I’ve Enjoyed

• Secure Remote Access: I can securely access my home network devices from anywhere.
• Consistent Environment: All my devices appear on the same network, simplifying file sharing and remote management.
• No Need for Complex VPN Setups: Tailscale eliminates the need for traditional VPN configurations, port forwarding, or dynamic DNS services.

Conclusion

Tailscale has transformed the way I interact with my devices across different locations. Its ease of use and robust feature set make it an excellent choice for anyone looking to create a personal, secure network.

If you’re interested in simplifying your network setup and want a hassle-free way to connect your devices, I highly recommend giving Tailscale a try.

Links:

• Tailscale Official Website
• WebVM Project on GitHub
• WireGuard-go on GitHub

Note: This post reflects my personal experiences with Tailscale. Performance may vary based on individual configurations and devices.

Internet Access and Blocked Websites

• Slow Access to Twitter: If you’re an avid Twitter user, be prepared for slower connection speeds. The platform doesn’t perform as well in Vietnam.
• Inaccessible Websites: Certain websites are blocked in Vietnam, including:
  • Medium
  • Some porn websites
• Tip: If you rely on any of these platforms, consider downloading content beforehand or using a VPN to access them during your stay.

Visa Process for Chinese Citizens

• E-Visa Conversion: Travelers from China holding an E-Visa need to convert it into a paper visa upon arrival.
  • Bring Two Photos: Ensure you have two passport-sized photos ready for the visa conversion process.
  • Carry Cash for Tips: It’s customary to offer tips during the visa processing, so have some cash (preferably small denominations) on hand.

Exploring Saigon (Ho Chi Minh City)

• City Vibes: Saigon reminded me of a blend between Shanghai and Qingdao.
  • Cleanliness: While the city is bustling with energy, it’s not as clean as Shanghai or Qingdao.
• Abundance of Coffee Shops: Vietnam is famous for its coffee culture. You’ll find numerous coffee shops on almost every street corner.
  • Must-Try: Don’t miss out on traditional Vietnamese iced coffee (Cà Phê Sữa Đá).

Affordability

• Cheaper Than Bangkok: Compared to cities like Bangkok, Saigon is significantly more affordable.
  • Hotels: Accommodation can be about half the price of equivalent hotels in Bangkok.
  • General Expenses: Food, transportation, and entertainment are also reasonably priced.

Currency and Payments

• Accepted Currencies:
  • Vietnamese Dong (VND) is the official currency.
  • US Dollars (USD) are widely accepted and sometimes preferred.
  • Chinese RMB can be used in some places, but USD is more commonly accepted.
• Tip: Carrying USD can be convenient, but always have some local currency for small purchases.

Local Customs and Observations

• Clothing Choices: You’ll notice many locals wearing long-sleeved clothing and pants, even in hot weather.
  • Reason: This is to prevent sunburn and protect their skin from the sun.
• Hardworking People: The Vietnamese are known for their strong work ethic. Streets are lively with activity from early morning until late at night.
• Affordable Goods: Many items, from street food to souvenirs, are priced affordably.
• Non-Spicy Cuisine:
  • Similar to Cantonese Food: The flavors are mild and focus on the freshness of ingredients.
  • Must-Try Dishes: Pho (noodle soup), Banh Mi (baguette sandwich), and fresh spring rolls.

Final Thoughts

Vietnam offers a captivating blend of cultural experiences, historical sites, and modern attractions. Whether you’re sipping coffee in a cozy café, exploring bustling markets, or simply taking in the sights and sounds of the city, there’s something for every traveler.

Travel Tips:

• Plan Ahead: Be aware of the internet limitations and plan accordingly if you need access to certain websites.
• Stay Hydrated: The climate can be hot and humid, so drink plenty of water.
• Respect Local Customs: Dress modestly when visiting temples or rural areas.

I hope these insights help you in planning your trip to Vietnam. It’s a destination filled with unforgettable experiences!

Safe travels!

Introduction

Over the years, I’ve gathered a lot of experience in fishing, especially in selecting the right gear. Choosing the appropriate rod, reel, line, and lure can significantly impact your fishing success. In this blog, I want to share my insights to help you make informed decisions when selecting fishing equipment.

Rod Selection

Understanding Rod Actions

• L (Light) Action Rods:
  • Best for casting lures weighing 4-5g.
  • Ideal for using lures within half of the rod’s lure weight range (e.g., for a rod rated 2-7g, the optimal lure weight is around 4-5g).
• UL (Ultra-Light) Action Rods:
  • Suitable for casting lures weighing 3g or less.

One-Piece vs. Multi-Piece Rods

• One-Piece Rods:
  • Offer better sensitivity and casting performance.
  • However, they are less portable and harder to transport.
  • Not as easy to sell if you decide to upgrade.
• Multi-Piece Rods:
  • More convenient for travel.
  • Slight trade-off in sensitivity compared to one-piece rods.

Recommended Action

• The best action for spinning cast rods is Light (L) action.

Reel Selection

Spinning Reels vs. Baitcasting Reels

• Spinning Reels:
  • Easier for casting, especially for beginners.
  • Better suited for saltwater fishing than baitcasting reels.
  • Can use heavier action rods to cast small lures effectively.
• Baitcasting Reels:
  • Look cool and offer precision, but have a steeper learning curve.
  • Not as user-friendly for casting light lures.

Reel Size and Line Compatibility

• For a 2500S reel:
  • 0.8 PE line is sufficient.
  • Using 0.6 PE line on a 2500S reel may require too much line to fill the spool.
• For a 2000S reel:
  • 0.6 PE line is appropriate.
• It’s not common to use PE lines smaller than 0.6, as they are less widely available.

Light Weight Reels

• Always choose lightweight reels for better balance and less fatigue during fishing.

Line Selection

PE Line vs. Fluorocarbon

• Casting Difficulty:
  • 1.5 PE and fluorocarbon lines offer similar casting challenges.
• PE Line Advantages:
  • Thinner diameter for the same strength compared to fluorocarbon.
  • Essential for longer casting distances.
• Avoiding Line Connections:
  • Use a Carolina rig to eliminate the need to connect PE line and fluorocarbon leader.

Recommended Lines

• YGK PE Line:
  • Highly recommended for its quality.
  • Performs much better than Sufix lines.
  • Consider using 0.6 or 0.8 PE from YGK.
• Fluorocarbon Line:
  • 1.5 Fluorocarbon can be used as a leader if necessary.

Rigs and Lures

• Carolina Rig:
  • Allows you to use PE line without needing to connect it to a fluorocarbon leader.
  • Effective for various fishing conditions.

Recommendations

Final Gear Choices

• Rod:
  • L Action Spinning Rod with multiple pieces for portability.
• Reel:
  • 2000S or 2500S spinning reel, prioritize lightweight models.
• Line:
  • 0.6 or 0.8 YGK PE line (lean towards 0.6 for more capacity).
  • Optional 1.5 Fluorocarbon leader.
• Rig:
  • Use a Carolina rig setup.

Brands and Models

Chinese Domestic Market (CDM) Rods

• Few high-quality CDM spinning rods are available.
• The best CDM spinning rod is the 翠鸟 (Kingfisher) from 钓之屋 (Fishing House).
• The 游侠 (the larger version of 翠鸟) is not as good.
  • Example: A 2500S reel on a 游侠 lacks proper rod skewness, affecting casting performance.

Shimano Rods and Reels

• Shimano spinning rods and reels are superior in performance.
• The reels have features that prevent accidental rolling after opening the bail, allowing you to stop the line with your finger for better control.

Reels I’ve Used

• Shimano Vanquish 2500S (2023 model):
  • Extremely lightweight and smooth.
  • Feels almost too slim, raising concerns about durability.
  • The infinite loop system is excellent.
• Shimano Stradic 2500S (2023 model):
  • Heavier compared to Vanquish.
  • Handle design is not as comfortable.
  • Appearance gives a sense of reliability.
• Shimano Vanford C2000SHG (2020 model):
  • Great performance but similar concerns about durability due to plastic components.
  • Feels less robust than the Stradic but is lighter.

Price Comparison (in RMB)

• 钓之屋 翠鸟 (Kingfisher): 360
• Shimano Stradic: 900
• Shimano Vanford: 1,100
• Shimano Vanquish: 2,600
• Shimano Stella: 3,800

Conclusion

Choosing the right fishing gear is crucial for an enjoyable and successful fishing experience. By considering the rod action, reel type and size, line selection, and the right rigs, you can optimize your setup for the best performance. While high-end brands like Shimano offer top-quality equipment, there are also cost-effective options available. Remember to prioritize what suits your fishing style and comfort.

Happy fishing!

After researching, I found that this problem is not unique. For instance, in a discussion on V2EX, many users encountered similar issues. The solution is to run warp-cli set-mode proxy before executing warp-cli connect to bypass the local address. Surprisingly, Cloudflare’s official documentation does not mention this crucial step, undoubtedly increasing the complexity and risk of configuration.

In the process of exploring solutions, I found that some users suggested repairing by rebuilding the instance or using VNC connection. However, since I am using AWS Lightsail, VNC is not applicable. Ultimately, I decided to try the method mentioned in this article: creating a snapshot backup of the current VPS, then creating a new instance from the snapshot, and loading a script to execute warp-cli set-mode proxy when the new instance starts.

After checking the existing instance, I found that no snapshot had been created. This discovery reminded me of the importance of regular backups. Without other options, I could only attempt a snapshot backup as guided by the aforementioned article. However, no matter what startup script command I tried, it failed to execute successfully. The execution result of AWS Lightsail’s startup script is not visible, making problem-solving more difficult.

In near desperation, I found an old snapshot dated 2022 on the snapshot page. Although this snapshot was created using old technology, and many important updates might be lost after recovery, it was my last hope. After starting the snapshot recovery process, I unexpectedly discovered through the history command that this snapshot contained all the important updates. This discovery allowed the entire recovery process to be completed smoothly.

This experience re-emphasized the importance of backups. Careful backups from the past ultimately avoided severe data loss. Furthermore, AWS’s static IP retention feature also played a crucial role. The new instance could immediately bind to the IP once the old instance released the static IP, achieving a seamless switch.

Conclusion

1. Backups are essential: Regular backups are key to ensuring stable system operations.
2. Operate with caution: Before executing critical commands, thoroughly review and understand relevant documentation and user feedback to avoid potential risks.
3. Trust your past self: Meticulous work done in the past can often prove invaluable at critical moments.

I hope this experience can serve as a reference and help for others, preventing similar issues from occurring.

Many customers have TLS enabled, which is different from the lab environment.

Curl

Curl requires a specified CA certificate, otherwise it will report an error.

curl --cacert ca.crt https://127.0.0.1:10080/status

Wget

Many containers do not have curl, so wget is used instead. Wget is better as it does not require a CA certificate.

wget --no-check-certificate http://127.0.0.1:10080/status

After moving house, there are many more devices at home that need internet access. However, I don’t want to configure a proxy on each device, so I thought of using a transparent gateway.

Transparent Gateway

After some research, I found that the easiest way is to use the premium version of Clash, although I didn’t know when Clash released a premium version. I mainly referred to this article. It’s much simpler than setting up iptables.

Network Topology

I have a 10-year-old Thinkpad x230 at home, which is perfect for this purpose. Here is a simple topology diagram.

Router1 is a fiber-optic modem with routing capabilities, Router2 is a regular router, with the gateway and DNS pointing to the Thinkpad, where Linux is running to act as a transparent gateway with Clash on top.

                                 +------------+
                                 |            |
                                 |  Internet  |
                                 |            |
                                 +-----+------+
                                       |
                                 +-----+------+
                                 |            |
                      +----------+  Router1   +-----------+
                      |          |            |           |
                      |          +------------+           |
                      |                                   |
                      |                                   |
                +-----+-----+                       +-----+------+
                |           |                       |            |
     +----------+  Router2  +----------+            |  Thinkpad  |
     |          |           |          |            |            |
     |          +-----+-----+          |            +------------+
     |                |                |
     |                |                |
     |                |                |
+----+-----+     +----+-----+    +-----+-----+
|          |     |          |    |           |
|   Mac    |     |  iPad    |    |  iPhone   |
|          |     |          |    |           |
+----------+     +----------+    +-----------+

Add DNS Section in Clash Configuration

dns:
enable: true
listen: 0.0.0.0:53
enhanced-mode: fake-ip
nameserver:
  - 114.114.114.114
fallback:
  - 8.8.8.8

Clash tun Feature Section

tun:
enable: true
stack: system # or gvisor
dns-hijack:
  - any:53
  - tcp://any:53
auto-route: true
auto-detect-interface: true

For traffic forwarding, simply edit /etc/sysctl.conf on the Thinkpad and add net.ipv4.ip_forward=1, then execute sysctl -p to apply it. After that, point the gateway and DNS of Router2 to the Thinkpad, and you’re done.

Network Protocols

Initially, I used native HTTP2 for unblocking, but it cannot proxy UDP. When only a few devices need unblocking, it doesn’t matter whether UDP is used, but with many devices at home, some of them can only use UDP. I considered socks + tls, but it didn’t feel secure and required opening odd ports like UDP 443. It felt like giving away my intentions. Eventually, I chose Trojan, which essentially mimics native HTTPS. Trojan has two versions; I used Trojan-go simply because I didn’t want to manage dependencies. Also, I’m more familiar with Go.

Trojan-go has a requirement for a genuinely accessible HTTP server, so I used the simplest Python http.server. Back in Python 2, it was called simplehttp. You can simply use python3 -m http.server 80 and optionally add --directory to specify a directory.

Additionally, Trojan-go requires the client to fill in the SNI, which means using the domain used during key application. Therefore, prerequisites like applying for the domain, applying for Let’s Encrypt certificates, and configuring crontab must all be completed. There’s a learning curve, but I had done it before, so I just skipped that part.

For the client part, you can use Clash directly, and refer to here for guidance.

Recently, both of my kids have become interested in Minecraft, and some of their peers are also playing it. We previously bought the Switch version, but its online capabilities are quite poor, and the device performance is subpar, resulting in a less-than-ideal experience. Thus, I began considering the idea of setting up our own server. Of course, you can play in multiplayer with friends, but the game ends as soon as the host goes offline, which is not as good as having a server that is always online.

For version selection, there is the NetEase version, the official Java version, and the Bedrock version. Based on my previous understanding, the NetEase version has all sorts of anti-addiction regulations, so that’s a no-go. The Java version server setup seems to rely on third-party launchers, resembling piracy to some extent. Therefore, I decided on the Bedrock version. Another reason for choosing the Bedrock version is its origins as the mobile version of Minecraft, and Microsoft later expanded support to more platforms, making it the most versatile. Additionally, its core is written in C++, which should offer better performance and resource efficiency.

Downloading the Server Software

I downloaded the Ubuntu version. While I also downloaded the Windows version and managed to run it, I’m not experienced in managing Windows Server, and the Windows version has some quirks. I won’t delve into those here.

Local Network Server

Initially, I intended to use an old Thinkpad, but couldn’t find it, and ended up using an old Macbook. After updating everything, I found that the old Macbook was still quite robust, outperforming the Thinkpad. So, I set up a virtual machine. I discovered that Ubuntu Server no longer supports direct ISO downloads, requiring Multipass to launch instead.

What is Multipass?

Developed by the Ubuntu team, it is a lightweight virtualization platform. Multipass consists of multipass and multipassd. The former provides both GUI and CLI, while the latter requires root permissions and runs in the background. It can be downloaded directly or installed via brew cask.

Multipass seems to only install Ubuntu Server, with some customization, such as automatically creating a user named “ubuntu” and key pairing. Once installed, you get a 1C 1GB virtual machine called “primary” that automatically mounts the user’s home directory.

multipassd is more of a pluggable virtual hypervisor supporting hyperkit and qemu by default, with hyperkit intended for Intel macOS and qemu for M1.

multipassd can also use Virtualbox as a backend hypervisor, which is more recommended because it offers bridging capabilities, whereas relying solely on port mapping would be cumbersome. It’s not that qemu doesn’t support bridging, but there’s an easy-to-follow Virtualbox bridging tutorial on Multipass’s official documentation.

I set up Virtualbox first. I must say, Oracle is generous here as it’s still free to use. Then, I followed the steps from the link above. It’s essential to reboot the machine after running the first step sudo multipass set local.driver=virtualbox, as the variable might not take effect immediately. Otherwise, the primary created will still use the old backend. Since there’s already a default virtual machine, I didn’t want to create another one to avoid extra resource usage. Additionally, note a few things:

1. For modifying primary configuration, I didn’t find a way to do it using Multipass, so I used vbox’s command sudo VBoxManage controlvm "primary" --cpus 2 --memory 4096 (where memory is in MB).
2. The primary mount point in Multipass can automatically unmount in some situations, resulting in errors. Therefore, Minecraft data shouldn’t be stored in the mount point permanently to avoid core dumps.

After extracting the server files, just start it in tmux/screen.

Lastly, I used Amphetamine from the App Store to ensure that the laptop continues working when closed by disabling the default sleep setting after the screen is closed. You’ll see some warnings when doing this, but just be aware of them.

Update:

For systemd startup, you can refer to this gist. The server’s console has some commands, like “stop”, that perform graceful shutdowns, so it’s necessary to rely on a screen session to create it. Additionally, the server startup has some environmental dependencies, so it’s best to write a small script for starting it, like:

#!/bin/bash

cd /home/user/bedrock-server
LD_LIBRARY_PATH=. ./bedrock_server

Using absolute paths can lead to core dumps.

Public Network Server

For public network servers, the choices are to map the internal network to the public network or purchase a cloud server. Due to security concerns, I opted to buy a cloud server. Although the official site mentioned Ubuntu support, I found that Debian servers also run without issues. The only thing to note is that the Bedrock version uses UDP, and China Unicom’s 5G network disables UDP, so without WiFi, you cannot connect.

Server Monitoring

To monitor the server’s status, monitoring is necessary, especially for cloud servers, as provider information might be insufficient. I used Grafana Cloud.

Using the free version is fine, just follow the guide to select a Linux server integration, then run the Grafana Agent installation and verification on the server to be monitored. Note to change the hostname in the Grafana Agent configuration file to differentiate between different servers, which acts as a label.

I understand that Grafana Agent is a lightweight node exporter with some Prometheus functionalities, but it can also remote write, so there’s no need to worry about the disk filling up.

One day, I impulsively turned on GitHub’s Vigilant mode.

As a result, all my commits started looking like this.

To figure out how to make them Verified, I found the following method.

Method

I actually referred to this link. However, it wasn’t quite enough, as there might be authentication-related issues on MacBooks that lead to commit errors. So, I found this solution.

In summary, to verify, you need to enter a password. The issue on a Mac is the prompt for entering the password, which needs to be replaced with pinentry-mac, which most people install via homebrew.

Moreover, this solution thoughtfully provides a way to verify:

echo "test" | gpg --clearsign

GPG Experience

1. It doesn’t replace the ssh key. After successfully setting it up, I deleted my GitHub ssh key and discovered that I couldn’t log in. Actually, it only verifies the legitimacy of commits.
2. On the local machine, in any repo, you only need to enter the password once, and that makes it a verified commit. It doesn’t affect daily use; it just adds a green check mark for verification.
3. Using the https protocol + token seems more reliable than this method, but I’m not sure if it provides a verified mark.

During the Chinese New Year holiday, I spent some time at home playing on the Switch with my child, and I ended up purchasing Minecraft (Bedrock Edition). In order to play together, I also bought the Windows version. Additionally, I wanted to test if the GitHub Actions for this blog are still working properly.

Preparation

You need two Microsoft accounts, so I dug out my wife’s long-unused Hotmail account.

Multiplayer

1. Restart the game on the Switch and log into the Microsoft account. It has to be restarted each time, and I’m not sure why.
2. Add each other as friends using the short ID.
3. Create a world on the Switch; Windows should join (it doesn’t work the other way around, and I’m not sure why).
4. Enjoy playing together!

This guide is brief and straightforward, with nothing much more to add.

Recently, I encountered a problem where a user wanted to synchronize data from PostgreSQL to TiDB (which uses the same protocol as MySQL) and wanted to know whether the data after synchronization is consistent. I hadn’t dealt with this kind of issue before, so I did a bit of research.

Typically, to verify data consistency, you compute a checksum on both sides and compare them.

TiDB (MySQL) Side

For the verification of a specific table, the following SQL is used:

SELECT bit_xor(
    CAST(crc32(
        concat_ws(',',
            col1, col2, col3, …, colN,
            concat(isnull(col1), isnull(col2), …, isnull(colN))
        )
    ) AS UNSIGNED)
)
FROM t;

Let’s look at a specific example:

DROP TABLE IF EXISTS t;
CREATE TABLE t (i INT, j INT);
INSERT INTO t VALUES (2, 3), (NULL, NULL);
SELECT bit_xor(
    CAST(crc32(
        concat_ws(',',
            i, j,
            concat(isnull(i), isnull(j))
        )
    ) AS UNSIGNED)
)
FROM t;

The result is:

+-------------------------------------------------------------------------------------------------------------------------------------------+
| bit_xor(
    CAST(crc32(
        concat_ws(',',
            i, j,
            concat(isnull(i), isnull(j))
        )
    ) AS UNSIGNED)
) |
+-------------------------------------------------------------------------------------------------------------------------------------------+
|                                                           5062371 |
+-------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

PostgreSQL Side

The goal is simply to write the same SQL as above, but PostgreSQL does not support bit_xor, crc32, isnull, nor does it have unsigned types. Therefore, the solution is relatively straightforward—relying on UDFs (User-Defined Functions).

After some research, the main missing functions can be addressed with a few custom implementations.

bit_xor:

CREATE OR REPLACE AGGREGATE bit_xor(IN v bigint) (SFUNC = int8xor, STYPE = bigint);

crc32:

CREATE OR REPLACE FUNCTION crc32(text_string text) RETURNS bigint AS $$
DECLARE
    tmp bigint;
    i int;
    j int;
    byte_length int;
    binary_string bytea;
BEGIN
    IF text_string = '' THEN
        RETURN 0;
    END IF;

    i = 0;
    tmp = 4294967295;
    byte_length = bit_length(text_string) / 8;
    binary_string = decode(replace(text_string, E'\\', E'\\\\'), 'escape');
    LOOP
        tmp = (tmp # get_byte(binary_string, i))::bigint;
        i = i + 1;
        j = 0;
        LOOP
            tmp = ((tmp >> 1) # (3988292384 * (tmp & 1)))::bigint;
            j = j + 1;
            IF j >= 8 THEN
                EXIT;
            END IF;
        END LOOP;
        IF i >= byte_length THEN
            EXIT;
        END IF;
    END LOOP;
    RETURN (tmp # 4294967295);
END
$$ IMMUTABLE LANGUAGE plpgsql;

isnull:

CREATE OR REPLACE FUNCTION isnull(anyelement) RETURNS int AS $$
BEGIN
    RETURN CAST(($1 IS NULL) AS INT);
END
$$ LANGUAGE plpgsql;

After creating the three UDFs above, let’s test the previous example. Note that UNSIGNED should be changed to BIGINT.

DROP TABLE IF EXISTS t;
CREATE TABLE t (i INT, j INT);
INSERT INTO t VALUES (2, 3), (NULL, NULL);
SELECT bit_xor(
    CAST(crc32(
        concat_ws(',',
            i, j,
            concat(isnull(i), isnull(j))
        )
    ) AS BIGINT)
)
FROM t;

The result:

 bit_xor
---------
 5062371
(1 row)

It’s exactly the same as on the TiDB (MySQL) side.

Postscript

1. I haven’t tested more extensively; this is just a simple test.
2. UDFs are indeed a great feature that greatly enhance flexibility.

Initially, I didn’t think this was something worth writing about, because CTex was previously working smoothly for everyone. However, it turns out that CTex hasn’t been updated since 2016. So, I wanted to find a replacement for Chinese LaTeX on Windows in 2021.

Configuration Method

1. Install MiKTeX (TexLive should work as well). MiKTeX can automatically download dependency packages and you can also proactively install the ctex package.
2. Install the VSCode LaTeX extension.
3. Configure the LaTeX extension. I found a powerful configuration on Zhihu, but it’s quite complex, so I simplified it a bit.

A brief explanation: the core components here are the recipes and tools. Tools are the compilation toolchain, specifying which tools to use for compilation and the options to use, without regard to order. Recipes define how to combine the above tools to generate the final document, where the order does matter. I’ve put XeLaTex first here because it’s the most compatible for compiling Chinese. If you use pdflatex to compile Chinese documents, you’re likely to encounter issues.

{
   "latex-workshop.latex.recipes": [
         {
            "name": "XeLaTeX",
            "tools": [
                "xelatex"
            ]
        },
        {
            "name": "latexmk 🔃",
            "tools": [
                "latexmk"
            ]
        },
        {
            "name": "latexmk (latexmkrc)",
            "tools": [
                "latexmk_rconly"
            ]
        },
        {
            "name": "latexmk (lualatex)",
            "tools": [
                "lualatexmk"
            ]
        },
        {
            "name": "pdflatex ➞ bibtex ➞ pdflatex × 2",
            "tools": [
                "pdflatex",
                "bibtex",
                "pdflatex",
                "pdflatex"
            ]
        },
        {
            "name": "Compile Rnw files",
            "tools": [
                "rnw2tex",
                "latexmk"
            ]
        },
        {
            "name": "Compile Jnw files",
            "tools": [
                "jnw2tex",
                "latexmk"
            ]
        },
        {
            "name": "tectonic",
            "tools": [
                "tectonic"
            ]
        }
    ],
    "latex-workshop.latex.tools": [
        {
            "name": "xelatex",
            "command": "xelatex",
            "args": [
                "-synctex=1",
                "-interaction=nonstopmode",
                "-file-line-error",
                "%DOCFILE%"
            ]
        },
        {
            "name": "latexmk",
            "command": "latexmk",
            "args": [
                "-synctex=1",
                "-interaction=nonstopmode",
                "-file-line-error",
                "-pdf",
                "-outdir=%OUTDIR%",
                "%DOC%"
            ],
            "env": {}
        },
        {
            "name": "lualatexmk",
            "command": "latexmk",
            "args": [
                "-synctex=1",
                "-interaction=nonstopmode",
                "-file-line-error",
                "-lualatex",
                "-outdir=%OUTDIR%",
                "%DOC%"
            ],
            "env": {}
        },
        {
            "name": "latexmk_rconly",
            "command": "latexmk",
            "args": [
                "%DOC%"
            ],
            "env": {}
        },
        {
            "name": "pdflatex",
            "command": "pdflatex",
            "args": [
                "-synctex=1",
                "-interaction=nonstopmode",
                "-file-line-error",
                "%DOC%"
            ],
            "env": {}
        },
        {
            "name": "bibtex",
            "command": "bibtex",
            "args": [
                "%DOCFILE%"
            ],
            "env": {}
        },
        {
            "name": "rnw2tex",
            "command": "Rscript",
            "args": [
                "-e",
                "knitr::opts_knit$set(concordance = TRUE); knitr::knit('%DOCFILE_EXT%')"
            ],
            "env": {}
        },
        {
            "name": "jnw2tex",
            "command": "julia",
            "args": [
                "-e",
                "using Weave; weave(\"%DOC_EXT%\", doctype=\"tex\")"
            ],
            "env": {}
        },
        {
            "name": "jnw2texmintex",
            "command": "julia",
            "args": [
                "-e",
                "using Weave; weave(\"%DOC_EXT%\", doctype=\"texminted\")"
            ],
            "env": {}
        },
        {
            "name": "tectonic",
            "command": "tectonic",
            "args": [
                "--synctex",
                "--keep-logs",
                "%DOC%.tex"
            ],
            "env": {}
        }
    ],
    "latex-workshop.view.pdf.viewer": "tab"
}

1. Create a folder.
2. Use VSCode to open this folder and create a .tex file with the following content:

\documentclass[UTF8]{ctexart}

\begin{document}
OK, it’s all set...
\end{document}

It should automatically start compiling. Confirm to download any required dependencies if prompted. The compiled effect is as follows:

Go Action

Click on Actions on the repository page, and then New Workflow to see recommended actions. Since this blog uses Go code, it shows the Go Action.

The rest involves following Travis’s approach to set up the workflow.

1. Check out the blog’s repo
2. Check out the publishing site repo
3. make
4. Commit the changes to the publishing site

Note that write permissions are required for the publishing site, so you need to configure a token, similar to Travis.

1. Generate a token with only repo permissions
2. Go to a particular repo and set up secrets (enter the token). Ideally, this should be set up on the publishing site, but it works when set in the blog repo. I haven’t explored why yet.

You will need two actions in total: one is GitHub’s own checkout, and the other is a third-party action called “Push directory to another repository”. There might be better options available, and I’ll explore them when I have more time.

Finally, here is my simple GitHub Action CI file:

name: CI

on:
  push:
    branches: [ master ]

jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    steps:

    - name: Set up Go 1.x
      uses: actions/setup-go@v2
      with:
        go-version: ^1.13

    - name: Check out code into the Go module directory
      uses: actions/checkout@v2

    - name: Get dependencies
      run: |
        go get -v -t -d ./...
        if [ -f Gopkg.toml ]; then
            curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
            dep ensure
        fi

    - name: Check out my other private repo
      uses: actions/checkout@v2
      with:
        repository: jackysp/jackysp.github.io
        token: ${{ secrets.UPDATE_BLOG }}
        path: public

    - name: Build
      run: make
    
    - name: Pushes to another repository
      id: public
      uses: cpina/github-action-push-to-another-repository@cp_instead_of_deleting
      env:
        API_TOKEN_GITHUB: ${{ secrets.UPDATE_BLOG }}
      with:
        source-directory: 'public'
        destination-github-username: 'jackysp'
        destination-repository-name: 'jackysp.github.io'
        user-email: your@email.com
        commit-message: See ORIGIN_COMMIT

A Simple Example

#!/usr/bin/env sysbench

require("oltp_common")

function prepare_statements()
end

function event()
    con:query("set autocommit = 1")
end

The first line require includes Sysbench’s built-in basic library; the empty prepare_statement is a callback function from oltp_common that must be present; the specific execution of a single load is implemented in the event function.

Save this script as a Lua file, for example, named set.lua, and then execute it using sysbench.

sysbench --config-file=config --threads=100 set.lua --tables=1 --table_size=1000000 run

You can use the above command. Of course, here --tables=1 and --table_size=1000000 are not useful for this load, so they are optional. --threads controls concurrency.

$ cat config
time=120
db-driver=mysql
mysql-host=172.16.5.33
mysql-port=34000
mysql-user=root
mysql-db=sbtest
report-interval=10

In the config file, parameters you don’t frequently adjust are written once to avoid having a long string of parameters in the command line. These are required fields: time represents the test duration, report-interval is used to observe real-time performance results, and the others pertain to how to connect to the database.

The running output generally looks like:

[ 10s ] thds: 100 tps: 94574.34 qps: 94574.34 (r/w/o: 0.00/0.00/94574.34) lat (ms,95%): 3.68 err/s: 0.00 reconn/s: 0.00
[ 20s ] thds: 100 tps: 77720.30 qps: 77720.30 (r/w/o: 0.00/0.00/77720.30) lat (ms,95%): 5.28 err/s: 0.00 reconn/s: 0.00
[ 30s ] thds: 100 tps: 56080.10 qps: 56080.10 (r/w/o: 0.00/0.00/56080.10) lat (ms,95%): 9.22 err/s: 0.00 reconn/s: 0.00
[ 40s ] thds: 100 tps: 93315.90 qps: 93315.90 (r/w/o: 0.00/0.00/93315.90) lat (ms,95%): 4.82 err/s: 0.00 reconn/s: 0.00
[ 50s ] thds: 100 tps: 97491.02 qps: 97491.02 (r/w/o: 0.00/0.00/97491.02) lat (ms,95%): 4.65 err/s: 0.00 reconn/s: 0.00
[ 60s ] thds: 100 tps: 94034.27 qps: 94034.27 (r/w/o: 0.00/0.00/94034.27) lat (ms,95%): 4.91 err/s: 0.00 reconn/s: 0.00
[ 70s ] thds: 100 tps: 74707.37 qps: 74707.37 (r/w/o: 0.00/0.00/74707.37) lat (ms,95%): 6.79 err/s: 0.00 reconn/s: 0.00
[ 80s ] thds: 100 tps: 89485.10 qps: 89485.10 (r/w/o: 0.00/0.00/89485.10) lat (ms,95%): 5.18 err/s: 0.00 reconn/s: 0.00
[ 90s ] thds: 100 tps: 109296.44 qps: 109296.44 (r/w/o: 0.00/0.00/109296.44) lat (ms,95%): 2.91 err/s: 0.00 reconn/s: 0.00

Finally, there will be a summary report.

SQL statistics:
    queries performed:
        read:                            0
        write:                           0
        other:                           10424012
        total:                           10424012
    transactions:                        10424012 (86855.65 per sec.)
    queries:                             10424012 (86855.65 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

Throughput:
    events/s (eps):                      86855.6517
    time elapsed:                        120.0154s
    total number of events:              10424012

Latency (ms):
         min:                                    0.09
         avg:                                    1.15
         max:                                 1527.74
         95th percentile:                        4.91
         sum:                             11994122.49

Threads fairness:
    events (avg/stddev):           104240.1200/600.21
    execution time (avg/stddev):   119.9412/0.01

Lost Connection

There are generally three reasons for a Lost Connection:

1. A timeout occurs either directly between the client and the database or at some point along the intermediate link, such as from the client to the Proxy or from the Proxy to the database.
2. A bug occurs during SQL execution, which can generally be recovered, thus preventing the TiDB server from crashing completely (panic).
3. TiDB itself crashes, often due to excessive memory use, causing an OOM (Out of Memory), or a user deliberately kills TiDB. Another possibility is an unrecovered bug, which typically appears more frequently in background threads.

Timeout

Direct Timeout

TiDB supports the MySQL-compatible wait_timeout variable, with a default value of 0, meaning no timeout is set, unlike MySQL’s default of 8 hours.

The only place it is used in the code is in getSessionVarsWaitTimeout. In the connection’s Run section, its value is set for packet IO. If the variable is non-zero, a timeout is set before each readPacket.

If the client does not send data beyond the specified time, the connection will be disconnected. At this time, a log message “read packet timeout, close this connection” will appear, along with the specific timeout duration.

Intermediate Link Timeout

Another scenario is an intermediate link timeout. A normal timeout in an intermediate link (proxy) typically returns an EOF error to the database. In older versions, at least a connection closed log would be output.

In the newer master version, product managers suggested changing this log to a debug level, so it is generally no longer output.

However, in the new version, a monitoring item called DisconnectionCounter has been added,

which records normal and abnormal disconnections as a supplement to downgraded logging.

Bugs that Are Recovered

TiDB “basically” can recover from panics caused by unknown bugs. However, if there is an array out-of-bounds, a null pointer reference, or intentional panic, it cannot guarantee correct results for the current and subsequent SQL, so terminating the current connection is a wise choice.

At this time, an error log “connection running loop panic” will appear, along with a lastSQL field that outputs the current erroneous SQL.

Panic Not Recovered

Whether it’s an unrecovered panic or a system-level OOM-induced panic, they do not leave a log in TiDB’s logs. TiDB clusters managed by deployment tools like Ansible or TiUP will automatically restart a crashed TiDB server. Consequently, the log will contain a new “Welcome” message, which might be overlooked. However, the Uptime in monitoring will show TiDB’s Uptime reset to zero, making this issue relatively easy to detect. Of course, it’s better to have accompanying alerts.

Unrecovered panic outputs are Golang’s default outputs, usually redirected to tidb_stderr.log by deployment tools. Older versions of Ansible overwrite this file every restart, but now use an append mode.

Nevertheless, it has some other drawbacks, like lacking timestamps. This makes it difficult to timestamp-match with TiDB logs. This PR implemented distinguishing tidb_stderr.log based on PID but hasn’t been coordinated with the deployment tools and is temporarily disabled.

To get this standard panic output, you can use the panicparse introduced in the previous article to parse the panic results. Typically, you can look at the topmost stack. The example in the image evidently shows an out-of-memory error, commonly referred to as OOM. To identify which SQL caused the OOM, check TiDB’s logs for resource-heavy SQL, which are usually logged with the expensive_query tag, and can be checked by grepping the logs. This will not be exemplified here.

Tracing

TiDB has supported tracing since version 2.1, but it hasn’t been widely used. I think there are two main reasons:

1. The initial version of tracing only supported the JSON format, requiring the output to be copied and pasted into a TiDB-specific web page at a special host port to view it. Although novel, the multiple steps involved prevented widespread adoption.

   

2. Another issue is that tracing provides insight only after a problem is known. If developers suspect a problem or slow execution in advance, they must proactively add events at those points. Often, unforeseen issues cannot be covered, leaving gaps.

Once the framework of tracing is in place, adding events is relatively straightforward and involves adding code like the snippet below at the desired points:

Interested individuals can add events to TiDB as needed, offering a good hands-on experience.

Eventually, tracing added format='row' and format='log' features. I personally favor format='log'.

Difference between Tracing and Explain (Analyze)

1. Tracing operates at the function level, while Explain operates at the operator level. Tracing is easier to add and more granular and does not need to be part of a plan.
2. Tracing can trace any SQL, while Explain only shows data reading parts. For example, with an Insert, Explain shows almost nothing, whereas tracing provides detailed insights from SQL parsing to the full transaction commit.

Key Functions

The definition of key functions varies from person to person, so the content of this section is subjective.

execute

The execute function is the necessary pathway for text protocol execution. It also nicely demonstrates the various processes of SQL handling.

1. ParseSQL analyzes the SQL. The final implementation is in the parser, where SQL is parsed according to the rules introduced in the second article. Note that the parsed SQL may be a single statement or multiple statements. TiDB itself supports the multi-SQL feature, allowing multiple SQL statements to be executed at once.

2. After parsing, a stmtNodes array is returned, which is processed one-by-one in the for loop below. The first step is to compile, where the core of compile is optimization, generating a plan. By following the Optimize function, you can find logic similar to logical and physical optimization found in other common databases.

   

3. The last part is execution, where executeStatement and particularly the runStmt function are key functions.

runStmt

Judging from the call graph of runStmt, this function is almost the mandatory pathway for all SQL execution. Except for point query statements using the binary protocol with automatic commit, all other statements go through this function. This function is responsible for executing SQL, excluding SQL parsing and compilation (the binary protocol does not need repeated SQL parsing, nor does SQL compilation require plan caching).

The core part of the runStmt function is as shown above. From top to bottom:

1. checkTxnAborted

   When a transaction is already corrupted and cannot be committed, the user must actively close the transaction to end the already corrupted transaction. During execution, transactions may encounter errors that cannot be handled and must be terminated. The transaction cannot be silently closed because the user may continue to execute SQL and assume it is still within the transaction. This function ensures that all subsequent SQL commands by the user are not executed and directly return an error until the user uses rollback or commit to explicitly close the transaction for normal execution.

2. Exec

   Execute the SQL and return the result set (rs).

3. IsReadOnly

   After executing a SQL, it’s necessary to determine whether it is a read-only SQL. If it is not read-only, it must be temporarily stored in the transaction’s execution history. This execution history is used when a transaction conflict or other errors require the transaction to be retried. Read-only SQL is bypassed because the retry of the transaction is done during the commit phase, and at this point, the only feedback to the client can be success or failure of the commit; reading results is meaningless.

   This section also includes StmtCommit and StmtRollback. TiDB supports MySQL-like statement commits and rollbacks—if a statement fails during a transaction, that single statement will be atomically rolled back, while other successfully executed statements will eventually commit with the transaction.

   In TiDB, the feature of statement commit is implemented with a two-layer buffer: both the transaction and the statement have their own buffers. After a statement executes successfully, the statement’s buffer is merged into the transaction buffer. If a statement fails, the statement’s buffer is discarded, thus ensuring the atomicity of statement commits. Of course, a statement commit may fail, in which case the entire transaction buffer becomes unusable, and the transaction can only be rolled back.

4. finishStmt

   Once a statement is executed, should it be committed? This depends on whether the transaction was explicitly started (i.e., with begin or start transaction) and whether autocommit is enabled. The role of finishStmt is to, post-execution, check if it should be committed based on the above conditions. It’s essentially for cleaning up and checking after each statement execution.

5. pending section

   Some SQLs in TiDB do not require a transaction (e.g., the set statement). However, before parsing, the database doesn’t know whether the statement requires a transaction. The latency of starting a transaction in TiDB is relatively high because it requires obtaining a TSO (timestamp oracle) from PD. TiDB has an optimization to asynchronously obtain a TSO, meaning a TSO is prepared regardless of whether a transaction is eventually needed. If a statement indeed doesn’t require a TSO and a transaction is not activated, remaining in a pending status, the pending transaction must be closed.

Logs

Let’s first look at a section of logs from TiDB at initial startup, divided into several parts:

[2020/08/12 16:12:07.282 +08:00] [INFO] [printer.go:42] ["Welcome to TiDB."] ["Release Version"=None] [Edition=None] ["Git Commit Hash"=None] ["Git Branch"=None] ["UTC Build Time"=None] [GoVersion=go1.15] ["Race Enabled"=false] ["Check Table Before Drop"=false] ["TiKV Min Version"=v3.0.0-60965b006877ca7234adaced7890d7b029ed1306]
[2020/08/12 16:12:07.300 +08:00] [INFO] [printer.go:56] ["loaded config"] [config="{\"host\":\"0.0.0.0\",\"advertise-address\":\"0.0.0.0\",\"port\":4000,\"cors\":\"\",\"store\":\"mocktikv\",\"path\":\"/tmp/tidb\",\"socket\":\"\",\"lease\":\"45s\",\"run-ddl\":true,\"split-table\":true,\"token-limit\":1000,\"oom-use-tmp-storage\":true,\"tmp-storage-path\":\"C:\\\\Users\\\\username\\\\AppData\\\\Local\\\\Temp\\\\tidb\\\\tmp-storage\",\"oom-action\":\"log\",\"mem-quota-query\":1073741824,\"tmp-storage-quota\":-1,\"enable-streaming\":false,\"enable-batch-dml\":false,\"lower-case-table-names\":2,\"server-version\":\"\",\"log\":{\"level\":\"info\",\"format\":\"text\",\"disable-timestamp\":null,\"enable-timestamp\":null,\"disable-error-stack\":null,\"enable-error-stack\":null,\"file\":{\"filename\":\"\",\"max-size\":300,\"max-days\":0,\"max-backups\":0},\"enable-slow-log\":true,\"slow-query-file\":\"tidb-slow.log\",\"slow-threshold\":300,\"expensive-threshold\":10000,\"query-log-max-len\":4096,\"record-plan-in-slow-log\":1},\"security\":{\"skip-grant-table\":false,\"ssl-ca\":\"\",\"ssl-cert\":\"\",\"ssl-key\":\"\",\"require-secure-transport\":false,\"cluster-ssl-ca\":\"\",\"cluster-ssl-cert\":\"\",\"cluster-ssl-key\":\"\",\"cluster-verify-cn\":null},\"status\":{\"status-host\":\"0.0.0.0\",\"metrics-addr\":\"\",\"status-port\":10080,\"metrics-interval\":15,\"report-status\":true,\"record-db-qps\":false},\"performance\":{\"max-procs\":0,\"max-memory\":0,\"stats-lease\":\"3s\",\"stmt-count-limit\":5000,\"feedback-probability\":0.05,\"query-feedback-limit\":1024,\"pseudo-estimate-ratio\":0.8,\"force-priority\":\"NO_PRIORITY\",\"bind-info-lease\":\"3s\",\"txn-total-size-limit\":104857600,\"tcp-keep-alive\":true,\"cross-join\":true,\"run-auto-analyze\":true,\"agg-push-down-join\":false,\"committer-concurrency\":16,\"max-txn-ttl\":600000},\"prepared-plan-cache\":{\"enabled\":false,\"capacity\":100,\"memory-guard-ratio\":0.1},\"opentracing\":{\"enable\":false,\"rpc-metrics\":false,\"sampler\":{\"type\":\"const\",\"param\":1,\"sampling-server-url\":\"\",\"max-operations\":0,\"sampling-refresh-interval\":0},\"reporter\":{\"queue-size\":0,\"buffer-flush-interval\":0,\"log-spans\":false,\"local-agent-host-port\":\"\"}},\"proxy-protocol\":{\"networks\":\"\",\"header-timeout\":5},\"tikv-client\":{\"grpc-connection-count\":4,\"grpc-keepalive-time\":10,\"grpc-keepalive-timeout\":3,\"commit-timeout\":\"41s\",\"max-batch-size\":128,\"overload-threshold\":200,\"max-batch-wait-time\":0,\"batch-wait-size\":8,\"enable-chunk-rpc\":true,\"region-cache-ttl\":600,\"store-limit\":0,\"store-liveness-timeout\":\"120s\",\"copr-cache\":{\"enable\":false,\"capacity-mb\":1000,\"admission-max-result-mb\":10,\"admission-min-process-ms\":5}},\"binlog\":{\"enable\":false,\"ignore-error\":false,\"write-timeout\":\"15s\",\"binlog-socket\":\"\",\"strategy\":\"range\"},\"compatible-kill-query\":false,\"plugin\":{\"dir\":\"\",\"load\":\"\"},\"pessimistic-txn\":{\"enable\":true,\"max-retry-count\":256},\"check-mb4-value-in-utf8\":true,\"max-index-length\":3072,\"alter-primary-key\":false,\"treat-old-version-utf8-as-utf8mb4\":true,\"enable-table-lock\":false,\"delay-clean-table-lock\":0,\"split-region-max-num\":1000,\"stmt-summary\":{\"enable\":true,\"enable-internal-query\":false,\"max-stmt-count\":200,\"max-sql-length\":4096,\"refresh-interval\":1800,\"history-size\":24},\"repair-mode\":false,\"repair-table-list\":[],\"isolation-read\":{\"engines\":[\"tikv\",\"tiflash\",\"tidb\"]},\"max-server-connections\":0,\"new_collations_enabled_on_first_bootstrap\":false,\"experimental\":{\"allow-auto-random\":false,\"allow-expression-index\":false}}"]

1. Mandatory startup outputs: “Welcome to TiDB,” git hash, Golang version, etc.
2. Actually loaded configuration (this section is somewhat difficult to read)

The remainder are some routine startup logs. The process can be referenced from the main function section introduced in the first article, mainly outputting the initial system table creation process.

[2020/08/12 16:12:07.300 +08:00] [INFO] [main.go:341] ["disable Prometheus push client"]
[2020/08/12 16:12:07.300 +08:00] [INFO] [store.go:68] ["new store"] [path=mocktikv:///tmp/tidb]
[2020/08/12 16:12:07.300 +08:00] [INFO] [systime_mon.go:25] ["start system time monitor"]
[2020/08/12 16:12:07.310 +08:00] [INFO] [store.go:74] ["new store with retry success"]
[2020/08/12 16:12:07.310 +08:00] [INFO] [tidb.go:71] ["new domain"] [store=8d19232e-a273-4e31-ba9b-a3467998345c] ["ddl lease"=45s] ["stats lease"=3s]
[2020/08/12 16:12:07.315 +08:00] [INFO] [ddl.go:321] ["[ddl] start DDL"] [ID=0e1bd28e-03ed-4900-bf71-f58b3b9d954a] [runWorker=true]
[2020/08/12 16:12:07.315 +08:00] [INFO] [ddl.go:309] ["[ddl] start delRangeManager OK"] ["is a emulator"=true]
[2020/08/12 16:12:07.315 +08:00] [INFO] [ddl_worker.go:130] ["[ddl] start DDL worker"] [worker="worker 1, tp general"]
[2020/08/12 16:12:07.315 +08:00] [INFO] [ddl_worker.go:130] ["[ddl] start DDL worker"] [worker="worker 2, tp add index"]
[2020/08/12 16:12:07.315 +08:00] [INFO] [delete_range.go:133] ["[ddl] start delRange emulator"]
[2020/08/12 16:12:07.317 +08:00] [INFO] [domain.go:144] ["full load InfoSchema success"] [usedSchemaVersion=0] [neededSchemaVersion=0] ["start time"=2.0015ms]
[2020/08/12 16:12:07.317 +08:00] [INFO] [domain.go:368] ["full load and reset schema validator"]
[2020/08/12 16:12:07.317 +08:00] [INFO] [tidb.go:199] ["rollbackTxn for ddl/autocommit failed"]

Because DDL logs are very numerous, the TiDB logs basically record each step of the DDL execution, so I’ve truncated this part of the log here. However, the basic outline can be sorted out. Firstly, the DDL execution is initiated from ddl_api, at this time recording ["CRUCIAL OPERATION"] style logs. DDL is a crucial operation, so it belongs to CRUCIAL type logs. Then, we can see a series of logs with the ddl keyword linked together, such as [ddl] add DDL jobs, [ddl] start DDL job, [ddl] run DDL job, [ddl] finish DDL job, and [ddl] DDL job is finished. These represent the process from when the DDL owner acquires a job to its final execution completion. Moreover, they have a unique job ID, which can be used to link a DDL in the log with something like jobs="ID:2.

[2020/08/12 16:12:07.518 +08:00] [INFO] [server.go:235] ["server is running MySQL protocol"] [addr=0.0.0.0:4000]
[2020/08/12 16:12:07.518 +08:00] [INFO] [http_status.go:80] ["for status and metrics report"] ["listening on addr"=0.0.0.0:10080]
[2020/08/12 16:12:07.520 +08:00] [INFO] [domain.go:1015] ["init stats info time"] ["take time"=3.0126ms]
[2020/08/12 16:15:41.482 +08:00] [INFO] [server.go:388] ["new connection"] [conn=1] [remoteAddr=127.0.0.1:64888]
[2020/08/12 21:03:19.954 +08:00] [INFO] [server.go:391] ["connection closed"] [conn=1]

Thereafter, the appearance of server is running MySQL protocol means that TiDB can provide services externally. Later, there are logs corresponding to the creation and closing of each connection, namely new connection and connection closed. Of course, they also have their corresponding connection ID, which is unique for a TiDB. You can use the keyword conn=1 in the log to contextually link them together.

Stack Logs

Most of TiDB’s SQL errors (except for duplicate entry and syntax errors) will output the complete stack information. Due to the requirements of unified log format, the stack now looks very unsightly…

For this stack trace, I believe no one really enjoys reading it. Therefore, we need to paste it into Vim and execute %s/\\n/\r/g and %s/\\t/ /g to turn it into a Golang-style stack.

When you see which module it’s stuck in, like the plan part here, you can find the corresponding colleague for support.

However, there is a more user-friendly tool for dealing with Golang’s lengthy stack called panicparse. To install it, simply run go get github.com/maruel/panicparse/v2/cmd/pp. The effect is as follows:

Whether it’s TiDB running goroutines or panic outputs, it can be parsed using this. It has several features:

1. It can display active and inactive goroutines.
2. It can show the relationships between goroutines.
3. Keyword highlighting.
4. Supports Windows.

The latest 2.0.0 version supports race detector and HTML formatted output.

This concludes the introduction to the analysis of key functions and logs (startup, DDL, connection, error stack).

System Variables

The system variable names in TiDB are defined in tidb_vars.go. This file also includes some default values for variables, but the place where they are actually assembled is defaultSysVars.

This large struct array defines the scope, variable names, and default values for all variables in TiDB. Besides TiDB’s own system variables, it also includes compatibility with MySQL’s system variables.

Scope

In TiDB, there are three types of variable scopes literally:

They are ScopeNone, ScopeGlobal, and ScopeSession. They represent:

• ScopeNone: Read-only variables
• ScopeGlobal: Global variables
• ScopeSession: Session variables

The actual effect of these scopes is that when you use SQL to read or write them, you need to use the corresponding syntax. If the SQL fails, the SQL operation does not take effect. If the SQL executes successfully, it merely means the setting is complete, but it does not mean that it takes effect according to the corresponding scope.

Let’s use the method mentioned in the first article to start a single-node TiDB for demonstration:

ScopeNone

Take performance_schema_max_mutex_classes as an example,

MySQL  127.0.0.1:4000  SQL > select @@performance_schema_max_mutex_classes;
+----------------------------------------+
| @@performance_schema_max_mutex_classes |
+----------------------------------------+
| 200                                    |
+----------------------------------------+
1 row in set (0.0002 sec)
MySQL  127.0.0.1:4000  SQL > select @@global.performance_schema_max_mutex_classes;
+-----------------------------------------------+
| @@global.performance_schema_max_mutex_classes |
+-----------------------------------------------+
| 200                                           |
+-----------------------------------------------+
1 row in set (0.0004 sec)
MySQL  127.0.0.1:4000  SQL > select @@session.performance_schema_max_mutex_classes;
ERROR: 1238 (HY000): Variable 'performance_schema_max_mutex_classes' is a GLOBAL variable

As you can see, the scope of ScopeNone can be read as a global variable,

MySQL  127.0.0.1:4000  SQL > set global performance_schema_max_mutex_classes = 1;
ERROR: 1105 (HY000): Variable 'performance_schema_max_mutex_classes' is a read-only variable
MySQL  127.0.0.1:4000  SQL > set performance_schema_max_mutex_classes = 1;
ERROR: 1105 (HY000): Variable 'performance_schema_max_mutex_classes' is a read-only variable
MySQL  127.0.0.1:4000  SQL > set session performance_schema_max_mutex_classes = 1;
ERROR: 1105 (HY000): Variable 'performance_schema_max_mutex_classes' is a read-only variable

But it cannot be set in any way.

To trace the usage of ScopeNone, you will see

In setSysVariable, when this type of scope variable is encountered, an error is directly returned.

In ValidateGetSystemVar, it is handled as a global variable. From a theoretical standpoint, these ScopeNone variables are essentially a single copy in the code. Once TiDB is started, they exist in memory as read-only and are not actually stored in TiKV.

ScopeGlobal

Using gtid_mode as an example,

MySQL  127.0.0.1:4000  SQL > select @@gtid_mode;
+-------------+
| @@gtid_mode |
+-------------+
| OFF         |
+-------------+
1 row in set (0.0003 sec)
MySQL  127.0.0.1:4000  SQL > select @@global.gtid_mode;
+--------------------+
| @@global.gtid_mode |
+--------------------+
| OFF                |
+--------------------+
1 row in set (0.0006 sec)
MySQL  127.0.0.1:4000  SQL > select @@session.gtid_mode;
ERROR: 1238 (HY000): Variable 'gtid_mode' is a GLOBAL variable

It works the same way as MySQL global variable reading,

MySQL  127.0.0.1:4000  SQL > set gtid_mode=on;
ERROR: 1105 (HY000): Variable 'gtid_mode' is a GLOBAL variable and should be set with SET GLOBAL
MySQL  127.0.0.1:4000  SQL > set session gtid_mode=on;
ERROR: 1105 (HY000): Variable 'gtid_mode' is a GLOBAL variable and should be set with SET GLOBAL
MySQL  127.0.0.1:4000  SQL > set global gtid_mode=on;
Query OK, 0 rows affected (0.0029 sec)
MySQL  127.0.0.1:4000  SQL > select @@global.gtid_mode;
+--------------------+
| @@global.gtid_mode |
+--------------------+
| ON                 |
+--------------------+
1 row in set (0.0005 sec)
MySQL  127.0.0.1:4000  SQL > select @@gtid_mode;
+-------------+
| @@gtid_mode |
+-------------+
| ON          |
+-------------+
1 row in set (0.0006 sec)

The setting method is also compatible with MySQL. At this point, we can shut down the single-instance TiDB and restart it,

MySQL  127.0.0.1:4000  SQL > select @@gtid_mode;
+-------------+
| @@gtid_mode |
+-------------+
| ON          |
+-------------+
1 row in set (0.0003 sec)

And you can see that the result can still be read, meaning that this setting was persisted to the storage engine. Looking closely at the code, you can see:

The actual implementation involves executing an internal replace statement to update the original value. This constitutes a complete transaction involving acquiring two TSOs and committing the entire process, making it slower compared to setting session variables.

ScopeSession

Using rand_seed2 as an example,

MySQL  127.0.0.1:4000  SQL > select @@rand_seed2;
+--------------+
| @@rand_seed2 |
+--------------+
|              |
+--------------+
1 row in set (0.0005 sec)
MySQL  127.0.0.1:4000  SQL > select @@session.rand_seed2;
+----------------------+
| @@session.rand_seed2 |
+----------------------+
|                      |
+----------------------+
1 row in set (0.0003 sec)
MySQL  127.0.0.1:4000  SQL > select @@global.rand_seed2;
ERROR: 1238 (HY000): Variable 'rand_seed2' is a SESSION variable

Reading is compatible with MySQL.

MySQL  127.0.0.1:4000  SQL > set rand_seed2='abc';
Query OK, 0 rows affected (0.0006 sec)
MySQL  127.0.0.1:4000  SQL > set session rand_seed2='bcd';
Query OK, 0 rows affected (0.0004 sec)
MySQL  127.0.0.1:4000  SQL > set global rand_seed2='cde';
ERROR: 1105 (HY000): Variable 'rand_seed2' is a SESSION variable and can't be used with SET GLOBAL
MySQL  127.0.0.1:4000  SQL > select @@rand_seed2;
+--------------+
| @@rand_seed2 |
+--------------+
| bcd          |
+--------------+

The setting is also compatible with MySQL. It can be simply observed that this operation only changes the session’s memory. The actual place where it finally takes effect is SetSystemVar.

There are some tricks here.

Actual Scope of Variables

The previous section covered setting session variables. Based on MySQL’s variable rules, setting a global variable does not affect the current session. Only newly created sessions will load global variables for session variable assignment. Ultimately, the active session variable take effect. Global variables without session properties still have unique characteristics, and this chapter will cover:

1. Activation of session variables
2. Activation of pure global variables
3. Mechanism of global variable function

These three aspects.

Activation of Session Variables

Whether a session variable is also a global variable only affects whether it needs to load global variable data from the storage engine when the session starts. The default value in the code is the initial value for eternity if no loading is required.

The actual range where a variable operates can only be observed in SetSystemVar.

For example, in this part, s.MemQuotaNestedLoopApply = tidbOptInt64(val, DefTiDBMemQuotaNestedLoopApply) changes the s structure, effectively changing the current session,

Whereas atomic.StoreUint32(&ProcessGeneralLog, uint32(tidbOptPositiveInt32(val, DefTiDBGeneralLog))) changes the value of the global variable ProcessGeneralLog, thereby affecting the entire TiDB instance when set tidb_general_log = 1 is executed.

Activation of Pure Global Variables

Pure global variables in current TiDB are used for background threads like DDL, statistics, etc.

Because only one TiDB server requires them, session-level variables hold no meaning for these.

Mechanism of Global Variable Function

Global variables in TiDB don’t activate immediately after setting. A connection fetches the latest global system variables from TiKV to assign them to the current session the first time it’s established. Concurrent connection creation results in frequent access to the TiKV node holding a few global variables. Thus, TiDB caches global variables, updating them every two seconds, significantly reducing TiKV load. The problem arises that after setting a global variable, a brief wait is necessary before creating a new connection, ensuring new connections will read the latest global variable. This is one of the few eventual consistency locations within TiDB.

For specific details, see this commentary in loadCommonGlobalVariablesIfNeeded.

Metrics

Compared to system variables, Metrics in TiDB are simpler, or straightforward. The most common Metrics are Histogram and Counter, the former is used to record actual values for an operation and the latter records occurrences of fixed events. All Metrics in TiDB are uniformly located here, with AlertManager and Grafana scripts also available separately under alertmanager and grafana.

There are many Metrics, and from a beginner’s perspective, it’s best to focus on a specific monitoring example. Let’s take the TPS (transactions per second) panel as an example.

Click EDIT and you will see the monitoring formula is:

The tidb_session_transaction_duration_seconds is the name of this specific metric. Since it is a histogram, it can actually be expressed as three types of values: sum, count, and bucket, which represent the total sum of values, the count (which functions the same as a counter), and the distribution by bucket, respectively.

In this context, [1m] represents a time window of 1 minute, indicating the precision of the measurement. The rate function calculates the slope, essentially the rate of change, indicating how many times something occurs per second. The sum function is used for aggregation, and when combined with by (type, txn_mode), it represents aggregation by the dimensions of type and txn_mode.

The Legend below displays the dimensions above using {{type}}-{{txn_mode}}. When surrounded by {{}}, it can display the actual label names.

In this representation, the final states of transactions are commit, abort, and rollback. A commit indicates a successful user-initiated transaction, rollback indicates a user-initiated rollback (which cannot fail), and abort indicates a user-initiated commit that failed.

The second label, txn_mode, refers to two modes: optimistic and pessimistic transactions. There’s nothing further to explain about these modes.

Corresponding to the code:

This segment of code shows that tidb_session_transaction_duration_seconds is divided into several parts, including namespace and subsystem. Generally, to find a variable in a formula like tidb_session_transaction_duration_seconds_count within TiDB code, you need to remove the first two words and the last word.

From this code snippet, you can see it’s a histogram, specifically a HistogramVec, which is an array of histograms because it records data with several different labels. The labels LblTxnMode and LblType are these two labels.

Checking the references, there is a place for registration, which is in the main function we discussed in the first article, where metrics are registered.

Other references show how metrics are instantiated. Why do we do this? Mainly because as the number of labels increases, the performance of metrics becomes poorer, which is related to Prometheus’s implementation. We had no choice but to create many instantiated global variables.

Taking the implementation of Rollback as an example, its essence is to record the actual execution time of a transaction when Rollback is truly executed. Since it’s a histogram, it is also used as a counter in this instance.

How to Check the Support Level of a Syntax

There are usually two methods:

1. Check through the parser repo
2. Directly check within the TiDB repo

Both of these methods require the environment setup from the previous article. If you haven’t tried that yet, give it a go.

Preparation

1. Install GoYacc Support

   

   The GoYacc Support plugin is a creation by a colleague at our company, a third-party plugin officially accepted by JetBrains, a well-regarded product. It includes syntax highlighting and intelligence, which is great!

2. Download parser repo

   If you’re checking syntax directly from the parser, you need to download it manually. If you’re navigating from TiDB, IDEA will automatically download the code, so no extra steps are needed.

Check via parser repo

Open the parser using IDEA, switch to the branch you need, and locate the parser.y file. However, it is more recommended to check from within TiDB.

Check via TiDB repo

1. Open the TiDB project with IDEA and switch to the required branch

   

2. Find the parser.y file; make sure to select the broadest search scope

   

   Alternatively, you can find it in the file list,

   

   

Let’s take checking the SHOW ENGINES SQL statement as an example.

The entry point for the entire statement parsing is Start. Below it is the StatementList, followed by Statement. Under the large list of Statements, you can find ShowStmt.

However, ShowStmt is actually quite complex. Another way is to directly search for ShowEngines within parser.y, since naming follows Golang conventions, with camel case and capitalized letters for public exposure. Naturally, if familiar with the code, you’d know ShowEngines is under ShowTargetFilterable. Its first branch is ShowEngines.

What is the level of support for SHOW ENGINES?

You can look at how ast.ShowEngines is processed. Here, you can’t just jump to it; you need to copy and search.

You only need to see how it’s processed under TiDB, and you can skip test files.

One is the actual implementation,

The other is the build schema, which you can ignore for now,

Entering fetchShowEngines, you can see the specific implementation is simple, running an internal SQL to read a system table.

Checking SHOW ENGINES ends here. You can see that it’s fully supported.

Which statements only have syntax support?

Taking the temporary table creation syntax as an example, find its position in the parser.y file.

It’s an option.

You can see that if the temporary table option is specified, it simply returns true with an attached warning, stating that the table is still treated as a regular table. Previously, the parser had a lot of operations that only returned without doing anything, not even a warning, but these are now rare.

Advantages of Querying via TiDB repo

You can see that checking via the TiDB repo allows you to find the parser’s detailed hash using IDEA. If you check directly via the parser, you need to first look up the parser’s hash in TiDB’s go.mod, then check out to the corresponding hash in the parser. If you need to check specific implementations, you have to go back to TiDB, making back-and-forth checks less convenient compared to looking within a single project. The only advantage is the ease of blaming commit history.

Viewing and Modifying Default Configuration

The default configurations can be easily viewed in TiDB, specifically the variable defaultConf. The configurations listed here are the actual default settings.

Taking the first Host configuration as an example, it has a mapping to toml and json files.

This essentially shows how it’s written in a toml file. The DefHost following it is the specific default value.

Something important to note is that configurations have a hierarchical relationship. For example, the log-related configuration in the configuration file is:

In the code, it is represented as:

This denotes a configuration called “level” under the log configuration.

What if you want to add more levels? For instance, the most complex configuration for CopCache adds another layer under tikv-client called copr-cache.

Since toml files do not support multi-level nesting, this leads to the most complex configuration writing in TiDB.

To use non-default configurations with the TiDB started through IDEA as mentioned above, the simplest way is to modify this defaultConf.

Summary

From this, you can see that checking whether a statement is supported, and whether it’s just syntax support or has a specific implementation, can be achieved with the described methods. You also learned how to view and modify default configurations, allowing you to conduct some verifications yourself. In the next article, I plan to introduce TiDB’s system variables.

There are many articles on reading the source code of TiDB, often referred to as the “Twenty-Four Chapters Scriptures”. However, these introductions typically proceed from a macro to a micro perspective. This series attempts to introduce how to read TiDB’s source code from an easier angle. The goals we aim to achieve are:

1. Enable readers to start reading TiDB’s code themselves, rather than understanding it passively through pre-written articles.
2. Provide some common examples of looking into the details of the code, such as examining the scope of a variable.

After all, teaching people to fish is better than giving them fish. While the code changes often, the methods remain mostly unchanged.

Why choose TiDB to read?

1. I am not familiar with TiKV or PD.

2. TiDB is the entry point directly interacting with users and is also the most likely to be questioned.

3. TiDB can run independently and be debugged. If you want to run some SQL after reading the code to verify your understanding, it can be easily done.

Preparations

1. A development machine

   TiDB is a pure Golang project. It can be conveniently developed on Linux, MacOS, and even Windows. My environment is Windows 10.

2. A copy of the TiDB source code, available for download at the official repo.

3. Golang environment, following the official guide is straightforward.

4. Goland or IntelliJ IDEA + Golang plugin

   I personally feel there’s no difference between the two. Why not recommend VSCode + Golang plugin? Mainly because I’m used to the JetBrains suite, and indeed commercial software tends to be higher quality than community software. For long-term use, it’s recommended to pay for it. Students can use it for free, but need to renew the license every year.

Environment Setup

1. After installing the Golang environment, remember to set the GOPATH.

2. The TiDB code doesn’t need to be developed under the GOPATH, so you can place it anywhere. I usually create a directory called work and throw various codes in there.

3. Open Goland/IDEA. I use IDEA because I often look at code in other languages.

4. Open with IDEA, select the tidb directory.

   

5. At this point, IDEA typically prompts you to set up GOROOT and enable Go Modules. Follow the recommendations.

The environment setup is now complete.

Entry Points

At the beginning, someone advised me to start with the session package. However, after some experience, I personally feel there are two better entry points: the main function and the dispatch function.

main Function

The main function of TiDB can be seen at link. You can roughly go through what happens when starting a tidb-server from top to bottom.

From top to bottom:

• Parse flags

• Output version information and exit

• Register store and monitoring

• Configuration file check

• Initialize temporary folders, etc.

• Set global variables, CPU affinity, log, trace, print server information, set binlog, set monitoring

• Create store and domain

  The createStoreAndDomain method is important, as critical background threads are created here.

• Create server and register stop signal function

• Start the server

  Within runServer, the srv.Run() actually brings up the tidb-server. In the Run() function here, the server continuously listens to network requests, creating a new connection for each new request and using a new goroutine to serve it continually.

• After this, cleanup work is done when the server needs to stop, ultimately writing out the logs.

Thus, the entire main function process ends. Through the main function, you can see the complete lifecycle of a server from creation to destruction.

Additionally, with IDEA, you can easily start and debug TiDB. Click on this triangle symbol as shown in the image below:

A pop-up with options to run and debug the main function will appear. Essentially, this starts a TiDB with default configurations. TiDB defaults to using mocktikv as the storage engine, so it can be started on a single machine for various testing and validation.

As for how to modify the configuration for starting and debugging, this will be introduced in subsequent articles in the series.

dispatch Function

From here, we can proceed further to another suitable entry point function, dispatch.

The dispatch function has several characteristics:

1. Requests coming from clients only enter the dispatch function, meaning from this point onward, user requests are executed. If you set breakpoints here, you can conveniently filter out SQL executed by internal threads.

2. From here, various requests are dispatched into different processing logic, ensuring you don’t miss any user requests. It avoids situations like spending significant time reading text protocol code only to find out the user is actually using a binary protocol.

3. dispatch itself is located at a very early stage, meaning its parameters mostly come directly from the client’s initial information. If it’s a text protocol, directly reading parameters can parse out the SQL text.

At the start, dispatch primarily focuses on obtaining tokens corresponding to the token-limit parameter. Requests that can’t get a token won’t execute, which explains why you can create many connections but only 1000 SQL executions are allowed simultaneously by default.

Next, we enter the most crucial switch case:

These commands are MySQL protocol commands, so it’s apparent from here exactly what TiDB implements. For comparison, you can refer to this link (this link is only for the text protocol). For full details, see the figure below:

Within dispatch, the most important are mysql.ComQuery, as well as the trio mysql.ComStmtPrepare, mysql.ComStmtExecute, and mysql.ComStmtClose. The latter trio is more frequently used in actual production, hence even more important. In contrast, mysql.ComQuery is generally used only for some simple tests and validations.

Since dispatch is the entry point for interfacing with clients, it can conveniently tally how many requests the database has handled. The so-called QPS derived from monitoring statistics is essentially the number of times this function executes per second. Here arises an issue: in cases like multi-query requests, such as select 1; select 1; select 1;, multiple statements sent together are regarded as a single request by dispatch, but as multiple by clients. While using the binary protocol, some clients prepare a statement, then execute, and finally close it. Seemingly equivalent to executing a single SQL from the client’s perspective, the database actually completes three requests.

In summary, users’ perceived QPS may not necessarily align with the number of dispatch function calls. In later versions, the QPS panel in TiDB’s monitoring was changed to CPS, which stands for Commands Per Second, representing the number of commands executed per second.

Looking at the callers of dispatch can also reveal information that helps explain some frequently asked questions:

1. An EOF error in dispatch typically means the client has actively disconnected, so there’s no need to maintain the database connection, and it is severed.

2. In case of an undetermined error (indicating a transaction’s commit is uncertain—whether it has succeeded or failed needs manual intervention for verification), manual intervention is required immediately, and the connection will be closed.

3. If writing binlog fails and ignore-error = false, previously the tidb-server process wouldn’t exit but couldn’t provide services. Now, the tidb-server will exit directly.

4. For all other dispatch errors, the connection will not be closed, allowing service to continue, but the failure information will be logged as “command dispatched failed”, which is arguably one of the most critical logs for TiDB.

Conclusion

This concludes the introduction from setting up the environment to finding a reasonable entry point to start reading code. Subsequent posts in the series will cover aspects such as configuration (adjustments, default values), variables (default values, scope, actual range, activation), supported syntax, etc. Stay tuned.

I had to install Docker on Windows to reproduce a bug.

Process

1. Using Windows 10 as an example, if you have the Home Basic version, you’ll need to pay to upgrade to the Pro version because you need to enable Hyper-V and Container features, which costs about 800 RMB.
2. Install everything with the default settings, and do not switch to Windows Containers, since most images are still under Linux. If you do switch, you can restore it after starting up.
3. If you encounter permission issues with shared folders, follow the instructions at link. However, this might not solve the problem, and you might encounter a sharing failure. In that case, go to the settings, troubleshoot, and reset to factory defaults. After resetting, ensure the shared folders are selected.
4. When you encounter errors during use, just try a few more times. It might work; if not, reset it.

Impressions

Initially, there were no issues using Docker on Linux; Docker itself was simple back then. Later, using it on Mac brought changes, including a user interface, various colors, and numerous bugs. Right from the start, I encountered bugs. Docker did not support Windows a long time ago, and given the various bugs on Mac, I didn’t have high expectations. The results were still quite surprising. In summary, here are a few points:

1. It’s easier to use.
2. There are more bugs. Do not expect much, and be prepared to reset at any time. Fortunately, resetting offers a shortcut, making it a pretty usable tool.
3. It’s incredibly slow.

At this point, I have no optimism for Docker, Kubernetes, or similar technologies. Done~

I tried this, but it didn’t work at all on my Thinkpad X230. I then tried changing some other options in the aforementioned file, but none worked, and surprisingly, Ubuntu even reported errors.

So, I reinstalled the more preferred Debian. Tried again, and it still didn’t work. Finally, I found a more brutal method.

systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target

This directly points these units to /dev/null…

To revert, simply use:

systemctl unmask sleep.target suspend.target hibernate.target hybrid-sleep.target

It’s simple and effective.

Update:

If you only mask them, the CPU usage of systemd-logind will be very high because it continuously attempts to sleep. Therefore, you also need to change HandleLidSwitch and others to ignore. As follows:

HandleSuspendKey=ignore
HandleHibernateKey=ignore
HandleLidSwitch=ignore
HandleLidSwitchExternalPower=ignore
HandleLidSwitchDocked=ignore

Then, execute systemctl restart systemd-logind. For more details, refer to this: SystemD-LoginD High CPU Usage.

The purpose of dynamic updates is pretty simple. As a long-term user of China Unicom’s broadband, although Unicom provides a public address, it is essentially a dynamic address, not a fixed one. If you want to access your home devices using an IP address from outside, you need to use dynamic DNS (DDNS).

Many routers come with built-in DDNS functionality, but they mostly wrap the interfaces of the commonly used service providers. These providers generally have a few characteristics: 1. Blocked by the Great Firewall; 2. Not cheap; 3. Domestic providers may have security issues; 4. The company might have gone out of business. Rather than relying on these unreliable services, it’s better to write your own script for updating.

Thus, the scripting approach comes into play. Initially, I planned to use Cloudflare, as it is the recommended way. Later, I discovered that my domain provider Namesilo offers an API to update DNS. However, it returns data in XML format, and I wasn’t sure how to parse this with shell scripts.

Then, I thought of using Go. Since this DDNS client would likely be deployed on a router-like device, languages like Python or Java require a runtime environment, and C might need some dynamic libraries to run, which I wasn’t sure how to handle. The fact that Go doesn’t require dynamic libraries was a significant advantage. So, I handwrote a tool called und.

1. First, create a DNS record in Namesilo.
2. Obtain the binary of und suitable for your platform. I only provide binaries for arm64|linux and amd64|three mainstream OS in this case. For other platforms, you’ll need to compile it yourself. You can refer to the Makefile.
3. Generate an API key from Namesilo, then start und according to its usage documentation. You might need to run it in the background, so use nohup.

GitHub Features Experience

GitHub Actions

It feels like a replacement for chaotic third-party CI services. I directly chose the Go option for und. By default, it simply runs go build -v . in Ubuntu.

Release

I used this feature when releasing TiDB before, but didn’t remember to upload/automatically generate binaries. Since TiDB is not something that can run completely with just one component, releasing a single binary doesn’t make much sense.

This time, the experience led me to believe:

1. When releasing, tagging is best done directly using the release feature.
2. After the release, since you can edit it, it’s a good time to make each binary and upload them. Based on the Makefile setup, you can generate a version. This is quite an important feature.

These two steps are quite convenient.

C:\Users\name\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\`  

Start-Process -FilePath "C:\Users\name\bin\gost-windows-amd64.exe" -ArgumentList "-L=", "-F=" -RedirectStandardOutput "C:\Users\name\bin\gost-windows-amd64.log" -RedirectStandardError "C:\Users\name\bin\gost-windows-amd64.err" -WindowStyle Hidden

Note: Start-Process seems to perform a fork-like action, and by default, it opens a new PowerShell window to execute. That’s why -WindowStyle Hidden is added at the end. You can’t use -NoNewWindow here because it only prevents the creation of a new window for executing Start-Process, but the old window will not exit.
Note 2: After the old window exits, the forked process seems to become an orphan and is managed elsewhere, so permissions, such as network connection permissions, might need to be requested again.

One day, I came across an article by Chen Hao on Twitter. Having benefited from several of his blog posts, I instinctively felt it was reliable, so I read it and decided to write this practical guide.

Why Use an HTTPS Proxy

In the guide, it’s clearly explained why, plus my own experiences of several shadowsocks being banned, I felt it was necessary to switch to a more secure proxy method.

How to Deploy an HTTPS Proxy

gost

gost is the tool most recommended in the guide. At first, I misunderstood it as a method similar to kcptun, still relying on shadowsocks. In fact, gost implements multiple proxy types, meaning you don’t need other proxies if you have it. I never liked the method of continuously wrapping to accelerate/obfuscate shadowsocks, always feeling that longer pathways bring more problems.

Steps

• Directly download the latest release from the gost repo. Although I have a Golang environment both locally and on the VPS, downloading directly is the easiest. I downloaded version 2.9.0 here.

• Following certbot on a bare VPS doesn’t work… it requires:

  1. Starting an nginx server, as referenced in this guide. Of course, this requires having a domain name pointing an A record to the VPS.
  2. Verifying access through the domain.
  3. Stopping nginx.
  4. Using certbot’s –standalone mode, which will generate the certificates upon success.

• Here, I didn’t use Docker for deployment but used systemd instead, directly creating a systemd unit similar to kcptun. The difference is, because the certificate needs updating, the unit requires a reload method. This tutorial teaches a lot about using systemd, and the author’s article quality is also high, highly recommended for subscription.

  1. Create a /lib/systemd/system/gost.service file with the following content, replacing the domain with your own:

  [Unit]
  Description=gost service
  After=network.target
  StartLimitIntervalSec=0
  
  [Service]
  Type=simple
  Restart=always
  RestartSec=1
  User=root
  PIDFile=/home/admin/gost.pid
  ExecStart=/home/admin/bin/gost -L "http2://xxx:yyy@0.0.0.0:443?cert=/etc/letsencrypt/live/example.com/fullchain.pem&key=/etc/letsencrypt/live/example.com/privkey.pem&probe_resist=code:404"
  ExecReload=/bin/kill -HUP $MAINPID
  
  [Install]
  WantedBy=multi-user.target
  

  ExecStart is a simplified version of the Docker method in the guide. ExecReload just kills the process.

  1. Test whether it’s successful using systemctl start|status|restart|enable gost.

  2. Configure crontab to update the certificate. I didn’t use systemd because I’m not familiar with it.

  0 0 1 * * /usr/bin/certbot renew --force-renewal
  5 0 1 * * systemctl restart gost
  

• After completing the above, nginx can be directly stopped and disabled.

• Configure the client. This is simple; just refer to the guide. The principle is straightforward because gost implements the shadowsocks protocol using the shadowsocks Golang version. Therefore, the following command starts a local shadowsocks server, and you configure your client to add a local server configuration that matches the password.

  .\bin\gost-windows-amd64.exe -L=ss://aes-128-cfb:passcode@:1984 -F=https://xxx:yyy@example.com:443
  

PS: I still don’t know how to configure a global HTTPS proxy on Android without root, or how to set it up on iOS without a U.S. account. Also, I’m unsure how to elegantly configure startup scripts on Windows 10. These are issues to explore further…

Continuation

Regarding the mobile problem mentioned above, I found that HTTPS proxy client support is generally poor. Gost itself seems to have problems, possibly due to my usage. In short, if not using a local gost to connect remotely, authentication errors occur.

During the holiday break, I tinkered a bit more. First, I deployed a gost HTTP proxy on my home NAS using the simplest nohup + ctrl-D method to maintain it. It’s compiled with GOARCH=arm64. After a trial run for a day, Android’s weak built-in HTTP proxy worked well, but globally routing through it wasn’t great. Hence, I switched from HTTP to using SS to connect to HTTPS remotely. I essentially moved the local service on Windows to my NAS. Additionally, through simple double-port forwarding from NAS -> internal router -> optical modem router, I could also use the NAS as an SS server via the public IP.

The remaining issue is the DDNS. After researching, it seems Cloudflare’s API is a more reliable option. Seeing an official flarectl, I compiled it to the NAS and wrote a small script, revisiting the various (pitfalls) wonders of bash, especially remembering special writing for string comparisons such as [ $a != $b ] to [ $a != $b* ] to handle trailing “\r” “\n” characters. However, detaching the name server still takes some time. The final effect is to be tested.

Additionally, on the NAS, I currently use curl to fetch my public IP from a third-party. I have a hunch that this method might not work someday or might cause issues.

When I first started writing Golang, I was always looking for a convenient debugging tool. Back then, I came across documentation about using gdb to debug and also tried delve, but neither felt easy to use. Later, on someone’s advice, I went back to the good old print statements…

Over the past couple of days, I was debugging go test and found that tests would always hang when run per package. I couldn’t think of a suitable method at first, so I thought of delve again. After giving it a try, I found it has become much more mature than before.

Usage

dlv attach ${pid} is the method I use most often. After attaching, you can use debugging commands similar to gdb. You can use help to view specific commands.

(dlv) help
The following commands are available:
    args ------------------------ Print function arguments.
    break (alias: b) ------------ Sets a breakpoint.
    breakpoints (alias: bp) ----- Print out info for active breakpoints.
    call ------------------------ Resumes process, injecting a function call (EXPERIMENTAL!!!)
    clear ----------------------- Deletes breakpoint.
    clearall -------------------- Deletes multiple breakpoints.
    condition (alias: cond) ----- Set breakpoint condition.
    config ---------------------- Changes configuration parameters.
    continue (alias: c) --------- Run until breakpoint or program termination.
    deferred -------------------- Executes command in the context of a deferred call.
    disassemble (alias: disass) - Disassembler.
    down ------------------------ Move the current frame down.
    edit (alias: ed) ------------ Open where you are in $DELVE_EDITOR or $EDITOR
    exit (alias: quit | q) ------ Exit the debugger.
    frame ----------------------- Set the current frame, or execute command on a different frame.
    funcs ----------------------- Print list of functions.
    goroutine (alias: gr) ------- Shows or changes current goroutine.
    goroutines (alias: grs) ----- List program goroutines.
    help (alias: h) ------------- Prints the help message.
    libraries ------------------- List loaded dynamic libraries.
    list (alias: ls | l) -------- Show source code.
    locals ---------------------- Print local variables.
    next (alias: n) ------------- Step over to next source line.
    on -------------------------- Executes a command when a breakpoint is hit.
    print (alias: p) ------------ Evaluate an expression.
    regs ------------------------ Print contents of CPU registers.
    restart (alias: r) ---------- Restart process.
    set ------------------------- Changes the value of a variable.
    source ---------------------- Executes a file containing a list of delve commands.
    sources --------------------- Print list of source files.
    stack (alias: bt) ----------- Print stack trace.
    step (alias: s) ------------- Single step through program.
    step-instruction (alias: si)  Single step a single CPU instruction.
    stepout (alias: so) --------- Step out of the current function.
    thread (alias: tr) ---------- Switch to the specified thread.
    threads --------------------- Print out info for every traced thread.
    trace (alias: t) ------------ Set tracepoint.
    types ----------------------- Print list of types.
    up -------------------------- Move the current frame up.
    vars ------------------------ Print package variables.
    whatis ---------------------- Prints type of an expression.
Type help followed by a command for full documentation.

Many of these commands are the same as those in gdb. Another command I use frequently is grs, which outputs all goroutines. You can also use grs -t, which is equivalent to gdb’s t a a bt. The only slight drawback is that it only outputs 10 stack frames, and any additional ones are truncated.

Additionally, it seems that processes forked by go test can’t be attached to. If you want to test, you must first compile it into a test file and then execute it. You can refer to this issue for more details.

$ dlv attach 19654
could not attach to pid 19654: decoding dwarf section info at offset 0x0: too short

Furthermore, by default, Go’s test caches results, which can be controlled via environment variables. However, with Go modules (go mod), it’s recommended to use ./ddl.test -test.count=1 to disable caching. It doesn’t feel very elegant.

Among all these implementations, I personally think the most reliable and stable one is the original Python version. The reason is simple - it has the most users. The Golang version is said to have the most features and also performs very well, making it quite powerful. This might be due to Golang’s inherent high performance and ease of implementation. There’s also an implementation using libev, a pure C implementation, which also offers good performance and is very lightweight.

Additionally, updating the server is a necessary task for Shadowsocks users due to well-known reasons. The server should be updated frequently. If you’re using the Python implementation, you might be able to install updates via pip, although I haven’t confirmed this. The Golang version may require a Golang build environment, and then you can use go get -u. For updating libev, you can use apt on Debian-based systems, as apt includes shadowsocks-libev. I haven’t checked if it is available in the Red Hat-based yum repositories.

After this introduction, let’s go over the deployment steps, which are quite straightforward:

1. Deploy a Debian 9 or Ubuntu 17 VPS. Mainstream providers like Vultr should have these options available. Assume we are using Debian 9 here.
2. Run apt install shadowsocks-libev to install.
3. Edit the configuration file using vim /etc/shadowsocks-libev/config.json. It’s best to set the Server IP to 0.0.0.0 to avoid IP issues similar to those on AWS Lightsail. *. For AWS Lightsail, you need to bind a static IP and open firewall ports. Specific steps can be found on Google.
4. Restart the service using systemctl restart shadowsocks-libev to apply the changes.
5. Enable TCP BBR. Specific instructions can be found on Google.

package main

import (
	"fmt"
	"os"
	"runtime"
	"time"
)

func main() {
	runtime.GOMAXPROCS(2)
	a := make(map[int]int)
	go func() {
		i := 0
		for {
			a[1] = i
			i++
			time.Sleep(1000)
		}
	}()
	for {
		if a[1] > 1000000 {
			fmt.Println(a[1])
			os.Exit(1)
		}
	}
}

After compiling and running it (assuming your machine has 2 or more cores), you will get the following error:

fatal error: concurrent map read and map write

goroutine 1 [running]:
runtime.throw(0x10c3e05, 0x21)
        /usr/local/Cellar/go/1.10.3/libexec/src/runtime/panic.go:616 +0x81 fp=0xc42004bf00 sp=0xc42004bee0 pc=0x10263f1
runtime.mapaccess1_fast64(0x10a5b60, 0xc42007e180, 0x1, 0xc42008e048)
        /usr/local/Cellar/go/1.10.3/libexec/src/runtime/hashmap_fast.go:101 +0x197 fp=0xc42004bf28 sp=0xc42004bf00 pc=0x1008d27
main.main()
        /Users/yusp/test/panic3/main.go:22 +0x7c fp=0xc42004bf88 sp=0xc42004bf28 pc=0x108e28c
runtime.main()
        /usr/local/Cellar/go/1.10.3/libexec/src/runtime/proc.go:198 +0x212 fp=0xc42004bfe0 sp=0xc42004bf88 pc=0x1027c62
runtime.goexit()
        /usr/local/Cellar/go/1.10.3/libexec/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42004bfe8 sp=0xc42004bfe0 pc=0x104e501

goroutine 5 [runnable]:
time.Sleep(0x3e8)
        /usr/local/Cellar/go/1.10.3/libexec/src/runtime/time.go:102 +0x166
main.main.func1(0xc42007e180)
        /Users/yusp/test/panic3/main.go:18 +0x61
created by main.main
        /Users/yusp/test/panic3/main.go:13 +0x59

It seems straightforward; Golang’s map is not thread-safe, and concurrent read and write cause a panic. However, look at the error information on line /Users/yusp/test/panic3/main.go:18 +0x61, which points to line 18 of main.go where Sleep is called, not the actual point of concurrency issue. In a vast stack trace, it becomes even harder to locate the problem.

A workaround that comes to mind is if you only see the read stack and want to see the write stack, set a variable at the read position, reset it after reading, and check the value of this variable at the write position. If reading is currently happening, panic will be triggered.

This article will first introduce the classification of INSERT statements in TiDB, along with the syntax and semantics of each statement, and then describe the source code implementation of the five types of INSERT statements.

Types of INSERT Statements

In broad terms, TiDB has the following six types of INSERT statements:

• Basic INSERT
• INSERT IGNORE
• INSERT ON DUPLICATE KEY UPDATE
• INSERT IGNORE ON DUPLICATE KEY UPDATE
• REPLACE
• LOAD DATA

In theory, all six statements belong to the category of INSERT statements.

The first one, Basic INSERT, is the most common INSERT statement, using the syntax INSERT INTO VALUES (). It implies inserting a record, and if a unique constraint conflict occurs (such as primary key conflict, unique index conflict), it returns an execution failure.

The second, with the syntax INSERT IGNORE INTO VALUES (), ignores the current INSERT row if a unique constraint conflict occurs and logs a warning. After the statement execution finishes, you can use SHOW WARNINGS to see which rows were not inserted.

The third one, with the syntax INSERT INTO VALUES () ON DUPLICATE KEY UPDATE, updates the conflicting row, then inserts data if there is a conflict. If the updated row conflicts with another row in the table, it returns an error.

The fourth one, similar to the previous case, if the updated row conflicts with another row, this does not insert the row and shows a warning.

The fifth one, with the syntax REPLACE INTO VALUES (), deletes the conflicting row in the table after a conflict and continues to attempt data insertion. If another conflict occurs again, it continues to delete conflicting data on the table until there is no conflicting data left in the table, then inserts the data.

The last one, using the syntax LOAD DATA INFILE INTO, has semantics similar to INSERT IGNORE, both ignoring conflicts. The difference is that LOAD DATA imports data files into a table, meaning its data source is a CSV data file.

Since INSERT IGNORE ON DUPLICATE KEY UPDATE involves special processing on INSERT ON DUPLICATE KEY UPDATE, it won’t be explained in detail separately but will be covered in the same section. Due to the unique nature of LOAD DATA, it will be discussed in other chapters.

Basic INSERT Statement

The major differences among the several INSERT statements lie in the execution level. Continuing from the “TiDB Source Code Reading Series (4) Overview of INSERT Statement”, here is the statement execution process. Those who do not remember the previous content can refer back to the original article.

INSERT’s execution logic is located in executor/insert.go. In fact, the execution logic for all four types of INSERT statements covered previously is in this file. Here, we first discuss the most basic Basic INSERT.

InsertExec is an implementation of the INSERT executor, conforming to the Executor interface. The most important methods are the following three interfaces:

• Open: Performs some initialization
• Next: Executes the write operation
• Close: Performs some cleanup tasks

Among them, the most important and complex is the Next method. Depending on whether a SELECT statement is used to retrieve data (INSERT SELECT FROM), the Next process is divided into two branches: insertRows and insertRowsFromSelect. Both processes eventually lead to the exec function to execute the INSERT.

In the exec function, the first four types of INSERT statements are processed together. The standard INSERT covered in this section directly enters insertOneRow.

Before discussing insertOneRow, let’s look at a segment of SQL.

CREATE TABLE t (i INT UNIQUE);
INSERT INTO t VALUES (1);
BEGIN;
INSERT INTO t VALUES (1);
COMMIT;

Paste these lines of SQL sequentially into MySQL and TiDB to see the results.

MySQL:

mysql> CREATE TABLE t (i INT UNIQUE);
Query OK, 0 rows affected (0.15 sec)
mysql> INSERT INTO t VALUES (1);
Query OK, 1 row affected (0.01 sec)
mysql> BEGIN;
Query OK, 0 rows affected (0.00 sec)
mysql> INSERT INTO t VALUES (1);
ERROR 1062 (23000): Duplicate entry '1' for key 'i'
mysql> COMMIT;
Query OK, 0 rows affected (0.11 sec)

TiDB:

mysql> CREATE TABLE t (i INT UNIQUE);
Query OK, 0 rows affected (1.04 sec)
mysql> INSERT INTO t VALUES (1);
Query OK, 1 row affected (0.12 sec)
mysql> BEGIN;
Query OK, 0 rows affected (0.01 sec)
mysql> INSERT INTO t VALUES (1);
Query OK, 1 row affected (0.00 sec)
mysql> COMMIT;
ERROR 1062 (23000): Duplicate entry '1' for key 'i'

As you can see, for INSERT statements, TiDB performs conflict detection at the time of transaction commit, whereas MySQL does it when the statement is executed. The reason for this is that TiDB is designed with a layered structure with TiKV; to ensure efficient execution, only read operations within a transaction must retrieve data from the storage engine, while all write operations are initially placed within the transaction’s own memDbBuffer in a single TiDB instance. The data is then written to TiKV as a batch during transaction commit. In the implementation, the PresumeKeyNotExists option is set within insertOneRow, assuming that insertions will not encounter conflicts if no conflicts are detected locally, without needing to check for conflicting data in TiKV. These data are marked as pending verification, and the BatchGet interface is used during the commit process to batch check the whole transaction’s pending data.

After all the data goes through insertOneRow and completes the insertion, the INSERT statement essentially concludes. The remaining tasks involve setting the lastInsertID and other return information, and then returning the results to the client.

INSERT IGNORE Statement

The semantics of INSERT IGNORE were introduced earlier. It was mentioned how a standard INSERT checks at the time of commit, but can INSERT IGNORE do the same? The answer is no, because:

1. If INSERT IGNORE is checked at the commit, the transaction module will need to know which rows should be ignored and which should immediately raise errors and roll back, undoubtedly increasing module coupling.
2. Users want to immediately know which rows were not inserted through INSERT IGNORE. In other words, they would like to see which rows were not actually inserted immediately through SHOW WARNINGS.

This requires checking data conflicts promptly when executing INSERT IGNORE. One obvious approach is to try reading the data intended for insertion, logging a warning when finding a conflict, and proceeding to the next row. However, if the statement inserts multiple rows, it would require repetitive reads from TiKV for conflict detection, which would be inefficient. Therefore, TiDB implements a batchChecker, with the code located in executor/batch_checker.go.

In the batchChecker, first, prepare the data for insertion, constructing possible conflicting unique constraints into a key within getKeysNeedCheck. TiDB implements unique constraints by constructing unique keys, as detailed in “Three Articles to Understand TiDB’s Technical Inside Story – On Computation”.

Then, pass the constructed keys through BatchGetValues to read them all at once, resulting in a key-value map where those read are the conflicting data.

Finally, check the keys of the data intended for insertion against the results from BatchGetValues. If a conflicting row is found, prepare a warning message and proceed to the next row. If a conflicting row isn’t found, a safe INSERT can proceed. The implementation of this portion is found in batchCheckAndInsert.

Similarly, after executing the insertion for all data, return information is set, and the execution results are returned to the client.

INSERT ON DUPLICATE KEY UPDATE Statement

INSERT ON DUPLICATE KEY UPDATE is the most complex among the INSERT statements. Its semantic essence includes both an INSERT and an UPDATE. The complexity arises since during an UPDATE, a row can be updated to any valid version.

In the previous section, it was discussed how TiDB uses batching to implement conflict checking for special INSERT statements. The same method is used for INSERT ON DUPLICATE KEY UPDATE, but the implementation process is somewhat more complex due to the semantic complexity.

Initially, similar to INSERT IGNORE, the keys constructed from the data to be inserted are read out at once using BatchGetValues, resulting in a key-value map. Then, all records corresponding to the read keys are again read using a batch BatchGetValues, prepared for possible future UPDATE operations. The specific implementation is in initDupOldRowValue.

Then, during conflict checking, if a conflict occurs, an UPDATE is performed first. As discussed in the Basic INSERT section earlier, TiDB executes INSERT in TiKV during commit. Similarly, UPDATE is also executed in TiKV during commit. In this UPDATE process, unique constraint conflicts might still occur. If so, then an error is returned. If the statement is INSERT IGNORE ON DUPLICATE KEY UPDATE, this error is ignored, and the next row proceeds.

In the UPDATE from the previous step, another scenario can occur, as in the SQL below:

CREATE TABLE t (i INT UNIQUE);
INSERT INTO t VALUES (1), (1) ON DUPLICATE KEY UPDATE i = i;

Here, it is clear that there are no original data in the table; the INSERT in the second line cannot read out possibly conflicting data, but there is a conflict between the two rows of data intended to be inserted themselves. Correct execution here should involve the first 1 being inserted normally, with the second 1 encountering conflict and updating the first 1. Thus, it is necessary to handle it as follows: remove the key-value of the data updated in the previous step from the initial step’s key-value map, reconstruct unique constraint keys and values for the data from the UPDATE based on table information, and add this key-value pair back into the initial key-value map for subsequent data conflict checking. The detail implementation is in fillBackKeys. This scenario also arises in other INSERT statements like INSERT IGNORE, REPLACE, and LOAD DATA. It is introduced here because INSERT ON DUPLICATE KEY UPDATE showcases the full functionality of the batchChecker.

Finally, after all data completes insertion/update, return information is set, and results are returned to the client.

REPLACE Statement

Although the REPLACE statement appears as a separate type of DML, in examining its syntax, it is merely replacing INSERT with REPLACE compared to a standard Basic INSERT. The difference is that REPLACE is a one-to-many statement. Briefly, for a typical INSERT statement which needs to INSERT a row and encounters a unique constraint conflict, various treatments are available:

• Abandon the insert and return an error: Basic INSERT
• Abandon the insert without error: INSERT IGNORE
• Abandon the insert, turning it into updating the conflicting row. If the updated value conflicts again,
• Return an error: INSERT ON DUPLICATE KEY UPDATE
• No error: INSERT IGNORE ON DUPLICATE KEY UPDATEThey all handle conflicts when a row of data conflicts with a row in the table differently. However, the REPLACE statement is distinct; it will delete all conflicting rows it encounters until there are no more conflicts, and then insert the data. If there are 5 unique indexes in the table, there could be 5 rows conflicting with the row waiting to be inserted. The REPLACE statement will delete these 5 rows all at once and then insert its own data. See the SQL below:

CREATE TABLE t (
i int unique,
j int unique,
k int unique,
l int unique,
m int unique);
INSERT INTO t VALUES
(1, 1, 1, 1, 1),
(2, 2, 2, 2, 2),
(3, 3, 3, 3, 3),
(4, 4, 4, 4, 4);
REPLACE INTO t VALUES (1, 2, 3, 4, 5);
SELECT * FROM t;
i j k l m
1 2 3 4 5

After execution, it actually affects 5 rows of data.

Once we understand the uniqueness of the REPLACE statement, we can more easily comprehend its specific implementation.

Similar to the INSERT statement, the main execution part of the REPLACE statement is also in its Next method. Unlike INSERT, it passes its own exec method through insertRowsFromSelect and insertRows. In exec, it calls replaceRow, which also uses batch conflict detection in batchChecker. The difference from INSERT is that all detected conflicts are deleted here, and finally, the row to be inserted is written in.

In Conclusion

The INSERT statement is among the most complex, versatile, and powerful of all DML statements. It includes statements like INSERT ON DUPLICATE UPDATE, which can perform both INSERT and UPDATE operations, and REPLACE, where a single row of data can impact many rows. The INSERT statement itself can be connected to a SELECT statement as input for the data to be inserted, thus its implementation is influenced by the planner (for more on the planner, see related source code reading articles: Part 7: Rule-Based Optimization and Part 8: Cost-Based Optimization). Familiarity with the implementation of various INSERT-related statements in TiDB can help readers use these statements more reasonably and efficiently in the future. Additionally, readers interested in contributing code to TiDB can also gain a quicker understanding of this part of the implementation through this article.

yum install haproxy is effective for CentOS 7. After installation, you can start the service using systemctl start haproxy. But don’t rush yet.

Configuring HAProxy

Add the following content to /etc/haproxy/haproxy.cfg.

global # The content of global is generally fixed and quite understandable.
        log 127.0.0.1   local2
        maxconn 4096
        user haproxy
        group haproxy
        chroot /var/lib/haproxy
        daemon
        pidfile /var/run/haproxy.pid
        stats socket /var/run/haproxy.sock         # Create a socket file for haproxy
        nbproc 40                                  # Start 40 processes to forward concurrently, higher versions can use nbthread, a threaded approach.

defaults # This section is mostly copied, not entirely clear on the options.
        log     global
        mode    http
        option  tcplog
        option  dontlognull
        retries 3
        option  redispatch
        maxconn 1024
        timeout connect 5000ms
        timeout client 50000ms
        timeout server 50000ms

listen cdb_cluster 0.0.0.0:3030  # The actual proxy name and address for receiving services.
## cdb balance leastconn - the cluster listening on port 3030.
        mode tcp
        balance leastconn  # This method is most suitable for databases; do not change.
        server cdb1 172.16.30.3:26257 check # Check seems to require a port for feedback status; without it, it might not work, but it doesn't matter.
        server cdb2 172.16.30.3:26258 check
        server cdb3 172.16.30.3:26259 check
        server cdb4 172.16.30.3:26260 check

Start and Connect

systemctl start haproxy to start the service.

psql -Uroot -h127.0.0.1 -p3030 test to connect to the database.

CockroachDB Official Recommendation

CockroachDB officially provided their recommended configuration. In this configuration, they use:

default
# TCP keep-alive on client side. Server already enables them.
    option              clitcpka

listen psql
    option httpchk GET /health?ready=1

These two configurations, the first is to keep the client connection alive, which seems very useful. The second is a status check port, which I understand might be an option to ensure the service is available before dispatching requests, and it also seems very useful. It is recommended to add them.

First of all, TPC-C is the de facto OLTP benchmark standard. It is a set of specifications, and any database can publish its test results under this standard, so there’s no issue of quarreling over the testing tools used.

Secondly, TPC-C is closer to real-world scenarios as it includes a transaction model within it. In the flow of this transaction model, there are both high-frequency simple transaction statements and low-frequency inventory query statements. Therefore, it tests the database more comprehensively and practically.

Testing TPC-C on CockroachDB

This year, CockroachDB released its TPC-C performance results. However, unfortunately, they did not use a tool recognized by the database industry that implements the TPC-C standard for testing. Instead, they used their own implementation of a TPC-C tool. The compliance level of this tool was not recognized. In the white paper officially released by them, it is also mentioned that this TPC-C cannot be compared with the TPC-C standard.

Therefore, I thought of using a highly recognized tool in the industry for testing. Here, I chose Benchmarksql version 5.0.

Benchmarksql 5.0 supports the PostgreSQL protocol, Oracle protocol, and MySQL protocol (the MySQL protocol is supported in the code, but the author hasn’t fully tested it, so the official documentation doesn’t mention MySQL). Among these, the PostgreSQL protocol is supported by CockroachDB.

Test Preparation

After preparing the Benchmarksql code, don’t rush into testing. There are three main pitfalls here that need to be addressed first.

1. CockroachDB does not support adding a primary key after table creation. Therefore, you need to include the primary key when creating the table. Specifically, in the run folder under the root directory of the Benchmarksql code, create a sql.cdb folder. Copy tableCreates.sql and indexCreates.sql from the sql.common folder at the same level into sql.cdb. Then move the primary keys in indexCreates.sql into the table creation statements in tableCreates.sql. For how to define indexes while creating tables, please refer to the database documentation syntax via Google.

2. CockroachDB is a “strongly typed” database. This is my own way of describing it. It has a rather peculiar behavior: when you add different data types (e.g., int + float), it will report an error saying, “InternalError: unsupported binary operator: <int> + <float>”. Generally, databases don’t behave like this; most would perform some implicit conversions, or in other words, they are very tolerant of SQL writers. But CockroachDB is unique in that if you don’t specify the type, it reports an error. This greatly reduces the burden of type inference in its internal implementation.

   This behavior causes Benchmarksql to fail to run the tests properly. The solution is to add the required type at the position where the error occurs. For example, change update t set i = i + ?; (the ? is generally filled in using prepare/execute) to update t set i = i + ?::DECIMAL;. Yes, CockroachDB specifies types explicitly by adding ::<type_name> at the end. But strangely, not all additions require type specification.

3. CockroachDB does not support SELECT FOR UPDATE. This is the easiest to solve: comment out all FOR UPDATE clauses in Benchmarksql. CockroachDB itself supports the serializable isolation level; lacking FOR UPDATE doesn’t affect consistency.

Starting the Test

After overcoming the pitfalls mentioned above, you can proceed with the normal testing process: creating the database, creating tables and indexes, importing data, and testing. You can refer to Benchmarksql’s HOW-TO-RUN.txt.

Test Results

On my test machine with 40 cores, 128 GB of memory, and SSD, under 100 warehouses, the tpmC is approximately 5,000. This is about one-tenth of PostgreSQL 10 on the same machine. PostgreSQL can reach around 500,000 tpmC.

CockroachDB uses the PostgreSQL protocol. If you want to use Sysbench for testing, you need to enable pg protocol support in Sysbench. Sysbench already supports the pg protocol, but it is not enabled by default during compilation. You can configure it with the following command:

./configure --with-pgsql

Of course, preliminary work involves downloading the Sysbench source code and installing the necessary PostgreSQL header files required for compilation (you can use yum or sudo to install them).

Testing

The testing method is no different from testing MySQL or PostgreSQL; you can test any of the create, read, update, delete (CRUD) operations you like. The only thing to note is to set auto_inc to off.

This is because CockroachDB’s auto-increment behavior is different from PostgreSQL’s. It generates a unique id, but it does not guarantee that the ids are sequential or incremental. This is fine when inserting data. However, during delete, update, or query operations, since all SQL statements use id as the condition for these operations, you may encounter situations where data cannot be found.

That is:

When auto_inc = on (which is the default value in Sysbench)

Table Structure

CREATE TABLE sbtest1 (
   id INT NOT NULL DEFAULT unique_rowid(),
   k INTEGER NOT NULL DEFAULT 0:::INT,
   c STRING(120) NOT NULL DEFAULT '':::STRING,
   pad STRING(60) NOT NULL DEFAULT '':::STRING,
   CONSTRAINT ""primary"" PRIMARY KEY (id ASC),
   INDEX k_1 (k ASC),
   FAMILY ""primary"" (id, k, c, pad)
)

Data

root@:26257/sbtest> SELECT id FROM sbtest1 ORDER BY id LIMIT 1;
+--------------------+
|         id         |
+--------------------+
| 354033003848892419 |
+--------------------+

As you can see, the data does not start from 1, nor is it sequential. Normally, the id in a Sysbench table should be within the range [1, table_size].

SQL

UPDATE sbtest%u SET k = k + 1 WHERE id = ?

Taking the UPDATE statement as an example, id is used as the query condition. Sysbench assumes that this id should be between [1, table_size], but in reality, it’s not.

Example of Correct Testing Command Line

sysbench --db-driver=pgsql --pgsql-host=127.0.0.1 --pgsql-port=26257 --pgsql-user=root --pgsql-db=sbtest \
        --time=180 --threads=50 --report-interval=10 --tables=32 --table-size=10000000 \
        oltp_update_index \
        --sum_ranges=50 --distinct_ranges=50 --range_size=100 --simple_ranges=100 --order_ranges=100 \
        --index_updates=100 --non_index_updates=10 --auto_inc=off prepare/run/cleanup

INSERT Testing

Let’s discuss the INSERT test separately. The INSERT test refers to Sysbench’s oltp_insert. The characteristic of this test is that when auto_inc is on, data is inserted during the prepare phase of the test; otherwise, only the table is created without inserting data. Because when auto_inc is on, after the prepare phase, during the run phase, the inserted data will not cause conflicts due to the guarantee of the auto-increment column. When auto_inc is off, the id of the data inserted during the run phase is randomly assigned, which aligns with some actual testing scenarios.

For CockroachDB, when testing INSERT operations with auto_inc set to off, after the prepare phase, during the run phase of data insertion, you can observe the monitoring metrics (by connecting to CockroachDB’s HTTP port) under the “Distribution” section in “KV Transactions”. You’ll notice a large number of “Fast-path Committed” transactions. This indicates that transactions are committed using one-phase commit (1PC). That is, the data involved in the transaction does not span across CockroachDB nodes, so there’s no need to ensure consistency through two-phase commit transactions. This is an optimization in CockroachDB, which is very effective in INSERT tests and can deliver excellent performance.

If auto_inc is on, although for other tests that require read-before-write operations, the results in CockroachDB might be inflated, it is still fair for the INSERT test. If time permits, you can supplement the tests to see the differences.

OLTP-Bench is an open-source benchmarking tool platform for OLTP scenarios from CMU’s DB Group. It was designed to provide a simple, easy-to-use, and extensible testing platform.

It connects to databases via the JDBC interface, supporting the following test suites:

• TPC-C
• Wikipedia
• Synthetic Resource Stresser
• Twitter
• Epinions.com
• TATP
• AuctionMark
• SEATS
• YCSB
• JPAB (Hibernate)
• CH-benCHmark
• Voter (Japanese “American Idol”)
• SIBench (Snapshot Isolation)
• SmallBank
• LinkBench

Detailed project information can be found here, and the GitHub page is here.

The project introduction page includes three papers published by the authors, with the one from 2013 being the most important, also linked on the GitHub page.

Based on the GitHub page, the project does not seem to have a high level of attention and has not been very active recently. Most issues and pull requests come from within CMU.

OLTP-Bench: An Extensible Testbed for Benchmarking Relational Databases

The paper “OLTP-Bench: An Extensible Testbed for Benchmarking Relational Databases” can be regarded as the most detailed introduction to this project.

In the first and second chapters, the authors introduce the motivation for creating this framework, which is to integrate multiple test sets and provide features that simple benchmarking tools do not have, while offering excellent extensibility to attract developers to support more databases.

From the activity on GitHub, it is evident that this extensibility is more about adding database support rather than test sets. However, the number of supported test suites is already quite extensive.

Chapter three introduces the architectural design, with a focus on test suite management, load generators, SQL syntax conversion, multi-client scenarios (similar to multiple sysbench instances stressing a single MySQL), and result collection.

Chapter four discusses the supported test suites. I’m only familiar with TPCC and YCSB. The authors classify them from three perspectives:

1. Transaction-focused, such as TPCC and SmallBank
2. Internet applications, like LinkBench and Wikipedia
3. Specialized tests, such as YCSB and SIBench

Further details can be seen in the table: [table]

Chapter five describes the demo deployment environment, with subsequent sections introducing the demo’s features.

Chapter six uses the demo from the previous chapter to introduce features, analyzed as follows:

1. Rate control. It seems odd for a benchmarking tool to perform rate control, as the conventional understanding is to push performance as high as possible to gauge system limits. The paper provides an example using the Wikipedia test suite, increasing by 25 TPS every 10 seconds to observe database latency changes.

2. Tagging different transactions in the same test suite for separate statistics – using TPCC as an example to statistically categorize transactions from different stages.

3. Modifying load content, like switching from read-only to write-only loads.

4. Changing the method for load randomness.

5. Monitoring server status alongside database monitoring by deploying an OLTP-Bench monitor on the server.

6. Running multiple test suites simultaneously, such as running TPCC and YCSB concurrently.

7. Multi-client usage, mentioned in chapter three.

8. Repeatability. To prove OLTP-Bench results are genuine and reliable, the authors tested PG’s SSI performance using SIBench from the tool on similarly configured machines, achieving results consistent with those in PG’s SSI paper.

In summary, rate control and transaction tagging stand out as novel features, while the rest are not particularly special.

Chapter seven is arguably the most valuable part of the article, discussing cloud environments where users might only have database access and not server control. Users may struggle to assess the cost-effectiveness of different cloud database services or configurations due to charges encompassing CPU, storage, network, and asynchronous sync in some architectures. Thus, using benchmarking tools to derive performance and subsequently calculate cost-effectiveness is particularly worthwhile. This chapter compares varying perspectives: different service providers, configurations, comparing databases on the same configuration, and presents the cost-effectiveness outcomes.

In chapter eight, the authors compare OLTP-Bench with other similar tools, providing a favorable self-assessment.

Chapter nine outlines the authors’ future plans, including support for pure NoSQL, additional databases’ proprietary SQL syntax, generating real-world load distributions from production data, and support for stored procedures.

In conclusion, as the authors mentioned, this is an integrative framework where ease of use and extensibility are key.

Usage Summary

OLTP-Bench is relatively simple to install and use, especially the deployment. Its cross-platform nature provides a better user experience compared to traditional tpcc and sysbench. Usage is relatively straightforward due to the plethora of test configuration templates provided, allowing easy initiation of tests with simple configuration file modifications. The test results are stable, although certain features mentioned in papers, like server status monitoring, still require exploration.

I tested all 15 test suites on MySQL 5.7 and TiDB, obtaining the following results: [table]

Its usability is quite evident. As for the ease of secondary development, it should be relatively simple, considering the entire OLTP-Bench project is not particularly large, with around 40,000 lines of code.

Other

• tpch: While the framework’s code appears to support tpch, it proved unusable during practical tests, likely due to incomplete implementation and thus excluded from the README.
• Referring to future work mentioned in chapter nine of the paper, especially “generating load to match production data distribution,” this remains unimplemented, as seen in the codebase.

The English font set in Microsoft’s VS Code is monospaced, while the Chinese font is not.

Method

1. Download and install the fonts from here. This font combines Source Han Sans and is said to align punctuation strictly.
2. Use the method described here to set up VS Code.

PS: It is indeed aligned, although the English text appears a bit narrow.

The DDL paper on F1 serves as the foundation for TiDB’s DDL implementation. There are two main papers on F1: one provides an overview of F1’s DDL, and the other specifically details the schema change method for DDL. I personally believe the second is key and more confusing to me. There is an introduction to the second paper here, which can help in understanding.

Understanding

Online DDL Concept

The DDL discussed here refers to online DDL. The concept of online DDL originates from databases like MySQL, whereas PostgreSQL and similar databases might not support it. This concept is also quite vague; the distinction is whether you need to use exclusive locks during DDL operations to block transactions. Therefore, all databases can perform online DDL; it just depends on whether they’re willing to put in the effort. For traditional businesses where 24/7 availability isn’t a priority, DDL operations can be performed during maintenance times or late at night. Even if a few users are online, at most, they might experience minor delays. However, modern internet businesses are strict about maintenance windows, creating a higher demand for non-blocking DDL, which MySQL, as a quintessential internet database, was first to support. The typical implementation involves creating a copy of the schema table, with operations being sent to both the new and old tables during the transition.

For MySQL’s supported online DDL, see this webpage. Primarily, it’s categorized into operations on indexes and columns. This explains my curiosity about why TiDB’s examples for implementation often involve adding indexes.

F1’s Method

Having worked on something similar to Aurora before, there were many issues with this area. If you’re only performing offline DDL, it doesn’t have to be this complicated. According to F1’s paper, it uses the following series of state changes to accomplish a DDL:

                                    (reorganization)
absent -> delete only -> write only ---------------> public

In this sequence, each node passes through these four states, transitioning to the next state upon receiving a command. The agreed-upon rule is that each state doesn’t persist for more than twice the lease time across all nodes. How is this ensured? Through the following rule: if a node takes too long to move to the next state after receiving the transition command (meaning a state exceeds twice the lease time), it means the node received the command too late and will stop providing services and shut down.

Within these four states, ‘absent’ indicates a state where the node hasn’t received instructions yet, and ‘public’ signifies the completion of the DDL. What about the two middle states? The background link described them as follows:

• A delete-only table or column can be modified only by delete operations.
• A delete-only index can be modified only by delete and update operations. Update operations can delete key-value pairs corresponding to updated index keys, but they cannot create any new ones.
• A write-only column or index can have their key-value pairs modified by insert, delete, and update operations, but none of their pairs can be read by user transactions.
• A write-only constraint is applied for all new insert, delete, and update operations, but it is not guaranteed to hold over all existing data.

Summarized in a table:

The first state is rather peculiar, and the second one even more so—“can not read” might have been a better name for it. It’s said that this design allows two concurrent states to behave consistently, specifically pairs like (absent and delete only), (delete only and write only), and (write only and public).

Examples:

Adding index idx to table t, the deployment environment consists of two databases, a and b.

1. a enters delete only and completes adding the index. b has yet to receive any instructions.
   • Insert operations on a and b: a’s idx is ignored, b is unaware and also ignores.
   • Read operations: a and b ignore idx.
   • Deletes and updates: a’s idx responds, b ignores.
2. a enters write only, b enters delete only, and indexing is completed.
   • Insert operations on a and b: a’s idx responds, b ignores (b loses index data?).
   • Read operations: a and b ignore idx.
   • Deletes and updates: both a and b’s idx respond.
3. a enters the public state, b enters the write-only state.
   • Insert operations on a and b: both a and b’s idx respond.
   • Read operations: a’s idx responds, b ignores.
   • Deletes and updates: both a and b’s idx respond.

Removing index idx from table t, with two databases a and b in the deployment environment.

1. a enters delete only and completes removing the index. b has yet to receive any instructions.
   • Insert operations on a and b: a’s idx is ignored, b has index idx and inserts into the index.
   • Read operations: a ignores idx, b uses it.
   • Deletes and updates: a’s idx responds (no-op if the index is removed), b responds.
2. a enters write only, b enters delete only, and fulfills the command.
   • Insert operations on a and b: a’s idx responds (no-op), b ignores.
   • Read operations: both a and b ignore idx.
   • Deletes and updates: both a’s and b’s idx respond (it’s a no-op as the index is already deleted).
3. a enters the public state, b enters write only.
   • Insert operations on a and b: both a and b respond (b no-op?).
   • Read operations: a’s idx responds, b ignores.
   • Deletes and updates: both a and b’s idx respond (b no-op?).

The CAP theorem has become one of the hottest theorems in recent years; when discussing distributed systems, CAP is inevitably mentioned. However, I feel that I haven’t thoroughly understood it, so I wanted to write a blog post to record my understanding. I will update the content as I gain new insights.

Understanding

I read the first part of this paper.

The CAP theorem [Bre12] says that you can only have two of the three desirable properties of:

• C: Consistency, which we can think of as serializability for this discussion;
• A: 100% availability, for both reads and updates;
• P: tolerance to network partitions.

This leads to three kinds of systems: CA, CP and AP, based on what letter you leave out.

Let me share my understanding, using a network composed of three machines (x, y, and z) as an example:

• C (Consistency): The three machines appear as one. Operations of addition, deletion, modification, and query on any one machine should always be consistent. That is, if you read data from x and then read from y, the results are the same. If you write data to x and then read from y, you should also read the newly written data. Wikipedia also specifically mentions that it’s acceptable to read the data just written to x from y after a short period of time (eventual consistency).

• A (Availability): The three machines, as a whole, must always be readable and writable; even if some parts fail, it must be readable and writable.

• P (Partition Tolerance): If the network between x, y, and z is broken, any machine cannot or refuses to provide services; it is neither readable nor writable.

Here, C is the easiest to understand. The concepts of A and P are somewhat vague and easy to confuse.

Now let’s discuss the three combinations:

If the network between x, y, and z is disconnected:

• CA: Ensure data consistency (C). When x writes data, y can read it (C). Allow the system to continue providing services—even if only x and y are operational—ensuring it is readable and writable (A). We can only tolerate z not providing service; it cannot read or write, nor return incorrect data (losing P).

• CP: Ensure data consistency (C). Allow all three machines to provide services (even if only for reads) (P). We can only tolerate that x, y, and z cannot write (losing A).

• AP: Allow all three machines to write (A). Allow all three machines to provide services (reads count) (P). We can only tolerate that the data written by x and y doesn’t reach z; z will return data inconsistent with x and y (losing C).

CA is exemplified by Paxos/Raft, which are majority protocols that sacrifice P; minority nodes remain completely silent. CP represents a read-only system; if a system is read-only, whether there’s a network partition doesn’t really matter—the tolerance to network partitions is infinitely large. AP is suitable for systems that only append and do not update—only inserts, no deletes or updates. Finally, by merging the results together, it can still function.

Travis CI is generally used for automating tests without needing to update the repository with the test outputs. This article explains how to automatically commit the results from Travis CI.

Process

The basic process references this gist.

Below is the .travis.yml from the gist.

language: ruby
rvm:
  - 2.0.0
env:
  global:
  - USER="username"
  - EMAIL="username@mail.com"
  - REPO="name of target repo"
  - FILES="README.md foo.txt bar.txt"
  - GH_REPO="github.com/${USER}/${REPO}.git"
  - secure: "put travis gem output here => http://docs.travis-ci.com/user/encryption-keys/"
script:
  - ruby test.rb
after_success:
  - MESSAGE=$(git log --format=%B -n 1 $TRAVIS_COMMIT)
  - git clone git://${GH_REPO}
  - mv -f ${FILES} ${REPO}
  - cd ${REPO}
  - git remote
  - git config user.email ${EMAIL}
  - git config user.name ${USER}
  - git add ${FILES}
  - git commit -m "${MESSAGE}"
  - git push "https://${GH_TOKEN}@${GH_REPO}" master > /dev/null 2>&1

Note here that MESSAGE should be quoted when committing, which the original gist did not include.

Original README:

# Travis-CI tested push
Sometimes we have a private repository to hold both problems and solutions as a reference for the class projects.
The students can see only the problems in a public repository where they are able to clone/fork and develop their own solutions.
We do not want the solution files in the public repository and each bug found/feature added in the project requires a push for each repository.
It would be cool to work only with the reference repo and use tests to see if our modification is good enough for the public release.
This is possible with Travis-CI following simple steps:
- Create private and public repos
- Download Ruby
- Install the travis gem ``gem install travis``
- Generate a token in the Github website to allow others to play with your repos (copy the hash)
- Log into your git account
- Generate a secure token with the Travis gem (copy long hash)
- Fill the environment variables in the ```.travis.yml``` file (USER, EMAIL, REPO, FILES)
- Replace the value of **secure** with your long hash
- Replace **GH_TOKEN** with your Travis token name
- Push ```.travis.yml``` to private repo
- Go to Travis to unlock your private repo tests
- Push your files to the private repo to test

Travis now has a [deployment](https://docs.travis-ci.com/user/deployment/) feature, which may be better for certain scenarios.

A simple translation:

1. Create a GitHub project. The original seems to have created two projects, one for updating another.
2. Install Ruby. Usually, gem is installed alongside.
3. Install travis via gem install travis.
4. Apply for a token for your GitHub account. You can Google the details. When selecting token permissions, only tick all related to repo; others can be omitted.
5. Copy the generated token.
6. In the root directory of the local machine’s repo (not sure if it must be the root) run travis encrypt GH_TOKEN="copied token". This creates an encrypted token to use as ${GH_TOKEN}, essentially an environment variable. The command output, a string on the screen, needs to be pasted into the travis config file after secure:. Use travis encrypt GH_TOKEN="copied token" --add to write directly into the config file.
7. Commit the modified configuration file.

This translation is not strictly literal. The above content is more suited to the following personal configuration:

language: ruby
branches:
  only:
  - master
rvm:
- 2.4.1
exclude:
- vendor
sudo: false
env:
  global:
    - secure: rxKkyttLE1L4VsVIhhDUYGoLlER33ijKbdAAJPE8vNDSHwyANYnsP1GXK/rcwQqsL/KcJa55wEjVwEBzTMCqZM4UYNVIWqrJepVYo4rL1WhO+jT5sCqVR3qxK9KbgodcSXbmySJnJs0iLGMIQ2bo8yE91OxIC/GKkLCIwr9x4EGwFd5EcE5bOqmVqoSRk1q/1/5yA0aVF+Pohc5ATCZGw9+IyprU2Dx7qbA7F/T/4FQTOQZ4CLZAgyh/Gp1P+uxf1OK4IMCc/P6jVeTmbzQIbUcX0uG09pR7F0GnlV1ZOutMjY7SF8tQ7LNv2Wf8iWdiqehcwKNe/4TFHjs6rm3lEc6F1ELB5s4Z+QXjIM70haENSwM1FI8K5biL7tndAC1TujKESm0XadxORy5yOz7TfQZDTuMXvmmH3j+NFL3vTYPyMwwFca+IQBwD67a4PKD0PWBgEFD9Kn3rAlAzhV5OYdUuxZhx5zuQjKX5szUbL166fgoRnUwDp8dsOjLgOUqQa47IRqR3CTPzbf3zZIxGuX5x6mWySezCNprnXKCpyCegJBLoxQusA+EEYkvl4AOzhnmkhxFbEbHp+DYBjcSEEgpd06l67l3KzjMkpF02vr9CHNj8r7lAtZxwBVxYmczk289D5csOVR1SZKxQLwhx7k+CuEcYds685tLjIMmB0ZU=
    - USER="username"
    - FULLNAME="Your Name"
    - EMAIL="your-email@example.com"
    - REPO="your-username.github.io"
    - GH_REPO="github.com/${USER}/${REPO}.git"
before_script:
  - git clone https://${GH_TOKEN}@${GH_REPO}
script: bundle exec jekyll b -d ${REPO}
after_success:
  - MESSAGE=$(git log --format=%B -n 1 $TRAVIS_COMMIT)
  - cd ${REPO}
  - git config user.email ${EMAIL}
  - git config user.name ${FULLNAME}
  - git add --all
  - git commit -m "${MESSAGE}"
  - git push --force origin master

This configuration is used for automatically updating a blog created with Jekyll. There are two projects, one for source files and another for compiled HTML files. The purpose of this setup is to allow updating the blog without having to set up a Jekyll environment, even allowing updates directly from the GitHub website.

• Client Usage Guide MySQL Shell User Guide
• Server Configuration Guide Using MySQL as a Document Store
• Application Development API Guide X DevAPI User Guide
• Introduction to Server Internal Implementation X Protocol.

Implementation Principle

• Communication between client and server is over TCP and the protocol uses protobuf.
• After the server receives a message, it decodes and analyzes it. The protocol includes a concept called namespace, which specifically refers to whether the namespace is empty or “sql”, in which case the message content is executed as a SQL statement; if it is “xplugin” or “mysqlx,” the message is handled in another way. The other ways can be divided into:
  • Administrative commands
  • CRUD operations
• “xplugin” and “mysqlx” have the same function, with the latter being the new name for the former, retained temporarily for compatibility.
• The content of “mysqlx” messages, apart from explicit command content like kill_client, are mostly transformed into SQL statements which the server processes, essentially turning most into a form where the namespace is “sql”.

Implementation Steps

1. Start a new server for TiDB. The relevant configuration parameters such as IP, port, and socket need to be set.
2. Implement the reading and writing functionality for message communication.
3. Write a process for this new server to establish connections, including authentication, that follows the protocol. Use tcpdump to capture messages between MySQL and the client to derive protocol content, implementing the process by understanding MySQL source code.
4. The server should include contents like the Query Context from the original TiDB server, as it primarily translates into SQL for execution.
5. Implement the decoding and handling of messages. Although only a sentence, the workload included is substantial.

In mysqlx_all_msgs.h, all messages are initialized

  init_message_factory()
  {
    server_message<Mysqlx::Connection::Capabilities>(Mysqlx::ServerMessages::CONN_CAPABILITIES, "CONN_CAPABILITIES", "Mysqlx.Connection.Capabilities");
    server_message<Mysqlx::Error>(Mysqlx::ServerMessages::ERROR, "ERROR", "Mysqlx.Error");
    server_message<Mysqlx::Notice::Frame>(Mysqlx::ServerMessages::NOTICE, "NOTICE", "Mysqlx.Notice.Frame");
    server_message<Mysqlx::Ok>(Mysqlx::ServerMessages::OK, "OK", "Mysqlx.Ok");
    server_message<Mysqlx::Resultset::ColumnMetaData>(Mysqlx::ServerMessages::RESULTSET_COLUMN_META_DATA, "RESULTSET_COLUMN_META_DATA", "Mysqlx.Resultset.ColumnMetaData");
    server_message<Mysqlx::Resultset::FetchDone>(Mysqlx::ServerMessages::RESULTSET_FETCH_DONE, "RESULTSET_FETCH_DONE", "Mysqlx.Resultset.FetchDone");
    server_message<Mysqlx::Resultset::FetchDoneMoreResultsets>(Mysqlx::ServerMessages::RESULTSET_FETCH_DONE_MORE_RESULTSETS, "RESULTSET_FETCH_DONE_MORE_RESULTSETS", "Mysqlx.Resultset.FetchDoneMoreResultsets");
    server_message<Mysqlx::Resultset::Row>(Mysqlx::ServerMessages::RESULTSET_ROW, "RESULTSET_ROW", "Mysqlx.Resultset.Row");
    server_message<Mysqlx::Session::AuthenticateOk>(Mysqlx::ServerMessages::SESS_AUTHENTICATE_OK, "SESS_AUTHENTICATE_OK", "Mysqlx.Session.AuthenticateOk");
    server_message<Mysqlx::Sql::StmtExecuteOk>(Mysqlx::ServerMessages::SQL_STMT_EXECUTE_OK, "SQL_STMT_EXECUTE_OK", "Mysqlx.Sql.StmtExecuteOk");

    client_message<Mysqlx::Connection::CapabilitiesGet>(Mysqlx::ClientMessages::CON_CAPABILITIES_GET, "CON_CAPABILITIES_GET", "Mysqlx.Connection.CapabilitiesGet");
    client_message<Mysqlx::Connection::CapabilitiesSet>(Mysqlx::ClientMessages::CON_CAPABILITIES_SET, "CON_CAPABILITIES_SET", "Mysqlx.Connection.CapabilitiesSet");
    client_message<Mysqlx::Connection::Close>(Mysqlx::ClientMessages::CON_CLOSE, "CON_CLOSE", "Mysqlx.Connection.Close");
    client_message<Mysqlx::Crud::Delete>(Mysqlx::ClientMessages::CRUD_DELETE, "CRUD_DELETE", "Mysqlx.Crud.Delete");
    client_message<Mysqlx::Crud::Find>(Mysqlx::ClientMessages::CRUD_FIND, "CRUD_FIND", "Mysqlx.Crud.Find");
    client_message<Mysqlx::Crud::Insert>(Mysqlx::ClientMessages::CRUD_INSERT, "CRUD_INSERT", "Mysqlx.Crud.Insert");
    client_message<Mysqlx::Crud::Update>(Mysqlx::ClientMessages::CRUD_UPDATE, "CRUD_UPDATE", "Mysqlx.Crud.Update");
    client_message<Mysqlx::Crud::CreateView>(Mysqlx::ClientMessages::CRUD_CREATE_VIEW, "CRUD_CREATE_VIEW", "Mysqlx.Crud.CreateView");
    client_message<Mysqlx::Crud::ModifyView>(Mysqlx::ClientMessages::CRUD_MODIFY_VIEW, "CRUD_MODIFY_VIEW", "Mysqlx.Crud.ModifyView");
    client_message<Mysqlx::Crud::DropView>(Mysqlx::ClientMessages::CRUD_DROP_VIEW, "CRUD_DROP_VIEW", "Mysqlx.Crud.DropView");
    client_message<Mysqlx::Expect::Close>(Mysqlx::ClientMessages::EXPECT_CLOSE, "EXPECT_CLOSE", "Mysqlx.Expect.Close");
    client_message<Mysqlx::Expect::Open>(Mysqlx::ClientMessages::EXPECT_OPEN, "EXPECT_OPEN", "Mysqlx.Expect.Open");
    client_message<Mysqlx::Session::AuthenticateContinue>(Mysqlx::ClientMessages::SESS_AUTHENTICATE_CONTINUE, "SESS_AUTHENTICATE_CONTINUE", "Mysqlx.Session.AuthenticateContinue");
    client_message<Mysqlx::Session::AuthenticateStart>(Mysqlx::ClientMessages::SESS_AUTHENTICATE_START, "SESS_AUTHENTICATE_START", "Mysqlx.Session.AuthenticateStart");
    client_message<Mysqlx::Session::Close>(Mysqlx::ClientMessages::SESS_CLOSE, "SESS_CLOSE", "Mysqlx.Session.Close");
    client_message<Mysqlx::Session::Reset>(Mysqlx::ClientMessages::SESS_RESET, "SESS_RESET", "Mysqlx.Session.Reset");
    client_message<Mysqlx::Sql::StmtExecute>(Mysqlx::ClientMessages::SQL_STMT_EXECUTE, "SQL_STMT_EXECUTE", "Mysqlx.Sql.StmtExecute");
  }

Server and client messages are that many. Client messages are dispatched in xpl_dispatcher.cc.

ngs::Error_code do_dispatch_command(xpl::Session &session, xpl::Crud_command_handler &crudh,
                                    xpl::Expectation_stack &expect, ngs::Request &command)
{
  switch (command.get_type())
  {
    case Mysqlx::ClientMessages::SQL_STMT_EXECUTE:
      return on_stmt_execute(session, static_cast<const Mysqlx::Sql::StmtExecute&>(*command.message()));

    case Mysqlx::ClientMessages::CRUD_FIND:
      return crudh.execute_crud_find(session, static_cast<const Mysqlx::Crud::Find&>(*command.message()));

    case Mysqlx::ClientMessages::CRUD_INSERT:
      return crudh.execute_crud_insert(session, static_cast<const Mysqlx::Crud::Insert&>(*command.message()));

    case Mysqlx::ClientMessages::CRUD_UPDATE:
      return crudh.execute_crud_update(session, static_cast<const Mysqlx::Crud::Update&>(*command.message()));

    case Mysqlx::ClientMessages::CRUD_DELETE:
      return crudh.execute_crud_delete(session, static_cast<const Mysqlx::Crud::Delete&>(*command.message()));

    case Mysqlx::ClientMessages::CRUD_CREATE_VIEW:
      return crudh.execute_create_view(session, static_cast<const Mysqlx::Crud::CreateView&>(*command.message()));

    case Mysqlx::ClientMessages::CRUD_MODIFY_VIEW:
      return crudh.execute_modify_view(session, static_cast<const Mysqlx::Crud::ModifyView&>(*command.message()));

    case Mysqlx::ClientMessages::CRUD_DROP_VIEW:
      return crudh.execute_drop_view(session, static_cast<const Mysqlx::Crud::DropView&>(*command.message()));

    case Mysqlx::ClientMessages::EXPECT_OPEN:
      return on_expect_open(session, expect, static_cast<const Mysqlx::Expect::Open&>(*command.message()));

    case Mysqlx::ClientMessages::EXPECT_CLOSE:
      return on_expect_close(session, expect, static_cast<const Mysqlx::Expect::Close&>(*command.message()));
  }

  session.proto().get_protocol_monitor().on_error_unknown_msg_type();
  return ngs::Error(ER_UNKNOWN_COM_ERROR, "Unexpected message received");
}

The rest is filling in the gaps.

Client::run => Client::handle_message => Session::handle_message => Session::handle_auth_message => some auth handlers
                                                                 => Session::handle_ready_message => xpl::dispatcher::dispatch_command => ngs::Error_code do_dispatch_command => some crud handlers

Mapping between MySQL type and X protocol type

//     ================= ============ ======= ========== ====== ========
//     SQL Type          .type        .length .frac_dig  .flags .charset
//     ================= ============ ======= ========== ====== ========
//     TINY              SINT         x
//     TINY UNSIGNED     UINT         x                  x
//     SHORT             SINT         x
//     SHORT UNSIGNED    UINT         x                  x
//     INT24             SINT         x
//     INT24 UNSIGNED    UINT         x                  x
//     INT               SINT         x
//     INT UNSIGNED      UINT         x                  x
//     LONGLONG          SINT         x
//     LONGLONG UNSIGNED UINT         x                  x
//     DOUBLE            DOUBLE       x       x          x
//     FLOAT             FLOAT        x       x          x
//     DECIMAL           DECIMAL      x       x          x
//     VARCHAR,CHAR,...  BYTES        x                  x      x
//     GEOMETRY          BYTES
//     TIME              TIME         x
//     DATE              DATETIME     x
//     DATETIME          DATETIME     x
//     YEAR              UINT         x                  x
//     TIMESTAMP         DATETIME     x
//     SET               SET                                    x
//     ENUM              ENUM                                   x
//     NULL              BYTES
//     BIT               BIT          x
//     ================= ============ ======= ========== ====== ========

The first SQL field information of MySQL:

Field   1:  `@@lower_case_table_names`
Catalog:    `def`
Database:   ``
Table:      ``
Org_table:  ``
Type:       LONGLONG
Collation:  binary (63)
Length:     21
Max_length: 1
Decimals:   0
Flags:      UNSIGNED BINARY NUM 

Field   2:  `connection_id()`
Catalog:    `def`
Database:   ``
Table:      ``
Org_table:  ``
Type:       LONGLONG
Collation:  binary (63)
Length:     21
Max_length: 1
Decimals:   0
Flags:      NOT_NULL UNSIGNED BINARY NUM 

Field   3:  `variable_value`
Catalog:    `def`
Database:   `performance_schema`
Table:      `session_status`
Org_table:  `session_status`
Type:       VAR_STRING
Collation:  utf8_general_ci (33)
Length:     3072
Max_length: 0
Decimals:   0
Flags:      

For TiDB:

Field   1:  `@@lower_case_table_names`
Catalog:    `def`
Database:   ``
Table:      ``
Org_table:  ``
Type:       STRING
Collation:  ? (0)
Length:     0
Max_length: 1
Decimals:   31
Flags:      

Field   2:  `connection_id()`
Catalog:    `def`
Database:   ``
Table:      ``
Org_table:  ``
Type:       LONGLONG
Collation:  binary (63)
Length:     20
Max_length: 1
Decimals:   0
Flags:      UNSIGNED BINARY NUM 

Field   3:  `variable_value`
Catalog:    `def`
Database:   ``
Table:      ``
Org_table:  ``
Type:       STRING
Collation:  utf8_general_ci (33)
Length:     1024
Max_length: 0
Decimals:   0
Flags:      

1. Disable SeLinux
   Edit the configuration file:

   vi /etc/selinux/config
   

   Modify as follows:

   #SELINUX=enforcing    # Comment out
   #SELINUXTYPE=targeted # Comment out
   SELINUX=disabled      # Add this line
   

   Then reboot the system:

   reboot  # Restart the system
   

2. Create a Directory
   Using the root user, create a directory named /nfs. Note: It’s best to check which partition has the most space by running df, as the root (/) partition may not have the most space. In some automatic partitioning setups, the /home partition may have the most space.

3. Install NFS Utilities and RPC Bind

   yum -y install nfs-utils rpcbind
   

4. Enable Services at Boot

   chkconfig nfs on
   chkconfig rpcbind on
   chkconfig nfslock on
   

5. Configure Exports
   Edit the NFS exports file:

   vi /etc/exports
   

   Add the following line:

   /home/nfs 192.168.1.0/24(rw,sync,no_all_squash)
   

6. Start NFS Services

   service rpcbind start
   service nfs start
   service nfslock start
   exportfs -a
   

7. Configure NFS Ports
   Edit the NFS configuration file:

   vi /etc/sysconfig/nfs
   

   Uncomment the following lines:

   LOCKD_TCPPORT=32803
   LOCKD_UDPPORT=32769
   MOUNTD_PORT=892
   

8. Restart NFS Services

   service rpcbind restart
   service nfs restart
   service nfslock restart
   

9. Verify RPC Services

   rpcinfo -p localhost
   

   Note down the ports and their types.

10. Configure Firewall Rules
    Adjust the IP range according to your network:

    iptables -I INPUT -m state --state NEW -p tcp -m multiport --dport 111,892,2049,32803 -s 192.168.0.0/24 -j ACCEPT
    iptables -I INPUT -m state --state NEW -p udp -m multiport --dport 111,892,2049,32769 -s 192.168.0.0/24 -j ACCEPT
    

11. Save Firewall Rules
    Test from the client side. If successful, save the iptables configuration:

    service iptables save
    

Client Side

1. Create Mount Point

   mkdir /nfs
   

2. Check RPC Services on Server

   rpcinfo -p [server_ip]
   

3. Show NFS Exports

   showmount -e [server_ip]
   

4. Mount NFS Share

   mount -t nfs -o soft,intr,bg,rw [server_ip]:/home/nfs /nfs
   

5. Unmount NFS Share

   umount /nfs
   

6. Configure Automatic Mounting
   Edit the fstab file:

   vi /etc/fstab
   

   Add the following line:

   [server_ip]:/home/nfs /nfs nfs soft,intr,bg,rw 0 0
   

Bridging highly simulates a network card, making the router believe that the virtual machine’s network card truly exists. Personally, I feel it’s similar to resistors connected in parallel, whereas NAT (another common virtual machine network connection method) is more like parasitizing on the host’s network card.

Why Use Bridging

It allows you to treat the virtual machine as a completely independent machine, enabling mutual access with the external network (which is not possible with NAT).

How to Configure Bridging

In CentOS 6, refer to the command-line method in this article.

We don’t use the GUI method because:

• We’re unsure which options to fill in on the last screen.
• We don’t know how to reset if we make a wrong selection.

Command-line steps:

1. Check if bridge-utils is installed:

   rpm -q bridge-utils
   

   Usually, it’s already installed. If not, install it:

   su -
   yum install bridge-utils
   

2. Verify your network interfaces:

   Run ifconfig to ensure you have at least three network interfaces:

   eth0      Link encap:Ethernet  HWaddr 00:18:E7:16:DA:65
             inet addr:192.168.0.117  Bcast:192.168.0.255  Mask:255.255.255.0
             inet6 addr: fe80::218:e7ff:fe16:da65/64 Scope:Link
             UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
             RX packets:556 errors:0 dropped:0 overruns:0 frame:0
             TX packets:414 errors:0 dropped:0 overruns:0 carrier:0
             collisions:0 txqueuelen:1000
             RX bytes:222834 (217.6 KiB)  TX bytes:48430 (47.2 KiB)
             Interrupt:16 Base address:0x4f00
   
   lo        Link encap:Local Loopback
             inet addr:127.0.0.1  Mask:255.0.0.0
             inet6 addr: ::1/128 Scope:Host
             UP LOOPBACK RUNNING  MTU:16436  Metric:1
             RX packets:8 errors:0 dropped:0 overruns:0 frame:0
             TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
             collisions:0 txqueuelen:0
             RX bytes:480 (480.0 b)  TX bytes:480 (480.0 b)
   
   virbr0    Link encap:Ethernet  HWaddr 52:54:00:2A:C1:7E
             inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
             UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
             RX packets:0 errors:0 dropped:0 overruns:0 frame:0
             TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
             collisions:0 txqueuelen:0
             RX bytes:0 (0.0 b)  TX bytes:2793 (2.7 KiB)
   

3. Navigate to the network scripts directory:

   su –
   cd /etc/sysconfig/network-scripts
   

4. Bring down the eth0 interface:

   ifdown eth0
   

   This step is crucial and must be performed locally. When I first configured this, I didn’t shut down the network (since I was working remotely). I didn’t realize that updating the ifcfg-eth0 configuration without restarting the network would immediately apply changes, resulting in loss of network connectivity.

5. Edit ifcfg-eth0:

   In the ifcfg-eth0 file, include:

   DEVICE=eth0
   ONBOOT=yes
   BRIDGE=br0
   

   Keep only these three lines in the file. There’s no need to configure an IP address here. Bridging seems to replace the original network card with the bridge, so you can delegate the configuration to the bridge.

6. Create a new file ifcfg-br0:

   DEVICE=br0
   ONBOOT=yes
   TYPE=Bridge
   BOOTPROTO=static
   IPADDR=xxx.xxx.xxx.xxx   # Use the IP you originally had in ifcfg-eth0
   GATEWAY=xxx.xxx.xxx.xxx  # Your gateway address
   NETMASK=255.255.255.0    # Your netmask
   DNS1=xxx.xxx.xxx.xxx     # Your primary DNS server
   DNS2=xxx.xxx.xxx.xxx     # Your secondary DNS server (if any)
   STP=on
   DELAY=0
   

   Note: Replace xxx.xxx.xxx.xxx with your actual network settings.

7. Bring up the interfaces:

   ifup br0
   ifup eth0
   

8. Verify the bridge interface:

   Check ifconfig to ensure that br0 is now present.

9. Update firewall rules:

   Edit /etc/sysconfig/iptables and add:

   -A INPUT -i br0 -j ACCEPT
   

   (This is a general example; you may need to adjust it based on your specific firewall configuration.)

10. Restart the firewall:

    service iptables restart
    

11. Configure bridging in virt-manager:

    When creating a new virtual machine using virt-manager, you can now select br0 for the network interface. Without this bridge, the bridging option would not be available.

Note: When configuring the IP inside the virtual machine, be sure to specify the GATEWAY. Otherwise, the virtual machine will only be able to access the internal network and not the external network. At this point, the virtual machine won’t automatically discover the gateway.

Installed Version: CentOS 6.3

1. Using Win32DiskImager to create a USB flash drive image was unsuccessful; installing from an external USB optical drive was successful.
2. During the installation process, make sure to select the “Virtual Host” installation mode.
3. The rest can be set to default or slightly modified, such as choosing the time zone.
4. After installation, it will include the KVM suite and SSH.

Installation Notes

• No internet connection is needed throughout the process, which is much better than Debian and Ubuntu.
• You’re not forced to set up a non-root user.
• Before installation, be sure to check whether your CPU supports virtualization and enable the motherboard’s virtualization setting. If the motherboard supports virtualization but doesn’t have a virtualization option, you can still use virtualization as it’s definitely enabled by default. There’s a saying that Intel CPUs with a ‘K’ cannot perform virtualization. ‘K’ means Intel CPUs that can be overclocked. It seems that faster and newer CPUs are not necessarily better.

Database Structure

The database consists of four tables:

1. STUDENT: Contains student number (SNO), name (SNAME), gender (SSEX), birthday (SBIRTHDAY), and class (CLASS).
2. COURSE: Includes course number (CNO), name (CNAME), and teacher number (TNO).
3. SCORE: Records student number (SNO), course number (CNO), and degree (DEGREE).
4. TEACHER: Holds teacher number (TNO), name (TNAME), gender (TSEX), birthday (TBIRTHDAY), professional title (PROF), and department (DEPART).

Sample Data

• Students such as Zeng Hua, Kang Ming, and Wang Fang are stored with specific details, including their class and gender.
• Courses like “Introduction to Computers” and “Operating Systems” are associated with teacher numbers.
• Scores are recorded for students across various courses.
• Teachers are described with their professional roles and departments.

Query Problems

Several SQL queries are suggested for practice, such as:

• Extracting specific columns like SNAME, SSEX, and CLASS from the STUDENT table.
• Listing distinct departments for teachers.
• Calculating and sorting grades within the SCORE table.
• Performing database operations to find student averages, count of students per class, and comparing scores.

Advanced Query Exercises

• Performing set operations and conditional joins to answer complex questions like finding students who scored more than others or comparing teachers’ scores.
• Use of SQL functions like DATE_PART, subqueries, and unions to gather specific data.

Additional Queries

• Techniques to refine queries for performance, like avoiding the NOT IN method.
• Handling conditions like age calculations using AGE(SBIRTHDAY) and filtering by name patterns.

Overall, these exercises provide a robust framework for practicing SQL skills on a structured set of sample data, focusing on various database manipulation and retrieval techniques.7. Query:

- `SELECT A.TNAME, B.CNAME FROM TEACHER A JOIN COURSE B ON A.TNO = B.TNO WHERE A.TSEX='男';`
- Explanation: Joins teacher and course tables to select male teachers and their course names.

1. Query:

   • SELECT A.* FROM SCORE A WHERE DEGREE=(SELECT MAX(DEGREE) FROM SCORE B);
   • Explanation: Selects all columns from the highest score in the score table.

2. Query:

   • SELECT SNAME FROM STUDENT A WHERE SSEX=(SELECT SSEX FROM STUDENT B WHERE B.SNAME='李军');
   • Explanation: Selects student names who have the same gender as the student named ‘Li Jun.’

3. Query:

   • SELECT SNAME FROM STUDENT A WHERE SSEX=(SELECT SSEX FROM STUDENT B WHERE B.SNAME='李军') AND CLASS=(SELECT CLASS FROM STUDENT C WHERE C.SNAME='李军');
   • Explanation: Selects student names who have the same gender and class as the student named ‘Li Jun.’

4. Two Answers:

   • SELECT A.* FROM SCORE A JOIN STUDENT B ON A.SNO = B.SNO JOIN COURSE C ON A.CNO = C.CNO WHERE B.SSEX='男' AND C.CNAME='计算机导论';
   • SELECT * FROM SCORE WHERE SNO IN(SELECT SNO FROM STUDENT WHERE SSEX='男') AND CNO=(SELECT CNO FROM COURSE WHERE CNAME='计算机导论');
   • Explanation: Both queries select scores of male students for the course ‘Introduction to Computer Science.’