It is not trying to solve music generation all at once. It is trying to make the loop inspectable.

Why symbolic MIDI

Audio generation is expensive and hard to inspect. MIDI is smaller, structured, and easier to evaluate. For early experiments, that matters.

The project starts with controls such as:

• mood
• energy
• density
• brightness

Those controls are simple, but they are useful enough to test whether the model responds to conditioning at all.

Training path

The first phase can generate toy piano MIDI without external datasets. It then tokenizes MIDI events, trains with MLX on Apple Silicon, and writes checkpoints plus training logs.

The repo also includes a MAESTRO import path. Since MAESTRO does not have mood labels, the importer creates heuristic weak labels from features such as tempo, density, velocity, silence, repetition, and pitch range.

That is the right level of honesty for the project: the labels are weak labels, not ground truth.

Generation and preview

Generated tokens can be exported back to MIDI. The project also has a local UI server that can generate and download MIDI files, with an optional audio-rendering path using FluidSynth and a local SoundFont.

This makes the experiment audible without turning it into a full production app.

What I learned

For music ML projects, the training loop is only one part of the system. The useful loop includes:

• dataset construction
• tokenization
• checkpointing
• generation controls
• structural evaluation
• audible preview

If any part is missing, it becomes hard to tell whether the model is improving or just producing plausible-looking tokens.

Current status

mood-midi-mlx is private and experimental. It is a research workbench for Apple Silicon rather than a finished listening product. That separation is useful: mood-midi-mlx can explore models, while driftloop can focus on a coherent listening experience.

The important constraint is that every note is composed and synthesized live in the browser. There are no prerecorded loops, no server, no account, and no backend generation job.

Why it is not mainly an AI app

The interesting origin story is that driftloop came after an earlier model-first attempt. That earlier direction tried to distill a multi-instrument music model and sample from it for the listening experience. The output was structurally chaotic in a way that sampling tweaks did not fix.

driftloop starts from a different premise: for continuous background music, structure matters more than model capacity.

Algorithmic composition handles the load-bearing parts:

• chord progression
• voice leading
• bass motion
• rhythm
• drum pattern
• section shape

A small model can still help with lead melody, but it should not be responsible for making the whole piece coherent.

Runtime shape

The app has two layers:

1. Vanilla JavaScript composition logic that keeps the music in key, on grid, and structurally coherent.
2. An optional MelodyRNN path through TensorFlow.js for lead melody.

Synthesis uses js-synthesizer, a WebAssembly port of FluidSynth, with a General MIDI SoundFont through the WebAudio API.

The SoundFont is large, so deployment has a practical constraint: Cloudflare Pages has a single-file size limit smaller than the SoundFont. The app fetches the SoundFont from a public upstream mirror on first launch, then the service worker caches it for offline reuse.

What I learned

Generative music is not only generation. It is arrangement, continuity, transitions, repetition management, and sound rendering.

The most important product question is not “is it AI?” It is “can I leave this playing without getting annoyed?” That favors simple, controllable structure over model novelty.

Current status

driftloop is private but product-shaped. It is a static PWA candidate, and the current architecture is intentionally deployable without backend secrets or runtime infrastructure.

That referee boundary is the most important design choice. The game engine holds the authoritative state and decides what each seat is allowed to see. The AI players only receive projected views.

Why the browser is the referee

Werewolf is an information game. The system must know the full state, but each player must see only a slice of it. If the AI seat receives hidden information by accident, the whole experiment is meaningless.

Nightfall keeps that boundary explicit:

• engine/ owns state, phase transitions, view projection, resolution, victory checks, and deterministic replay
• orchestrator/ runs the environment-independent game loop
• agentd-client/ calls POST /v1/turns
• app/ renders the spectator UI without owning game rules

The same engine can drive a Node CLI transcript or the browser spectator.

agentd integration

Nightfall does not embed agent logic. It calls an external agentd runtime. The werewolf, seer, and villager agents live on the agentd side as personas and model choices. Nightfall only maps roles to agent_ref values and sends the projected view for the current seat and phase.

That separation matters. The game should own rules; agentd should own agent execution.

Game shape

The current board is intentionally small:

• 6 seats
• 2 wolves
• 1 seer
• 3 villagers
• no role reveal on death
• loop: night seer, night wolf, day discussion, day vote

Victory is checked after deaths: wolves eliminated means good wins; all villagers dead means wolves win. Killing the seer does not end the game, it only removes the good side’s information source.

What I learned

Games are useful agent testbeds because they force state discipline. A chat bot can hide vague behavior behind language. A game cannot. The engine needs phases, legal actions, projected views, victory rules, and reproducible transcripts.

Nightfall is also a reminder that a good agent integration is often just a clean boundary. The AI seat should not know the whole game. It should know what that player would know.

Current status

Nightfall is private and experimental. It is useful as an AI interaction lab, especially for testing information hiding, role-specific personas, and multi-agent turn orchestration.

The useful distinction is that Canopy does not try to be a map app. It is a route-planning capability.

Why this exists

Singapore cycling routes are not only about shortest path. A good route often means:

• prefer PCN-like paths
• avoid high-car roads
• keep loops close to a target distance
• export a track that can be followed elsewhere
• explain how much of the route is car-free or on-road

General routing APIs are not always good at expressing those preferences. Canopy uses a self-hosted GraphHopper service with a custom bike profile that strongly prefers cycleways, PCN-like paths, and low-car links while penalizing motor roads.

OneMap is used for geocoding and POIs, not routing.

Tool surface

The MCP server exposes tools such as:

• geocode(query)
• plan_loop(start, distance_km, direction?, prefer="pcn")
• route(origin, destination, prefer="pcn")
• pois_along(geometry, types=[...])
• audit_route(geometry)
• export_gpx(geometry, name)

That gives an agent a clean workflow: resolve a place, plan a route, audit the result, find POIs along the corridor, and export GPX.

Honest boundaries

Canopy optimizes for low-car and away-from-traffic routing. It does not avoid pedestrians, because Singapore PCNs are shared paths.

Round-trip distance is also approximate in GraphHopper. Canopy tries multiple seeds and scaled requests, then returns the actual distance instead of pretending the target was exact.

The project also intentionally avoids building turn-by-turn navigation. A downstream app can load the generated GPX track into something like OsmAnd or CoMaps.

What I learned

MCP is a good fit when the tool is specific and verifiable. Route planning should not be hidden inside a chat answer. The agent should call a tool that returns geometry, distance, audit percentages, and a GPX path.

For location work, the agent’s job is orchestration. The route engine should still be a route engine.

Current status

Canopy is private and active. It is tied to local Singapore routing data, OneMap credentials, and a local GraphHopper runtime, so the repo is not a clean general-purpose hosted service.

It is a small application, but it has the shape of a real product: backend data fetching, caching, scoring, frontend display, and deployment.

Why build it

Fishing decisions depend on several signals:

• tide timing
• weather
• wind
• rain
• forecast window
• saved locations

Generic weather apps show the raw pieces, but they do not answer the combined question I care about. A personal dashboard can make that tradeoff directly.

Architecture

The backend is a Cloudflare Worker. It fetches marine/tide data and weather forecasts, computes a score, and caches results in Workers KV.

The frontend is an Astro and React app deployed to Cloudflare Pages. Shared TypeScript code keeps constants and data shapes aligned.

This is a good fit for Cloudflare because the workload is small, read-heavy, and cacheable.

Privacy and scope

The app is intentionally personal and ad-free. It does not need accounts, social features, or public rankings. It needs to answer a small question reliably.

The main external dependency risk is data quality and API availability. That makes caching and clear failure states important.

What I learned

Personal dashboards are worth building when they compress repeated decision-making. The value is not that the app is complicated. The value is that it turns several tabs and mental calculations into one screen.

The scoring model should also stay humble. A “good fishing day” score is a decision aid, not a guarantee. The app should make the inputs visible enough that I can override the score with judgment.

Current status

good-fishing-day is private because it is tied to personal usage and deployment details. The architecture pattern is reusable: small Worker backend, cached external data, static frontend, and a domain-specific score.

The interesting constraint is that it aims to use open data and avoid API keys where possible.

Why maps are hard for language models

Map questions sound simple:

• find coffee near me
• show quiet parks nearby
• route to a place with shade
• find restaurants within walking distance

But those questions mix language, location, distance, categories, freshness, and user context. A language model can understand the request, but it should not invent geography.

The system needs tools.

Open-data shape

The project uses sources such as OpenStreetMap, Nominatim, Overpass, Wikimedia REST, and related open endpoints. The map UI is built around Leaflet and React.

The model’s role is to translate intent into geospatial operations:

• what is the user looking for?
• what radius or location matters?
• is this a route, POI search, or environment query?
• what enrichment should be added?

The tools then fetch actual data.

Product boundary

The app should be honest about uncertainty. Open geospatial data is uneven. Names change, POI tags are inconsistent, and “quiet” or “hipster” are semantic hints rather than guaranteed database fields.

That means the UI should show results as interpreted matches, not absolute truth.

What I learned

Natural-language interfaces work best when they produce inspectable structured plans. If the app can show the interpreted action, it becomes much easier to debug:

• searched category
• center point
• radius
• data source
• filters
• enrichment steps

This is especially important for maps because wrong answers can look visually convincing.

Current status

map-ask is private and prototype-stage. It is safe to discuss as an architecture pattern: use an LLM for intent shaping, then rely on real geospatial tools for facts.

The whole point is that it should keep working with no network.

Why build it

Airplane mode is a useful product constraint. It removes a lot of lazy assumptions:

• no backend calls
• no CDN dependency during play
• no login
• no remote save
• no ads
• no analytics requirement

What remains is the app itself: local assets, local state, and games that are worth playing in short sessions.

Template shape

AeroPlay is also designed as a template. The repo can be used to bootstrap another offline game hub without carrying old issues or unrelated history.

That makes the project more useful than a single app. It becomes a base pattern:

• Vite and TypeScript
• local game modules
• service worker
• manifest
• localStorage progress
• mobile-first layout

Product boundary

The project should not become a general gaming platform. Its strength is the opposite: a small set of local games, predictable input, and no network requirement.

For this kind of app, “more features” can easily make the experience worse. If the user opens it in flight mode, the app should not show half-broken online features.

What I learned

Offline-first is easiest when it is a requirement from the beginning. Retrofitting offline behavior onto an app that assumed servers and remote assets is much harder.

AeroPlay also shows that simple games still need polish:

• touch controls
• stable layout
• saved state
• fast startup
• no accidental scroll
• clear pause and restart behavior

Those details matter more on a phone than in a desktop demo.

Open source status

AeroPlay is public because it is self-contained, harmless to share, and useful as a template. It is a good example of a small project whose constraints make it more reusable.

The project is not meant to be a large mod. It is a rules sandbox.

What it changes

The pack includes features such as:

• sprint regeneration
• random player effects
• torch drops on non-player entity death
• local snow effect on player death
• random material drops from broken blocks
• adjusted fishing loot weights

These are intentionally playful mechanics. The point is to test how small rule changes affect the feel of a world.

Why behavior-only matters

Keeping the pack behavior-only lowers the setup cost. There is no custom client asset pipeline and no experimental toggle requirement.

That makes it easier to share across devices:

• package as .mcpack
• import on iOS or Windows
• activate in world settings
• play immediately

For a small personal pack, install friction matters more than architectural elegance.

The design lesson

Game modding is a good reminder that systems can be fun without being complicated. A simple rule like “breaking blocks produces random materials” changes the entire resource economy.

The engineering lesson is to make each rule easy to remove. If a mechanic stops being fun, it should not be tangled with the rest of the pack.

What I learned

Small game experiments should be packaged early. A mechanic that only works in a local development folder is not really tested. It needs to be imported, activated, and played in the actual environment.

Bedrock Lab’s value is that it turns ideas into a package quickly. That makes it easier to decide whether a mechanic is worth keeping.

Current status

Bedrock Lab is private and small. It is the kind of project that is more useful as a personal sandbox than as a maintained public mod.

The project is small, but the architectural jump is large.

Why add a server

An embedded database can keep many assumptions local:

• one process owns access
• file paths are local
• query state is in-process
• errors are library errors
• clients are application code

A server changes the boundary. Now there are connections, protocol messages, authentication questions, configuration, network errors, allocation pressure, and client compatibility expectations.

That is why this project exists: to study that boundary.

PostgreSQL compatibility

PostgreSQL compatibility is useful because it gives the project a familiar client surface. If a normal client can connect, the database becomes easier to test and demonstrate.

But compatibility is also a trap. The PostgreSQL ecosystem expects a lot:

• protocol behavior
• type handling
• error responses
• transaction semantics
• SQL behavior
• metadata queries

The right goal for a small project is progressive compatibility, not pretending to be PostgreSQL.

Storage boundary

tegdb-server keeps TegDB as the storage engine and adds a server layer in front. That separation is useful:

• TegDB owns on-disk behavior
• the server owns connections and protocol handling
• configuration stays explicit
• future distribution experiments have a place to attach

The project README frames this as progressive scalability from single-node to more distributed deployment shapes. That is ambitious, but it starts with a concrete first step: one server in front of one storage engine.

What I learned

The protocol path has different performance concerns from the storage path. Per-query allocations, message parsing, response construction, and connection lifecycle can dominate before the storage engine becomes interesting.

This is a useful reminder: a database is not only its storage engine. It is also a wire protocol, client contract, operations surface, and set of compatibility promises.

Open source status

tegdb-server is public because it pairs naturally with the public TegDB experiment. It is best read as a learning project about database server boundaries.

This is the kind of project where the most useful public topic is not the trading strategy. It is the guardrail design.

Why risk first

Trading code has a dangerous failure mode: a bug can become a position.

That changes the engineering priority. Before clever strategy logic, the system needs operational controls:

• dry-run mode
• explicit venue adapters
• heartbeat checks
• halted state
• read-only mode
• data freshness checks
• position and exposure constraints
• clear separation between research and live execution

The project name reflects that priority. The guard matters before the CTA.

System shape

The maintained path is a modular trader stack:

• runtime orchestration
• risk app service
• strategy app service
• backtest app service
• domain models and ports
• SQLite repositories
• venue adapters and stubs

There are separate binaries for live or dry-run runtime, health checks, ingestion, historical import, backtest, and walk-forward validation.

That separation is important. A backtest binary should not have the same operational authority as a live execution process.

What should stay private

Some details are not appropriate for a public blog:

• strategy parameters
• production configuration
• account or venue details
• exact deployment topology
• operational thresholds
• anything that could imply a trade recommendation

The safe public layer is architecture: how to keep execution constrained, observable, and stoppable.

What I learned

Risk control is not one module. It is a state machine that should shape the whole runtime.

A good trading system needs to answer boring questions quickly:

• is data fresh?
• is the venue reachable?
• is the runtime allowed to trade?
• what happens after an error?
• can the system stop without making things worse?
• can research code accidentally reach execution?

If those answers are unclear, the system is not ready for live use.

Current status

cta-guard is private. I may continue writing about its engineering lessons, but only at the level of risk design, runtime isolation, and validation workflow.

The most useful lesson from the project is not a feature. It is how quickly storage complexity expands.

What it explores

The codebase touches several storage-system ideas:

• transactional KV
• page-based B+tree structures
• secondary indexes
• WAL encoding and replay
• embedded compute paths
• quorum commit experiments
• RPC schema generation
• redb and sled comparisons
• performance baselines

Each of these is reasonable on its own. Together, they create a system that needs strong boundaries to stay understandable.

Why this is hard

Storage systems are full of cross-cutting constraints:

• write durability affects read visibility
• page layout affects compaction and recovery
• transactions affect indexing
• indexes affect write amplification
• replication affects latency and failure handling
• observability affects whether any benchmark can be trusted

It is easy to add one more subsystem and hard to know when the conceptual model has become too large.

Documents as part of the project

One good part of scale-kv is that it includes design notes: API drafts, capacity models, fault drills, observability notes, and performance snapshots.

For this kind of experiment, documents are not decoration. They are the only way to keep track of what the system is supposed to prove.

What I learned

The project reinforced a simple rule: decide what a storage experiment is testing before building the next layer.

If the goal is page structure, do not also invent distributed commit. If the goal is quorum behavior, use the simplest storage backend that can support the test. If the goal is API design, avoid mixing it with WAL recovery work.

That does not mean the project failed. It means the project taught the lesson it was supposed to teach: complexity is the main resource being spent.

Current status

scale-kv is private and experimental. It may remain a lab rather than a product, which is a valid outcome for a systems project.

The project is deliberately prototype-shaped. It asks what an analytics stack looks like when browser or WASM compute is treated as a real part of the system rather than only a demo surface.

Components

The local stack is split into small services:

• API control plane
• HTTP range gateway over object storage
• Node runner using DuckDB-WASM
• browser query UI
• MinIO for S3-compatible storage

That separation makes the data flow easier to inspect:

1. data lives in object storage
2. range requests expose only needed bytes
3. DuckDB-WASM runs queries
4. lineage or job metadata can be tracked by the control plane

Why DuckDB-WASM

DuckDB is already a strong fit for local analytics. DuckDB-WASM adds a different deployment shape: query execution can move closer to the browser, a worker, or a constrained runtime.

The attraction is not that every analytics job should run in the browser. The attraction is that the compute boundary becomes flexible.

What the prototype tests

QuackLake is useful for testing system questions:

• how painful are range requests in practice?
• where should metadata live?
• what should the API own?
• when does browser compute become awkward?
• how much can object storage simplify the backend?

These are architectural questions, not UI questions.

What I learned

WASM analytics is promising, but the hard parts move around. Query execution can be surprisingly capable, while data access, file layout, caching, and observability become the real design surface.

The range gateway is a good example. It sounds like plumbing, but it shapes whether the system feels smooth or fragile.

Current status

QuackLake is private and experimental. I would not present it as a finished product. Its value is as a small lab for object-storage-first analytics and DuckDB-WASM deployment patterns.

The project exists because raw prompting is not enough for repeated technical work.

What is inside

The repo contains skills such as:

• OceanBase solution architect guidance
• OceanBase DDL syntax checking
• seekdb-backed DDL validation

Each skill tries to narrow the agent’s behavior for a specific kind of work. Instead of asking a general model to “help with OceanBase,” the skill defines what kind of answer is expected, which constraints matter, and which validation path should be used.

Why skills instead of notes

Traditional notes help humans. Skills help agents act more consistently.

That difference matters. A good skill should:

• define the task boundary
• mention common failure modes
• route to validation where possible
• make assumptions explicit
• avoid generic advice when concrete checks exist

For database work, this is especially useful because the cost of vague answers is high. A DDL suggestion that sounds plausible but violates grammar or deployment constraints is worse than no suggestion.

The useful pattern

The best skills are not huge knowledge dumps. They are small operational wrappers around repeatable judgment.

For example, a DDL validation skill should not merely explain DDL. It should push the agent toward an actual validation path and make clear when offline validation is possible.

That is the difference between “LLM as autocomplete” and “LLM as a guided operator.”

What I learned

Agent quality improves when domain knowledge is packaged close to the workflow. It is not enough to rely on the model remembering a product correctly.

Skills are also easier to maintain than long prompts embedded in many tools. If a rule changes, update the skill. If a validation command improves, update the skill. The agent workflows that use it inherit the better behavior.

Open source status

ob-skills is public because the project is mostly guidance and validation workflow. It is a good fit for sharing: others can inspect the assumptions, reuse the structure, or adapt the skill pattern to their own database work.

The project is not trying to compete with SQLite. It is a learning and systems-design project: what happens when you build the parts of a small database yourself?

Design philosophy

The most important choice is simplicity. TegDB uses a deliberately constrained architecture:

• embedded engine
• single-process access
• write-ahead logging
• crash recovery
• B+tree-style indexing
• schema and constraint checks
• SQL-like query surface

Avoiding broad concurrency is not a weakness for this kind of project. It reduces the number of failure modes and makes the storage path easier to reason about.

Why build a database

Database internals are easier to understand when you have to implement the boring parts:

• how records are serialized
• how pages are found
• how an index points to storage
• when a transaction becomes durable
• how recovery decides what to replay
• how a query stops early with LIMIT

Reading database code is useful. Writing a small one makes the tradeoffs harder to ignore.

Reliability work

The most interesting code in a database is often not the happy path. TegDB spends attention on:

• write-ahead log commit markers
• rollback behavior
• partial write handling
• file locking
• recovery after crash
• corruption boundaries

These are the parts that turn “data structure on disk” into “database.”

What I learned

The database abstraction is expensive because it hides many coordinated guarantees. Even a small embedded engine has to care about durability, serialization, schema behavior, query planning shortcuts, and operational failure.

The lesson is not that every app should have a custom database. The lesson is the opposite: after building even a small one, you gain more respect for mature storage engines.

Open source status

TegDB is public because the project is mainly educational and architectural. It is useful as a record of experiments in storage design, not as a claim that it should replace established embedded databases.

That is a different direction from the first version of this note, which described agentd mainly as a Wasm-native harness. The pushed version is now more concrete: an Agent is a manifest with a system prompt, visible tool catalog, resource limits, and a model choice. The actual handler is the built-in generic agent.

Turn API

A caller submits:

POST /v1/turns

with a tenant, agent_ref, scope, and payload. The runtime creates an AgentRun, resolves a lane, assembles the visible tools, invokes the generic agent, and records a structured final_decision.

There is no push side. A Telegram bot, a browser game, or another transport bridge should call the API and render the result itself. That keeps agentd from becoming coupled to a specific messaging system.

Capability surface

The current public tool surface includes:

• plan.*
• run.*
• schedule.*
• artifact.*
• memory.*
• context.*
• web.*
• clock.*
• llm.*
• output.*

The important pattern is that tools carry reliability metadata: maturity, idempotency, side-effect scope, failure classes, replay expectations, and delivery semantics. That metadata is part of making agent behavior inspectable instead of magical.

Storage model

Durable state now lives in SeekDB, a MySQL-compatible OceanBase-derived database. Control-plane state, context, artifacts, and memory all use that one database boundary. Memory is per tenant and uses native vector plus full-text retrieval.

This is a pragmatic simplification. There is no MinIO, S3, Litestream, or separate object store in the core path.

What I learned

Agents need less magic and more runtime discipline. Even a personal agent runtime has to answer boring operational questions:

• context scopes matter
• retries can duplicate work
• memory needs indexing and pruning
• tools need permission boundaries
• failures need artifacts and logs
• callers need a stable way to wait for output

The useful boundary is no longer just Wasm. It is the turn contract plus explicit host capabilities.

Current status

agentd is experimental and private. It is useful as a lab for runtime shape, but I do not consider it a public platform yet. The public lesson is the architecture: keep transports outside the core runtime, keep tool capabilities explicit, and make every run produce inspectable output.

It is a pure frontend tool. All calculations run in the browser, and the estimate can be shared through encoded URLs.

Why build it

Sizing discussions often begin with incomplete information:

• source database type
• data size
• growth expectations
• workload style
• read/write ratio
• availability requirement
• target deployment shape

The first estimate is rarely final. But without a structured model, the conversation quickly becomes a spreadsheet with hidden assumptions.

OB Sizer makes the assumptions visible.

Supported shape

The tool covers common migration sources:

• MySQL
• Oracle
• PostgreSQL
• TiDB
• Aurora
• SQL Server

It models OceanBase replica choices such as 2F1A and 3F, real OB Cloud SKU shapes, and workload differences across OLTP, HTAP, and mixed patterns.

The important design choice is that it is not only a calculator. It includes methodology in the app, so users can see why a number appears.

Product boundary

OB Sizer should not pretend to replace a real sizing engagement. The correct role is earlier:

• make first-pass estimates consistent
• expose assumptions
• help compare scenarios
• create a common language for follow-up questions

That boundary keeps the tool useful without making it misleading.

Why browser-only

A browser-only tool is easy to share and easy to run in customer-facing discussions. It also avoids collecting workload information on a backend.

The tradeoff is that the model and SKU tables have to ship with the app. That is acceptable because the tool is a discussion aid, not a real-time pricing or provisioning system.

What I learned

Capacity estimation tools are part engineering and part communication. A technically clever formula is not useful if the user cannot see its assumptions.

The best parts of OB Sizer are the ones that make uncertainty explicit:

• workload multipliers
• replica mode impact
• CPU and memory assumptions
• storage overhead
• write amplification
• safety margins

For database migration work, that transparency matters more than pretending the first number is exact.

The product sounds tiny: count taps for 10, 30, or 60 seconds. The interesting part is making that count credible.

Why a custom tool

Most tap-speed apps are built for casual play. Competition use has different requirements:

• work offline
• run well on an iPad
• avoid accidental browser gestures
• count multi-finger input correctly
• preserve enough raw data to audit a run
• keep history locally without accounts

For this use case, simplicity matters more than visual novelty.

Input model

TapSurge uses pointerdown and active pointer tracking. The key rule is that holding a finger down should not count repeatedly. A tap is a new press, not a frame, animation event, or repeated touch state.

The app records:

• total taps
• average CPS
• one-second live CPS
• maximum live CPS
• maximum simultaneous fingers
• raw timestamps and pointer IDs

That raw data is important. It lets a run be inspected after the fact instead of trusting only the final number.

Offline design

The app is intended to work in a PWA-like mode. It has a local app shell, a service worker, and no backend dependency during play.

This is one of those projects where offline support is not a nice extra. It is part of the user story. If the tool is used at a booth, in a classroom, or during a small event, network dependency is unnecessary risk.

UI constraints

The UI has to be obvious under pressure. A player should not be reading instructions while the timer is running.

The controls are intentionally few:

• choose duration
• start
• tap
• see result
• review history
• export evidence if needed

The harder work is preventing the browser from interfering: double-tap zoom, scroll gestures, safe-area issues, and fullscreen behavior all matter more on a real iPad than in a desktop browser.

What I learned

Small event tools benefit from being boring. The best version is not the most animated one; it is the one that produces trustworthy results with minimal setup.

TapSurge is a good example of a private tool that could be public, but does not need a large roadmap. Its scope is the feature.

The idea is simple: keep local inference usable by normal OpenAI SDK clients.

Why this exists

Local models are most useful when they can plug into existing tools. A model that only works through a special command is interesting, but a model that looks like an OpenAI-compatible service can be used by editors, agents, scripts, and gateways.

That is the purpose of qwen-local. It is not a model research project. It is an adapter and runtime boundary.

The default shape is:

• MLX for Apple Silicon inference
• Qwen for chat
• Qwen embeddings
• Qwen3 TTS through mlx-audio
• MLX Whisper for speech-to-text
• one /v1 API surface

Once models are cached, inference should not require external API calls.

The useful constraint: 16 GB

The project targets a realistic personal machine rather than a workstation with huge memory. That constraint forces decisions:

• prefer quantized models
• keep concurrency conservative
• avoid loading every capability eagerly if it hurts responsiveness
• make stuck inference and runtime locks visible

Local AI services fail in different ways from hosted APIs. A hosted provider returns rate-limit errors or provider errors. A local process can get memory pressure, model load stalls, file cache problems, or long single-user queues.

That makes operational behavior part of the product.

Relationship to tailgate

qwen-local is the local model service. Tailgate is the gateway that decides when to use it.

Keeping those roles separate matters. The local service should focus on model loading, request compatibility, and media endpoints. The gateway can handle policy, provider selection, fallback, and external clients.

That split keeps qwen-local from becoming a general AI router.

What I learned

OpenAI-compatible does not mean full OpenAI clone. The useful target is compatibility for the clients I actually use:

• chat completions
• embeddings
• speech generation
• transcription
• predictable model IDs
• normal error shapes where possible

The second lesson is that local inference needs a health model. It is not enough to expose an endpoint. I need to know whether the service is loaded, busy, stuck, or unavailable, especially when another tool is routing requests into it.

June 2026 update

The pushed version has moved the local speech stack from Kokoro to Qwen3 TTS. The default TTS model is now mlx-community/Qwen3-TTS-12Hz-0.6B-CustomVoice-4bit, served through mlx-audio with local-tts as the API model alias. The default voice is vivian, and "default" maps to that voice.

This keeps the project aligned around Qwen for chat, embeddings, and speech while still keeping Whisper as the ASR backend. It also reinforces the single-worker assumption: chat, embeddings, TTS, and ASR all share one serialized local inference worker so the Mac does not get overloaded by parallel requests.

Open source status

This project is private because it includes local operational assumptions and is tuned for my own machine. The general pattern is public enough to discuss: make local models boring by putting them behind familiar API contracts.

It is not meant to be a public model marketplace. The point is not to replace OpenRouter or any other provider. The point is to make my own AI workflow less scattered.

The problem

Once you use multiple model providers, the configuration spreads quickly:

• local model endpoint
• hosted model provider keys
• fallback behavior
• model names
• pricing assumptions
• tool-specific environment variables
• different capabilities for chat, embeddings, speech, and transcription

Every client wants a slightly different setup. That is annoying for normal use and worse for agents, because agent configuration should be boring and repeatable.

Tailgate puts that complexity behind one OpenAI-compatible surface.

Design shape

The core API follows familiar endpoints:

• GET /v1/models
• POST /v1/chat/completions
• POST /v1/embeddings
• POST /v1/audio/speech
• POST /v1/audio/transcriptions

Behind that surface, the gateway can route requests to local qwen-local, DeepSeek, OpenRouter, or future compatible providers. It tracks provider health, supports streaming passthrough, and can apply simple route selection rules.

The most useful rule is not fancy AI logic. It is policy:

• prefer local when the task fits
• keep secrets off client machines
• avoid sending private work to external providers accidentally
• fall back only when the route explicitly allows it

Why private

Tailgate contains too many assumptions about my own environment to be a clean open source project. It is shaped around private networking, provider credentials, model preferences, and operational defaults.

The public lesson is still useful: an AI gateway does not need to start as a large platform. For one person, it can simply be a policy boundary.

What I learned

The biggest value of a gateway is not only key management. It is reducing mental overhead.

Before the gateway, every tool needed to know too much. After the gateway, tools only need:

• one base URL
• one API key or private network policy
• normal OpenAI-compatible request shapes

That makes experiments cheaper. I can change the provider map without editing every client.

The second lesson is that local models need protection. A small local model service may only handle one heavy inference at a time. A gateway can enforce concurrency and fallback rules so clients do not accidentally overload the local runtime.

June 2026 update

The pushed version has a clearer premium route. premium/chat now prefers the configured DeepSeek premium model before falling back to the configured OpenRouter premium model. Config order is also used as a tiebreaker when candidates have the same price ranking, which keeps routing behavior predictable instead of surprising.

This is a good example of the gateway’s real job: not to be clever, but to make model policy explicit. A client can ask for premium/chat; Tailgate decides which concrete upstream model should satisfy that tier.

Current status

Tailgate is active and private. I expect it to stay private unless the configuration model becomes generic enough to be useful outside my own setup.

The important constraint is that there is no backend inference service. The browser is not just the UI; it is also the runtime.

Why build it

Most translation tools are service-shaped: text goes to a server, the server runs a model, and the result comes back. That is fine for many cases, but it is also more infrastructure than I wanted for a narrow personal tool.

PocketBabel asks a smaller question: if the scope is only English to Chinese and Chinese to English, can the whole product be a static site?

That decision makes the system easier to reason about:

• no API key
• no account system
• no backend deployment
• no request logging
• no server-side scaling problem

The cost is that the first model download matters and browser performance becomes part of the product.

Product boundary

The project intentionally avoids becoming a general translation platform. The current shape is narrow:

• English to Chinese
• Chinese to English
• text input and output
• desktop and mobile browser support
• PWA shell and offline reuse

It does not try to support OCR, speech input, synced history, arbitrary language pairs, or collaborative workflows. Those features are tempting, but they would change the project from a useful small app into a maintenance surface.

Technical shape

The implementation is a React and Vite app. Translation is handled by transformers.js with browser-side model loading and caching. Deployment is static, which makes Cloudflare Pages a natural fit.

The UI challenge is not only making a textarea and a button. The app needs to explain model state without becoming noisy:

• model not downloaded yet
• model loading
• translation running
• offline reuse available
• failure state when the browser cannot support the runtime well

For an app like this, good status text is part of the architecture. If users do not know whether the model is downloading, warming up, or stuck, the app feels broken even when the code is doing the right thing.

What I learned

Browser AI is practical when the product scope is narrow. It is much less practical when you pretend the browser is a free replacement for a server.

The useful pattern is:

1. choose a task with a clear input and output
2. constrain the model set
3. make loading and caching behavior visible
4. avoid account and sync features unless they are essential

PocketBabel works because it does not try to be Google Translate. It is a small translation surface for a small set of language directions.

June 2026 update

The pushed version has grown in two small but useful directions without changing the core product boundary.

First, short word or phrase inputs can now use dictionary-style prompts. That makes PocketBabel more useful for vocabulary lookup: instead of treating every short input as a sentence, the app can ask for meanings, part-of-speech style structure, and usage notes.

Second, translated output can be read aloud through the browser’s speechSynthesis API when the browser supports it. This is deliberately output speech, not speech input or speech translation. It keeps the app frontend-only while making the translated result easier to check.

Open source status

PocketBabel is public because the architecture is self-contained and does not rely on private infrastructure. It is also a good example of the kind of project that benefits from being inspectable: the privacy story is stronger when the code path is visible.

The goal of this series is not to pretend every small repository should become a product. The goal is to leave a useful engineering record: why each project exists, what design choices mattered, what I learned from it, and where the boundary is between a good private tool and a project worth publishing.

I will keep this page updated as projects change.

Project map

Why write about private projects?

Open source is not the only useful form of sharing. A private project can still contain lessons worth publishing:

• how the problem was framed
• which constraints shaped the architecture
• what was deliberately left out
• where a prototype became too complex
• what I would do differently next time

For private repositories, these notes avoid secrets, private network details, exact production configuration, personal data, and sensitive business or trading parameters. The public artifact is the engineering story, not a dump of implementation details.

How I classify these projects

Some projects are small tools that should stay small. tapsurge, good-fishing-day, and bedrock-lab are in that category. Their value comes from being built for one specific situation.

Some are experiments in system shape. agentd, scale-kv, quacklake, tegdb, and tegdb-server are more about understanding boundaries than shipping a polished product.

Some are workbench infrastructure. qwen-local, tailgate, PocketBabel, canopy, and ob-skills exist because I want AI tools to fit into my own environment instead of depending entirely on remote services.

Some are creative-system prototypes. driftloop, mood-midi-mlx, and nightfall are less about CRUD-product shape and more about how rules, local models, and browser runtimes can create a coherent experience.

Some projects sit close to professional or sensitive domains. ob-sizer is safe to discuss at the methodology level. cta-guard needs more caution: the interesting public topic is risk control and operational design, not trading edge.

Maintenance rule

When a project changes meaningfully, I will update this index first. If the change is only implementation detail, the individual post can stay as a snapshot. If the project changes direction, graduates into a product, or gets retired, the index should say so plainly.

That makes this page the map, and each project post a field note.

This post is intentionally sanitized. I am not including tokens, local machine details, private paths, secrets, or any internal repository configuration that should not be published.

Official Symphony resources

If you are trying to reproduce or extend this setup, start from the upstream project rather than this blog post alone:

• Repository: github.com/openai/symphony — source, issues, and release notes.
• Specification: SPEC.md — describes the intended behavior and interfaces.
• Reference implementation: elixir/ — Elixir-based reference; follow the README there for build and run details.

Symphony is positioned as experimental or preview-quality software; run it only in environments and repositories you trust, and read the repo README for current limitations and safety expectations.

Hands-on: from clone to first run

Everything below follows the Elixir reference README. If a step fails, fix that step before tweaking your narrative expectations—the runtime is strict about valid WORKFLOW.md YAML at startup.

0. Prerequisites

• Linear: a workspace where you can create a project and issues inside that project.

• Linear API key: Settings → Security & access → Personal API keys. Export it in your shell (do not commit it):

  export LINEAR_API_KEY="your_linear_personal_api_key"
  

• Codex CLI with codex app-server available (Symphony launches Codex in App Server mode). Ensure codex is on PATH when the codex.command in WORKFLOW.md runs.

• Runtime for the reference implementation: the upstream docs recommend mise for Erlang/Elixir versions; use mise install in symphony/elixir as documented.

1. Linear setup: workspace, team, project, workflow, and API key

Symphony only sees what Linear exposes through the API. If the board is wrong, every later step looks like a Symphony bug. Configure Linear before you wire project_slug into WORKFLOW.md.

1.1 Workspace and team

• Use a Linear workspace you control (personal or org). You need permission to create projects and issues on a team.
• Pick the team that will own the automated work. Issues are always tied to a team; your project will live under that team’s context. For a solo setup, one dedicated team per “product line” or per repo is enough.

1.2 Create a project (this is the Symphony queue boundary)

1. In Linear, open Projects (or the team’s project list) and create a new project for the repository you are automating (for example one GitHub repo ↔ one Linear project).
2. Give it a clear name so you do not file issues into the wrong queue later.
3. Open the project itself—not only the team backlog. Symphony’s tracker.project_slug refers to this project.

1.3 Read the project_slug correctly

1. With the project open, copy the page URL from the browser address bar (or use “Copy link” if Linear offers it for the project).
2. The slug is the identifier in that URL that points at this project. Paste it into WORKFLOW.md as project_slug exactly—same spelling, same segment the URL uses.
3. If you rename the project or move it, re-check the URL and update WORKFLOW.md; a stale slug is an instant “nothing happens” failure.

1.4 Align workflow states with WORKFLOW.md

Your simplified WORKFLOW.md lists active_states and terminal_states (for example Todo, In Progress, Rework, Merging, and terminals like Done, Canceled, Cancelled, Duplicate).

1. In Linear, open Team settings → Workflow (wording may vary slightly by plan and UI version).
2. Ensure the team that owns this project actually has status names that match what you put in YAML—including spelling (Canceled vs Cancelled are different strings).
3. If a state is missing, add it to the team workflow. If Linear ships a default you do not use (for example an extra backlog column), you can leave it unused; what matters is that every state your agent and YAML mention exists on the board.
4. Decide how issues enter the pipeline: many setups use Todo or In Progress as the first “Symphony should care” state. Put that state in active_states so polling can pick the issue up.

1.5 Personal API key (Symphony uses LINEAR_API_KEY)

1. Open your user Settings → Security & access (or API / Personal API keys, depending on Linear’s UI).
2. Create a new personal API key, give it a label you will recognize (for example symphony-local).
3. Copy the key once, set export LINEAR_API_KEY="..." on the machine that runs Symphony, and never commit it to git or paste it into WORKFLOW.md unless you intentionally use env indirection like tracker.api_key: $LINEAR_API_KEY (still keep secrets out of the repo).

1.6 Creating issues the way Symphony expects

1. Create the issue inside the project: from the project view, use New issue (or equivalent) so the issue is associated with that project. Creating an issue only on the team backlog without attaching the project is the classic “Symphony is idle” mistake.
2. Set title and description to something actionable; the Markdown body of WORKFLOW.md passes issue.title and issue.description into Codex.
3. Move the issue to a state listed under active_states (for example Todo or In Progress) so it is not sitting in a column Symphony does not poll.

1.7 Optional but useful

• Templates: a small issue template (context, acceptance criteria, “how to validate”) makes agent runs less ambiguous.
• Labels: optional; Symphony does not require them unless you add logic elsewhere.
• Permissions: if the API key belongs to a restricted user, confirm that user can read and update issues in the target project.

After this, you can copy project_slug into WORKFLOW.md with confidence. If anything in this section is skipped, revisit 1.3 (slug) and 1.6 (issue in project) first when debugging.

2. Build the Symphony binary (reference implementation)

git clone https://github.com/openai/symphony
cd symphony/elixir
mise trust
mise install
mise exec -- mix setup
mise exec -- mix build

After this, the launcher is ./bin/symphony inside symphony/elixir (see the same README). You can start it with an absolute path to any WORKFLOW.md you maintain:

mise exec -- ./bin/symphony /absolute/path/to/your/repo/WORKFLOW.md

If you omit the path, it defaults to ./WORKFLOW.md in the current directory—useful when you are iterating inside a single checkout.

Optional flags from upstream:

• --logs-root — log directory (default: ./log relative to how you invoke the binary).
• --port — also starts the optional Phoenix observability UI (dashboard/API as described in the Elixir README).

3. Add WORKFLOW.md to the repository you want automated

1. Copy the template from the Symphony repo: elixir/WORKFLOW.md → your target repo (often repo root).

2. Edit the YAML front matter for your world:

   • tracker.project_slug: in Linear, open your project, copy its URL from the browser, and take the slug segment (the README describes this explicitly).
   • workspace.root: a directory on disk where Symphony may create one workspace per issue (large disk is fine; this is not your git clone root—it is a parent for per-issue workspaces).
   • hooks.after_create: typically git clone ... . into that workspace so Codex works on a fresh copy of your code. Use the clone URL and branch you actually use (HTTPS or SSH is your choice; private repos need credentials on the machine running Symphony).
   • codex.command: must match how you invoke App Server locally (model flags, config, etc.). If this command is wrong, the agent never comes up cleanly.

3. Align Linear workflow states with what WORKFLOW.md expects. The stock template references states such as Todo, In Progress, Rework, Human Review, and Merging. If your team uses different names, either rename states in Linear (Team Settings → Workflow) or edit active_states / terminal_states and the Markdown “status map” in WORKFLOW.md so they match reality.

4. Optionally copy the skills from the Symphony repo (commit, push, pull, land, linear, etc.) into your repo if your workflow prompt expects them—the Elixir README calls this out.

Symphony does not boot if WORKFLOW.md is missing or the YAML front matter is invalid; fix the file and restart.

4. Run and sanity-check before opening a ticket

export LINEAR_API_KEY="..."   # if not already in your shell profile
cd /path/to/openai/symphony/elixir
mise exec -- ./bin/symphony /path/to/your/automated/repo/WORKFLOW.md

Then verify:

• The process stays running and polls on the interval you set (polling.interval_ms in the template).
• If you passed --port, you can hit the dashboard/API URLs documented in the Elixir README for live state.

5. First Linear issue (the mistake that looks like “Symphony is broken”)

Do this in order or you will get silent no-ops:

1. Create or pick a Linear project whose slug matches tracker.project_slug exactly.
2. Create the issue inside that project, not as a free-floating team issue.
3. Put the issue in an active state listed under active_states in WORKFLOW.md (for the default template, something like Todo or In Progress—not Backlog if your prompt tells the agent to ignore Backlog).

If Symphony polls successfully but your issue never enters the watched project, you will see healthy logs and zero useful work—this is the project_slug lesson from later in this post.

6. Two repos (repeat the pattern)

For each codebase, maintain its own WORKFLOW.md, its own Linear project (and slug), its own workspace.root, and run its own ./bin/symphony .../WORKFLOW.md process. Trying to multiplex multiple repositories through one workflow file is how you get accidental coupling and confusing failures.

If you want the upstream one-liner to bootstrap with Codex inside your repo, the FAQ in the Elixir README suggests pointing Codex at elixir/README.md and asking it to wire files for your codebase—still verify project_slug, workspace paths, and git remotes yourself.

Why I tried this

What interested me most about Symphony was not “AI that writes code” in isolation. I already have coding tools for that. The interesting part was orchestration:

• a task source
• a state machine
• an isolated workspace per task
• an agent runtime
• a repeatable loop from issue to code change

That is a different shape of workflow from normal editor-assisted coding.

Why I used Linear instead of GitHub Issues

One thing became clear very quickly: Symphony is designed around Linear as the source of truth for work. It does not naturally start from GitHub Issues. Instead, the workflow looks more like this:

1. Create a Linear issue
2. Symphony polls the configured Linear project
3. Symphony creates a dedicated workspace for that issue
4. Codex works inside that workspace
5. The workflow advances by issue state

At first this felt a little strange, because I am used to GitHub Issues being the center of project work. But after testing it, I could see the logic. Linear is the task system. GitHub is the code system.

The first practical lesson: project scoping matters

A surprisingly easy mistake was creating an issue in the wrong place.

I had a Linear workspace and a correctly configured project, but the first issue I created was not actually attached to the project that Symphony was watching. From the outside it looked like “nothing is happening,” but the real problem was much simpler: Symphony was correctly polling the configured project and my issue was outside that scope.

That was a good reminder that in this setup, project_slug is not a decorative field. It is the queue boundary.

Making WORKFLOW.md actually usable

The stock elixir/WORKFLOW.md in the Symphony repository is intentionally large: long status maps, PR sweeps, workpad templates, and guardrails meant for serious team-style orchestration. For solo maintenance on a small repo, that is often more surface area than you want to own on day one.

What I actually wanted was a small YAML front matter plus a short agent brief that still respects Linear state and runs a tight validate loop.

The elements I kept in practice:

• one Linear project per repository
• one Symphony process per repository
• one workspace root per repository
• explicit active and terminal states (only the ones I really use)
• explicit install/setup commands in after_create
• explicit validation before completion (npm in my case)
• codex app-server with sandbox left at workspace write, approval policy set explicitly so the run does not stall on prompts

A simplified WORKFLOW.md (sanitized)

Below is the shape of the workflow file I run. Values such as the Linear project slug, workspace directory, and git remote are placeholders—replace them with your own. Do not copy real identifiers from this post into production without checking them in Linear and Git.

---
tracker:
  kind: linear
  project_slug: "your-linear-project-slug"
  active_states:
    - Todo
    - In Progress
    - Rework
    - Merging
  terminal_states:
    - Done
    - Canceled
    - Cancelled
    - Duplicate

polling:
  interval_ms: 5000

workspace:
  root: ~/symphony-workspaces/your-repo-short-name

hooks:
  after_create: |
    git clone --depth 1 https://github.com/your-org/your-repo.git .
    npm install

agent:
  max_concurrent_agents: 1
  max_turns: 20

codex:
  command: codex app-server
  approval_policy: never
  thread_sandbox: workspace-write
  turn_sandbox_policy:
    type: workspaceWrite
---

You are working on a Linear issue {{ issue.identifier }}.

Title: {{ issue.title }}
Body: {{ issue.description }}

Rules:

- Always start by understanding the current state of the issue.
- If state is Todo, move it to In Progress.
- If state is Rework, review existing changes and fix issues.
- If state is Merging, finalize merge (do not keep coding).

Execution:

1. Understand the task
2. Reproduce or reason about current behavior
3. Make minimal safe changes
4. Run:
   - npm run build OR npm test
5. If success:
   - commit changes
   - push branch or merge directly according to repository flow

Do not:

- Ask humans for help
- Modify files outside workspace
- Skip validation

Goal:

Deliver working code change with valid validation and keep the Linear issue state accurate.

How to fill this in safely

• project_slug: in Linear, open the project and copy its URL; the slug is the path segment that identifies the project (see the Elixir README). It must match the project where you create issues.
• workspace.root: any empty-friendly parent directory on the machine that runs Symphony; Symphony creates a subdirectory per issue under this root.
• after_create: use your real git clone URL and package install command (npm install, pnpm install, make, and so on).
• Linear states: your team must actually define or use states compatible with active_states / terminal_states. If Linear uses different names, edit the lists to match.

This was a much better balance for me than copying the entire official workflow verbatim, while still staying inside Symphony’s YAML + Markdown contract.

The first real run

Once the workflow was wired correctly, the first successful run was a great moment. The basic flow worked:

1. Create a Linear issue
2. Symphony picks it up
3. A workspace is created
4. The repository is cloned
5. Dependencies are installed
6. Codex makes a change
7. Validation runs
8. The issue moves forward in the workflow

That first time matters because it changes the whole thing from “interesting repo I am reading” into “real tool I can use.”

A workflow surprise: no PR, direct merge

One unexpected result was that the run did not produce a pull request. Instead, it created a branch and then merged directly into main.

For a team workflow, that would be a problem. For my personal setup, I actually found it acceptable.

Because I am the only person using this flow right now, direct merge is not automatically bad. It is fast, and it fits a solo maintenance loop. The tradeoff is obvious: less review structure, more need for good validation and discipline.

If I later want a stricter process, the right fix is probably branch protection plus a stronger PR gate in the workflow.

Why I accepted a “solo mode”

After thinking about it, I realized there are really two different modes here:

Team mode

• branch protection
• pull requests
• human review gates
• merge discipline

Solo mode

• fast issue pickup
• direct code change
• direct landing when validation passes

For now I am explicitly leaning toward solo mode. That is not because it is universally better. It is just a better fit for a single developer trying to reduce friction on personal repos.

Scaling from one repo to two

After getting the first repository working, I wanted to know whether I could use Symphony across more than one repo.

The answer was yes, but not by forcing one workflow to manage everything. The cleaner model was:

• one Linear project per repository
• one WORKFLOW file per repository
• one Symphony process per repository
• one workspace root per repository

That means each repo gets its own queue, workspace, and execution loop. The result is much easier to reason about than trying to multiplex multiple repos through a single workflow.

Conceptually, the setup became:

Repo A -> Linear Project A -> Symphony Process A
Repo B -> Linear Project B -> Symphony Process B

That separation made the system feel much more stable.

Things that felt weird

A few things still feel unusual in this setup:

1. Linear is the real driver

If you are used to GitHub-centric project flow, it takes a minute to reset your intuition.

2. The workflow file is closer to an operating manual than a config file

It is not just about parameters. It strongly shapes agent behavior.

3. Small scope mistakes create “silent failures”

If the wrong project is watched, or the issue is created in the wrong place, everything can look healthy while nothing useful happens.

4. Defaults are often too implicit

Model choice, reasoning depth, safety behavior, and merge style all become much clearer once they are explicitly set instead of left to defaults.

What I would improve next

There are a few upgrades that would make this setup stronger without making it too heavyweight:

• make validation stricter before landing changes
• make commit messages more informative
• optionally require PRs for selected repos
• capture a better audit trail of what the agent actually did
• design a lightweight rollback path for bad automated changes

That would preserve the speed of solo mode while reducing the risk of bad direct merges.

Final take

My main takeaway is that Symphony becomes much more interesting once it is treated as a workflow runtime, not just a coding demo.

The useful mental model is not “an AI that edits files.” It is closer to this:

• work arrives through a queue
• each task gets an isolated environment
• the agent runs inside a bounded workflow
• the repo is just one part of the system

For a solo developer, that can actually be a very comfortable way to work, as long as the workflow is shaped carefully enough.

It is still early, still a little rough, and definitely not something I would blindly trust everywhere. But for small personal projects, it already feels surprisingly real.

Future potential: kill the relay

Symphony is not competing with tab completion. It is a probe for a nastier question: if work is just queue + policy + execution, why would you keep a permanent class of people whose main job is to sit between a customer sentence and a git merge?

Here is the version I actually believe.

B2B should look like a pipe, not a committee. Whoever hears the customer—sales engineer, CS, onboarding, whoever—opens the issue. That issue is the contract. Behind it, Symphony-grade orchestration does the rest: clone, implement, test, merge, release. Not “faster Jira.” Not “AI assists your sprint.” The default path is machine throughput; humans are for edge cases, politics, and blame.

Does that erase humans? No—it erases the middle. The classic career ladder where “product” rewrites reality for “engineering” so engineering can rewrite it again for Git is not destiny. It is coordination rent. Orchestration is a wrecking ball aimed at that rent. If your value is mostly translating between tools and meetings, the stack is not coming to help you—it is coming to delete the slot.

You can list risks forever—compliance, security, hallucinations, bad merges—and you should. But risk is not a moral argument for headcount. It is an argument for thinner, sharper ownership: a tiny number of people who set policy and own catastrophes, plus a machine that does the boring middle at machine speed.

Yes, today’s tools are still a preview: flaky, embarrassing, unsafe if you are lazy about validation. Irrelevant to the direction. The direction is first-hand demand in, shipped software out, with as few interpreters as the market will tolerate. In ten years, “we need more PMs and more engineers because that is how software is made” will read like “we need more telephone switchboard operators because calls exist.”

My two-repo setup is a toy. The logic is not.

Appendix: sanitized lessons learned

• Configure Linear (project, slug, workflow states, API key, issues inside the project) before blaming Symphony
• Start with one repo, not many
• Keep one workflow per repo
• Use one Linear project per repo
• Make state transitions explicit
• Do not rely too much on defaults
• Validate aggressively before allowing automated landing
• Expect the first “nothing happened” failure to be a scoping mistake

We’ve all been there. You’re deep in a technical discussion with an AI assistant—analyzing code, exploring architecture, or debugging a complex issue. The conversation is rich with insights, and you think: “This would make a great blog post.”

But then reality hits: you need to switch to your blog repository, format the content, commit it, push it, and wait for the build. By the time you’re back, the original context is gone, and the momentum is lost.

What if you could publish directly from where you are?

Building on Existing Automation

In my previous post about automatically publishing a blog using GitHub Actions, I set up a workflow where pushing to the blog repository triggers an automatic build and deployment to GitHub Pages. This solved the build and deployment automation, but there was still one manual step remaining: creating the post file itself.

The workflow I described there handles:

1. Checking out the blog repository
2. Building the Hugo site with make
3. Deploying to jackysp.github.io

But you still needed to be in the blog repository to create the post. That’s where MCP changes everything.

Enter MCP: The Missing Link

The Model Context Protocol (MCP) is revolutionizing how AI agents interact with external systems. Instead of treating AI as a passive tool, MCP enables agents to act as autonomous agents with direct access to your tools and workflows.

In my setup, I’ve connected MCP-enabled agents (like Cursor) directly to my blog repository via GitHub MCP. This means:

• No context switching: Stay in your current working directory, whether it’s a random project folder or a deep codebase exploration
• Preserve conversation flow: The AI maintains the full context of your discussion
• Direct publishing: Create and publish posts without leaving your IDE

The Architecture: Seamless Integration

Here’s how the complete workflow operates:

┌─────────────────────────────────────────────────────────┐
│  AI Agent (Cursor/Claude) with MCP enabled              │
│  - Context: Any code repository or discussion            │
│  - Tool: GitHub MCP Server                               │
└──────────────────┬──────────────────────────────────────┘
                   │
                   │ Creates post via GitHub MCP
                   │
                   ▼
┌─────────────────────────────────────────────────────────┐
│  Blog Repository (jackysp/blog)                         │
│  - content/posts/<slug>/index.md                        │
│  - Commit: "Publish: [title]"                           │
└──────────────────┬──────────────────────────────────────┘
                   │
                   │ Push to master branch
                   │
                   ▼
┌─────────────────────────────────────────────────────────┐
│  GitHub Actions (from previous post)                     │
│  - Build: Hugo static site generation                   │
│  - Deploy: Push to jackysp.github.io                    │
└──────────────────┬──────────────────────────────────────┘
                   │
                   │ Published
                   │
                   ▼
┌─────────────────────────────────────────────────────────┐
│  Live Site (jackysp.github.io)                          │
│  - Post is live and accessible                           │
└─────────────────────────────────────────────────────────┘

The GitHub Actions part remains exactly as described in the previous post—no changes needed there. The MCP layer adds the ability to trigger it from anywhere.

The Workflow in Action

1. AI-Powered Content Creation

When you’re discussing a technical topic with an AI agent, you can simply ask:

“Turn this discussion into a blog post and publish it.”

The AI agent, with access to GitHub via MCP, can:

• Extract key insights from your conversation
• Format content according to Hugo front matter requirements
• Create properly structured markdown files
• Handle images and assets
• Commit and push to the repository

2. Automated Build & Deploy

The moment a post is pushed to the master branch, the same GitHub Actions workflow from the previous post kicks in:

on:
  push:
    branches: [ master ]

The workflow (as detailed in the previous post):

1. Checks out the blog repository with submodules
2. Builds the Hugo site using make
3. Deploys the built artifacts to jackysp.github.io

All without manual intervention.

3. Governance Through Contracts

To ensure quality and prevent accidents, I’ve implemented an AI Publishing Contract (PUBLISHING.md) that defines:

• Allowed paths: Only content/** and static/** can be modified
• Post format: Required front matter fields (title, date, tags, slug, summary)
• Image handling: Standardized location and reference format
• Commit conventions: Single commit per post with descriptive messages

This contract ensures that AI agents can publish content while respecting the repository structure and quality standards.

Why This Matters: The Developer Experience Revolution

Zero Context Switching

Traditional workflow:

1. Copy conversation → Switch to blog repo → Format → Commit → Push → Wait
2. Context lost, momentum broken

New workflow:

1. Ask AI to publish → Done
2. Context preserved, workflow continuous

Capturing Technical Insights

The best technical insights often emerge during active problem-solving. With this workflow, you can:

• Document discoveries in real-time
• Turn debugging sessions into tutorials
• Transform architecture discussions into deep-dives
• Share codebase explorations as learning resources

Scaling Knowledge Sharing

Previously, the friction of publishing meant many valuable insights were never written down. Now, the barrier to publishing is minimal, making it easier to:

• Share learnings with your team
• Build a personal knowledge base
• Contribute to the developer community
• Document your problem-solving journey

Technical Implementation Details

MCP Server Configuration

The GitHub MCP server provides the AI agent with:

• Repository read/write access
• File creation and modification
• Commit and push capabilities
• Branch management

GitHub Actions Workflow

The CI/CD pipeline (as described in the previous post) handles:

• Go environment setup (for Hugo builds)
• Repository checkout with submodules
• Site generation via make
• Deployment to GitHub Pages repository

No changes needed to the existing workflow—it just gets triggered from a new entry point.

Hugo Site Configuration

Posts follow Hugo’s standard structure:

• Location: content/posts/<slug>/index.md
• Format: YAML front matter + Markdown content
• Images: Stored in each post bundle and referenced with relative filenames
• Draft control: draft: true/false for preview/publish

The Future: AI-Augmented Documentation

This workflow represents a shift toward AI-augmented documentation. Instead of treating AI as a writing assistant, we’re treating it as a publishing agent that can:

• Understand context from code discussions
• Extract technical insights automatically
• Format and structure content appropriately
• Publish without breaking workflow

As MCP and similar protocols mature, we’ll see more sophisticated capabilities:

• Automatic code analysis and explanation
• Multi-post series generation from extended discussions
• Cross-referencing with existing content
• SEO and metadata optimization

Getting Started

If you want to set up a similar workflow:

1. Set up automated publishing (see my previous post)
2. Enable MCP in your AI agent (Cursor, Claude Desktop, etc.)
3. Configure GitHub MCP server with repository access
4. Define publishing contracts for governance
5. Start publishing from your conversations

The technical details are straightforward, but the impact on productivity and knowledge capture is profound.

Conclusion

The intersection of AI agents, MCP protocols, and automated CI/CD creates a new paradigm for technical publishing. By building on the existing GitHub Actions automation and adding MCP as the entry point, we eliminate context switching and reduce friction.

This isn’t just about automating blog posts—it’s about preserving the flow state of technical discovery and making knowledge sharing as natural as having a conversation.

The future of technical documentation is here, and it’s conversational.

This post was created and published using the exact workflow described above—from a discussion about workflow automation to a live blog post, all without leaving the conversation context.

Most Moka pot guides treat brewing as a recipe problem: grind size, water temperature, and heat level.

That framing breaks down completely when you introduce:

• A Brikka (pressure valve + burst extraction)
• An induction hob
• A steel induction adapter plate

At this point, you are no longer “brewing coffee”. You are operating a multi-stage thermal + pressure system with delayed feedback.

This article reframes Brikka-on-induction as a control problem, not a recipe.

Brikka vs Classic Moka: A Structural Difference

Classic Moka pots operate as continuous-flow systems.

Brikka is fundamentally different.

Brikka is a Triggered System

• No flow occurs until a pressure threshold is reached
• Once triggered, the valve opens abruptly
• Extraction happens in a very short, high-energy window

From a systems perspective:

Classic Moka = streaming pipeline
Brikka = edge-triggered event

Induction + Adapter Plate: Where the Model Changes

With an induction hob and the official Bialetti adapter plate, the heat path becomes:

Induction coil
→ Steel adapter plate
→ Aluminum boiler
→ Water

The adapter plate introduces significant thermal inertia.

Low or medium power levels often never leave the heat accumulation phase.

Engineering Goal: Reach the Trigger, Then Stop

Because Brikka extracts only after the valve opens, the primary goal is:

Drive the system to the trigger point as efficiently as possible — then stop.

Any energy added after triggering only increases bitterness.

Parameter Design

Water

• Cold water to the fill line

Cold water ensures a linear pressure ramp.

Beans

• Medium to medium-light roast

Brikka amplifies front-loaded flavors.

Grind

• Timemore C5 ESP: baseline 1.2.0
• Adjust only ±0.0.5

Brikka has a narrow operating window.

Heat Strategy

Preheating Phase (Before Grinding):

• Start heating the adapter plate at Level 9 while measuring beans and grinding
• The official stainless steel adapter plate is thick and conducts heat slowly
• Preheating ensures the plate reaches operating temperature before brewing begins

Brewing Phase:

• Phase 1: Level 9 until trigger
• Phase 2: Cut power immediately at first continuous output
• Rinse boiler bottom with cold water

Residual heat is sufficient.

Timing as Validation

• Trigger time: 3–5 minutes
• Much longer indicates system inefficiency

Time validates the system, not flavor.

Design Trade-offs

High initial power stresses equipment but exits the thermal dead zone.

Immediate cutoff sacrifices volume but preserves flavor boundaries.

Final Mental Model

Brikka is not a brewer.
It is a pressure-triggered extraction event.

Once the event fires, the system should coast.

Respecting that boundary makes the system predictable and repeatable.

OceanBase targets high availability and scalability in a shared-nothing cluster. The core engineering challenge is to make four critical subsystems work together with predictable latency and correctness:

• Write transactions must be durable, replicated, and efficiently committed.
• Tablet replay must recover state quickly and safely.
• SQL parse to execute must optimize well while respecting multi-tenant constraints.
• Unit placement must map tenants to physical resources without fragmentation.

This article focuses on motivation, design, implementation highlights, and tradeoffs, using concrete code entry points from the OceanBase codebase.

System Architecture

OceanBase adopts a shared-nothing architecture where each node is equal and runs its own SQL engine, storage engine, and transaction engine. Understanding the overall architecture is essential before diving into implementation details.

Cluster, Zone, and Node Organization

graph TB
    subgraph Cluster["OceanBase Cluster"]
        subgraph Z1["Zone 1"]
            N1["OBServer Node 1"]
        end
        subgraph Z2["Zone 2"]
            N2["OBServer Node 2"]
        end
        subgraph Z3["Zone 3"]
            N3["OBServer Node 3"]
        end
    end
    
    subgraph Proxy["obproxy Layer"]
        P1["obproxy 1"]
        P2["obproxy 2"]
    end
    
    Client["Client Applications"] --> Proxy
    Proxy --> N1
    Proxy --> N2
    Proxy --> N3

Key Concepts:

• Cluster: A collection of nodes working together
• Zone: Logical availability zones for high availability and disaster recovery
• OBServer: Service process on each node handling SQL, storage, and transactions
• obproxy: Stateless proxy layer routing SQL requests to appropriate OBServer nodes

Data Organization: Partition, Tablet, and Log Stream

graph TB
    subgraph Table["Table"]
        P1["Partition 1"]
        P2["Partition 2"]
        P3["Partition 3"]
    end
    
    subgraph LS1["Log Stream 1"]
        T1["Tablet 1"]
        T2["Tablet 2"]
    end
    
    subgraph LS2["Log Stream 2"]
        T3["Tablet 3"]
    end
    
    subgraph LS3["Log Stream 3"]
        T4["Tablet 4"]
    end
    
    P1 --> T1
    P1 --> T2
    P2 --> T3
    P3 --> T4
    
    T1 --> LS1
    T2 --> LS1
    T3 --> LS2
    T4 --> LS3

Key Concepts:

• Partition: Logical shard of a table (hash, range, list partitioning)
• Tablet: Physical storage object storing ordered data records for a partition
• Log Stream (LS): Replication unit using Multi-Paxos for data consistency
• Replication: Each tablet has multiple replicas across zones, with one leader accepting writes. Log streams replicate data via Multi-Paxos protocol across different zones.

Multi-Tenant Resource Model

graph TB
    subgraph Tenant["Tenant"]
        T1["Tenant 1 MySQL Mode"]
        T2["Tenant 2 Oracle Mode"]
        T3["System Tenant"]
    end
    
    subgraph Pool["Resource Pool"]
        RP1["Pool 1"]
        RP2["Pool 2"]
        RP3["Pool 3"]
    end
    
    subgraph Unit["Resource Unit"]
        U1["Unit 1 CPU Memory Disk"]
        U2["Unit 2 CPU Memory Disk"]
        U3["Unit 3 CPU Memory Disk"]
    end
    
    subgraph Server["Physical Server"]
        S1["Server 1"]
        S2["Server 2"]
        S3["Server 3"]
    end
    
    T1 --> RP1
    T2 --> RP2
    T3 --> RP3
    
    RP1 --> U1
    RP2 --> U2
    RP3 --> U3
    
    U1 --> S1
    U2 --> S2
    U3 --> S3

Key Concepts:

• Tenant: Isolated database instance (MySQL or Oracle compatibility)
• Resource Pool: Groups resource units for a tenant across zones
• Resource Unit: Virtual container with CPU, memory, and disk resources
• Unit Placement: Rootserver schedules units to physical servers based on resource constraints

Layered Architecture

graph TB
    subgraph App["Application Layer"]
        Client["Client Applications"]
    end
    
    subgraph Proxy["Proxy Layer"]
        ODP["obproxy Router"]
    end
    
    subgraph OBServer["OBServer Layer"]
        subgraph Node["OBServer Node"]
            SQL["SQL Engine"]
            TX["Transaction Engine"]
            ST["Storage Engine"]
        end
    end
    
    subgraph Storage["Storage Layer"]
        subgraph LS["Log Stream"]
            Tablet["Tablet"]
            Memtable["Memtable"]
            SSTable["SSTable"]
        end
        Palf["Paxos Log Service"]
    end
    
    subgraph Resource["Resource Layer"]
        Tenant["Tenant"]
        Unit["Resource Unit"]
        Pool["Resource Pool"]
        RS["Rootserver"]
    end
    
    Client --> ODP
    ODP --> SQL
    SQL --> TX
    TX --> ST
    ST --> LS
    LS --> Palf
    Tenant --> Unit
    Unit --> Pool
    Pool --> RS
    RS --> Node

Key Concepts:

• SQL Engine: Parses, optimizes, and executes SQL statements
• Transaction Engine: Manages transaction lifecycle, commit protocols, and consistency
• Storage Engine: Handles data organization, memtables, and SSTables
• Log Service: Provides Paxos-based replication and durability
• Rootserver: Manages cluster metadata, resource scheduling, and placement

Design Overview

At a high level, each node runs a full SQL engine, storage engine, and transaction engine. Data is organized into tablets, which belong to log streams. Log streams replicate changes using Paxos-based log service. Tenants slice resources via unit configurations and pools, while rootserver components place those units on servers.

The following sections walk through each path with the relevant implementation anchors.

Architecture Diagrams

Transaction Write Path

flowchart LR
  subgraph A["Transaction Write Path"]
    C["Client"] --> S["SQL Engine"]
    S --> T["Transaction Context"]
    T --> M["Memtable Write"]
    M --> R["Redo Buffer"]
    R --> L["Log Service"]
    L --> P["Replicated Log"]
    P --> K["Commit Result"]
  end

Tablet Replay Path

flowchart LR
  subgraph B["Tablet Replay Path"]
    L["Log Service"] --> RS["Replay Service"]
    RS --> E["Tablet Replay Executor"]
    E --> LS["Log Stream"]
    LS --> TB["Tablet"]
    TB --> CK["Replay Checks"]
    CK --> AP["Apply Operation"]
    AP --> ST["Updated Tablet State"]
  end

SQL Compile and Execute

flowchart LR
  subgraph C["SQL Compile and Execute"]
    Q["SQL Text"] --> P["Parser"]
    P --> R["Resolver"]
    R --> W["Rewriter"]
    W --> O["Optimizer"]
    O --> LP["Logical Plan"]
    LP --> CG["Code Generator"]
    CG --> PP["Physical Plan"]
    PP --> EX["Executor"]
  end

Unit Placement

flowchart LR
  subgraph D["Unit Placement"]
    UC["Unit Config"] --> RP["Resource Pool"]
    RP --> PS["Placement Strategy"]
    PS --> CS["Candidate Servers"]
    CS --> CH["Chosen Server"]
    CH --> UN["Unit Instance"]
  end

Write Transaction: From Memtable to Replicated Log

Motivation

A write transaction must be both fast and durable. OceanBase uses memtables for in-memory writes, and a log stream for redo replication. The design must allow low-latency commit while supporting parallel redo submission and multi-participant (2PC) transactions.

Design Sketch

• Each transaction is represented by a per-LS context (ObPartTransCtx).
• Redo is flushed based on pressure or explicit triggers.
• Commit chooses one-phase or two-phase based on participants.
• Logs are submitted via a log adapter backed by logservice.

Implementation Highlights

• Transaction context lifecycle and commit logic are in src/storage/tx/ob_trans_part_ctx.cpp.
• Redo submission is driven by submit_redo_after_write, which switches between serial and parallel logging based on thresholds.
• Commit decides between one-phase and two-phase commit depending on participant count.
• The log writer (ObTxLSLogWriter) submits serialized logs via ObITxLogAdapter, which is wired to logservice (ObLogHandler).

Tradeoffs

• Serial vs parallel redo: Serial logging is simpler and cheaper for small transactions, but parallel logging reduces tail latency for large transactions at the cost of more coordination.
• 1PC vs 2PC: 1PC is fast for single-participant transactions; 2PC is required for distributed consistency but increases coordination overhead.
• In-memory batching vs durability: Larger batching improves throughput but can delay durability and increase replay time.

Tablet Replay: Reconstructing State Safely

Motivation

Recovery needs to be deterministic and safe: the system must replay logs to reconstruct tablet state without violating invariants or applying obsolete data.

Design Sketch

• Logservice schedules replay tasks per log stream.
• Tablet replay executor fetches the LS, locates the tablet, validates replay conditions, and applies the log.
• Specialized replay executors handle different log types (e.g., schema updates, split operations).

Implementation Highlights

• Replay orchestration lives in src/logservice/replayservice/ob_log_replay_service.cpp.
• Tablet replay logic is in src/logservice/replayservice/ob_tablet_replay_executor.cpp.
• Specific tablet operations are applied in dedicated executors, such as ObTabletServiceClogReplayExecutor in src/storage/tablet/ob_tablet_service_clog_replay_executor.cpp.

Tradeoffs

• Strictness vs throughput: Replay barriers enforce ordering for correctness but can reduce parallelism.
• Tablet existence checks: Allowing missing tablets can speed recovery but requires careful validation to avoid partial state.
• MDS synchronization: Metadata state updates improve correctness but add contention via locks.

SQL Parse to Execute: Compile Pipeline for Performance

Motivation

OceanBase supports MySQL and Oracle compatibility with rich SQL features. The compile pipeline must be fast, cache-friendly, and yield efficient execution plans.

Design Sketch

• SQL text enters the engine via ObSql::stmt_query.
• Parsing produces a parse tree.
• Resolution turns the parse tree into a typed statement tree.
• Rewrite and optimization generate a logical plan.
• Code generation produces a physical plan and execution context.

Implementation Highlights

• Entry and query handling: src/sql/ob_sql.cpp (stmt_query, handle_text_query).
• Resolver: ObResolver in src/sql/resolver/ob_resolver.h.
• Transform and optimize: ObSql::transform_stmt and ObSql::optimize_stmt in src/sql/ob_sql.cpp.
• Code generation: ObSql::code_generate in src/sql/ob_sql.cpp.

Tradeoffs

• Plan cache vs compile accuracy: Plan caching reduces latency but may reuse suboptimal plans under changing data distributions.
• Rewrite aggressiveness: More transformations can yield better plans but increase compile cost.
• JIT and rich formats: Faster execution for some workloads, but added complexity and memory pressure.

Unit Placement: Scheduling Tenant Resources

Motivation

Multi-tenancy requires predictable isolation and efficient resource utilization. Unit placement must respect CPU, memory, and disk constraints while minimizing fragmentation.

Design Sketch

• Unit config defines resource demands.
• Resource pool groups units by tenant and zone.
• Placement strategy scores candidate servers to pick a host for each unit.

Implementation Highlights

• Resource types and pools: src/share/unit/ob_unit_config.h, src/share/unit/ob_resource_pool.h, src/share/unit/ob_unit_info.h.
• Placement policy: src/rootserver/ob_unit_placement_strategy.cpp uses a weighted dot-product of remaining resources to choose a server.
• Orchestration: src/rootserver/ob_unit_manager.cpp handles creation, alteration, and migration of units and pools.

Tradeoffs

• Greedy placement vs global optimality: Dot-product scoring is efficient and practical but may not be globally optimal.
• Capacity normalization: Assuming comparable server capacities simplifies scoring but may bias placement in heterogeneous clusters.
• Latency vs stability: Fast placement decisions can lead to more churn; conservative placement improves stability but can reduce utilization.

Closing Thoughts

These four paths demonstrate how OceanBase balances correctness, performance, and operability. The code structure follows clear separation of responsibilities: transaction logic is in storage/tx, replication and replay are in logservice, SQL compilation is in sql, and scheduling is in rootserver and share/unit. The tradeoffs are explicit and largely encoded in thresholds and policies, which makes tuning feasible without invasive rewrites.

If you are extending OceanBase, start with the entry points highlighted above and follow the call chains into the relevant subsystem. It is the fastest way to build a mental model grounded in the actual implementation.

WebVM is a virtual machine (VM) that executes entirely within a web browser. It’s an innovative project that brings the power of a Linux environment straight to your browser, eliminating the need for traditional virtual machine setups. WebVM operates within a sandboxed environment, ensuring secure execution of applications without affecting the host system.

Understanding WebVM

The source repository provides a frontend for the WebVM demo. By forking the repository and following the instructions outlined in the GitHub Actions, you can build an image using the Dockerfile located at:

• Debian Large Dockerfile

This image can then be hosted on GitHub Pages, as demonstrated in my demo.

WebVM’s primary functionality is to stream resources to the browser, minimizing client-side resource consumption. It enables the execution of various applications that are typically restricted to virtual machines, all within a browser. Additionally, WebVM allows for the embedding of these applications through custom front-ends.

Default Image and Capabilities

The default WebVM image is Debian-mini, which may have limited capabilities. To enhance its functionality, I have opted for the Debian-large image, which has been extended to a 2GB disk capacity. This provides a more robust environment with additional tools and packages.

Usage and Benefits

WebVM offers a range of applications and capabilities:

1. Custom Image Creation: Create custom images tailored to your specific requirements, allowing for a personalized virtual environment.

2. Web-Based Linux Terminal: Access a web-based Linux terminal to execute Linux commands directly within the browser. This includes:

   • SSH/SCP File Transfers: Securely transfer files using SSH and SCP protocols.
   • HTTP Server Initiation: Start an HTTP server using python3 -m http.server.

3. Sandboxed Security: Operates within a sandbox environment, ensuring secure execution of applications without affecting the host system.

4. Serverless Architecture: Embraces a serverless architecture by executing entirely on the client side. Running a Linux server within a browser presents a unique and innovative approach to virtualization.

Alternative Options

Yes, there are alternative options. JSLinux is a preferred and faster option. However, it does not allow modifications to the image, which can be a limitation if you require a customized environment.

Additional Tips

• Internet Connectivity via Tailscale:

  WebVM can connect to the internet via Tailscale. It utilizes the first available node as an exit node. If you execute:

  curl https://ifconfig.me
  

  You will obtain your current node’s IP address.

• DNS Functionality:

  The DNS functionality of Tailscale is currently experiencing issues. It’s recommended to use IP addresses to connect to other nodes within your Tailscale network instead of domain names.

• Default Credentials:

  You can obtain the default user:password and root:password credentials by checking the Dockerfile:

  • Default Credentials in Dockerfile

Conclusion

WebVM is a powerful tool that brings the versatility of a Linux environment to your browser. Whether you’re looking to experiment with Linux commands, develop applications, or require a portable and sandboxed environment, WebVM offers a serverless and secure solution. Its ability to create custom images and operate entirely on the client side sets it apart from other web-based virtual machines.

Feel free to explore WebVM and customize it to suit your needs. Happy coding!

What Modifications Did I Make?

1. Handlebar Grips

   The original grips were glued on and became loose over time, so I replaced them with new, more secure ones.

2. Bell

   Safety first! I added a bell to alert pedestrians and other cyclists.

3. CNC Mount Base for Action Cameras

   Installed at the center of the handlebar, it’s perfect for mounting my DJI OSMO Pocket 1 or a phone holder.

4. Front Bottle Bag from Decathlon Riverside

   A convenient spot for a water bottle and small items.

5. Front Light

   Mounted just above the front wheel to illuminate the path without blinding anyone.

6. Soft Saddle from Decathlon

   For a more comfortable ride.

7. Four Universal Easy Wheels

   These make the bike easier to push when folded. The set of four cost me 100 SGD.

8. Taillight

   For visibility and safety during night rides.

9. Front Bag

   Provides additional storage space for essentials.

10. Anti-Loose Accessory

    Helps keep components tight and secure.

    

11. Tool Set

    Essential for on-the-go repairs and adjustments.

    

Why Did I Choose These Modifications?

I experimented with mounting accessories on the handlebar, like the bottle holder and front light, but it made the handlebar too crowded and unstable. For instance, the bottle holder wasn’t secure when attached to the handlebar. I decided to utilize existing mounts, such as the taillight bracket, to streamline the setup.

The additional accessories are intended to make the bike even easier to push when folded. It’s already convenient, but I thought, why not enhance it further?

Areas for Improvement

• Anti-Loose Accessory: It’s not working as well as I’d hoped and might need to be replaced.

• Front Bag Straps: They’re a bit long, and I’m concerned they could get caught in the front wheel. I might need to shorten them.

Cost Breakdown

Most accessories cost between 5–50 SGD, except for the universal wheels, which were 100 SGD for all four. Initially, I wasn’t sure if I needed all four wheels—maybe two would have sufficed—but I ultimately decided to get all four for better stability.

Anything Else?

I can’t think of anything else at the moment. If you have any suggestions or feedback, I’d love to hear them!

Tips for Buying Accessories in Singapore

• Decathlon: A great place to find affordable accessories, and they have a 365-day return policy!

• Shopee: Good for finding unique or hard-to-find items.

• Taobao: Offers a wide range of accessories, but shipping can be expensive. You can ask a friend in China to help you buy and ship the items to Singapore.

I don’t recommend buying from bike shops in Singapore, as they tend to be more expensive—about twice the price.

Let me know if there’s anything else you’d like to know or discuss!

As a long-time podcaster, I’ve always enjoyed sharing my thoughts and ideas through audio. While the world of video content—and the role of a YouTuber—has its allure, the complexities of video editing have kept me anchored in the realm of podcasting. My journey has involved leveraging platforms like Spotify Creator (formerly Anchor) for hosting and distributing my recordings. This platform offers a wide array of features for free, including audio recording, editing capabilities, and automatic promotion to Spotify.

However, I sought a more comprehensive solution, one that would allow me to listen to my own podcast while driving, using the Podcast app on my CarPlay device. To achieve this, I ventured into publishing on Apple Podcasts (podcastsconnect.apple.com), which also offers free hosting. With a self-designed cover and episodes uploaded, I was set—or so I thought.

The Challenges of Traditional Podcasting

Despite having the technical setup, I faced significant challenges:

• Consistency: Maintaining a regular publishing schedule proved difficult.
• Voice Quality: My voice quality was inconsistent, affecting listener engagement.
• Content Preparation: Crafting well-structured episodes without improvisation was challenging.
• Enhancements: Incorporating background music and other audio elements to enrich the listening experience required additional effort.

These hurdles led to my podcast being suspended for approximately two years. I found myself in need of a solution that could simplify the process and revitalize my passion for podcasting.

Discovering NotebookLM: An AI-Powered Podcasting Tool

Recently, I stumbled upon NotebookLM (notebooklm.google.com), an innovative application developed by Google. NotebookLM harnesses the power of artificial intelligence to generate podcast content. Users can provide a topic and related documents, and the AI takes over, creating engaging podcast episodes.

My Experience with NotebookLM

Intrigued, I decided to give NotebookLM a try. The results were nothing short of astounding:

• Effortless Production: The AI effortlessly generated a half-hour episode featuring two speakers discussing the topics in English.
• Enhanced Content: It went beyond the provided information, utilizing search engines to gather additional relevant data from the internet.
• Quality Output: The quality of the generated content was exceptionally high, surpassing what I could produce on my own.
• Incorporated Music: Appropriate background music was added, enhancing the overall listening experience.
• Cost-Free: All these features were available entirely for free.

A Case Study: Deep Dive into TiDB

To put NotebookLM to the test, I created an episode about TiDB, a product developed by my current employer. The process was seamless, and the final product was impressive. You can listen to the episode here: Deep Dive into TiDB.

Conclusion

The integration of AI into podcast creation through tools like NotebookLM has the potential to revolutionize the way we produce content. It removes many of the barriers that podcasters face, such as time constraints, technical challenges, and the need for consistent quality.

For anyone looking to start or rejuvenate their podcast without the traditional hassles, I highly recommend giving NotebookLM a try. It’s remarkable to see how AI can not only match but enhance human capabilities in creative endeavors.

I hope this helps! Let me know if there’s anything you’d like to add or modify in your blog post.

Key Features

• Compact Size: The trifold design results in a very small folded size.
• Lightweight: Weighing between 10 to 11 kg, making them easy to carry.
• Easy Transportation: Suitable for air travel, and can be taken on public transport like MRT and buses.
• Thin Tires: Facilitates easy movement and efficient riding.
• Front Bracket: Allows for quick changes of front bags.
• Popularity: Highly popular in Singapore.
• Non-standard Accessories: May require specific accessories due to unique design.

My bike is a Brompton clone, and I’ll share some insights on the various brands and models available.

Brompton Bikes

Brompton is the original brand offering high-quality trifold bikes.

• Price Range: Approximately 10,000 RMB to 40,000 RMB, with common models around 15,000 RMB.
• Models:
  • C Line: Features 3-speed internal hub gears.
  • P Line: Equipped with 4-speed external shift gears.

Brompton Clone Brands

For those seeking more affordable options, several brands offer Brompton-like bikes ranging from 3,000 to 7,000 RMB.

Aceoffix

• Gears: Available in 3 or 5-speed external shift gears.
• Frame Material: Options of steel or aluminum frames.
• Compatibility: Standard size compatible with Brompton accessories.

Cran

• Gears: Offers 7 or 9-speed external shift gears.
• Sizes:
  • 7-Speed: Standard size.
  • 9-Speed: Non-standard size.
• Frame Material: Primarily steel; aluminum frames may have stability issues.

3Sixty

• One of the earliest Brompton clones.
• Features internal shift gears.

Yubu (鱼布)

• Custom assembled by a Brompton enthusiast in Hangzhou.
• Gears: 3-speed external shift gears.
• Reputation: Known for high quality and reliability.

Dagger

• A new entrant in the market.
• Price: Offers bikes at 3,700 RMB with features comparable to higher-priced models (>5,000 RMB).
• Value: High cost-performance ratio.

Choosing the Right Bike

• Avoid Internal Shift Gears: While reliable, they add weight to the bike.
• Optimal Gears: 3 or 5-speed external gears are stable and sufficient for most needs.
• Portability Concerns: 9-speed models might be less convenient for transportation due to size.
• Standard Size Preference: Ensures compatibility with standard transfer cases and accessories.
• Pedals: Consider non-foldable pedals with quick-release options for convenience.

Essential Accessories

• Lighting: Front and rear lights are mandatory by law in Singapore.
• Additional Gear:
  • Bottle Bracket
  • Bell
  • Phone Bracket
  • Helmet
  • Front and Rear Lights
• Bag: A Brompton two-strap bag for carrying essentials.

Riding Tips

• UV Protection: Use sun cream or wear sun-protective clothing.
• Saddle Upgrade:

  • Wide Saddle: For rides longer than 1 hour. 90 degrees is ideal for the saddle angle.

    

  • Thin Saddle: Suitable for shorter rides. 60 degrees is ideal for the saddle angle.

    

• Accessory Shopping: Most accessories can be found at stores like Decathlon, though some items may require sourcing elsewhere due to sizing.

Note: While many accessories are readily available at retailers like Decathlon, certain items like helmets may have limited sizing options.

Sanrenmu 9201

• Price: 198 Yuan
• Lock Type: Axis Lock
• Handle Material: Plastic
• Blade Material: D2 Steel

First Impressions

The 9201 features an axis lock mechanism, which I found to be very smooth and easy to open and close. The D2 steel blade offers good edge retention, and the plastic handle keeps the knife lightweight.

Pros

• Ease of Use: The axis lock allows for ambidextrous operation, making it convenient for both left and right-handed users.
• Lightweight: The plastic handle reduces overall weight, making it comfortable to carry.

Cons

• Poor Detailing: Some edges and surfaces were rough, causing minor cuts to my hand during use.
• Finish Quality: The attention to detail was lacking, which detracted from the overall experience.

Personal Modifications

I used sandpaper and a rotary tool to smooth out the rough edges. While this improved the handling, I accidentally scratched the blade slightly during the process.

Sanrenmu 9008

• Price: 168 Yuan
• Lock Type: Frame Lock with Safety Lock
• Handle Material: Steel
• Blade Material: Unknown
• Features: Quick-opening flipper

First Impressions

Out of the box, the 9008 showcased excellent workmanship. The quick-opening flipper worked flawlessly, and the safety lock adds an extra layer of security when the knife is not in use.

Pros

• Build Quality: The knife feels solid and well-constructed.
• Quick-opening Flipper: Provides rapid deployment of the blade.
• Safety Lock: Prevents accidental opening.

Cons

• Weight: The steel handle makes it quite heavy, which isn’t ideal for everyday carry.
• Button Quality: The button mechanism isn’t as smooth as I would like.
• Blade Material: Unspecified steel that doesn’t hold an edge as well as higher-end materials.

Usage

Due to its weight, I decided to keep the 9008 in my car. It features a window-breaking hammer, making it a practical tool for emergencies.

Sanrenmu 820

• Price: 298 Yuan
• Lock Type: Frame Lock
• Handle Material: Titanium
• Blade Material: VG10 Steel
• Features: Ball bearings (no safety lock or flipper)

First Impressions

The 820 stands out with its titanium handle and VG10 blade, offering a premium feel. It operates on ball bearings, ensuring a smooth action when opening and closing.

Pros

• Material Quality: The VG10 blade and titanium handle are high-quality materials that enhance durability and aesthetics.
• Workmanship: Excellent fit and finish.

Cons

1. Opening Mechanism Issues:

   • The stop pin is too small.
   • The lock interface is large and deep, making one-handed opening difficult.

2. Pocket Clip:

   • The clip is excessively tight, making it hard to attach to pockets easily.

Personal Modifications

I disassembled the 820 and polished the lock interface, significantly improving the ease of one-handed opening. Post-modification, the knife opens smoothly, though there’s a slight looseness when closed.

Reflections and Insights

• Axis Lock vs. Frame Lock:

  • Axis Lock (9201): Offers ambidextrous use and works well with a plastic handle but has a complex structure that’s challenging to disassemble.
  • Frame Lock (820 & 9008): Simpler design and easier to disassemble (especially the 820 after some practice). However, it’s more suited to one-handed use and relies heavily on precise craftsmanship for smooth operation.

• Handle Materials:

  • Titanium Handle (820): Initially underestimated, the titanium handle provides a superior feel compared to plastic or steel. Once accustomed to it, switching back to other materials feels like a downgrade.
  • Steel Handle (9008): Durable but adds unnecessary weight, making it less ideal for carrying around.
  • Plastic Handle (9201): Lightweight but doesn’t offer the same premium feel as metal handles.

• Practicality vs. Aesthetics:

  • The 9008, while heavy, serves a practical purpose in the car for emergencies.
  • The 820, after modifications, has become my preferred everyday carry despite the tight pocket clip.

• DIY Modifications:

  • Personalizing and improving the knives was a rewarding experience, even if there were minor mishaps like scratching the blade on the 9201.
  • Disassembling and reassembling the 820 has turned it into not just a tool but a hobbyist’s toy.

Conclusion

Each knife has its place:

• Sanrenmu 9008: Stays in the car for emergency situations, thanks to its sturdy build and window-breaking feature.
• Sanrenmu 820: Becomes my go-to carry knife after modifications. Its high-quality materials and improved functionality make it stand out.
• Sanrenmu 9201: Although I’m unsure of its future use, it might serve as an additional car knife or a backup.

If you’re considering a Sanrenmu folding knife, think about what features matter most to you—be it the locking mechanism, handle material, or ease of carry. Also, don’t shy away from making personal adjustments to tailor the knife to your preferences.

Note: This review is based on personal experiences and modifications. Results may vary depending on individual usage and skills.

Recommended Usage

• Compile all Go files in the current directory:

  go build
  

• Compile all Go files explicitly:

  go build *.go
  

Common Pitfalls

Compiling a Single File

Running:

go build main.go

will only build main.go. This can lead to errors if main.go depends on other Go files in the same package, as those files won’t be included in the build process.

Including Non-Go Files

Using:

go build *

will cause an error if the * wildcard includes non-Go files. The compiler will output an error message stating that only Go files can be compiled. This serves as a precise reminder to exclude non-Go files from the build command.

By using go build correctly, you can ensure that all necessary files in your package are compiled together, avoiding missing dependencies and other common compilation issues.

Table of Contents

• Introduction to Ollama
  • A Popular Choice
  • Ease of Use
  • Built with Golang
• My Practices with Ollama
  • Preferred Models
    • Llama 3.1
    • Mistral
    • Phi-3
    • Qwen-2
  • Hardware Constraints
• Exploring UIs for Ollama
  • OpenWebUI
  • Page Assist
  • Enchanted
  • AnythingLLM
  • Dify
• Diving into llamaindex
• Experimenting with Candle
• Conclusion

Introduction to Ollama

A Popular Choice

Ollama has rapidly become a favorite among developers interested in local LLMs. Within a year, it has garnered significant attention on GitHub, reflecting its growing user base and community support.

Ease of Use

One of Ollama’s standout features is its simplicity. It’s as easy to use as Docker, making it accessible even to those who may not be deeply familiar with machine learning frameworks. The straightforward command-line interface allows users to download and run models with minimal setup.

Built with Golang

Ollama is written in Golang, ensuring performance and efficiency. Golang’s concurrency features contribute to Ollama’s ability to handle tasks effectively, which is crucial when working with resource-intensive LLMs.

My Practices with Ollama

Preferred Models

Llama 3.1

I’ve found that Llama 3.1 works exceptionally well with Ollama. It’s my go-to choice due to its performance and compatibility.

Mistral

While Mistral also performs well, it hasn’t gained as much popularity as Llama. Nevertheless, it’s a solid option worth exploring.

Phi-3

Developed by Microsoft, Phi-3 is both fast and efficient. The 2B parameter model strikes a balance between size and performance, making it one of the best small-sized LLMs available.

Qwen-2

Despite its impressive benchmarks, Qwen-2 didn’t meet my expectations in practice. It might work well in certain contexts, but it didn’t suit my specific needs.

Hardware Constraints

Running large models on hardware with limited resources can be challenging. On my 16GB MacBook, models around 7B to 8B parameters are the upper limit. Attempting to run larger models results in performance issues.

Exploring UIs for Ollama

Enhancing the user experience with UIs can make interacting with local LLMs more intuitive. Here’s a look at some UIs I’ve tried:

OpenWebUI

OpenWebUI offers a smooth and user-friendly interface similar to Ollama’s default UI. It requires Docker to run efficiently, which might be a barrier for some users.

• Features:
  • Basic Retrieval-Augmented Generation (RAG) capabilities.
  • Connection to OpenAI APIs.

Page Assist

Page Assist is a Chrome extension that I’ve chosen for its simplicity and convenience.

• Advantages:
  • No requirement for Docker.
  • Accesses the current browser page as input, enabling context-aware interactions.

Enchanted

Enchanted is unique as it provides an iOS UI for local LLMs with support for Ollama.

• Usage:
  • By using Tailscale, I can connect it to Ollama running on my MacBook.
  • Serves as an alternative to Apple’s native intelligence features.

AnythingLLM

AnythingLLM offers enhanced RAG capabilities. However, in my experience, it hasn’t performed consistently well enough for regular use.

Dify

Dify is a powerful and feature-rich option.

• Pros:
  • Easy to set up with an extensive feature set.
• Cons:
  • Resource-intensive, requiring Docker and running multiple containers like Redis and PostgreSQL.

Diving into llamaindex

llamaindex is geared towards developers who are comfortable writing code. While it offers robust functionalities, it does have a learning curve.

• Observations:
  • Documentation is somewhat limited, often necessitating diving into the source code.
  • The llamaindex-cli tool aims to simplify getting started but isn’t entirely stable.
    • Works seamlessly with OpenAI.
    • Requires code modifications to function with Ollama.

Experimenting with Candle

Candle is an intriguing project written in Rust.

• Features:

  • Uses Hugging Face to download models.
  • Simple to run but exhibits slower performance compared to Ollama.

• Additional Tools:

  • Cake: A distributed solution based on Candle, Cake opens up possibilities for scaling and extending use cases.

Conclusion

Exploring local LLMs has been an exciting journey filled with learning and experimentation. Tools like Ollama, llamaindex, and Candle offer various pathways to harnessing the power of LLMs on personal hardware. While there are challenges, especially with hardware limitations and setup complexities, the control and privacy afforded by local models make the effort worthwhile.

Feel free to share your experiences or ask questions in the comments below!

Discovering Tailscale Through WebVM

My journey with Tailscale began when I came across WebVM, an impressive project that allows you to run a virtual machine directly in your browser. Intrigued by the possibilities, I delved deeper and discovered that Tailscale could help me create a seamless, private network across all my devices.

What is Tailscale?

Tailscale is a mesh VPN network built on top of WireGuard, specifically using the WireGuard-go implementation. It allows you to create a secure, encrypted network between your devices, no matter where they are located.

Key Features

• Free Plan Available: Tailscale offers a free plan that is sufficient for personal use, allowing up to 20 devices.
• Ease of Use: Setting up Tailscale is straightforward. With minimal configuration, you can have your own network up and running quickly.
• Cross-Platform Support: Tailscale works exceptionally well across the Apple ecosystem, including iOS, tvOS, and macOS.
• Magic DNS Service: It provides a built-in DNS service that makes it easy to address your devices by name.

Performance on Different Platforms

While Tailscale shines on Apple devices, in my experience, it hasn’t performed as well on Windows. I encountered some connectivity and stability issues on Windows machines, which may vary based on individual setups.

My Tailscale Setup

Here’s how I leveraged Tailscale to connect my devices and access my home network seamlessly.

Running Tailscale on Apple TV

I installed Tailscale on my Apple TV, which stays online 24/7. This makes it an excellent candidate for a consistently available node in my network.

• Enabling Subnet Routing: By enabling subnet routing on the Apple TV, I can access other devices on the same local network, such as my NAS and router, as if I were connected locally.
• Setting Up an Exit Node: I configured the Apple TV as an exit node, allowing me to route internet traffic through my home network. This is useful when I need to access geo-restricted content or ensure a secure connection.

Connecting Other Devices

I also installed Tailscale on my MacBook and iPhone, which allows all my personal devices to communicate over the secure network, no matter where I am.

Benefits I’ve Enjoyed

• Secure Remote Access: I can securely access my home network devices from anywhere.
• Consistent Environment: All my devices appear on the same network, simplifying file sharing and remote management.
• No Need for Complex VPN Setups: Tailscale eliminates the need for traditional VPN configurations, port forwarding, or dynamic DNS services.

Conclusion

Tailscale has transformed the way I interact with my devices across different locations. Its ease of use and robust feature set make it an excellent choice for anyone looking to create a personal, secure network.

If you’re interested in simplifying your network setup and want a hassle-free way to connect your devices, I highly recommend giving Tailscale a try.

Links:

• Tailscale Official Website
• WebVM Project on GitHub
• WireGuard-go on GitHub

Note: This post reflects my personal experiences with Tailscale. Performance may vary based on individual configurations and devices.

Internet Access and Blocked Websites

• Slow Access to Twitter: If you’re an avid Twitter user, be prepared for slower connection speeds. The platform doesn’t perform as well in Vietnam.
• Inaccessible Websites: Certain websites are blocked in Vietnam, including:
  • Medium
  • Some porn websites
• Tip: If you rely on any of these platforms, consider downloading content beforehand or using a VPN to access them during your stay.

Visa Process for Chinese Citizens

• E-Visa Conversion: Travelers from China holding an E-Visa need to convert it into a paper visa upon arrival.
  • Bring Two Photos: Ensure you have two passport-sized photos ready for the visa conversion process.
  • Carry Cash for Tips: It’s customary to offer tips during the visa processing, so have some cash (preferably small denominations) on hand.

Exploring Saigon (Ho Chi Minh City)

• City Vibes: Saigon reminded me of a blend between Shanghai and Qingdao.
  • Cleanliness: While the city is bustling with energy, it’s not as clean as Shanghai or Qingdao.
• Abundance of Coffee Shops: Vietnam is famous for its coffee culture. You’ll find numerous coffee shops on almost every street corner.
  • Must-Try: Don’t miss out on traditional Vietnamese iced coffee (Cà Phê Sữa Đá).

Affordability

• Cheaper Than Bangkok: Compared to cities like Bangkok, Saigon is significantly more affordable.
  • Hotels: Accommodation can be about half the price of equivalent hotels in Bangkok.
  • General Expenses: Food, transportation, and entertainment are also reasonably priced.

Currency and Payments

• Accepted Currencies:
  • Vietnamese Dong (VND) is the official currency.
  • US Dollars (USD) are widely accepted and sometimes preferred.
  • Chinese RMB can be used in some places, but USD is more commonly accepted.
• Tip: Carrying USD can be convenient, but always have some local currency for small purchases.

Local Customs and Observations

• Clothing Choices: You’ll notice many locals wearing long-sleeved clothing and pants, even in hot weather.
  • Reason: This is to prevent sunburn and protect their skin from the sun.
• Hardworking People: The Vietnamese are known for their strong work ethic. Streets are lively with activity from early morning until late at night.
• Affordable Goods: Many items, from street food to souvenirs, are priced affordably.
• Non-Spicy Cuisine:
  • Similar to Cantonese Food: The flavors are mild and focus on the freshness of ingredients.
  • Must-Try Dishes: Pho (noodle soup), Banh Mi (baguette sandwich), and fresh spring rolls.

Final Thoughts

Vietnam offers a captivating blend of cultural experiences, historical sites, and modern attractions. Whether you’re sipping coffee in a cozy café, exploring bustling markets, or simply taking in the sights and sounds of the city, there’s something for every traveler.

Travel Tips:

• Plan Ahead: Be aware of the internet limitations and plan accordingly if you need access to certain websites.
• Stay Hydrated: The climate can be hot and humid, so drink plenty of water.
• Respect Local Customs: Dress modestly when visiting temples or rural areas.

I hope these insights help you in planning your trip to Vietnam. It’s a destination filled with unforgettable experiences!

Safe travels!

Introduction

Over the years, I’ve gathered a lot of experience in fishing, especially in selecting the right gear. Choosing the appropriate rod, reel, line, and lure can significantly impact your fishing success. In this blog, I want to share my insights to help you make informed decisions when selecting fishing equipment.

Rod Selection

Understanding Rod Actions

• L (Light) Action Rods:
  • Best for casting lures weighing 4-5g.
  • Ideal for using lures within half of the rod’s lure weight range (e.g., for a rod rated 2-7g, the optimal lure weight is around 4-5g).
• UL (Ultra-Light) Action Rods:
  • Suitable for casting lures weighing 3g or less.

One-Piece vs. Multi-Piece Rods

• One-Piece Rods:
  • Offer better sensitivity and casting performance.
  • However, they are less portable and harder to transport.
  • Not as easy to sell if you decide to upgrade.
• Multi-Piece Rods:
  • More convenient for travel.
  • Slight trade-off in sensitivity compared to one-piece rods.

Recommended Action

• The best action for spinning cast rods is Light (L) action.

Reel Selection

Spinning Reels vs. Baitcasting Reels

• Spinning Reels:
  • Easier for casting, especially for beginners.
  • Better suited for saltwater fishing than baitcasting reels.
  • Can use heavier action rods to cast small lures effectively.
• Baitcasting Reels:
  • Look cool and offer precision, but have a steeper learning curve.
  • Not as user-friendly for casting light lures.

Reel Size and Line Compatibility

• For a 2500S reel:
  • 0.8 PE line is sufficient.
  • Using 0.6 PE line on a 2500S reel may require too much line to fill the spool.
• For a 2000S reel:
  • 0.6 PE line is appropriate.
• It’s not common to use PE lines smaller than 0.6, as they are less widely available.

Light Weight Reels

• Always choose lightweight reels for better balance and less fatigue during fishing.

Line Selection

PE Line vs. Fluorocarbon

• Casting Difficulty:
  • 1.5 PE and fluorocarbon lines offer similar casting challenges.
• PE Line Advantages:
  • Thinner diameter for the same strength compared to fluorocarbon.
  • Essential for longer casting distances.
• Avoiding Line Connections:
  • Use a Carolina rig to eliminate the need to connect PE line and fluorocarbon leader.

Recommended Lines

• YGK PE Line:
  • Highly recommended for its quality.
  • Performs much better than Sufix lines.
  • Consider using 0.6 or 0.8 PE from YGK.
• Fluorocarbon Line:
  • 1.5 Fluorocarbon can be used as a leader if necessary.

Rigs and Lures

• Carolina Rig:
  • Allows you to use PE line without needing to connect it to a fluorocarbon leader.
  • Effective for various fishing conditions.

Recommendations

Final Gear Choices

• Rod:
  • L Action Spinning Rod with multiple pieces for portability.
• Reel:
  • 2000S or 2500S spinning reel, prioritize lightweight models.
• Line:
  • 0.6 or 0.8 YGK PE line (lean towards 0.6 for more capacity).
  • Optional 1.5 Fluorocarbon leader.
• Rig:
  • Use a Carolina rig setup.

Brands and Models

Chinese Domestic Market (CDM) Rods

• Few high-quality CDM spinning rods are available.
• The best CDM spinning rod is the 翠鸟 (Kingfisher) from 钓之屋 (Fishing House).
• The 游侠 (the larger version of 翠鸟) is not as good.
  • Example: A 2500S reel on a 游侠 lacks proper rod skewness, affecting casting performance.

Shimano Rods and Reels

• Shimano spinning rods and reels are superior in performance.
• The reels have features that prevent accidental rolling after opening the bail, allowing you to stop the line with your finger for better control.

Reels I’ve Used

• Shimano Vanquish 2500S (2023 model):
  • Extremely lightweight and smooth.
  • Feels almost too slim, raising concerns about durability.
  • The infinite loop system is excellent.
• Shimano Stradic 2500S (2023 model):
  • Heavier compared to Vanquish.
  • Handle design is not as comfortable.
  • Appearance gives a sense of reliability.
• Shimano Vanford C2000SHG (2020 model):
  • Great performance but similar concerns about durability due to plastic components.
  • Feels less robust than the Stradic but is lighter.

Price Comparison (in RMB)

• 钓之屋 翠鸟 (Kingfisher): 360
• Shimano Stradic: 900
• Shimano Vanford: 1,100
• Shimano Vanquish: 2,600
• Shimano Stella: 3,800

Conclusion

Choosing the right fishing gear is crucial for an enjoyable and successful fishing experience. By considering the rod action, reel type and size, line selection, and the right rigs, you can optimize your setup for the best performance. While high-end brands like Shimano offer top-quality equipment, there are also cost-effective options available. Remember to prioritize what suits your fishing style and comfort.

Happy fishing!

After researching, I found that this problem is not unique. For instance, in a discussion on V2EX, many users encountered similar issues. The solution is to run warp-cli set-mode proxy before executing warp-cli connect to bypass the local address. Surprisingly, Cloudflare’s official documentation does not mention this crucial step, undoubtedly increasing the complexity and risk of configuration.

In the process of exploring solutions, I found that some users suggested repairing by rebuilding the instance or using VNC connection. However, since I am using AWS Lightsail, VNC is not applicable. Ultimately, I decided to try the method mentioned in this article: creating a snapshot backup of the current VPS, then creating a new instance from the snapshot, and loading a script to execute warp-cli set-mode proxy when the new instance starts.

After checking the existing instance, I found that no snapshot had been created. This discovery reminded me of the importance of regular backups. Without other options, I could only attempt a snapshot backup as guided by the aforementioned article. However, no matter what startup script command I tried, it failed to execute successfully. The execution result of AWS Lightsail’s startup script is not visible, making problem-solving more difficult.

In near desperation, I found an old snapshot dated 2022 on the snapshot page. Although this snapshot was created using old technology, and many important updates might be lost after recovery, it was my last hope. After starting the snapshot recovery process, I unexpectedly discovered through the history command that this snapshot contained all the important updates. This discovery allowed the entire recovery process to be completed smoothly.

This experience re-emphasized the importance of backups. Careful backups from the past ultimately avoided severe data loss. Furthermore, AWS’s static IP retention feature also played a crucial role. The new instance could immediately bind to the IP once the old instance released the static IP, achieving a seamless switch.

Conclusion

1. Backups are essential: Regular backups are key to ensuring stable system operations.
2. Operate with caution: Before executing critical commands, thoroughly review and understand relevant documentation and user feedback to avoid potential risks.
3. Trust your past self: Meticulous work done in the past can often prove invaluable at critical moments.

I hope this experience can serve as a reference and help for others, preventing similar issues from occurring.

Many customers have TLS enabled, which is different from the lab environment.

Curl

Curl requires a specified CA certificate, otherwise it will report an error.

curl --cacert ca.crt https://127.0.0.1:10080/status

Wget

Many containers do not have curl, so wget is used instead. Wget is better as it does not require a CA certificate.

wget --no-check-certificate http://127.0.0.1:10080/status

After moving house, there are many more devices at home that need internet access. However, I don’t want to configure a proxy on each device, so I thought of using a transparent gateway.

Transparent Gateway

After some research, I found that the easiest way is to use the premium version of Clash, although I didn’t know when Clash released a premium version. I mainly referred to this article. It’s much simpler than setting up iptables.

Network Topology

I have a 10-year-old Thinkpad x230 at home, which is perfect for this purpose. Here is a simple topology diagram.

Router1 is a fiber-optic modem with routing capabilities, Router2 is a regular router, with the gateway and DNS pointing to the Thinkpad, where Linux is running to act as a transparent gateway with Clash on top.

                                 +------------+
                                 |            |
                                 |  Internet  |
                                 |            |
                                 +-----+------+
                                       |
                                 +-----+------+
                                 |            |
                      +----------+  Router1   +-----------+
                      |          |            |           |
                      |          +------------+           |
                      |                                   |
                      |                                   |
                +-----+-----+                       +-----+------+
                |           |                       |            |
     +----------+  Router2  +----------+            |  Thinkpad  |
     |          |           |          |            |            |
     |          +-----+-----+          |            +------------+
     |                |                |
     |                |                |
     |                |                |
+----+-----+     +----+-----+    +-----+-----+
|          |     |          |    |           |
|   Mac    |     |  iPad    |    |  iPhone   |
|          |     |          |    |           |
+----------+     +----------+    +-----------+

Add DNS Section in Clash Configuration

dns:
enable: true
listen: 0.0.0.0:53
enhanced-mode: fake-ip
nameserver:
  - 114.114.114.114
fallback:
  - 8.8.8.8

Clash tun Feature Section

tun:
enable: true
stack: system # or gvisor
dns-hijack:
  - any:53
  - tcp://any:53
auto-route: true
auto-detect-interface: true

For traffic forwarding, simply edit /etc/sysctl.conf on the Thinkpad and add net.ipv4.ip_forward=1, then execute sysctl -p to apply it. After that, point the gateway and DNS of Router2 to the Thinkpad, and you’re done.

Network Protocols

Initially, I used native HTTP2 for unblocking, but it cannot proxy UDP. When only a few devices need unblocking, it doesn’t matter whether UDP is used, but with many devices at home, some of them can only use UDP. I considered socks + tls, but it didn’t feel secure and required opening odd ports like UDP 443. It felt like giving away my intentions. Eventually, I chose Trojan, which essentially mimics native HTTPS. Trojan has two versions; I used Trojan-go simply because I didn’t want to manage dependencies. Also, I’m more familiar with Go.

Trojan-go has a requirement for a genuinely accessible HTTP server, so I used the simplest Python http.server. Back in Python 2, it was called simplehttp. You can simply use python3 -m http.server 80 and optionally add --directory to specify a directory.

Additionally, Trojan-go requires the client to fill in the SNI, which means using the domain used during key application. Therefore, prerequisites like applying for the domain, applying for Let’s Encrypt certificates, and configuring crontab must all be completed. There’s a learning curve, but I had done it before, so I just skipped that part.

For the client part, you can use Clash directly, and refer to here for guidance.

Recently, both of my kids have become interested in Minecraft, and some of their peers are also playing it. We previously bought the Switch version, but its online capabilities are quite poor, and the device performance is subpar, resulting in a less-than-ideal experience. Thus, I began considering the idea of setting up our own server. Of course, you can play in multiplayer with friends, but the game ends as soon as the host goes offline, which is not as good as having a server that is always online.

For version selection, there is the NetEase version, the official Java version, and the Bedrock version. Based on my previous understanding, the NetEase version has all sorts of anti-addiction regulations, so that’s a no-go. The Java version server setup seems to rely on third-party launchers, resembling piracy to some extent. Therefore, I decided on the Bedrock version. Another reason for choosing the Bedrock version is its origins as the mobile version of Minecraft, and Microsoft later expanded support to more platforms, making it the most versatile. Additionally, its core is written in C++, which should offer better performance and resource efficiency.

Downloading the Server Software

I downloaded the Ubuntu version. While I also downloaded the Windows version and managed to run it, I’m not experienced in managing Windows Server, and the Windows version has some quirks. I won’t delve into those here.

Local Network Server

Initially, I intended to use an old Thinkpad, but couldn’t find it, and ended up using an old Macbook. After updating everything, I found that the old Macbook was still quite robust, outperforming the Thinkpad. So, I set up a virtual machine. I discovered that Ubuntu Server no longer supports direct ISO downloads, requiring Multipass to launch instead.

What is Multipass?

Developed by the Ubuntu team, it is a lightweight virtualization platform. Multipass consists of multipass and multipassd. The former provides both GUI and CLI, while the latter requires root permissions and runs in the background. It can be downloaded directly or installed via brew cask.

Multipass seems to only install Ubuntu Server, with some customization, such as automatically creating a user named “ubuntu” and key pairing. Once installed, you get a 1C 1GB virtual machine called “primary” that automatically mounts the user’s home directory.

multipassd is more of a pluggable virtual hypervisor supporting hyperkit and qemu by default, with hyperkit intended for Intel macOS and qemu for M1.

multipassd can also use Virtualbox as a backend hypervisor, which is more recommended because it offers bridging capabilities, whereas relying solely on port mapping would be cumbersome. It’s not that qemu doesn’t support bridging, but there’s an easy-to-follow Virtualbox bridging tutorial on Multipass’s official documentation.

I set up Virtualbox first. I must say, Oracle is generous here as it’s still free to use. Then, I followed the steps from the link above. It’s essential to reboot the machine after running the first step sudo multipass set local.driver=virtualbox, as the variable might not take effect immediately. Otherwise, the primary created will still use the old backend. Since there’s already a default virtual machine, I didn’t want to create another one to avoid extra resource usage. Additionally, note a few things:

1. For modifying primary configuration, I didn’t find a way to do it using Multipass, so I used vbox’s command sudo VBoxManage controlvm "primary" --cpus 2 --memory 4096 (where memory is in MB).
2. The primary mount point in Multipass can automatically unmount in some situations, resulting in errors. Therefore, Minecraft data shouldn’t be stored in the mount point permanently to avoid core dumps.

After extracting the server files, just start it in tmux/screen.

Lastly, I used Amphetamine from the App Store to ensure that the laptop continues working when closed by disabling the default sleep setting after the screen is closed. You’ll see some warnings when doing this, but just be aware of them.

Update:

For systemd startup, you can refer to this gist. The server’s console has some commands, like “stop”, that perform graceful shutdowns, so it’s necessary to rely on a screen session to create it. Additionally, the server startup has some environmental dependencies, so it’s best to write a small script for starting it, like:

#!/bin/bash

cd /home/user/bedrock-server
LD_LIBRARY_PATH=. ./bedrock_server

Using absolute paths can lead to core dumps.

Public Network Server

For public network servers, the choices are to map the internal network to the public network or purchase a cloud server. Due to security concerns, I opted to buy a cloud server. Although the official site mentioned Ubuntu support, I found that Debian servers also run without issues. The only thing to note is that the Bedrock version uses UDP, and China Unicom’s 5G network disables UDP, so without WiFi, you cannot connect.

Server Monitoring

To monitor the server’s status, monitoring is necessary, especially for cloud servers, as provider information might be insufficient. I used Grafana Cloud.

Using the free version is fine, just follow the guide to select a Linux server integration, then run the Grafana Agent installation and verification on the server to be monitored. Note to change the hostname in the Grafana Agent configuration file to differentiate between different servers, which acts as a label.

I understand that Grafana Agent is a lightweight node exporter with some Prometheus functionalities, but it can also remote write, so there’s no need to worry about the disk filling up.

One day, I impulsively turned on GitHub’s Vigilant mode.

As a result, all my commits started looking like this.

To figure out how to make them Verified, I found the following method.

Method

I actually referred to this link. However, it wasn’t quite enough, as there might be authentication-related issues on MacBooks that lead to commit errors. So, I found this solution.

In summary, to verify, you need to enter a password. The issue on a Mac is the prompt for entering the password, which needs to be replaced with pinentry-mac, which most people install via homebrew.

Moreover, this solution thoughtfully provides a way to verify:

echo "test" | gpg --clearsign

GPG Experience

1. It doesn’t replace the ssh key. After successfully setting it up, I deleted my GitHub ssh key and discovered that I couldn’t log in. Actually, it only verifies the legitimacy of commits.
2. On the local machine, in any repo, you only need to enter the password once, and that makes it a verified commit. It doesn’t affect daily use; it just adds a green check mark for verification.
3. Using the https protocol + token seems more reliable than this method, but I’m not sure if it provides a verified mark.

During the Chinese New Year holiday, I spent some time at home playing on the Switch with my child, and I ended up purchasing Minecraft (Bedrock Edition). In order to play together, I also bought the Windows version. Additionally, I wanted to test if the GitHub Actions for this blog are still working properly.

Preparation

You need two Microsoft accounts, so I dug out my wife’s long-unused Hotmail account.

Multiplayer

1. Restart the game on the Switch and log into the Microsoft account. It has to be restarted each time, and I’m not sure why.
2. Add each other as friends using the short ID.
3. Create a world on the Switch; Windows should join (it doesn’t work the other way around, and I’m not sure why).
4. Enjoy playing together!

This guide is brief and straightforward, with nothing much more to add.

Recently, I encountered a problem where a user wanted to synchronize data from PostgreSQL to TiDB (which uses the same protocol as MySQL) and wanted to know whether the data after synchronization is consistent. I hadn’t dealt with this kind of issue before, so I did a bit of research.

Typically, to verify data consistency, you compute a checksum on both sides and compare them.

TiDB (MySQL) Side

For the verification of a specific table, the following SQL is used:

SELECT bit_xor(
    CAST(crc32(
        concat_ws(',',
            col1, col2, col3, …, colN,
            concat(isnull(col1), isnull(col2), …, isnull(colN))
        )
    ) AS UNSIGNED)
)
FROM t;

Let’s look at a specific example:

DROP TABLE IF EXISTS t;
CREATE TABLE t (i INT, j INT);
INSERT INTO t VALUES (2, 3), (NULL, NULL);
SELECT bit_xor(
    CAST(crc32(
        concat_ws(',',
            i, j,
            concat(isnull(i), isnull(j))
        )
    ) AS UNSIGNED)
)
FROM t;

The result is:

+-------------------------------------------------------------------------------------------------------------------------------------------+
| bit_xor(
    CAST(crc32(
        concat_ws(',',
            i, j,
            concat(isnull(i), isnull(j))
        )
    ) AS UNSIGNED)
) |
+-------------------------------------------------------------------------------------------------------------------------------------------+
|                                                           5062371 |
+-------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

PostgreSQL Side

The goal is simply to write the same SQL as above, but PostgreSQL does not support bit_xor, crc32, isnull, nor does it have unsigned types. Therefore, the solution is relatively straightforward—relying on UDFs (User-Defined Functions).

After some research, the main missing functions can be addressed with a few custom implementations.

bit_xor:

CREATE OR REPLACE AGGREGATE bit_xor(IN v bigint) (SFUNC = int8xor, STYPE = bigint);

crc32:

CREATE OR REPLACE FUNCTION crc32(text_string text) RETURNS bigint AS $$
DECLARE
    tmp bigint;
    i int;
    j int;
    byte_length int;
    binary_string bytea;
BEGIN
    IF text_string = '' THEN
        RETURN 0;
    END IF;

    i = 0;
    tmp = 4294967295;
    byte_length = bit_length(text_string) / 8;
    binary_string = decode(replace(text_string, E'\\', E'\\\\'), 'escape');
    LOOP
        tmp = (tmp # get_byte(binary_string, i))::bigint;
        i = i + 1;
        j = 0;
        LOOP
            tmp = ((tmp >> 1) # (3988292384 * (tmp & 1)))::bigint;
            j = j + 1;
            IF j >= 8 THEN
                EXIT;
            END IF;
        END LOOP;
        IF i >= byte_length THEN
            EXIT;
        END IF;
    END LOOP;
    RETURN (tmp # 4294967295);
END
$$ IMMUTABLE LANGUAGE plpgsql;

isnull:

CREATE OR REPLACE FUNCTION isnull(anyelement) RETURNS int AS $$
BEGIN
    RETURN CAST(($1 IS NULL) AS INT);
END
$$ LANGUAGE plpgsql;

After creating the three UDFs above, let’s test the previous example. Note that UNSIGNED should be changed to BIGINT.

DROP TABLE IF EXISTS t;
CREATE TABLE t (i INT, j INT);
INSERT INTO t VALUES (2, 3), (NULL, NULL);
SELECT bit_xor(
    CAST(crc32(
        concat_ws(',',
            i, j,
            concat(isnull(i), isnull(j))
        )
    ) AS BIGINT)
)
FROM t;

The result:

 bit_xor
---------
 5062371
(1 row)

It’s exactly the same as on the TiDB (MySQL) side.

Postscript

1. I haven’t tested more extensively; this is just a simple test.
2. UDFs are indeed a great feature that greatly enhance flexibility.

Initially, I didn’t think this was something worth writing about, because CTex was previously working smoothly for everyone. However, it turns out that CTex hasn’t been updated since 2016. So, I wanted to find a replacement for Chinese LaTeX on Windows in 2021.

Configuration Method

1. Install MiKTeX (TexLive should work as well). MiKTeX can automatically download dependency packages and you can also proactively install the ctex package.
2. Install the VSCode LaTeX extension.
3. Configure the LaTeX extension. I found a powerful configuration on Zhihu, but it’s quite complex, so I simplified it a bit.

A brief explanation: the core components here are the recipes and tools. Tools are the compilation toolchain, specifying which tools to use for compilation and the options to use, without regard to order. Recipes define how to combine the above tools to generate the final document, where the order does matter. I’ve put XeLaTex first here because it’s the most compatible for compiling Chinese. If you use pdflatex to compile Chinese documents, you’re likely to encounter issues.

{
   "latex-workshop.latex.recipes": [
         {
            "name": "XeLaTeX",
            "tools": [
                "xelatex"
            ]
        },
        {
            "name": "latexmk 🔃",
            "tools": [
                "latexmk"
            ]
        },
        {
            "name": "latexmk (latexmkrc)",
            "tools": [
                "latexmk_rconly"
            ]
        },
        {
            "name": "latexmk (lualatex)",
            "tools": [
                "lualatexmk"
            ]
        },
        {
            "name": "pdflatex ➞ bibtex ➞ pdflatex × 2",
            "tools": [
                "pdflatex",
                "bibtex",
                "pdflatex",
                "pdflatex"
            ]
        },
        {
            "name": "Compile Rnw files",
            "tools": [
                "rnw2tex",
                "latexmk"
            ]
        },
        {
            "name": "Compile Jnw files",
            "tools": [
                "jnw2tex",
                "latexmk"
            ]
        },
        {
            "name": "tectonic",
            "tools": [
                "tectonic"
            ]
        }
    ],
    "latex-workshop.latex.tools": [
        {
            "name": "xelatex",
            "command": "xelatex",
            "args": [
                "-synctex=1",
                "-interaction=nonstopmode",
                "-file-line-error",
                "%DOCFILE%"
            ]
        },
        {
            "name": "latexmk",
            "command": "latexmk",
            "args": [
                "-synctex=1",
                "-interaction=nonstopmode",
                "-file-line-error",
                "-pdf",
                "-outdir=%OUTDIR%",
                "%DOC%"
            ],
            "env": {}
        },
        {
            "name": "lualatexmk",
            "command": "latexmk",
            "args": [
                "-synctex=1",
                "-interaction=nonstopmode",
                "-file-line-error",
                "-lualatex",
                "-outdir=%OUTDIR%",
                "%DOC%"
            ],
            "env": {}
        },
        {
            "name": "latexmk_rconly",
            "command": "latexmk",
            "args": [
                "%DOC%"
            ],
            "env": {}
        },
        {
            "name": "pdflatex",
            "command": "pdflatex",
            "args": [
                "-synctex=1",
                "-interaction=nonstopmode",
                "-file-line-error",
                "%DOC%"
            ],
            "env": {}
        },
        {
            "name": "bibtex",
            "command": "bibtex",
            "args": [
                "%DOCFILE%"
            ],
            "env": {}
        },
        {
            "name": "rnw2tex",
            "command": "Rscript",
            "args": [
                "-e",
                "knitr::opts_knit$set(concordance = TRUE); knitr::knit('%DOCFILE_EXT%')"
            ],
            "env": {}
        },
        {
            "name": "jnw2tex",
            "command": "julia",
            "args": [
                "-e",
                "using Weave; weave(\"%DOC_EXT%\", doctype=\"tex\")"
            ],
            "env": {}
        },
        {
            "name": "jnw2texmintex",
            "command": "julia",
            "args": [
                "-e",
                "using Weave; weave(\"%DOC_EXT%\", doctype=\"texminted\")"
            ],
            "env": {}
        },
        {
            "name": "tectonic",
            "command": "tectonic",
            "args": [
                "--synctex",
                "--keep-logs",
                "%DOC%.tex"
            ],
            "env": {}
        }
    ],
    "latex-workshop.view.pdf.viewer": "tab"
}

1. Create a folder.
2. Use VSCode to open this folder and create a .tex file with the following content:

\documentclass[UTF8]{ctexart}

\begin{document}
OK, it’s all set...
\end{document}

It should automatically start compiling. Confirm to download any required dependencies if prompted. The compiled effect is as follows:

Go Action

Click on Actions on the repository page, and then New Workflow to see recommended actions. Since this blog uses Go code, it shows the Go Action.

The rest involves following Travis’s approach to set up the workflow.

1. Check out the blog’s repo
2. Check out the publishing site repo
3. make
4. Commit the changes to the publishing site

Note that write permissions are required for the publishing site, so you need to configure a token, similar to Travis.

1. Generate a token with only repo permissions
2. Go to a particular repo and set up secrets (enter the token). Ideally, this should be set up on the publishing site, but it works when set in the blog repo. I haven’t explored why yet.

You will need two actions in total: one is GitHub’s own checkout, and the other is a third-party action called “Push directory to another repository”. There might be better options available, and I’ll explore them when I have more time.

Finally, here is my simple GitHub Action CI file:

name: CI

on:
  push:
    branches: [ master ]

jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    steps:

    - name: Set up Go 1.x
      uses: actions/setup-go@v2
      with:
        go-version: ^1.13

    - name: Check out code into the Go module directory
      uses: actions/checkout@v2

    - name: Get dependencies
      run: |
        go get -v -t -d ./...
        if [ -f Gopkg.toml ]; then
            curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
            dep ensure
        fi

    - name: Check out my other private repo
      uses: actions/checkout@v2
      with:
        repository: jackysp/jackysp.github.io
        token: ${{ secrets.UPDATE_BLOG }}
        path: public

    - name: Build
      run: make
    
    - name: Pushes to another repository
      id: public
      uses: cpina/github-action-push-to-another-repository@cp_instead_of_deleting
      env:
        API_TOKEN_GITHUB: ${{ secrets.UPDATE_BLOG }}
      with:
        source-directory: 'public'
        destination-github-username: 'jackysp'
        destination-repository-name: 'jackysp.github.io'
        user-email: your@email.com
        commit-message: See ORIGIN_COMMIT

A Simple Example

#!/usr/bin/env sysbench

require("oltp_common")

function prepare_statements()
end

function event()
    con:query("set autocommit = 1")
end

The first line require includes Sysbench’s built-in basic library; the empty prepare_statement is a callback function from oltp_common that must be present; the specific execution of a single load is implemented in the event function.

Save this script as a Lua file, for example, named set.lua, and then execute it using sysbench.

sysbench --config-file=config --threads=100 set.lua --tables=1 --table_size=1000000 run

You can use the above command. Of course, here --tables=1 and --table_size=1000000 are not useful for this load, so they are optional. --threads controls concurrency.

$ cat config
time=120
db-driver=mysql
mysql-host=172.16.5.33
mysql-port=34000
mysql-user=root
mysql-db=sbtest
report-interval=10

In the config file, parameters you don’t frequently adjust are written once to avoid having a long string of parameters in the command line. These are required fields: time represents the test duration, report-interval is used to observe real-time performance results, and the others pertain to how to connect to the database.

The running output generally looks like:

[ 10s ] thds: 100 tps: 94574.34 qps: 94574.34 (r/w/o: 0.00/0.00/94574.34) lat (ms,95%): 3.68 err/s: 0.00 reconn/s: 0.00
[ 20s ] thds: 100 tps: 77720.30 qps: 77720.30 (r/w/o: 0.00/0.00/77720.30) lat (ms,95%): 5.28 err/s: 0.00 reconn/s: 0.00
[ 30s ] thds: 100 tps: 56080.10 qps: 56080.10 (r/w/o: 0.00/0.00/56080.10) lat (ms,95%): 9.22 err/s: 0.00 reconn/s: 0.00
[ 40s ] thds: 100 tps: 93315.90 qps: 93315.90 (r/w/o: 0.00/0.00/93315.90) lat (ms,95%): 4.82 err/s: 0.00 reconn/s: 0.00
[ 50s ] thds: 100 tps: 97491.02 qps: 97491.02 (r/w/o: 0.00/0.00/97491.02) lat (ms,95%): 4.65 err/s: 0.00 reconn/s: 0.00
[ 60s ] thds: 100 tps: 94034.27 qps: 94034.27 (r/w/o: 0.00/0.00/94034.27) lat (ms,95%): 4.91 err/s: 0.00 reconn/s: 0.00
[ 70s ] thds: 100 tps: 74707.37 qps: 74707.37 (r/w/o: 0.00/0.00/74707.37) lat (ms,95%): 6.79 err/s: 0.00 reconn/s: 0.00
[ 80s ] thds: 100 tps: 89485.10 qps: 89485.10 (r/w/o: 0.00/0.00/89485.10) lat (ms,95%): 5.18 err/s: 0.00 reconn/s: 0.00
[ 90s ] thds: 100 tps: 109296.44 qps: 109296.44 (r/w/o: 0.00/0.00/109296.44) lat (ms,95%): 2.91 err/s: 0.00 reconn/s: 0.00

Finally, there will be a summary report.

SQL statistics:
    queries performed:
        read:                            0
        write:                           0
        other:                           10424012
        total:                           10424012
    transactions:                        10424012 (86855.65 per sec.)
    queries:                             10424012 (86855.65 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

Throughput:
    events/s (eps):                      86855.6517
    time elapsed:                        120.0154s
    total number of events:              10424012

Latency (ms):
         min:                                    0.09
         avg:                                    1.15
         max:                                 1527.74
         95th percentile:                        4.91
         sum:                             11994122.49

Threads fairness:
    events (avg/stddev):           104240.1200/600.21
    execution time (avg/stddev):   119.9412/0.01

Lost Connection

There are generally three reasons for a Lost Connection:

1. A timeout occurs either directly between the client and the database or at some point along the intermediate link, such as from the client to the Proxy or from the Proxy to the database.
2. A bug occurs during SQL execution, which can generally be recovered, thus preventing the TiDB server from crashing completely (panic).
3. TiDB itself crashes, often due to excessive memory use, causing an OOM (Out of Memory), or a user deliberately kills TiDB. Another possibility is an unrecovered bug, which typically appears more frequently in background threads.

Timeout

Direct Timeout

TiDB supports the MySQL-compatible wait_timeout variable, with a default value of 0, meaning no timeout is set, unlike MySQL’s default of 8 hours.

The only place it is used in the code is in getSessionVarsWaitTimeout. In the connection’s Run section, its value is set for packet IO. If the variable is non-zero, a timeout is set before each readPacket.

If the client does not send data beyond the specified time, the connection will be disconnected. At this time, a log message “read packet timeout, close this connection” will appear, along with the specific timeout duration.

Intermediate Link Timeout

Another scenario is an intermediate link timeout. A normal timeout in an intermediate link (proxy) typically returns an EOF error to the database. In older versions, at least a connection closed log would be output.

In the newer master version, product managers suggested changing this log to a debug level, so it is generally no longer output.

However, in the new version, a monitoring item called DisconnectionCounter has been added,

which records normal and abnormal disconnections as a supplement to downgraded logging.

Bugs that Are Recovered

TiDB “basically” can recover from panics caused by unknown bugs. However, if there is an array out-of-bounds, a null pointer reference, or intentional panic, it cannot guarantee correct results for the current and subsequent SQL, so terminating the current connection is a wise choice.

At this time, an error log “connection running loop panic” will appear, along with a lastSQL field that outputs the current erroneous SQL.

Panic Not Recovered

Whether it’s an unrecovered panic or a system-level OOM-induced panic, they do not leave a log in TiDB’s logs. TiDB clusters managed by deployment tools like Ansible or TiUP will automatically restart a crashed TiDB server. Consequently, the log will contain a new “Welcome” message, which might be overlooked. However, the Uptime in monitoring will show TiDB’s Uptime reset to zero, making this issue relatively easy to detect. Of course, it’s better to have accompanying alerts.

Unrecovered panic outputs are Golang’s default outputs, usually redirected to tidb_stderr.log by deployment tools. Older versions of Ansible overwrite this file every restart, but now use an append mode.

Nevertheless, it has some other drawbacks, like lacking timestamps. This makes it difficult to timestamp-match with TiDB logs. This PR implemented distinguishing tidb_stderr.log based on PID but hasn’t been coordinated with the deployment tools and is temporarily disabled.

To get this standard panic output, you can use the panicparse introduced in the previous article to parse the panic results. Typically, you can look at the topmost stack. The example in the image evidently shows an out-of-memory error, commonly referred to as OOM. To identify which SQL caused the OOM, check TiDB’s logs for resource-heavy SQL, which are usually logged with the expensive_query tag, and can be checked by grepping the logs. This will not be exemplified here.

Tracing

TiDB has supported tracing since version 2.1, but it hasn’t been widely used. I think there are two main reasons:

1. The initial version of tracing only supported the JSON format, requiring the output to be copied and pasted into a TiDB-specific web page at a special host port to view it. Although novel, the multiple steps involved prevented widespread adoption.

   

2. Another issue is that tracing provides insight only after a problem is known. If developers suspect a problem or slow execution in advance, they must proactively add events at those points. Often, unforeseen issues cannot be covered, leaving gaps.

Once the framework of tracing is in place, adding events is relatively straightforward and involves adding code like the snippet below at the desired points:

Interested individuals can add events to TiDB as needed, offering a good hands-on experience.

Eventually, tracing added format='row' and format='log' features. I personally favor format='log'.

Difference between Tracing and Explain (Analyze)

1. Tracing operates at the function level, while Explain operates at the operator level. Tracing is easier to add and more granular and does not need to be part of a plan.
2. Tracing can trace any SQL, while Explain only shows data reading parts. For example, with an Insert, Explain shows almost nothing, whereas tracing provides detailed insights from SQL parsing to the full transaction commit.

Key Functions

The definition of key functions varies from person to person, so the content of this section is subjective.

execute

The execute function is the necessary pathway for text protocol execution. It also nicely demonstrates the various processes of SQL handling.

1. ParseSQL analyzes the SQL. The final implementation is in the parser, where SQL is parsed according to the rules introduced in the second article. Note that the parsed SQL may be a single statement or multiple statements. TiDB itself supports the multi-SQL feature, allowing multiple SQL statements to be executed at once.

2. After parsing, a stmtNodes array is returned, which is processed one-by-one in the for loop below. The first step is to compile, where the core of compile is optimization, generating a plan. By following the Optimize function, you can find logic similar to logical and physical optimization found in other common databases.

   

3. The last part is execution, where executeStatement and particularly the runStmt function are key functions.

runStmt

Judging from the call graph of runStmt, this function is almost the mandatory pathway for all SQL execution. Except for point query statements using the binary protocol with automatic commit, all other statements go through this function. This function is responsible for executing SQL, excluding SQL parsing and compilation (the binary protocol does not need repeated SQL parsing, nor does SQL compilation require plan caching).

The core part of the runStmt function is as shown above. From top to bottom:

1. checkTxnAborted

   When a transaction is already corrupted and cannot be committed, the user must actively close the transaction to end the already corrupted transaction. During execution, transactions may encounter errors that cannot be handled and must be terminated. The transaction cannot be silently closed because the user may continue to execute SQL and assume it is still within the transaction. This function ensures that all subsequent SQL commands by the user are not executed and directly return an error until the user uses rollback or commit to explicitly close the transaction for normal execution.

2. Exec

   Execute the SQL and return the result set (rs).

3. IsReadOnly

   After executing a SQL, it’s necessary to determine whether it is a read-only SQL. If it is not read-only, it must be temporarily stored in the transaction’s execution history. This execution history is used when a transaction conflict or other errors require the transaction to be retried. Read-only SQL is bypassed because the retry of the transaction is done during the commit phase, and at this point, the only feedback to the client can be success or failure of the commit; reading results is meaningless.

   This section also includes StmtCommit and StmtRollback. TiDB supports MySQL-like statement commits and rollbacks—if a statement fails during a transaction, that single statement will be atomically rolled back, while other successfully executed statements will eventually commit with the transaction.

   In TiDB, the feature of statement commit is implemented with a two-layer buffer: both the transaction and the statement have their own buffers. After a statement executes successfully, the statement’s buffer is merged into the transaction buffer. If a statement fails, the statement’s buffer is discarded, thus ensuring the atomicity of statement commits. Of course, a statement commit may fail, in which case the entire transaction buffer becomes unusable, and the transaction can only be rolled back.

4. finishStmt

   Once a statement is executed, should it be committed? This depends on whether the transaction was explicitly started (i.e., with begin or start transaction) and whether autocommit is enabled. The role of finishStmt is to, post-execution, check if it should be committed based on the above conditions. It’s essentially for cleaning up and checking after each statement execution.

5. pending section

   Some SQLs in TiDB do not require a transaction (e.g., the set statement). However, before parsing, the database doesn’t know whether the statement requires a transaction. The latency of starting a transaction in TiDB is relatively high because it requires obtaining a TSO (timestamp oracle) from PD. TiDB has an optimization to asynchronously obtain a TSO, meaning a TSO is prepared regardless of whether a transaction is eventually needed. If a statement indeed doesn’t require a TSO and a transaction is not activated, remaining in a pending status, the pending transaction must be closed.

Logs

Let’s first look at a section of logs from TiDB at initial startup, divided into several parts:

[2020/08/12 16:12:07.282 +08:00] [INFO] [printer.go:42] ["Welcome to TiDB."] ["Release Version"=None] [Edition=None] ["Git Commit Hash"=None] ["Git Branch"=None] ["UTC Build Time"=None] [GoVersion=go1.15] ["Race Enabled"=false] ["Check Table Before Drop"=false] ["TiKV Min Version"=v3.0.0-60965b006877ca7234adaced7890d7b029ed1306]
[2020/08/12 16:12:07.300 +08:00] [INFO] [printer.go:56] ["loaded config"] [config="{\"host\":\"0.0.0.0\",\"advertise-address\":\"0.0.0.0\",\"port\":4000,\"cors\":\"\",\"store\":\"mocktikv\",\"path\":\"/tmp/tidb\",\"socket\":\"\",\"lease\":\"45s\",\"run-ddl\":true,\"split-table\":true,\"token-limit\":1000,\"oom-use-tmp-storage\":true,\"tmp-storage-path\":\"C:\\\\Users\\\\username\\\\AppData\\\\Local\\\\Temp\\\\tidb\\\\tmp-storage\",\"oom-action\":\"log\",\"mem-quota-query\":1073741824,\"tmp-storage-quota\":-1,\"enable-streaming\":false,\"enable-batch-dml\":false,\"lower-case-table-names\":2,\"server-version\":\"\",\"log\":{\"level\":\"info\",\"format\":\"text\",\"disable-timestamp\":null,\"enable-timestamp\":null,\"disable-error-stack\":null,\"enable-error-stack\":null,\"file\":{\"filename\":\"\",\"max-size\":300,\"max-days\":0,\"max-backups\":0},\"enable-slow-log\":true,\"slow-query-file\":\"tidb-slow.log\",\"slow-threshold\":300,\"expensive-threshold\":10000,\"query-log-max-len\":4096,\"record-plan-in-slow-log\":1},\"security\":{\"skip-grant-table\":false,\"ssl-ca\":\"\",\"ssl-cert\":\"\",\"ssl-key\":\"\",\"require-secure-transport\":false,\"cluster-ssl-ca\":\"\",\"cluster-ssl-cert\":\"\",\"cluster-ssl-key\":\"\",\"cluster-verify-cn\":null},\"status\":{\"status-host\":\"0.0.0.0\",\"metrics-addr\":\"\",\"status-port\":10080,\"metrics-interval\":15,\"report-status\":true,\"record-db-qps\":false},\"performance\":{\"max-procs\":0,\"max-memory\":0,\"stats-lease\":\"3s\",\"stmt-count-limit\":5000,\"feedback-probability\":0.05,\"query-feedback-limit\":1024,\"pseudo-estimate-ratio\":0.8,\"force-priority\":\"NO_PRIORITY\",\"bind-info-lease\":\"3s\",\"txn-total-size-limit\":104857600,\"tcp-keep-alive\":true,\"cross-join\":true,\"run-auto-analyze\":true,\"agg-push-down-join\":false,\"committer-concurrency\":16,\"max-txn-ttl\":600000},\"prepared-plan-cache\":{\"enabled\":false,\"capacity\":100,\"memory-guard-ratio\":0.1},\"opentracing\":{\"enable\":false,\"rpc-metrics\":false,\"sampler\":{\"type\":\"const\",\"param\":1,\"sampling-server-url\":\"\",\"max-operations\":0,\"sampling-refresh-interval\":0},\"reporter\":{\"queue-size\":0,\"buffer-flush-interval\":0,\"log-spans\":false,\"local-agent-host-port\":\"\"}},\"proxy-protocol\":{\"networks\":\"\",\"header-timeout\":5},\"tikv-client\":{\"grpc-connection-count\":4,\"grpc-keepalive-time\":10,\"grpc-keepalive-timeout\":3,\"commit-timeout\":\"41s\",\"max-batch-size\":128,\"overload-threshold\":200,\"max-batch-wait-time\":0,\"batch-wait-size\":8,\"enable-chunk-rpc\":true,\"region-cache-ttl\":600,\"store-limit\":0,\"store-liveness-timeout\":\"120s\",\"copr-cache\":{\"enable\":false,\"capacity-mb\":1000,\"admission-max-result-mb\":10,\"admission-min-process-ms\":5}},\"binlog\":{\"enable\":false,\"ignore-error\":false,\"write-timeout\":\"15s\",\"binlog-socket\":\"\",\"strategy\":\"range\"},\"compatible-kill-query\":false,\"plugin\":{\"dir\":\"\",\"load\":\"\"},\"pessimistic-txn\":{\"enable\":true,\"max-retry-count\":256},\"check-mb4-value-in-utf8\":true,\"max-index-length\":3072,\"alter-primary-key\":false,\"treat-old-version-utf8-as-utf8mb4\":true,\"enable-table-lock\":false,\"delay-clean-table-lock\":0,\"split-region-max-num\":1000,\"stmt-summary\":{\"enable\":true,\"enable-internal-query\":false,\"max-stmt-count\":200,\"max-sql-length\":4096,\"refresh-interval\":1800,\"history-size\":24},\"repair-mode\":false,\"repair-table-list\":[],\"isolation-read\":{\"engines\":[\"tikv\",\"tiflash\",\"tidb\"]},\"max-server-connections\":0,\"new_collations_enabled_on_first_bootstrap\":false,\"experimental\":{\"allow-auto-random\":false,\"allow-expression-index\":false}}"]

1. Mandatory startup outputs: “Welcome to TiDB,” git hash, Golang version, etc.
2. Actually loaded configuration (this section is somewhat difficult to read)

The remainder are some routine startup logs. The process can be referenced from the main function section introduced in the first article, mainly outputting the initial system table creation process.

[2020/08/12 16:12:07.300 +08:00] [INFO] [main.go:341] ["disable Prometheus push client"]
[2020/08/12 16:12:07.300 +08:00] [INFO] [store.go:68] ["new store"] [path=mocktikv:///tmp/tidb]
[2020/08/12 16:12:07.300 +08:00] [INFO] [systime_mon.go:25] ["start system time monitor"]
[2020/08/12 16:12:07.310 +08:00] [INFO] [store.go:74] ["new store with retry success"]
[2020/08/12 16:12:07.310 +08:00] [INFO] [tidb.go:71] ["new domain"] [store=8d19232e-a273-4e31-ba9b-a3467998345c] ["ddl lease"=45s] ["stats lease"=3s]
[2020/08/12 16:12:07.315 +08:00] [INFO] [ddl.go:321] ["[ddl] start DDL"] [ID=0e1bd28e-03ed-4900-bf71-f58b3b9d954a] [runWorker=true]
[2020/08/12 16:12:07.315 +08:00] [INFO] [ddl.go:309] ["[ddl] start delRangeManager OK"] ["is a emulator"=true]
[2020/08/12 16:12:07.315 +08:00] [INFO] [ddl_worker.go:130] ["[ddl] start DDL worker"] [worker="worker 1, tp general"]
[2020/08/12 16:12:07.315 +08:00] [INFO] [ddl_worker.go:130] ["[ddl] start DDL worker"] [worker="worker 2, tp add index"]
[2020/08/12 16:12:07.315 +08:00] [INFO] [delete_range.go:133] ["[ddl] start delRange emulator"]
[2020/08/12 16:12:07.317 +08:00] [INFO] [domain.go:144] ["full load InfoSchema success"] [usedSchemaVersion=0] [neededSchemaVersion=0] ["start time"=2.0015ms]
[2020/08/12 16:12:07.317 +08:00] [INFO] [domain.go:368] ["full load and reset schema validator"]
[2020/08/12 16:12:07.317 +08:00] [INFO] [tidb.go:199] ["rollbackTxn for ddl/autocommit failed"]

Because DDL logs are very numerous, the TiDB logs basically record each step of the DDL execution, so I’ve truncated this part of the log here. However, the basic outline can be sorted out. Firstly, the DDL execution is initiated from ddl_api, at this time recording ["CRUCIAL OPERATION"] style logs. DDL is a crucial operation, so it belongs to CRUCIAL type logs. Then, we can see a series of logs with the ddl keyword linked together, such as [ddl] add DDL jobs, [ddl] start DDL job, [ddl] run DDL job, [ddl] finish DDL job, and [ddl] DDL job is finished. These represent the process from when the DDL owner acquires a job to its final execution completion. Moreover, they have a unique job ID, which can be used to link a DDL in the log with something like jobs="ID:2.

[2020/08/12 16:12:07.518 +08:00] [INFO] [server.go:235] ["server is running MySQL protocol"] [addr=0.0.0.0:4000]
[2020/08/12 16:12:07.518 +08:00] [INFO] [http_status.go:80] ["for status and metrics report"] ["listening on addr"=0.0.0.0:10080]
[2020/08/12 16:12:07.520 +08:00] [INFO] [domain.go:1015] ["init stats info time"] ["take time"=3.0126ms]
[2020/08/12 16:15:41.482 +08:00] [INFO] [server.go:388] ["new connection"] [conn=1] [remoteAddr=127.0.0.1:64888]
[2020/08/12 21:03:19.954 +08:00] [INFO] [server.go:391] ["connection closed"] [conn=1]

Thereafter, the appearance of server is running MySQL protocol means that TiDB can provide services externally. Later, there are logs corresponding to the creation and closing of each connection, namely new connection and connection closed. Of course, they also have their corresponding connection ID, which is unique for a TiDB. You can use the keyword conn=1 in the log to contextually link them together.

Stack Logs

Most of TiDB’s SQL errors (except for duplicate entry and syntax errors) will output the complete stack information. Due to the requirements of unified log format, the stack now looks very unsightly…

For this stack trace, I believe no one really enjoys reading it. Therefore, we need to paste it into Vim and execute %s/\\n/\r/g and %s/\\t/ /g to turn it into a Golang-style stack.

When you see which module it’s stuck in, like the plan part here, you can find the corresponding colleague for support.

However, there is a more user-friendly tool for dealing with Golang’s lengthy stack called panicparse. To install it, simply run go get github.com/maruel/panicparse/v2/cmd/pp. The effect is as follows:

Whether it’s TiDB running goroutines or panic outputs, it can be parsed using this. It has several features:

1. It can display active and inactive goroutines.
2. It can show the relationships between goroutines.
3. Keyword highlighting.
4. Supports Windows.

The latest 2.0.0 version supports race detector and HTML formatted output.

This concludes the introduction to the analysis of key functions and logs (startup, DDL, connection, error stack).

System Variables

The system variable names in TiDB are defined in tidb_vars.go. This file also includes some default values for variables, but the place where they are actually assembled is defaultSysVars.

This large struct array defines the scope, variable names, and default values for all variables in TiDB. Besides TiDB’s own system variables, it also includes compatibility with MySQL’s system variables.

Scope

In TiDB, there are three types of variable scopes literally:

They are ScopeNone, ScopeGlobal, and ScopeSession. They represent:

• ScopeNone: Read-only variables
• ScopeGlobal: Global variables
• ScopeSession: Session variables

The actual effect of these scopes is that when you use SQL to read or write them, you need to use the corresponding syntax. If the SQL fails, the SQL operation does not take effect. If the SQL executes successfully, it merely means the setting is complete, but it does not mean that it takes effect according to the corresponding scope.

Let’s use the method mentioned in the first article to start a single-node TiDB for demonstration:

ScopeNone

Take performance_schema_max_mutex_classes as an example,

MySQL  127.0.0.1:4000  SQL > select @@performance_schema_max_mutex_classes;
+----------------------------------------+
| @@performance_schema_max_mutex_classes |
+----------------------------------------+
| 200                                    |
+----------------------------------------+
1 row in set (0.0002 sec)
MySQL  127.0.0.1:4000  SQL > select @@global.performance_schema_max_mutex_classes;
+-----------------------------------------------+
| @@global.performance_schema_max_mutex_classes |
+-----------------------------------------------+
| 200                                           |
+-----------------------------------------------+
1 row in set (0.0004 sec)
MySQL  127.0.0.1:4000  SQL > select @@session.performance_schema_max_mutex_classes;
ERROR: 1238 (HY000): Variable 'performance_schema_max_mutex_classes' is a GLOBAL variable

As you can see, the scope of ScopeNone can be read as a global variable,

MySQL  127.0.0.1:4000  SQL > set global performance_schema_max_mutex_classes = 1;
ERROR: 1105 (HY000): Variable 'performance_schema_max_mutex_classes' is a read-only variable
MySQL  127.0.0.1:4000  SQL > set performance_schema_max_mutex_classes = 1;
ERROR: 1105 (HY000): Variable 'performance_schema_max_mutex_classes' is a read-only variable
MySQL  127.0.0.1:4000  SQL > set session performance_schema_max_mutex_classes = 1;
ERROR: 1105 (HY000): Variable 'performance_schema_max_mutex_classes' is a read-only variable

But it cannot be set in any way.

To trace the usage of ScopeNone, you will see

In setSysVariable, when this type of scope variable is encountered, an error is directly returned.

In ValidateGetSystemVar, it is handled as a global variable. From a theoretical standpoint, these ScopeNone variables are essentially a single copy in the code. Once TiDB is started, they exist in memory as read-only and are not actually stored in TiKV.

ScopeGlobal

Using gtid_mode as an example,

MySQL  127.0.0.1:4000  SQL > select @@gtid_mode;
+-------------+
| @@gtid_mode |
+-------------+
| OFF         |
+-------------+
1 row in set (0.0003 sec)
MySQL  127.0.0.1:4000  SQL > select @@global.gtid_mode;
+--------------------+
| @@global.gtid_mode |
+--------------------+
| OFF                |
+--------------------+
1 row in set (0.0006 sec)
MySQL  127.0.0.1:4000  SQL > select @@session.gtid_mode;
ERROR: 1238 (HY000): Variable 'gtid_mode' is a GLOBAL variable

It works the same way as MySQL global variable reading,

MySQL  127.0.0.1:4000  SQL > set gtid_mode=on;
ERROR: 1105 (HY000): Variable 'gtid_mode' is a GLOBAL variable and should be set with SET GLOBAL
MySQL  127.0.0.1:4000  SQL > set session gtid_mode=on;
ERROR: 1105 (HY000): Variable 'gtid_mode' is a GLOBAL variable and should be set with SET GLOBAL
MySQL  127.0.0.1:4000  SQL > set global gtid_mode=on;
Query OK, 0 rows affected (0.0029 sec)
MySQL  127.0.0.1:4000  SQL > select @@global.gtid_mode;
+--------------------+
| @@global.gtid_mode |
+--------------------+
| ON                 |
+--------------------+
1 row in set (0.0005 sec)
MySQL  127.0.0.1:4000  SQL > select @@gtid_mode;
+-------------+
| @@gtid_mode |
+-------------+
| ON          |
+-------------+
1 row in set (0.0006 sec)

The setting method is also compatible with MySQL. At this point, we can shut down the single-instance TiDB and restart it,

MySQL  127.0.0.1:4000  SQL > select @@gtid_mode;
+-------------+
| @@gtid_mode |
+-------------+
| ON          |
+-------------+
1 row in set (0.0003 sec)

And you can see that the result can still be read, meaning that this setting was persisted to the storage engine. Looking closely at the code, you can see:

The actual implementation involves executing an internal replace statement to update the original value. This constitutes a complete transaction involving acquiring two TSOs and committing the entire process, making it slower compared to setting session variables.

ScopeSession

Using rand_seed2 as an example,

MySQL  127.0.0.1:4000  SQL > select @@rand_seed2;
+--------------+
| @@rand_seed2 |
+--------------+
|              |
+--------------+
1 row in set (0.0005 sec)
MySQL  127.0.0.1:4000  SQL > select @@session.rand_seed2;
+----------------------+
| @@session.rand_seed2 |
+----------------------+
|                      |
+----------------------+
1 row in set (0.0003 sec)
MySQL  127.0.0.1:4000  SQL > select @@global.rand_seed2;
ERROR: 1238 (HY000): Variable 'rand_seed2' is a SESSION variable

Reading is compatible with MySQL.

MySQL  127.0.0.1:4000  SQL > set rand_seed2='abc';
Query OK, 0 rows affected (0.0006 sec)
MySQL  127.0.0.1:4000  SQL > set session rand_seed2='bcd';
Query OK, 0 rows affected (0.0004 sec)
MySQL  127.0.0.1:4000  SQL > set global rand_seed2='cde';
ERROR: 1105 (HY000): Variable 'rand_seed2' is a SESSION variable and can't be used with SET GLOBAL
MySQL  127.0.0.1:4000  SQL > select @@rand_seed2;
+--------------+
| @@rand_seed2 |
+--------------+
| bcd          |
+--------------+

The setting is also compatible with MySQL. It can be simply observed that this operation only changes the session’s memory. The actual place where it finally takes effect is SetSystemVar.

There are some tricks here.

Actual Scope of Variables

The previous section covered setting session variables. Based on MySQL’s variable rules, setting a global variable does not affect the current session. Only newly created sessions will load global variables for session variable assignment. Ultimately, the active session variable take effect. Global variables without session properties still have unique characteristics, and this chapter will cover:

1. Activation of session variables
2. Activation of pure global variables
3. Mechanism of global variable function

These three aspects.

Activation of Session Variables

Whether a session variable is also a global variable only affects whether it needs to load global variable data from the storage engine when the session starts. The default value in the code is the initial value for eternity if no loading is required.

The actual range where a variable operates can only be observed in SetSystemVar.

For example, in this part, s.MemQuotaNestedLoopApply = tidbOptInt64(val, DefTiDBMemQuotaNestedLoopApply) changes the s structure, effectively changing the current session,

Whereas atomic.StoreUint32(&ProcessGeneralLog, uint32(tidbOptPositiveInt32(val, DefTiDBGeneralLog))) changes the value of the global variable ProcessGeneralLog, thereby affecting the entire TiDB instance when set tidb_general_log = 1 is executed.

Activation of Pure Global Variables

Pure global variables in current TiDB are used for background threads like DDL, statistics, etc.

Because only one TiDB server requires them, session-level variables hold no meaning for these.

Mechanism of Global Variable Function

Global variables in TiDB don’t activate immediately after setting. A connection fetches the latest global system variables from TiKV to assign them to the current session the first time it’s established. Concurrent connection creation results in frequent access to the TiKV node holding a few global variables. Thus, TiDB caches global variables, updating them every two seconds, significantly reducing TiKV load. The problem arises that after setting a global variable, a brief wait is necessary before creating a new connection, ensuring new connections will read the latest global variable. This is one of the few eventual consistency locations within TiDB.

For specific details, see this commentary in loadCommonGlobalVariablesIfNeeded.

Metrics

Compared to system variables, Metrics in TiDB are simpler, or straightforward. The most common Metrics are Histogram and Counter, the former is used to record actual values for an operation and the latter records occurrences of fixed events. All Metrics in TiDB are uniformly located here, with AlertManager and Grafana scripts also available separately under alertmanager and grafana.

There are many Metrics, and from a beginner’s perspective, it’s best to focus on a specific monitoring example. Let’s take the TPS (transactions per second) panel as an example.

Click EDIT and you will see the monitoring formula is:

The tidb_session_transaction_duration_seconds is the name of this specific metric. Since it is a histogram, it can actually be expressed as three types of values: sum, count, and bucket, which represent the total sum of values, the count (which functions the same as a counter), and the distribution by bucket, respectively.

In this context, [1m] represents a time window of 1 minute, indicating the precision of the measurement. The rate function calculates the slope, essentially the rate of change, indicating how many times something occurs per second. The sum function is used for aggregation, and when combined with by (type, txn_mode), it represents aggregation by the dimensions of type and txn_mode.

The Legend below displays the dimensions above using {{type}}-{{txn_mode}}. When surrounded by {{}}, it can display the actual label names.

In this representation, the final states of transactions are commit, abort, and rollback. A commit indicates a successful user-initiated transaction, rollback indicates a user-initiated rollback (which cannot fail), and abort indicates a user-initiated commit that failed.

The second label, txn_mode, refers to two modes: optimistic and pessimistic transactions. There’s nothing further to explain about these modes.

Corresponding to the code:

This segment of code shows that tidb_session_transaction_duration_seconds is divided into several parts, including namespace and subsystem. Generally, to find a variable in a formula like tidb_session_transaction_duration_seconds_count within TiDB code, you need to remove the first two words and the last word.

From this code snippet, you can see it’s a histogram, specifically a HistogramVec, which is an array of histograms because it records data with several different labels. The labels LblTxnMode and LblType are these two labels.

Checking the references, there is a place for registration, which is in the main function we discussed in the first article, where metrics are registered.

Other references show how metrics are instantiated. Why do we do this? Mainly because as the number of labels increases, the performance of metrics becomes poorer, which is related to Prometheus’s implementation. We had no choice but to create many instantiated global variables.

Taking the implementation of Rollback as an example, its essence is to record the actual execution time of a transaction when Rollback is truly executed. Since it’s a histogram, it is also used as a counter in this instance.

How to Check the Support Level of a Syntax

There are usually two methods:

1. Check through the parser repo
2. Directly check within the TiDB repo

Both of these methods require the environment setup from the previous article. If you haven’t tried that yet, give it a go.

Preparation

1. Install GoYacc Support

   

   The GoYacc Support plugin is a creation by a colleague at our company, a third-party plugin officially accepted by JetBrains, a well-regarded product. It includes syntax highlighting and intelligence, which is great!

2. Download parser repo

   If you’re checking syntax directly from the parser, you need to download it manually. If you’re navigating from TiDB, IDEA will automatically download the code, so no extra steps are needed.

Check via parser repo

Open the parser using IDEA, switch to the branch you need, and locate the parser.y file. However, it is more recommended to check from within TiDB.

Check via TiDB repo

1. Open the TiDB project with IDEA and switch to the required branch

   

2. Find the parser.y file; make sure to select the broadest search scope

   

   Alternatively, you can find it in the file list,

   

   

Let’s take checking the SHOW ENGINES SQL statement as an example.

The entry point for the entire statement parsing is Start. Below it is the StatementList, followed by Statement. Under the large list of Statements, you can find ShowStmt.

However, ShowStmt is actually quite complex. Another way is to directly search for ShowEngines within parser.y, since naming follows Golang conventions, with camel case and capitalized letters for public exposure. Naturally, if familiar with the code, you’d know ShowEngines is under ShowTargetFilterable. Its first branch is ShowEngines.

What is the level of support for SHOW ENGINES?

You can look at how ast.ShowEngines is processed. Here, you can’t just jump to it; you need to copy and search.

You only need to see how it’s processed under TiDB, and you can skip test files.

One is the actual implementation,

The other is the build schema, which you can ignore for now,

Entering fetchShowEngines, you can see the specific implementation is simple, running an internal SQL to read a system table.

Checking SHOW ENGINES ends here. You can see that it’s fully supported.

Which statements only have syntax support?

Taking the temporary table creation syntax as an example, find its position in the parser.y file.

It’s an option.

You can see that if the temporary table option is specified, it simply returns true with an attached warning, stating that the table is still treated as a regular table. Previously, the parser had a lot of operations that only returned without doing anything, not even a warning, but these are now rare.

Advantages of Querying via TiDB repo

You can see that checking via the TiDB repo allows you to find the parser’s detailed hash using IDEA. If you check directly via the parser, you need to first look up the parser’s hash in TiDB’s go.mod, then check out to the corresponding hash in the parser. If you need to check specific implementations, you have to go back to TiDB, making back-and-forth checks less convenient compared to looking within a single project. The only advantage is the ease of blaming commit history.

Viewing and Modifying Default Configuration

The default configurations can be easily viewed in TiDB, specifically the variable defaultConf. The configurations listed here are the actual default settings.

Taking the first Host configuration as an example, it has a mapping to toml and json files.

This essentially shows how it’s written in a toml file. The DefHost following it is the specific default value.

Something important to note is that configurations have a hierarchical relationship. For example, the log-related configuration in the configuration file is:

In the code, it is represented as:

This denotes a configuration called “level” under the log configuration.

What if you want to add more levels? For instance, the most complex configuration for CopCache adds another layer under tikv-client called copr-cache.

Since toml files do not support multi-level nesting, this leads to the most complex configuration writing in TiDB.

To use non-default configurations with the TiDB started through IDEA as mentioned above, the simplest way is to modify this defaultConf.

Summary

From this, you can see that checking whether a statement is supported, and whether it’s just syntax support or has a specific implementation, can be achieved with the described methods. You also learned how to view and modify default configurations, allowing you to conduct some verifications yourself. In the next article, I plan to introduce TiDB’s system variables.

There are many articles on reading the source code of TiDB, often referred to as the “Twenty-Four Chapters Scriptures”. However, these introductions typically proceed from a macro to a micro perspective. This series attempts to introduce how to read TiDB’s source code from an easier angle. The goals we aim to achieve are:

1. Enable readers to start reading TiDB’s code themselves, rather than understanding it passively through pre-written articles.
2. Provide some common examples of looking into the details of the code, such as examining the scope of a variable.

After all, teaching people to fish is better than giving them fish. While the code changes often, the methods remain mostly unchanged.

Why choose TiDB to read?

1. I am not familiar with TiKV or PD.

2. TiDB is the entry point directly interacting with users and is also the most likely to be questioned.

3. TiDB can run independently and be debugged. If you want to run some SQL after reading the code to verify your understanding, it can be easily done.

Preparations

1. A development machine

   TiDB is a pure Golang project. It can be conveniently developed on Linux, MacOS, and even Windows. My environment is Windows 10.

2. A copy of the TiDB source code, available for download at the official repo.

3. Golang environment, following the official guide is straightforward.

4. Goland or IntelliJ IDEA + Golang plugin

   I personally feel there’s no difference between the two. Why not recommend VSCode + Golang plugin? Mainly because I’m used to the JetBrains suite, and indeed commercial software tends to be higher quality than community software. For long-term use, it’s recommended to pay for it. Students can use it for free, but need to renew the license every year.

Environment Setup

1. After installing the Golang environment, remember to set the GOPATH.

2. The TiDB code doesn’t need to be developed under the GOPATH, so you can place it anywhere. I usually create a directory called work and throw various codes in there.

3. Open Goland/IDEA. I use IDEA because I often look at code in other languages.

4. Open with IDEA, select the tidb directory.

   

5. At this point, IDEA typically prompts you to set up GOROOT and enable Go Modules. Follow the recommendations.

The environment setup is now complete.

Entry Points

At the beginning, someone advised me to start with the session package. However, after some experience, I personally feel there are two better entry points: the main function and the dispatch function.

main Function

The main function of TiDB can be seen at link. You can roughly go through what happens when starting a tidb-server from top to bottom.

From top to bottom:

• Parse flags

• Output version information and exit

• Register store and monitoring

• Configuration file check

• Initialize temporary folders, etc.

• Set global variables, CPU affinity, log, trace, print server information, set binlog, set monitoring

• Create store and domain

  The createStoreAndDomain method is important, as critical background threads are created here.

• Create server and register stop signal function

• Start the server

  Within runServer, the srv.Run() actually brings up the tidb-server. In the Run() function here, the server continuously listens to network requests, creating a new connection for each new request and using a new goroutine to serve it continually.

• After this, cleanup work is done when the server needs to stop, ultimately writing out the logs.

Thus, the entire main function process ends. Through the main function, you can see the complete lifecycle of a server from creation to destruction.

Additionally, with IDEA, you can easily start and debug TiDB. Click on this triangle symbol as shown in the image below:

A pop-up with options to run and debug the main function will appear. Essentially, this starts a TiDB with default configurations. TiDB defaults to using mocktikv as the storage engine, so it can be started on a single machine for various testing and validation.

As for how to modify the configuration for starting and debugging, this will be introduced in subsequent articles in the series.

dispatch Function

From here, we can proceed further to another suitable entry point function, dispatch.

The dispatch function has several characteristics:

1. Requests coming from clients only enter the dispatch function, meaning from this point onward, user requests are executed. If you set breakpoints here, you can conveniently filter out SQL executed by internal threads.

2. From here, various requests are dispatched into different processing logic, ensuring you don’t miss any user requests. It avoids situations like spending significant time reading text protocol code only to find out the user is actually using a binary protocol.

3. dispatch itself is located at a very early stage, meaning its parameters mostly come directly from the client’s initial information. If it’s a text protocol, directly reading parameters can parse out the SQL text.

At the start, dispatch primarily focuses on obtaining tokens corresponding to the token-limit parameter. Requests that can’t get a token won’t execute, which explains why you can create many connections but only 1000 SQL executions are allowed simultaneously by default.

Next, we enter the most crucial switch case:

These commands are MySQL protocol commands, so it’s apparent from here exactly what TiDB implements. For comparison, you can refer to this link (this link is only for the text protocol). For full details, see the figure below:

Within dispatch, the most important are mysql.ComQuery, as well as the trio mysql.ComStmtPrepare, mysql.ComStmtExecute, and mysql.ComStmtClose. The latter trio is more frequently used in actual production, hence even more important. In contrast, mysql.ComQuery is generally used only for some simple tests and validations.

Since dispatch is the entry point for interfacing with clients, it can conveniently tally how many requests the database has handled. The so-called QPS derived from monitoring statistics is essentially the number of times this function executes per second. Here arises an issue: in cases like multi-query requests, such as select 1; select 1; select 1;, multiple statements sent together are regarded as a single request by dispatch, but as multiple by clients. While using the binary protocol, some clients prepare a statement, then execute, and finally close it. Seemingly equivalent to executing a single SQL from the client’s perspective, the database actually completes three requests.

In summary, users’ perceived QPS may not necessarily align with the number of dispatch function calls. In later versions, the QPS panel in TiDB’s monitoring was changed to CPS, which stands for Commands Per Second, representing the number of commands executed per second.

Looking at the callers of dispatch can also reveal information that helps explain some frequently asked questions:

1. An EOF error in dispatch typically means the client has actively disconnected, so there’s no need to maintain the database connection, and it is severed.

2. In case of an undetermined error (indicating a transaction’s commit is uncertain—whether it has succeeded or failed needs manual intervention for verification), manual intervention is required immediately, and the connection will be closed.

3. If writing binlog fails and ignore-error = false, previously the tidb-server process wouldn’t exit but couldn’t provide services. Now, the tidb-server will exit directly.

4. For all other dispatch errors, the connection will not be closed, allowing service to continue, but the failure information will be logged as “command dispatched failed”, which is arguably one of the most critical logs for TiDB.

Conclusion

This concludes the introduction from setting up the environment to finding a reasonable entry point to start reading code. Subsequent posts in the series will cover aspects such as configuration (adjustments, default values), variables (default values, scope, actual range, activation), supported syntax, etc. Stay tuned.

I had to install Docker on Windows to reproduce a bug.

Process

1. Using Windows 10 as an example, if you have the Home Basic version, you’ll need to pay to upgrade to the Pro version because you need to enable Hyper-V and Container features, which costs about 800 RMB.
2. Install everything with the default settings, and do not switch to Windows Containers, since most images are still under Linux. If you do switch, you can restore it after starting up.
3. If you encounter permission issues with shared folders, follow the instructions at link. However, this might not solve the problem, and you might encounter a sharing failure. In that case, go to the settings, troubleshoot, and reset to factory defaults. After resetting, ensure the shared folders are selected.
4. When you encounter errors during use, just try a few more times. It might work; if not, reset it.

Impressions

Initially, there were no issues using Docker on Linux; Docker itself was simple back then. Later, using it on Mac brought changes, including a user interface, various colors, and numerous bugs. Right from the start, I encountered bugs. Docker did not support Windows a long time ago, and given the various bugs on Mac, I didn’t have high expectations. The results were still quite surprising. In summary, here are a few points:

1. It’s easier to use.
2. There are more bugs. Do not expect much, and be prepared to reset at any time. Fortunately, resetting offers a shortcut, making it a pretty usable tool.
3. It’s incredibly slow.

At this point, I have no optimism for Docker, Kubernetes, or similar technologies. Done~

I tried this, but it didn’t work at all on my Thinkpad X230. I then tried changing some other options in the aforementioned file, but none worked, and surprisingly, Ubuntu even reported errors.

So, I reinstalled the more preferred Debian. Tried again, and it still didn’t work. Finally, I found a more brutal method.

systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target

This directly points these units to /dev/null…

To revert, simply use:

systemctl unmask sleep.target suspend.target hibernate.target hybrid-sleep.target

It’s simple and effective.

Update:

If you only mask them, the CPU usage of systemd-logind will be very high because it continuously attempts to sleep. Therefore, you also need to change HandleLidSwitch and others to ignore. As follows:

HandleSuspendKey=ignore
HandleHibernateKey=ignore
HandleLidSwitch=ignore
HandleLidSwitchExternalPower=ignore
HandleLidSwitchDocked=ignore

Then, execute systemctl restart systemd-logind. For more details, refer to this: SystemD-LoginD High CPU Usage.

The purpose of dynamic updates is pretty simple. As a long-term user of China Unicom’s broadband, although Unicom provides a public address, it is essentially a dynamic address, not a fixed one. If you want to access your home devices using an IP address from outside, you need to use dynamic DNS (DDNS).

Many routers come with built-in DDNS functionality, but they mostly wrap the interfaces of the commonly used service providers. These providers generally have a few characteristics: 1. Blocked by the Great Firewall; 2. Not cheap; 3. Domestic providers may have security issues; 4. The company might have gone out of business. Rather than relying on these unreliable services, it’s better to write your own script for updating.

Thus, the scripting approach comes into play. Initially, I planned to use Cloudflare, as it is the recommended way. Later, I discovered that my domain provider Namesilo offers an API to update DNS. However, it returns data in XML format, and I wasn’t sure how to parse this with shell scripts.

Then, I thought of using Go. Since this DDNS client would likely be deployed on a router-like device, languages like Python or Java require a runtime environment, and C might need some dynamic libraries to run, which I wasn’t sure how to handle. The fact that Go doesn’t require dynamic libraries was a significant advantage. So, I handwrote a tool called und.

1. First, create a DNS record in Namesilo.
2. Obtain the binary of und suitable for your platform. I only provide binaries for arm64|linux and amd64|three mainstream OS in this case. For other platforms, you’ll need to compile it yourself. You can refer to the Makefile.
3. Generate an API key from Namesilo, then start und according to its usage documentation. You might need to run it in the background, so use nohup.

GitHub Features Experience

GitHub Actions

It feels like a replacement for chaotic third-party CI services. I directly chose the Go option for und. By default, it simply runs go build -v . in Ubuntu.

Release

I used this feature when releasing TiDB before, but didn’t remember to upload/automatically generate binaries. Since TiDB is not something that can run completely with just one component, releasing a single binary doesn’t make much sense.

This time, the experience led me to believe:

1. When releasing, tagging is best done directly using the release feature.
2. After the release, since you can edit it, it’s a good time to make each binary and upload them. Based on the Makefile setup, you can generate a version. This is quite an important feature.

These two steps are quite convenient.

C:\Users\name\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\`  

Start-Process -FilePath "C:\Users\name\bin\gost-windows-amd64.exe" -ArgumentList "-L=", "-F=" -RedirectStandardOutput "C:\Users\name\bin\gost-windows-amd64.log" -RedirectStandardError "C:\Users\name\bin\gost-windows-amd64.err" -WindowStyle Hidden

Note: Start-Process seems to perform a fork-like action, and by default, it opens a new PowerShell window to execute. That’s why -WindowStyle Hidden is added at the end. You can’t use -NoNewWindow here because it only prevents the creation of a new window for executing Start-Process, but the old window will not exit.
Note 2: After the old window exits, the forked process seems to become an orphan and is managed elsewhere, so permissions, such as network connection permissions, might need to be requested again.

One day, I came across an article by Chen Hao on Twitter. Having benefited from several of his blog posts, I instinctively felt it was reliable, so I read it and decided to write this practical guide.

Why Use an HTTPS Proxy

In the guide, it’s clearly explained why, plus my own experiences of several shadowsocks being banned, I felt it was necessary to switch to a more secure proxy method.

How to Deploy an HTTPS Proxy

gost

gost is the tool most recommended in the guide. At first, I misunderstood it as a method similar to kcptun, still relying on shadowsocks. In fact, gost implements multiple proxy types, meaning you don’t need other proxies if you have it. I never liked the method of continuously wrapping to accelerate/obfuscate shadowsocks, always feeling that longer pathways bring more problems.

Steps

• Directly download the latest release from the gost repo. Although I have a Golang environment both locally and on the VPS, downloading directly is the easiest. I downloaded version 2.9.0 here.

• Following certbot on a bare VPS doesn’t work… it requires:

  1. Starting an nginx server, as referenced in this guide. Of course, this requires having a domain name pointing an A record to the VPS.
  2. Verifying access through the domain.
  3. Stopping nginx.
  4. Using certbot’s –standalone mode, which will generate the certificates upon success.

• Here, I didn’t use Docker for deployment but used systemd instead, directly creating a systemd unit similar to kcptun. The difference is, because the certificate needs updating, the unit requires a reload method. This tutorial teaches a lot about using systemd, and the author’s article quality is also high, highly recommended for subscription.

  1. Create a /lib/systemd/system/gost.service file with the following content, replacing the domain with your own:

  [Unit]
  Description=gost service
  After=network.target
  StartLimitIntervalSec=0
  
  [Service]
  Type=simple
  Restart=always
  RestartSec=1
  User=root
  PIDFile=/home/admin/gost.pid
  ExecStart=/home/admin/bin/gost -L "http2://xxx:yyy@0.0.0.0:443?cert=/etc/letsencrypt/live/example.com/fullchain.pem&key=/etc/letsencrypt/live/example.com/privkey.pem&probe_resist=code:404"
  ExecReload=/bin/kill -HUP $MAINPID
  
  [Install]
  WantedBy=multi-user.target
  

  ExecStart is a simplified version of the Docker method in the guide. ExecReload just kills the process.

  1. Test whether it’s successful using systemctl start|status|restart|enable gost.

  2. Configure crontab to update the certificate. I didn’t use systemd because I’m not familiar with it.

  0 0 1 * * /usr/bin/certbot renew --force-renewal
  5 0 1 * * systemctl restart gost
  

• After completing the above, nginx can be directly stopped and disabled.

• Configure the client. This is simple; just refer to the guide. The principle is straightforward because gost implements the shadowsocks protocol using the shadowsocks Golang version. Therefore, the following command starts a local shadowsocks server, and you configure your client to add a local server configuration that matches the password.

  .\bin\gost-windows-amd64.exe -L=ss://aes-128-cfb:passcode@:1984 -F=https://xxx:yyy@example.com:443
  

PS: I still don’t know how to configure a global HTTPS proxy on Android without root, or how to set it up on iOS without a U.S. account. Also, I’m unsure how to elegantly configure startup scripts on Windows 10. These are issues to explore further…

Continuation

Regarding the mobile problem mentioned above, I found that HTTPS proxy client support is generally poor. Gost itself seems to have problems, possibly due to my usage. In short, if not using a local gost to connect remotely, authentication errors occur.

During the holiday break, I tinkered a bit more. First, I deployed a gost HTTP proxy on my home NAS using the simplest nohup + ctrl-D method to maintain it. It’s compiled with GOARCH=arm64. After a trial run for a day, Android’s weak built-in HTTP proxy worked well, but globally routing through it wasn’t great. Hence, I switched from HTTP to using SS to connect to HTTPS remotely. I essentially moved the local service on Windows to my NAS. Additionally, through simple double-port forwarding from NAS -> internal router -> optical modem router, I could also use the NAS as an SS server via the public IP.

The remaining issue is the DDNS. After researching, it seems Cloudflare’s API is a more reliable option. Seeing an official flarectl, I compiled it to the NAS and wrote a small script, revisiting the various (pitfalls) wonders of bash, especially remembering special writing for string comparisons such as [ $a != $b ] to [ $a != $b* ] to handle trailing “\r” “\n” characters. However, detaching the name server still takes some time. The final effect is to be tested.

Additionally, on the NAS, I currently use curl to fetch my public IP from a third-party. I have a hunch that this method might not work someday or might cause issues.

When I first started writing Golang, I was always looking for a convenient debugging tool. Back then, I came across documentation about using gdb to debug and also tried delve, but neither felt easy to use. Later, on someone’s advice, I went back to the good old print statements…

Over the past couple of days, I was debugging go test and found that tests would always hang when run per package. I couldn’t think of a suitable method at first, so I thought of delve again. After giving it a try, I found it has become much more mature than before.

Usage

dlv attach ${pid} is the method I use most often. After attaching, you can use debugging commands similar to gdb. You can use help to view specific commands.

(dlv) help
The following commands are available:
    args ------------------------ Print function arguments.
    break (alias: b) ------------ Sets a breakpoint.
    breakpoints (alias: bp) ----- Print out info for active breakpoints.
    call ------------------------ Resumes process, injecting a function call (EXPERIMENTAL!!!)
    clear ----------------------- Deletes breakpoint.
    clearall -------------------- Deletes multiple breakpoints.
    condition (alias: cond) ----- Set breakpoint condition.
    config ---------------------- Changes configuration parameters.
    continue (alias: c) --------- Run until breakpoint or program termination.
    deferred -------------------- Executes command in the context of a deferred call.
    disassemble (alias: disass) - Disassembler.
    down ------------------------ Move the current frame down.
    edit (alias: ed) ------------ Open where you are in $DELVE_EDITOR or $EDITOR
    exit (alias: quit | q) ------ Exit the debugger.
    frame ----------------------- Set the current frame, or execute command on a different frame.
    funcs ----------------------- Print list of functions.
    goroutine (alias: gr) ------- Shows or changes current goroutine.
    goroutines (alias: grs) ----- List program goroutines.
    help (alias: h) ------------- Prints the help message.
    libraries ------------------- List loaded dynamic libraries.
    list (alias: ls | l) -------- Show source code.
    locals ---------------------- Print local variables.
    next (alias: n) ------------- Step over to next source line.
    on -------------------------- Executes a command when a breakpoint is hit.
    print (alias: p) ------------ Evaluate an expression.
    regs ------------------------ Print contents of CPU registers.
    restart (alias: r) ---------- Restart process.
    set ------------------------- Changes the value of a variable.
    source ---------------------- Executes a file containing a list of delve commands.
    sources --------------------- Print list of source files.
    stack (alias: bt) ----------- Print stack trace.
    step (alias: s) ------------- Single step through program.
    step-instruction (alias: si)  Single step a single CPU instruction.
    stepout (alias: so) --------- Step out of the current function.
    thread (alias: tr) ---------- Switch to the specified thread.
    threads --------------------- Print out info for every traced thread.
    trace (alias: t) ------------ Set tracepoint.
    types ----------------------- Print list of types.
    up -------------------------- Move the current frame up.
    vars ------------------------ Print package variables.
    whatis ---------------------- Prints type of an expression.
Type help followed by a command for full documentation.

Many of these commands are the same as those in gdb. Another command I use frequently is grs, which outputs all goroutines. You can also use grs -t, which is equivalent to gdb’s t a a bt. The only slight drawback is that it only outputs 10 stack frames, and any additional ones are truncated.

Additionally, it seems that processes forked by go test can’t be attached to. If you want to test, you must first compile it into a test file and then execute it. You can refer to this issue for more details.

$ dlv attach 19654
could not attach to pid 19654: decoding dwarf section info at offset 0x0: too short

Furthermore, by default, Go’s test caches results, which can be controlled via environment variables. However, with Go modules (go mod), it’s recommended to use ./ddl.test -test.count=1 to disable caching. It doesn’t feel very elegant.

Among all these implementations, I personally think the most reliable and stable one is the original Python version. The reason is simple - it has the most users. The Golang version is said to have the most features and also performs very well, making it quite powerful. This might be due to Golang’s inherent high performance and ease of implementation. There’s also an implementation using libev, a pure C implementation, which also offers good performance and is very lightweight.

Additionally, updating the server is a necessary task for Shadowsocks users due to well-known reasons. The server should be updated frequently. If you’re using the Python implementation, you might be able to install updates via pip, although I haven’t confirmed this. The Golang version may require a Golang build environment, and then you can use go get -u. For updating libev, you can use apt on Debian-based systems, as apt includes shadowsocks-libev. I haven’t checked if it is available in the Red Hat-based yum repositories.

After this introduction, let’s go over the deployment steps, which are quite straightforward:

1. Deploy a Debian 9 or Ubuntu 17 VPS. Mainstream providers like Vultr should have these options available. Assume we are using Debian 9 here.
2. Run apt install shadowsocks-libev to install.
3. Edit the configuration file using vim /etc/shadowsocks-libev/config.json. It’s best to set the Server IP to 0.0.0.0 to avoid IP issues similar to those on AWS Lightsail. *. For AWS Lightsail, you need to bind a static IP and open firewall ports. Specific steps can be found on Google.
4. Restart the service using systemctl restart shadowsocks-libev to apply the changes.
5. Enable TCP BBR. Specific instructions can be found on Google.

package main

import (
	"fmt"
	"os"
	"runtime"
	"time"
)

func main() {
	runtime.GOMAXPROCS(2)
	a := make(map[int]int)
	go func() {
		i := 0
		for {
			a[1] = i
			i++
			time.Sleep(1000)
		}
	}()
	for {
		if a[1] > 1000000 {
			fmt.Println(a[1])
			os.Exit(1)
		}
	}
}

After compiling and running it (assuming your machine has 2 or more cores), you will get the following error:

fatal error: concurrent map read and map write

goroutine 1 [running]:
runtime.throw(0x10c3e05, 0x21)
        /usr/local/Cellar/go/1.10.3/libexec/src/runtime/panic.go:616 +0x81 fp=0xc42004bf00 sp=0xc42004bee0 pc=0x10263f1
runtime.mapaccess1_fast64(0x10a5b60, 0xc42007e180, 0x1, 0xc42008e048)
        /usr/local/Cellar/go/1.10.3/libexec/src/runtime/hashmap_fast.go:101 +0x197 fp=0xc42004bf28 sp=0xc42004bf00 pc=0x1008d27
main.main()
        /Users/yusp/test/panic3/main.go:22 +0x7c fp=0xc42004bf88 sp=0xc42004bf28 pc=0x108e28c
runtime.main()
        /usr/local/Cellar/go/1.10.3/libexec/src/runtime/proc.go:198 +0x212 fp=0xc42004bfe0 sp=0xc42004bf88 pc=0x1027c62
runtime.goexit()
        /usr/local/Cellar/go/1.10.3/libexec/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42004bfe8 sp=0xc42004bfe0 pc=0x104e501

goroutine 5 [runnable]:
time.Sleep(0x3e8)
        /usr/local/Cellar/go/1.10.3/libexec/src/runtime/time.go:102 +0x166
main.main.func1(0xc42007e180)
        /Users/yusp/test/panic3/main.go:18 +0x61
created by main.main
        /Users/yusp/test/panic3/main.go:13 +0x59

It seems straightforward; Golang’s map is not thread-safe, and concurrent read and write cause a panic. However, look at the error information on line /Users/yusp/test/panic3/main.go:18 +0x61, which points to line 18 of main.go where Sleep is called, not the actual point of concurrency issue. In a vast stack trace, it becomes even harder to locate the problem.

A workaround that comes to mind is if you only see the read stack and want to see the write stack, set a variable at the read position, reset it after reading, and check the value of this variable at the write position. If reading is currently happening, panic will be triggered.

This article will first introduce the classification of INSERT statements in TiDB, along with the syntax and semantics of each statement, and then describe the source code implementation of the five types of INSERT statements.

Types of INSERT Statements

In broad terms, TiDB has the following six types of INSERT statements:

• Basic INSERT
• INSERT IGNORE
• INSERT ON DUPLICATE KEY UPDATE
• INSERT IGNORE ON DUPLICATE KEY UPDATE
• REPLACE
• LOAD DATA

In theory, all six statements belong to the category of INSERT statements.

The first one, Basic INSERT, is the most common INSERT statement, using the syntax INSERT INTO VALUES (). It implies inserting a record, and if a unique constraint conflict occurs (such as primary key conflict, unique index conflict), it returns an execution failure.

The second, with the syntax INSERT IGNORE INTO VALUES (), ignores the current INSERT row if a unique constraint conflict occurs and logs a warning. After the statement execution finishes, you can use SHOW WARNINGS to see which rows were not inserted.

The third one, with the syntax INSERT INTO VALUES () ON DUPLICATE KEY UPDATE, updates the conflicting row, then inserts data if there is a conflict. If the updated row conflicts with another row in the table, it returns an error.

The fourth one, similar to the previous case, if the updated row conflicts with another row, this does not insert the row and shows a warning.

The fifth one, with the syntax REPLACE INTO VALUES (), deletes the conflicting row in the table after a conflict and continues to attempt data insertion. If another conflict occurs again, it continues to delete conflicting data on the table until there is no conflicting data left in the table, then inserts the data.

The last one, using the syntax LOAD DATA INFILE INTO, has semantics similar to INSERT IGNORE, both ignoring conflicts. The difference is that LOAD DATA imports data files into a table, meaning its data source is a CSV data file.

Since INSERT IGNORE ON DUPLICATE KEY UPDATE involves special processing on INSERT ON DUPLICATE KEY UPDATE, it won’t be explained in detail separately but will be covered in the same section. Due to the unique nature of LOAD DATA, it will be discussed in other chapters.

Basic INSERT Statement

The major differences among the several INSERT statements lie in the execution level. Continuing from the “TiDB Source Code Reading Series (4) Overview of INSERT Statement”, here is the statement execution process. Those who do not remember the previous content can refer back to the original article.

INSERT’s execution logic is located in executor/insert.go. In fact, the execution logic for all four types of INSERT statements covered previously is in this file. Here, we first discuss the most basic Basic INSERT.

InsertExec is an implementation of the INSERT executor, conforming to the Executor interface. The most important methods are the following three interfaces:

• Open: Performs some initialization
• Next: Executes the write operation
• Close: Performs some cleanup tasks

Among them, the most important and complex is the Next method. Depending on whether a SELECT statement is used to retrieve data (INSERT SELECT FROM), the Next process is divided into two branches: insertRows and insertRowsFromSelect. Both processes eventually lead to the exec function to execute the INSERT.

In the exec function, the first four types of INSERT statements are processed together. The standard INSERT covered in this section directly enters insertOneRow.

Before discussing insertOneRow, let’s look at a segment of SQL.

CREATE TABLE t (i INT UNIQUE);
INSERT INTO t VALUES (1);
BEGIN;
INSERT INTO t VALUES (1);
COMMIT;

Paste these lines of SQL sequentially into MySQL and TiDB to see the results.

MySQL:

mysql> CREATE TABLE t (i INT UNIQUE);
Query OK, 0 rows affected (0.15 sec)
mysql> INSERT INTO t VALUES (1);
Query OK, 1 row affected (0.01 sec)
mysql> BEGIN;
Query OK, 0 rows affected (0.00 sec)
mysql> INSERT INTO t VALUES (1);
ERROR 1062 (23000): Duplicate entry '1' for key 'i'
mysql> COMMIT;
Query OK, 0 rows affected (0.11 sec)

TiDB:

mysql> CREATE TABLE t (i INT UNIQUE);
Query OK, 0 rows affected (1.04 sec)
mysql> INSERT INTO t VALUES (1);
Query OK, 1 row affected (0.12 sec)
mysql> BEGIN;
Query OK, 0 rows affected (0.01 sec)
mysql> INSERT INTO t VALUES (1);
Query OK, 1 row affected (0.00 sec)
mysql> COMMIT;
ERROR 1062 (23000): Duplicate entry '1' for key 'i'

As you can see, for INSERT statements, TiDB performs conflict detection at the time of transaction commit, whereas MySQL does it when the statement is executed. The reason for this is that TiDB is designed with a layered structure with TiKV; to ensure efficient execution, only read operations within a transaction must retrieve data from the storage engine, while all write operations are initially placed within the transaction’s own memDbBuffer in a single TiDB instance. The data is then written to TiKV as a batch during transaction commit. In the implementation, the PresumeKeyNotExists option is set within insertOneRow, assuming that insertions will not encounter conflicts if no conflicts are detected locally, without needing to check for conflicting data in TiKV. These data are marked as pending verification, and the BatchGet interface is used during the commit process to batch check the whole transaction’s pending data.

After all the data goes through insertOneRow and completes the insertion, the INSERT statement essentially concludes. The remaining tasks involve setting the lastInsertID and other return information, and then returning the results to the client.

INSERT IGNORE Statement

The semantics of INSERT IGNORE were introduced earlier. It was mentioned how a standard INSERT checks at the time of commit, but can INSERT IGNORE do the same? The answer is no, because:

1. If INSERT IGNORE is checked at the commit, the transaction module will need to know which rows should be ignored and which should immediately raise errors and roll back, undoubtedly increasing module coupling.
2. Users want to immediately know which rows were not inserted through INSERT IGNORE. In other words, they would like to see which rows were not actually inserted immediately through SHOW WARNINGS.

This requires checking data conflicts promptly when executing INSERT IGNORE. One obvious approach is to try reading the data intended for insertion, logging a warning when finding a conflict, and proceeding to the next row. However, if the statement inserts multiple rows, it would require repetitive reads from TiKV for conflict detection, which would be inefficient. Therefore, TiDB implements a batchChecker, with the code located in executor/batch_checker.go.

In the batchChecker, first, prepare the data for insertion, constructing possible conflicting unique constraints into a key within getKeysNeedCheck. TiDB implements unique constraints by constructing unique keys, as detailed in “Three Articles to Understand TiDB’s Technical Inside Story – On Computation”.

Then, pass the constructed keys through BatchGetValues to read them all at once, resulting in a key-value map where those read are the conflicting data.

Finally, check the keys of the data intended for insertion against the results from BatchGetValues. If a conflicting row is found, prepare a warning message and proceed to the next row. If a conflicting row isn’t found, a safe INSERT can proceed. The implementation of this portion is found in batchCheckAndInsert.

Similarly, after executing the insertion for all data, return information is set, and the execution results are returned to the client.

INSERT ON DUPLICATE KEY UPDATE Statement

INSERT ON DUPLICATE KEY UPDATE is the most complex among the INSERT statements. Its semantic essence includes both an INSERT and an UPDATE. The complexity arises since during an UPDATE, a row can be updated to any valid version.

In the previous section, it was discussed how TiDB uses batching to implement conflict checking for special INSERT statements. The same method is used for INSERT ON DUPLICATE KEY UPDATE, but the implementation process is somewhat more complex due to the semantic complexity.

Initially, similar to INSERT IGNORE, the keys constructed from the data to be inserted are read out at once using BatchGetValues, resulting in a key-value map. Then, all records corresponding to the read keys are again read using a batch BatchGetValues, prepared for possible future UPDATE operations. The specific implementation is in initDupOldRowValue.

Then, during conflict checking, if a conflict occurs, an UPDATE is performed first. As discussed in the Basic INSERT section earlier, TiDB executes INSERT in TiKV during commit. Similarly, UPDATE is also executed in TiKV during commit. In this UPDATE process, unique constraint conflicts might still occur. If so, then an error is returned. If the statement is INSERT IGNORE ON DUPLICATE KEY UPDATE, this error is ignored, and the next row proceeds.

In the UPDATE from the previous step, another scenario can occur, as in the SQL below:

CREATE TABLE t (i INT UNIQUE);
INSERT INTO t VALUES (1), (1) ON DUPLICATE KEY UPDATE i = i;

Here, it is clear that there are no original data in the table; the INSERT in the second line cannot read out possibly conflicting data, but there is a conflict between the two rows of data intended to be inserted themselves. Correct execution here should involve the first 1 being inserted normally, with the second 1 encountering conflict and updating the first 1. Thus, it is necessary to handle it as follows: remove the key-value of the data updated in the previous step from the initial step’s key-value map, reconstruct unique constraint keys and values for the data from the UPDATE based on table information, and add this key-value pair back into the initial key-value map for subsequent data conflict checking. The detail implementation is in fillBackKeys. This scenario also arises in other INSERT statements like INSERT IGNORE, REPLACE, and LOAD DATA. It is introduced here because INSERT ON DUPLICATE KEY UPDATE showcases the full functionality of the batchChecker.

Finally, after all data completes insertion/update, return information is set, and results are returned to the client.

REPLACE Statement

Although the REPLACE statement appears as a separate type of DML, in examining its syntax, it is merely replacing INSERT with REPLACE compared to a standard Basic INSERT. The difference is that REPLACE is a one-to-many statement. Briefly, for a typical INSERT statement which needs to INSERT a row and encounters a unique constraint conflict, various treatments are available:

• Abandon the insert and return an error: Basic INSERT
• Abandon the insert without error: INSERT IGNORE
• Abandon the insert, turning it into updating the conflicting row. If the updated value conflicts again,
• Return an error: INSERT ON DUPLICATE KEY UPDATE
• No error: INSERT IGNORE ON DUPLICATE KEY UPDATEThey all handle conflicts when a row of data conflicts with a row in the table differently. However, the REPLACE statement is distinct; it will delete all conflicting rows it encounters until there are no more conflicts, and then insert the data. If there are 5 unique indexes in the table, there could be 5 rows conflicting with the row waiting to be inserted. The REPLACE statement will delete these 5 rows all at once and then insert its own data. See the SQL below:

CREATE TABLE t (
i int unique,
j int unique,
k int unique,
l int unique,
m int unique);
INSERT INTO t VALUES
(1, 1, 1, 1, 1),
(2, 2, 2, 2, 2),
(3, 3, 3, 3, 3),
(4, 4, 4, 4, 4);
REPLACE INTO t VALUES (1, 2, 3, 4, 5);
SELECT * FROM t;
i j k l m
1 2 3 4 5

After execution, it actually affects 5 rows of data.

Once we understand the uniqueness of the REPLACE statement, we can more easily comprehend its specific implementation.

Similar to the INSERT statement, the main execution part of the REPLACE statement is also in its Next method. Unlike INSERT, it passes its own exec method through insertRowsFromSelect and insertRows. In exec, it calls replaceRow, which also uses batch conflict detection in batchChecker. The difference from INSERT is that all detected conflicts are deleted here, and finally, the row to be inserted is written in.

In Conclusion

The INSERT statement is among the most complex, versatile, and powerful of all DML statements. It includes statements like INSERT ON DUPLICATE UPDATE, which can perform both INSERT and UPDATE operations, and REPLACE, where a single row of data can impact many rows. The INSERT statement itself can be connected to a SELECT statement as input for the data to be inserted, thus its implementation is influenced by the planner (for more on the planner, see related source code reading articles: Part 7: Rule-Based Optimization and Part 8: Cost-Based Optimization). Familiarity with the implementation of various INSERT-related statements in TiDB can help readers use these statements more reasonably and efficiently in the future. Additionally, readers interested in contributing code to TiDB can also gain a quicker understanding of this part of the implementation through this article.

yum install haproxy is effective for CentOS 7. After installation, you can start the service using systemctl start haproxy. But don’t rush yet.

Configuring HAProxy

Add the following content to /etc/haproxy/haproxy.cfg.

global # The content of global is generally fixed and quite understandable.
        log 127.0.0.1   local2
        maxconn 4096
        user haproxy
        group haproxy
        chroot /var/lib/haproxy
        daemon
        pidfile /var/run/haproxy.pid
        stats socket /var/run/haproxy.sock         # Create a socket file for haproxy
        nbproc 40                                  # Start 40 processes to forward concurrently, higher versions can use nbthread, a threaded approach.

defaults # This section is mostly copied, not entirely clear on the options.
        log     global
        mode    http
        option  tcplog
        option  dontlognull
        retries 3
        option  redispatch
        maxconn 1024
        timeout connect 5000ms
        timeout client 50000ms
        timeout server 50000ms

listen cdb_cluster 0.0.0.0:3030  # The actual proxy name and address for receiving services.
## cdb balance leastconn - the cluster listening on port 3030.
        mode tcp
        balance leastconn  # This method is most suitable for databases; do not change.
        server cdb1 172.16.30.3:26257 check # Check seems to require a port for feedback status; without it, it might not work, but it doesn't matter.
        server cdb2 172.16.30.3:26258 check
        server cdb3 172.16.30.3:26259 check
        server cdb4 172.16.30.3:26260 check

Start and Connect

systemctl start haproxy to start the service.

psql -Uroot -h127.0.0.1 -p3030 test to connect to the database.

CockroachDB Official Recommendation

CockroachDB officially provided their recommended configuration. In this configuration, they use:

default
# TCP keep-alive on client side. Server already enables them.
    option              clitcpka

listen psql
    option httpchk GET /health?ready=1

These two configurations, the first is to keep the client connection alive, which seems very useful. The second is a status check port, which I understand might be an option to ensure the service is available before dispatching requests, and it also seems very useful. It is recommended to add them.

First of all, TPC-C is the de facto OLTP benchmark standard. It is a set of specifications, and any database can publish its test results under this standard, so there’s no issue of quarreling over the testing tools used.

Secondly, TPC-C is closer to real-world scenarios as it includes a transaction model within it. In the flow of this transaction model, there are both high-frequency simple transaction statements and low-frequency inventory query statements. Therefore, it tests the database more comprehensively and practically.

Testing TPC-C on CockroachDB

This year, CockroachDB released its TPC-C performance results. However, unfortunately, they did not use a tool recognized by the database industry that implements the TPC-C standard for testing. Instead, they used their own implementation of a TPC-C tool. The compliance level of this tool was not recognized. In the white paper officially released by them, it is also mentioned that this TPC-C cannot be compared with the TPC-C standard.

Therefore, I thought of using a highly recognized tool in the industry for testing. Here, I chose Benchmarksql version 5.0.

Benchmarksql 5.0 supports the PostgreSQL protocol, Oracle protocol, and MySQL protocol (the MySQL protocol is supported in the code, but the author hasn’t fully tested it, so the official documentation doesn’t mention MySQL). Among these, the PostgreSQL protocol is supported by CockroachDB.

Test Preparation

After preparing the Benchmarksql code, don’t rush into testing. There are three main pitfalls here that need to be addressed first.

1. CockroachDB does not support adding a primary key after table creation. Therefore, you need to include the primary key when creating the table. Specifically, in the run folder under the root directory of the Benchmarksql code, create a sql.cdb folder. Copy tableCreates.sql and indexCreates.sql from the sql.common folder at the same level into sql.cdb. Then move the primary keys in indexCreates.sql into the table creation statements in tableCreates.sql. For how to define indexes while creating tables, please refer to the database documentation syntax via Google.

2. CockroachDB is a “strongly typed” database. This is my own way of describing it. It has a rather peculiar behavior: when you add different data types (e.g., int + float), it will report an error saying, “InternalError: unsupported binary operator: <int> + <float>”. Generally, databases don’t behave like this; most would perform some implicit conversions, or in other words, they are very tolerant of SQL writers. But CockroachDB is unique in that if you don’t specify the type, it reports an error. This greatly reduces the burden of type inference in its internal implementation.

   This behavior causes Benchmarksql to fail to run the tests properly. The solution is to add the required type at the position where the error occurs. For example, change update t set i = i + ?; (the ? is generally filled in using prepare/execute) to update t set i = i + ?::DECIMAL;. Yes, CockroachDB specifies types explicitly by adding ::<type_name> at the end. But strangely, not all additions require type specification.

3. CockroachDB does not support SELECT FOR UPDATE. This is the easiest to solve: comment out all FOR UPDATE clauses in Benchmarksql. CockroachDB itself supports the serializable isolation level; lacking FOR UPDATE doesn’t affect consistency.

Starting the Test

After overcoming the pitfalls mentioned above, you can proceed with the normal testing process: creating the database, creating tables and indexes, importing data, and testing. You can refer to Benchmarksql’s HOW-TO-RUN.txt.

Test Results

On my test machine with 40 cores, 128 GB of memory, and SSD, under 100 warehouses, the tpmC is approximately 5,000. This is about one-tenth of PostgreSQL 10 on the same machine. PostgreSQL can reach around 500,000 tpmC.

CockroachDB uses the PostgreSQL protocol. If you want to use Sysbench for testing, you need to enable pg protocol support in Sysbench. Sysbench already supports the pg protocol, but it is not enabled by default during compilation. You can configure it with the following command:

./configure --with-pgsql

Of course, preliminary work involves downloading the Sysbench source code and installing the necessary PostgreSQL header files required for compilation (you can use yum or sudo to install them).

Testing

The testing method is no different from testing MySQL or PostgreSQL; you can test any of the create, read, update, delete (CRUD) operations you like. The only thing to note is to set auto_inc to off.

This is because CockroachDB’s auto-increment behavior is different from PostgreSQL’s. It generates a unique id, but it does not guarantee that the ids are sequential or incremental. This is fine when inserting data. However, during delete, update, or query operations, since all SQL statements use id as the condition for these operations, you may encounter situations where data cannot be found.

That is:

When auto_inc = on (which is the default value in Sysbench)

Table Structure

CREATE TABLE sbtest1 (
   id INT NOT NULL DEFAULT unique_rowid(),
   k INTEGER NOT NULL DEFAULT 0:::INT,
   c STRING(120) NOT NULL DEFAULT '':::STRING,
   pad STRING(60) NOT NULL DEFAULT '':::STRING,
   CONSTRAINT ""primary"" PRIMARY KEY (id ASC),
   INDEX k_1 (k ASC),
   FAMILY ""primary"" (id, k, c, pad)
)

Data

root@:26257/sbtest> SELECT id FROM sbtest1 ORDER BY id LIMIT 1;
+--------------------+
|         id         |
+--------------------+
| 354033003848892419 |
+--------------------+

As you can see, the data does not start from 1, nor is it sequential. Normally, the id in a Sysbench table should be within the range [1, table_size].

SQL

UPDATE sbtest%u SET k = k + 1 WHERE id = ?

Taking the UPDATE statement as an example, id is used as the query condition. Sysbench assumes that this id should be between [1, table_size], but in reality, it’s not.

Example of Correct Testing Command Line

sysbench --db-driver=pgsql --pgsql-host=127.0.0.1 --pgsql-port=26257 --pgsql-user=root --pgsql-db=sbtest \
        --time=180 --threads=50 --report-interval=10 --tables=32 --table-size=10000000 \
        oltp_update_index \
        --sum_ranges=50 --distinct_ranges=50 --range_size=100 --simple_ranges=100 --order_ranges=100 \
        --index_updates=100 --non_index_updates=10 --auto_inc=off prepare/run/cleanup

INSERT Testing

Let’s discuss the INSERT test separately. The INSERT test refers to Sysbench’s oltp_insert. The characteristic of this test is that when auto_inc is on, data is inserted during the prepare phase of the test; otherwise, only the table is created without inserting data. Because when auto_inc is on, after the prepare phase, during the run phase, the inserted data will not cause conflicts due to the guarantee of the auto-increment column. When auto_inc is off, the id of the data inserted during the run phase is randomly assigned, which aligns with some actual testing scenarios.

For CockroachDB, when testing INSERT operations with auto_inc set to off, after the prepare phase, during the run phase of data insertion, you can observe the monitoring metrics (by connecting to CockroachDB’s HTTP port) under the “Distribution” section in “KV Transactions”. You’ll notice a large number of “Fast-path Committed” transactions. This indicates that transactions are committed using one-phase commit (1PC). That is, the data involved in the transaction does not span across CockroachDB nodes, so there’s no need to ensure consistency through two-phase commit transactions. This is an optimization in CockroachDB, which is very effective in INSERT tests and can deliver excellent performance.

If auto_inc is on, although for other tests that require read-before-write operations, the results in CockroachDB might be inflated, it is still fair for the INSERT test. If time permits, you can supplement the tests to see the differences.

OLTP-Bench is an open-source benchmarking tool platform for OLTP scenarios from CMU’s DB Group. It was designed to provide a simple, easy-to-use, and extensible testing platform.

It connects to databases via the JDBC interface, supporting the following test suites:

• TPC-C
• Wikipedia
• Synthetic Resource Stresser
• Twitter
• Epinions.com
• TATP
• AuctionMark
• SEATS
• YCSB
• JPAB (Hibernate)
• CH-benCHmark
• Voter (Japanese “American Idol”)
• SIBench (Snapshot Isolation)
• SmallBank
• LinkBench

Detailed project information can be found here, and the GitHub page is here.

The project introduction page includes three papers published by the authors, with the one from 2013 being the most important, also linked on the GitHub page.

Based on the GitHub page, the project does not seem to have a high level of attention and has not been very active recently. Most issues and pull requests come from within CMU.

OLTP-Bench: An Extensible Testbed for Benchmarking Relational Databases

The paper “OLTP-Bench: An Extensible Testbed for Benchmarking Relational Databases” can be regarded as the most detailed introduction to this project.

In the first and second chapters, the authors introduce the motivation for creating this framework, which is to integrate multiple test sets and provide features that simple benchmarking tools do not have, while offering excellent extensibility to attract developers to support more databases.

From the activity on GitHub, it is evident that this extensibility is more about adding database support rather than test sets. However, the number of supported test suites is already quite extensive.

Chapter three introduces the architectural design, with a focus on test suite management, load generators, SQL syntax conversion, multi-client scenarios (similar to multiple sysbench instances stressing a single MySQL), and result collection.

Chapter four discusses the supported test suites. I’m only familiar with TPCC and YCSB. The authors classify them from three perspectives:

1. Transaction-focused, such as TPCC and SmallBank
2. Internet applications, like LinkBench and Wikipedia
3. Specialized tests, such as YCSB and SIBench

Further details can be seen in the table: [table]

Chapter five describes the demo deployment environment, with subsequent sections introducing the demo’s features.

Chapter six uses the demo from the previous chapter to introduce features, analyzed as follows:

1. Rate control. It seems odd for a benchmarking tool to perform rate control, as the conventional understanding is to push performance as high as possible to gauge system limits. The paper provides an example using the Wikipedia test suite, increasing by 25 TPS every 10 seconds to observe database latency changes.

2. Tagging different transactions in the same test suite for separate statistics – using TPCC as an example to statistically categorize transactions from different stages.

3. Modifying load content, like switching from read-only to write-only loads.

4. Changing the method for load randomness.

5. Monitoring server status alongside database monitoring by deploying an OLTP-Bench monitor on the server.

6. Running multiple test suites simultaneously, such as running TPCC and YCSB concurrently.

7. Multi-client usage, mentioned in chapter three.

8. Repeatability. To prove OLTP-Bench results are genuine and reliable, the authors tested PG’s SSI performance using SIBench from the tool on similarly configured machines, achieving results consistent with those in PG’s SSI paper.

In summary, rate control and transaction tagging stand out as novel features, while the rest are not particularly special.

Chapter seven is arguably the most valuable part of the article, discussing cloud environments where users might only have database access and not server control. Users may struggle to assess the cost-effectiveness of different cloud database services or configurations due to charges encompassing CPU, storage, network, and asynchronous sync in some architectures. Thus, using benchmarking tools to derive performance and subsequently calculate cost-effectiveness is particularly worthwhile. This chapter compares varying perspectives: different service providers, configurations, comparing databases on the same configuration, and presents the cost-effectiveness outcomes.

In chapter eight, the authors compare OLTP-Bench with other similar tools, providing a favorable self-assessment.

Chapter nine outlines the authors’ future plans, including support for pure NoSQL, additional databases’ proprietary SQL syntax, generating real-world load distributions from production data, and support for stored procedures.

In conclusion, as the authors mentioned, this is an integrative framework where ease of use and extensibility are key.

Usage Summary

OLTP-Bench is relatively simple to install and use, especially the deployment. Its cross-platform nature provides a better user experience compared to traditional tpcc and sysbench. Usage is relatively straightforward due to the plethora of test configuration templates provided, allowing easy initiation of tests with simple configuration file modifications. The test results are stable, although certain features mentioned in papers, like server status monitoring, still require exploration.

I tested all 15 test suites on MySQL 5.7 and TiDB, obtaining the following results: [table]

Its usability is quite evident. As for the ease of secondary development, it should be relatively simple, considering the entire OLTP-Bench project is not particularly large, with around 40,000 lines of code.

Other

• tpch: While the framework’s code appears to support tpch, it proved unusable during practical tests, likely due to incomplete implementation and thus excluded from the README.
• Referring to future work mentioned in chapter nine of the paper, especially “generating load to match production data distribution,” this remains unimplemented, as seen in the codebase.

The English font set in Microsoft’s VS Code is monospaced, while the Chinese font is not.

Method

1. Download and install the fonts from here. This font combines Source Han Sans and is said to align punctuation strictly.
2. Use the method described here to set up VS Code.

PS: It is indeed aligned, although the English text appears a bit narrow.

The DDL paper on F1 serves as the foundation for TiDB’s DDL implementation. There are two main papers on F1: one provides an overview of F1’s DDL, and the other specifically details the schema change method for DDL. I personally believe the second is key and more confusing to me. There is an introduction to the second paper here, which can help in understanding.

Understanding

Online DDL Concept

The DDL discussed here refers to online DDL. The concept of online DDL originates from databases like MySQL, whereas PostgreSQL and similar databases might not support it. This concept is also quite vague; the distinction is whether you need to use exclusive locks during DDL operations to block transactions. Therefore, all databases can perform online DDL; it just depends on whether they’re willing to put in the effort. For traditional businesses where 24/7 availability isn’t a priority, DDL operations can be performed during maintenance times or late at night. Even if a few users are online, at most, they might experience minor delays. However, modern internet businesses are strict about maintenance windows, creating a higher demand for non-blocking DDL, which MySQL, as a quintessential internet database, was first to support. The typical implementation involves creating a copy of the schema table, with operations being sent to both the new and old tables during the transition.

For MySQL’s supported online DDL, see this webpage. Primarily, it’s categorized into operations on indexes and columns. This explains my curiosity about why TiDB’s examples for implementation often involve adding indexes.

F1’s Method

Having worked on something similar to Aurora before, there were many issues with this area. If you’re only performing offline DDL, it doesn’t have to be this complicated. According to F1’s paper, it uses the following series of state changes to accomplish a DDL:

                                    (reorganization)
absent -> delete only -> write only ---------------> public

In this sequence, each node passes through these four states, transitioning to the next state upon receiving a command. The agreed-upon rule is that each state doesn’t persist for more than twice the lease time across all nodes. How is this ensured? Through the following rule: if a node takes too long to move to the next state after receiving the transition command (meaning a state exceeds twice the lease time), it means the node received the command too late and will stop providing services and shut down.

Within these four states, ‘absent’ indicates a state where the node hasn’t received instructions yet, and ‘public’ signifies the completion of the DDL. What about the two middle states? The background link described them as follows:

• A delete-only table or column can be modified only by delete operations.
• A delete-only index can be modified only by delete and update operations. Update operations can delete key-value pairs corresponding to updated index keys, but they cannot create any new ones.
• A write-only column or index can have their key-value pairs modified by insert, delete, and update operations, but none of their pairs can be read by user transactions.
• A write-only constraint is applied for all new insert, delete, and update operations, but it is not guaranteed to hold over all existing data.

Summarized in a table:

The first state is rather peculiar, and the second one even more so—“can not read” might have been a better name for it. It’s said that this design allows two concurrent states to behave consistently, specifically pairs like (absent and delete only), (delete only and write only), and (write only and public).

Examples:

Adding index idx to table t, the deployment environment consists of two databases, a and b.

1. a enters delete only and completes adding the index. b has yet to receive any instructions.
   • Insert operations on a and b: a’s idx is ignored, b is unaware and also ignores.
   • Read operations: a and b ignore idx.
   • Deletes and updates: a’s idx responds, b ignores.
2. a enters write only, b enters delete only, and indexing is completed.
   • Insert operations on a and b: a’s idx responds, b ignores (b loses index data?).
   • Read operations: a and b ignore idx.
   • Deletes and updates: both a and b’s idx respond.
3. a enters the public state, b enters the write-only state.
   • Insert operations on a and b: both a and b’s idx respond.
   • Read operations: a’s idx responds, b ignores.
   • Deletes and updates: both a and b’s idx respond.

Removing index idx from table t, with two databases a and b in the deployment environment.

1. a enters delete only and completes removing the index. b has yet to receive any instructions.
   • Insert operations on a and b: a’s idx is ignored, b has index idx and inserts into the index.
   • Read operations: a ignores idx, b uses it.
   • Deletes and updates: a’s idx responds (no-op if the index is removed), b responds.
2. a enters write only, b enters delete only, and fulfills the command.
   • Insert operations on a and b: a’s idx responds (no-op), b ignores.
   • Read operations: both a and b ignore idx.
   • Deletes and updates: both a’s and b’s idx respond (it’s a no-op as the index is already deleted).
3. a enters the public state, b enters write only.
   • Insert operations on a and b: both a and b respond (b no-op?).
   • Read operations: a’s idx responds, b ignores.
   • Deletes and updates: both a and b’s idx respond (b no-op?).

The CAP theorem has become one of the hottest theorems in recent years; when discussing distributed systems, CAP is inevitably mentioned. However, I feel that I haven’t thoroughly understood it, so I wanted to write a blog post to record my understanding. I will update the content as I gain new insights.

Understanding

I read the first part of this paper.

The CAP theorem [Bre12] says that you can only have two of the three desirable properties of:

• C: Consistency, which we can think of as serializability for this discussion;
• A: 100% availability, for both reads and updates;
• P: tolerance to network partitions.

This leads to three kinds of systems: CA, CP and AP, based on what letter you leave out.

Let me share my understanding, using a network composed of three machines (x, y, and z) as an example:

• C (Consistency): The three machines appear as one. Operations of addition, deletion, modification, and query on any one machine should always be consistent. That is, if you read data from x and then read from y, the results are the same. If you write data to x and then read from y, you should also read the newly written data. Wikipedia also specifically mentions that it’s acceptable to read the data just written to x from y after a short period of time (eventual consistency).

• A (Availability): The three machines, as a whole, must always be readable and writable; even if some parts fail, it must be readable and writable.

• P (Partition Tolerance): If the network between x, y, and z is broken, any machine cannot or refuses to provide services; it is neither readable nor writable.

Here, C is the easiest to understand. The concepts of A and P are somewhat vague and easy to confuse.

Now let’s discuss the three combinations:

If the network between x, y, and z is disconnected:

• CA: Ensure data consistency (C). When x writes data, y can read it (C). Allow the system to continue providing services—even if only x and y are operational—ensuring it is readable and writable (A). We can only tolerate z not providing service; it cannot read or write, nor return incorrect data (losing P).

• CP: Ensure data consistency (C). Allow all three machines to provide services (even if only for reads) (P). We can only tolerate that x, y, and z cannot write (losing A).

• AP: Allow all three machines to write (A). Allow all three machines to provide services (reads count) (P). We can only tolerate that the data written by x and y doesn’t reach z; z will return data inconsistent with x and y (losing C).

CA is exemplified by Paxos/Raft, which are majority protocols that sacrifice P; minority nodes remain completely silent. CP represents a read-only system; if a system is read-only, whether there’s a network partition doesn’t really matter—the tolerance to network partitions is infinitely large. AP is suitable for systems that only append and do not update—only inserts, no deletes or updates. Finally, by merging the results together, it can still function.

Travis CI is generally used for automating tests without needing to update the repository with the test outputs. This article explains how to automatically commit the results from Travis CI.

Process

The basic process references this gist.

Below is the .travis.yml from the gist.

language: ruby
rvm:
  - 2.0.0
env:
  global:
  - USER="username"
  - EMAIL="username@mail.com"
  - REPO="name of target repo"
  - FILES="README.md foo.txt bar.txt"
  - GH_REPO="github.com/${USER}/${REPO}.git"
  - secure: "put travis gem output here => http://docs.travis-ci.com/user/encryption-keys/"
script:
  - ruby test.rb
after_success:
  - MESSAGE=$(git log --format=%B -n 1 $TRAVIS_COMMIT)
  - git clone git://${GH_REPO}
  - mv -f ${FILES} ${REPO}
  - cd ${REPO}
  - git remote
  - git config user.email ${EMAIL}
  - git config user.name ${USER}
  - git add ${FILES}
  - git commit -m "${MESSAGE}"
  - git push "https://${GH_TOKEN}@${GH_REPO}" master > /dev/null 2>&1

Note here that MESSAGE should be quoted when committing, which the original gist did not include.

Original README:

# Travis-CI tested push
Sometimes we have a private repository to hold both problems and solutions as a reference for the class projects.
The students can see only the problems in a public repository where they are able to clone/fork and develop their own solutions.
We do not want the solution files in the public repository and each bug found/feature added in the project requires a push for each repository.
It would be cool to work only with the reference repo and use tests to see if our modification is good enough for the public release.
This is possible with Travis-CI following simple steps:
- Create private and public repos
- Download Ruby
- Install the travis gem ``gem install travis``
- Generate a token in the Github website to allow others to play with your repos (copy the hash)
- Log into your git account
- Generate a secure token with the Travis gem (copy long hash)
- Fill the environment variables in the ```.travis.yml``` file (USER, EMAIL, REPO, FILES)
- Replace the value of **secure** with your long hash
- Replace **GH_TOKEN** with your Travis token name
- Push ```.travis.yml``` to private repo
- Go to Travis to unlock your private repo tests
- Push your files to the private repo to test

Travis now has a [deployment](https://docs.travis-ci.com/user/deployment/) feature, which may be better for certain scenarios.

A simple translation:

1. Create a GitHub project. The original seems to have created two projects, one for updating another.
2. Install Ruby. Usually, gem is installed alongside.
3. Install travis via gem install travis.
4. Apply for a token for your GitHub account. You can Google the details. When selecting token permissions, only tick all related to repo; others can be omitted.
5. Copy the generated token.
6. In the root directory of the local machine’s repo (not sure if it must be the root) run travis encrypt GH_TOKEN="copied token". This creates an encrypted token to use as ${GH_TOKEN}, essentially an environment variable. The command output, a string on the screen, needs to be pasted into the travis config file after secure:. Use travis encrypt GH_TOKEN="copied token" --add to write directly into the config file.
7. Commit the modified configuration file.