agentd is my attempt to put those questions behind a small runtime contract. It is an experimental, multi-tenant, single-agent runtime for one host. The model still decides what to do. The host runtime owns the less glamorous parts: persistence, scheduling, scoped serialization, capability boundaries, raw traces, and a pull delivery outbox.

The goal is not to build a universal agent framework. The narrower goal is to make one turn durable and inspectable without coupling execution to Telegram, a browser, a game, or any other user interface.

The boundary starts with a turn

External systems submit work through one transport-neutral endpoint:

POST /v1/tenants/:tenant/turns

A request identifies an agent, a scope, a payload, and optionally a delivery destination and request_id. Submission returns 202 with a queued run_id. The caller can wait for the canonical result, inspect the trace, cancel the run, or consume delivery through a separate adapter.

REST turn / due schedule
          |
          v
      queued run -- transactional (tenant, agent, scope) claim
          |
          v
  context -> model <-> allowed built-in/MCP tools
          |
          v
 transaction: output + terminal trace + context + optional delivery

This separation matters because a transport and a runtime answer different questions. Telegram knows webhook authentication, chat identifiers, message formatting, and Telegram’s retry behavior. The runtime knows run state, context, capabilities, and final output. Neither component needs to absorb the other’s failure model.

Scope is both context ownership and a concurrency key

Every run moves through five states:

The tuple (tenant, agent, scope) is the serialization boundary. Runs for the same tuple execute one at a time; unrelated scopes can run concurrently. The claim is transactional, and the database also enforces that at most one run for a scope is running.

Using scope for both rolling context and serialization is deliberate. If two turns can modify the same conversation history concurrently, the runtime needs a merge policy or a clear winner. agentd avoids inventing one: a scope has one active owner. A Telegram chat can use tg:<chat_id>; a game can use a match and seat identifier; another caller can define a different stable boundary.

This is not distributed locking. It is single-database coordination for one host. There is no multi-process lease protocol, failover, or replication claim.

A process restart also has an explicit, conservative outcome. On startup, persisted runs that were running become failed with an agentd restarted error, while durable queued work remains available to the dispatcher. The runtime does not try to reconstruct the interrupted model/tool loop or silently execute it again. That avoids pretending an in-flight external effect is safe to repeat, at the cost of requiring a caller or operator to decide whether a failed turn should be resubmitted. More sophisticated recovery would need an explicit side-effect protocol, not just a different status update.

Submission idempotency is not replay

An optional request_id is unique within a tenant. Repeating the same submission returns the existing run instead of creating another one. This is useful when an HTTP client cannot tell whether its first request succeeded.

But request deduplication does not make the whole agent execution replayable. During a run, the model can call tools, an MCP server can affect another system, and a transport can deliver a message. Those effects do not become reversible because the initial request had an idempotency key.

The raw run_log records model, tool, output, status, and error observations. It is an inspection trace, not a deterministic re-execution engine. There is no side-effect ledger, rollback protocol, or replay simulator. Calling these different properties by different names prevents a comforting but false exactly-once story.

The model chooses; the host constrains

An agent is configuration rather than executable plugin code: persona, model, allowed capability families, limits, temperature, token budget, and context window. The native loop sends visible schemas to an OpenAI-compatible provider, validates the returned tool call, executes it in host code, records the result, and continues until the model produces final JSON or reaches a boundary.

The public alpha has 15 built-in capabilities across artifacts, memory, schedules, clock, bounded public-web access, and arithmetic. It intentionally has no generic shell, arbitrary HTTP request, hidden filesystem handle, approval queue, or operator execute endpoint.

MCP extends the tool catalog without changing that ownership model. Servers belong to a tenant, discovery is explicit, exposed names must be unique, and an allowlist can narrow the discovered catalog. Secret configuration stores the names of environment variables rather than their values.

That is a capability boundary, not a sandbox. An operator-configured stdio MCP process runs with the agentd OS user’s host and network permissions. A model can also be persuaded by a malicious prompt or tool result to call any capability it can see. The safe rule is therefore simple: expose only side effects that are acceptable without human approval, and isolate untrusted MCP servers outside the runtime.

Successful finalization has one transaction boundary

The most important write path happens after the model loop has produced a successful result. One database transaction commits:

• the canonical output and succeeded run status;
• the terminal trace event;
• the bounded rolling context;
• and, when explicitly requested, one pending delivery referencing the run.

If cancellation has already changed the run state, successful finalization is rejected. A cancelled run cannot leave behind successful output, updated context, or a delivery row. A second finalization is rejected as well.

This transaction does not roll back external tool effects that occurred before it. Its guarantee is narrower: readers do not observe a successful run whose output, terminal trace, context, and delivery intent disagree with one another. That is the consistency boundary the runtime can actually own.

Delivery is durable intent, not a callback hidden in execution

Every successful result is pullable. Delivery exists only when the caller provides an explicit destination such as tg:42; scope is never silently treated as an address.

Telegram -> webhook -> turn with explicit tg:<chat_id> destination
agentd    -> atomic run finalization -> delivery outbox
adapter   -> claim -> Telegram API -> acknowledge

Delivery rows reference the run instead of copying its output. An adapter claims pending work with a random token and expiry, performs the remote side effect, and acknowledges one of delivered, retry, or failed. An expired claim can be issued again; a retry updates the same row rather than inserting a new delivery.

The independent agentd-telegram-adapter uses exactly this interface. Its webhook path submits a turn and asks for an explicit Telegram destination. It does not send a direct reply from the webhook handler and does not fall back to polling run output as a second delivery path. This removes two easy sources of duplicate messages.

The outbox still cannot prove that a remote system performed an effect if the adapter crashes before acknowledgement. Claim expiry and retry make recovery possible; they do not manufacture exactly-once semantics from an external API.

State is separated by meaning, not by service count

The runtime uses one libSQL database. The schema separates responsibilities:

• runs own activation, status, canonical output, and requested destination;
• run_log owns raw execution observations;
• contexts own recent messages for a scope;
• memory owns explicit durable facts plus lexical and semantic indexes;
• artifacts own larger payloads;
• deliveries own remote-delivery state and retry fields;
• schedules and MCP registrations remain tenant-scoped resources.

This is intentionally less infrastructure than earlier prototypes. There is no separate object store, vector service, audit database, or replay database. The memory implementation keeps canonical text, an FTS5 index, and a pinned 384-dimension E5 embedding together; lexical and semantic ranks are fused with RRF.

The trade-off is equally explicit: this pre-1.0 runtime does not preserve schema or HTTP compatibility. Runtime data is treated as disposable. A schema mismatch requires --reset-data, not a migration. That is acceptable for the current personal-runtime scope, but it would not be an acceptable default for a mature multi-user platform.

Design claims should point to executable evidence

The repository includes a reliability matrix that maps each important property to a named test. The current suite covers tenant-scoped request deduplication, same-scope serialization, failed-agent lane release, atomic finalization, cancellation races, pull-only success, delivery reclaim and acknowledgement, MCP scoping, public-web network boundaries, memory scoping, schema mismatch, and non-loopback authentication requirements.

There is also a credential-free deterministic demo:

git clone https://github.com/minifish-org/agentd.git
cd agentd
./scripts/demo-e2e.sh

It starts a loopback OpenAI-compatible fixture, starts agentd with a fresh database, creates a tenant and agent, submits a turn, waits for the result, inspects the trace, verifies pull-only behavior, and exercises artifact storage. The first run downloads about 448 MiB of revision- and checksum-pinned embedding assets.

The fixture proves the documented runtime and API path. It does not prove model quality, real-provider tool compatibility, prompt-injection resistance, streaming behavior, or production latency. The release workflow separately runs the real embedding smoke test and builds and inspects the complete container image.

The gaps are part of the design document

The public alpha does not claim evidence for:

• process-kill fault injection at every transaction boundary;
• multi-process or multi-host coordination, failover, or replication;
• compatibility across database schema or HTTP versions;
• load, soak, latency, quota, or denial-of-service limits;
• deterministic replay or rollback of external tool effects;
• sandboxing of operator-configured MCP processes;
• behavior against every provider, MCP server, or transport adapter;
• model safety or factual correctness.

Its bearer token is an instance-wide operator credential, not end-user authentication. Tenants provide resource namespaces; they do not isolate mutually distrustful users who share that token. TLS termination, rate limits, backups, and host-level resource isolation remain deployment responsibilities. The full boundary is documented in the threat model.

Publishing these gaps is useful because an experimental system is easiest to overstate precisely where its tests end. A reliability matrix should show both the properties under test and the empty cells.

What building it changed for me

My distributed-database background makes me look for ownership, monotonic boundaries, and failure states before adding features. agentd applies those habits at a smaller scale:

1. A conversation scope needs an execution owner, not just a string in a prompt.
2. Idempotent submission, deterministic replay, and external-effect recovery are separate problems.
3. Successful output is not complete until related durable state agrees at one transaction boundary.
4. Transport integration belongs outside the runtime, but its delivery intent must still participate in finalization.
5. A tool catalog is a security boundary only when the host validates it and the deployment respects what the host cannot isolate.
6. Tests and explicit omissions are stronger project documentation than a long feature list.

agentd remains experimental. It is not a general workflow engine, an HA agent platform, or a sandbox for untrusted code. It is a public attempt to make one personal-agent turn durable, scoped, inspectable, and deliverable—and to state precisely where those guarantees stop.

Source, release notes, and the reproducible demo are available in the minifish-org/agentd repository under Apache-2.0.

The most useful lesson from the project is not a feature. It is how quickly storage complexity expands.

What it explores

The codebase touches several storage-system ideas:

• transactional KV
• page-based B+tree structures
• secondary indexes
• WAL encoding and replay
• embedded compute paths
• quorum commit experiments
• RPC schema generation
• redb and sled comparisons
• performance baselines

Each of these is reasonable on its own. Together, they create a system that needs strong boundaries to stay understandable.

Why this is hard

Storage systems are full of cross-cutting constraints:

• write durability affects read visibility
• page layout affects compaction and recovery
• transactions affect indexing
• indexes affect write amplification
• replication affects latency and failure handling
• observability affects whether any benchmark can be trusted

It is easy to add one more subsystem and hard to know when the conceptual model has become too large.

Documents as part of the project

One good part of scale-kv is that it includes design notes: API drafts, capacity models, fault drills, observability notes, and performance snapshots.

For this kind of experiment, documents are not decoration. They are the only way to keep track of what the system is supposed to prove.

What I learned

The project reinforced a simple rule: decide what a storage experiment is testing before building the next layer.

If the goal is page structure, do not also invent distributed commit. If the goal is quorum behavior, use the simplest storage backend that can support the test. If the goal is API design, avoid mixing it with WAL recovery work.

That does not mean the project failed. It means the project taught the lesson it was supposed to teach: complexity is the main resource being spent.

Current status

scale-kv is private and experimental. It may remain a lab rather than a product, which is a valid outcome for a systems project.

OceanBase targets high availability and scalability in a shared-nothing cluster. The core engineering challenge is to make four critical subsystems work together with predictable latency and correctness:

• Write transactions must be durable, replicated, and efficiently committed.
• Tablet replay must recover state quickly and safely.
• SQL parse to execute must optimize well while respecting multi-tenant constraints.
• Unit placement must map tenants to physical resources without fragmentation.

This article focuses on motivation, design, implementation highlights, and tradeoffs, using concrete code entry points from the OceanBase codebase.

System Architecture

OceanBase adopts a shared-nothing architecture where each node is equal and runs its own SQL engine, storage engine, and transaction engine. Understanding the overall architecture is essential before diving into implementation details.

Cluster, Zone, and Node Organization

graph TB
    subgraph Cluster["OceanBase Cluster"]
        subgraph Z1["Zone 1"]
            N1["OBServer Node 1"]
        end
        subgraph Z2["Zone 2"]
            N2["OBServer Node 2"]
        end
        subgraph Z3["Zone 3"]
            N3["OBServer Node 3"]
        end
    end
    
    subgraph Proxy["obproxy Layer"]
        P1["obproxy 1"]
        P2["obproxy 2"]
    end
    
    Client["Client Applications"] --> Proxy
    Proxy --> N1
    Proxy --> N2
    Proxy --> N3

Key Concepts:

• Cluster: A collection of nodes working together
• Zone: Logical availability zones for high availability and disaster recovery
• OBServer: Service process on each node handling SQL, storage, and transactions
• obproxy: Stateless proxy layer routing SQL requests to appropriate OBServer nodes

Data Organization: Partition, Tablet, and Log Stream

graph TB
    subgraph Table["Table"]
        P1["Partition 1"]
        P2["Partition 2"]
        P3["Partition 3"]
    end
    
    subgraph LS1["Log Stream 1"]
        T1["Tablet 1"]
        T2["Tablet 2"]
    end
    
    subgraph LS2["Log Stream 2"]
        T3["Tablet 3"]
    end
    
    subgraph LS3["Log Stream 3"]
        T4["Tablet 4"]
    end
    
    P1 --> T1
    P1 --> T2
    P2 --> T3
    P3 --> T4
    
    T1 --> LS1
    T2 --> LS1
    T3 --> LS2
    T4 --> LS3

Key Concepts:

• Partition: Logical shard of a table (hash, range, list partitioning)
• Tablet: Physical storage object storing ordered data records for a partition
• Log Stream (LS): Replication unit using Multi-Paxos for data consistency
• Replication: Each tablet has multiple replicas across zones, with one leader accepting writes. Log streams replicate data via Multi-Paxos protocol across different zones.

Multi-Tenant Resource Model

graph TB
    subgraph Tenant["Tenant"]
        T1["Tenant 1 MySQL Mode"]
        T2["Tenant 2 Oracle Mode"]
        T3["System Tenant"]
    end
    
    subgraph Pool["Resource Pool"]
        RP1["Pool 1"]
        RP2["Pool 2"]
        RP3["Pool 3"]
    end
    
    subgraph Unit["Resource Unit"]
        U1["Unit 1 CPU Memory Disk"]
        U2["Unit 2 CPU Memory Disk"]
        U3["Unit 3 CPU Memory Disk"]
    end
    
    subgraph Server["Physical Server"]
        S1["Server 1"]
        S2["Server 2"]
        S3["Server 3"]
    end
    
    T1 --> RP1
    T2 --> RP2
    T3 --> RP3
    
    RP1 --> U1
    RP2 --> U2
    RP3 --> U3
    
    U1 --> S1
    U2 --> S2
    U3 --> S3

Key Concepts:

• Tenant: Isolated database instance (MySQL or Oracle compatibility)
• Resource Pool: Groups resource units for a tenant across zones
• Resource Unit: Virtual container with CPU, memory, and disk resources
• Unit Placement: Rootserver schedules units to physical servers based on resource constraints

Layered Architecture

graph TB
    subgraph App["Application Layer"]
        Client["Client Applications"]
    end
    
    subgraph Proxy["Proxy Layer"]
        ODP["obproxy Router"]
    end
    
    subgraph OBServer["OBServer Layer"]
        subgraph Node["OBServer Node"]
            SQL["SQL Engine"]
            TX["Transaction Engine"]
            ST["Storage Engine"]
        end
    end
    
    subgraph Storage["Storage Layer"]
        subgraph LS["Log Stream"]
            Tablet["Tablet"]
            Memtable["Memtable"]
            SSTable["SSTable"]
        end
        Palf["Paxos Log Service"]
    end
    
    subgraph Resource["Resource Layer"]
        Tenant["Tenant"]
        Unit["Resource Unit"]
        Pool["Resource Pool"]
        RS["Rootserver"]
    end
    
    Client --> ODP
    ODP --> SQL
    SQL --> TX
    TX --> ST
    ST --> LS
    LS --> Palf
    Tenant --> Unit
    Unit --> Pool
    Pool --> RS
    RS --> Node

Key Concepts:

• SQL Engine: Parses, optimizes, and executes SQL statements
• Transaction Engine: Manages transaction lifecycle, commit protocols, and consistency
• Storage Engine: Handles data organization, memtables, and SSTables
• Log Service: Provides Paxos-based replication and durability
• Rootserver: Manages cluster metadata, resource scheduling, and placement

Design Overview

At a high level, each node runs a full SQL engine, storage engine, and transaction engine. Data is organized into tablets, which belong to log streams. Log streams replicate changes using Paxos-based log service. Tenants slice resources via unit configurations and pools, while rootserver components place those units on servers.

The following sections walk through each path with the relevant implementation anchors.

Architecture Diagrams

Transaction Write Path

flowchart LR
  subgraph A["Transaction Write Path"]
    C["Client"] --> S["SQL Engine"]
    S --> T["Transaction Context"]
    T --> M["Memtable Write"]
    M --> R["Redo Buffer"]
    R --> L["Log Service"]
    L --> P["Replicated Log"]
    P --> K["Commit Result"]
  end

Tablet Replay Path

flowchart LR
  subgraph B["Tablet Replay Path"]
    L["Log Service"] --> RS["Replay Service"]
    RS --> E["Tablet Replay Executor"]
    E --> LS["Log Stream"]
    LS --> TB["Tablet"]
    TB --> CK["Replay Checks"]
    CK --> AP["Apply Operation"]
    AP --> ST["Updated Tablet State"]
  end

SQL Compile and Execute

flowchart LR
  subgraph C["SQL Compile and Execute"]
    Q["SQL Text"] --> P["Parser"]
    P --> R["Resolver"]
    R --> W["Rewriter"]
    W --> O["Optimizer"]
    O --> LP["Logical Plan"]
    LP --> CG["Code Generator"]
    CG --> PP["Physical Plan"]
    PP --> EX["Executor"]
  end

Unit Placement

flowchart LR
  subgraph D["Unit Placement"]
    UC["Unit Config"] --> RP["Resource Pool"]
    RP --> PS["Placement Strategy"]
    PS --> CS["Candidate Servers"]
    CS --> CH["Chosen Server"]
    CH --> UN["Unit Instance"]
  end

Write Transaction: From Memtable to Replicated Log

Motivation

A write transaction must be both fast and durable. OceanBase uses memtables for in-memory writes, and a log stream for redo replication. The design must allow low-latency commit while supporting parallel redo submission and multi-participant (2PC) transactions.

Design Sketch

• Each transaction is represented by a per-LS context (ObPartTransCtx).
• Redo is flushed based on pressure or explicit triggers.
• Commit chooses one-phase or two-phase based on participants.
• Logs are submitted via a log adapter backed by logservice.

Implementation Highlights

• Transaction context lifecycle and commit logic are in src/storage/tx/ob_trans_part_ctx.cpp.
• Redo submission is driven by submit_redo_after_write, which switches between serial and parallel logging based on thresholds.
• Commit decides between one-phase and two-phase commit depending on participant count.
• The log writer (ObTxLSLogWriter) submits serialized logs via ObITxLogAdapter, which is wired to logservice (ObLogHandler).

Tradeoffs

• Serial vs parallel redo: Serial logging is simpler and cheaper for small transactions, but parallel logging reduces tail latency for large transactions at the cost of more coordination.
• 1PC vs 2PC: 1PC is fast for single-participant transactions; 2PC is required for distributed consistency but increases coordination overhead.
• In-memory batching vs durability: Larger batching improves throughput but can delay durability and increase replay time.

Tablet Replay: Reconstructing State Safely

Motivation

Recovery needs to be deterministic and safe: the system must replay logs to reconstruct tablet state without violating invariants or applying obsolete data.

Design Sketch

• Logservice schedules replay tasks per log stream.
• Tablet replay executor fetches the LS, locates the tablet, validates replay conditions, and applies the log.
• Specialized replay executors handle different log types (e.g., schema updates, split operations).

Implementation Highlights

• Replay orchestration lives in src/logservice/replayservice/ob_log_replay_service.cpp.
• Tablet replay logic is in src/logservice/replayservice/ob_tablet_replay_executor.cpp.
• Specific tablet operations are applied in dedicated executors, such as ObTabletServiceClogReplayExecutor in src/storage/tablet/ob_tablet_service_clog_replay_executor.cpp.

Tradeoffs

• Strictness vs throughput: Replay barriers enforce ordering for correctness but can reduce parallelism.
• Tablet existence checks: Allowing missing tablets can speed recovery but requires careful validation to avoid partial state.
• MDS synchronization: Metadata state updates improve correctness but add contention via locks.

SQL Parse to Execute: Compile Pipeline for Performance

Motivation

OceanBase supports MySQL and Oracle compatibility with rich SQL features. The compile pipeline must be fast, cache-friendly, and yield efficient execution plans.

Design Sketch

• SQL text enters the engine via ObSql::stmt_query.
• Parsing produces a parse tree.
• Resolution turns the parse tree into a typed statement tree.
• Rewrite and optimization generate a logical plan.
• Code generation produces a physical plan and execution context.

Implementation Highlights

• Entry and query handling: src/sql/ob_sql.cpp (stmt_query, handle_text_query).
• Resolver: ObResolver in src/sql/resolver/ob_resolver.h.
• Transform and optimize: ObSql::transform_stmt and ObSql::optimize_stmt in src/sql/ob_sql.cpp.
• Code generation: ObSql::code_generate in src/sql/ob_sql.cpp.

Tradeoffs

• Plan cache vs compile accuracy: Plan caching reduces latency but may reuse suboptimal plans under changing data distributions.
• Rewrite aggressiveness: More transformations can yield better plans but increase compile cost.
• JIT and rich formats: Faster execution for some workloads, but added complexity and memory pressure.

Unit Placement: Scheduling Tenant Resources

Motivation

Multi-tenancy requires predictable isolation and efficient resource utilization. Unit placement must respect CPU, memory, and disk constraints while minimizing fragmentation.

Design Sketch

• Unit config defines resource demands.
• Resource pool groups units by tenant and zone.
• Placement strategy scores candidate servers to pick a host for each unit.

Implementation Highlights

• Resource types and pools: src/share/unit/ob_unit_config.h, src/share/unit/ob_resource_pool.h, src/share/unit/ob_unit_info.h.
• Placement policy: src/rootserver/ob_unit_placement_strategy.cpp uses a weighted dot-product of remaining resources to choose a server.
• Orchestration: src/rootserver/ob_unit_manager.cpp handles creation, alteration, and migration of units and pools.

Tradeoffs

• Greedy placement vs global optimality: Dot-product scoring is efficient and practical but may not be globally optimal.
• Capacity normalization: Assuming comparable server capacities simplifies scoring but may bias placement in heterogeneous clusters.
• Latency vs stability: Fast placement decisions can lead to more churn; conservative placement improves stability but can reduce utilization.

Closing Thoughts

These four paths demonstrate how OceanBase balances correctness, performance, and operability. The code structure follows clear separation of responsibilities: transaction logic is in storage/tx, replication and replay are in logservice, SQL compilation is in sql, and scheduling is in rootserver and share/unit. The tradeoffs are explicit and largely encoded in thresholds and policies, which makes tuning feasible without invasive rewrites.

If you are extending OceanBase, start with the entry points highlighted above and follow the call chains into the relevant subsystem. It is the fastest way to build a mental model grounded in the actual implementation.

The CAP theorem has become one of the hottest theorems in recent years; when discussing distributed systems, CAP is inevitably mentioned. However, I feel that I haven’t thoroughly understood it, so I wanted to write a blog post to record my understanding. I will update the content as I gain new insights.

Understanding

I read the first part of this paper.

The CAP theorem [Bre12] says that you can only have two of the three desirable properties of:

• C: Consistency, which we can think of as serializability for this discussion;
• A: 100% availability, for both reads and updates;
• P: tolerance to network partitions.

This leads to three kinds of systems: CA, CP and AP, based on what letter you leave out.

Let me share my understanding, using a network composed of three machines (x, y, and z) as an example:

• C (Consistency): The three machines appear as one. Operations of addition, deletion, modification, and query on any one machine should always be consistent. That is, if you read data from x and then read from y, the results are the same. If you write data to x and then read from y, you should also read the newly written data. Wikipedia also specifically mentions that it’s acceptable to read the data just written to x from y after a short period of time (eventual consistency).

• A (Availability): The three machines, as a whole, must always be readable and writable; even if some parts fail, it must be readable and writable.

• P (Partition Tolerance): If the network between x, y, and z is broken, any machine cannot or refuses to provide services; it is neither readable nor writable.

Here, C is the easiest to understand. The concepts of A and P are somewhat vague and easy to confuse.

Now let’s discuss the three combinations:

If the network between x, y, and z is disconnected:

• CA: Ensure data consistency (C). When x writes data, y can read it (C). Allow the system to continue providing services—even if only x and y are operational—ensuring it is readable and writable (A). We can only tolerate z not providing service; it cannot read or write, nor return incorrect data (losing P).

• CP: Ensure data consistency (C). Allow all three machines to provide services (even if only for reads) (P). We can only tolerate that x, y, and z cannot write (losing A).

• AP: Allow all three machines to write (A). Allow all three machines to provide services (reads count) (P). We can only tolerate that the data written by x and y doesn’t reach z; z will return data inconsistent with x and y (losing C).

CA is exemplified by Paxos/Raft, which are majority protocols that sacrifice P; minority nodes remain completely silent. CP represents a read-only system; if a system is read-only, whether there’s a network partition doesn’t really matter—the tolerance to network partitions is infinitely large. AP is suitable for systems that only append and do not update—only inserts, no deletes or updates. Finally, by merging the results together, it can still function.