Building Relay

Phần 4 · Chương 4.2

Kho dữ liệu chưa từng lắng nghe

Bạn sẽ tạo ra: Một ClickHouse mà truy vấn chạm tới được, một schema phân tích với sổ ghi của riêng nó, và cùng một câu hỏi được trả lời trong 13,22 ms so với 585,9 ms của Postgres — cùng một ngày trong chín mươi mốt ngày không bao giờ đối soát được · khoảng 60 phút, bao gồm bài tập

Tài liệu gốc: SRS — Đặc tả yêu cầu phần mềm · SAD — Tài liệu kiến trúc phần mềm (tiếng Anh)

Bản dịch đang được chuẩn bị. Phần diễn giải của chương này chưa được dịch sang tiếng Việt. Các khối mã bên dưới là bản gốc tiếng Anh và giống hệt bản tiếng Anh của chương — bạn có thể gõ theo chúng ngay bây giờ. Bản dịch đầy đủ sẽ thay thế trang này.

The check that could not fail

curl -s "http://localhost:${RELAY_CLICKHOUSE_HTTP_PORT:-8123}/ping"
curl -s "http://localhost:${RELAY_CLICKHOUSE_HTTP_PORT:-8123}/?query=SELECT+1"
Code: 194. DB::Exception: default: Authentication failed: password is incorrect,
or there is no user with such name. ... (REQUIRED_PASSWORD)
docker compose exec clickhouse cat /etc/clickhouse-server/users.d/default-user.xml
<clickhouse>
  <users>
    <default>
      <!-- User default is available only locally -->
      <networks>
        <ip>::1</ip>
        <ip>127.0.0.1</ip>
      </networks>
    </default>
  </users>
</clickhouse>
compose.yaml
@@ -61,16 +61,35 @@
     image: clickhouse/clickhouse-server:25.3
     # The analytical store rides its own path (CON-01): single node in v1,
     # schema already cluster-shaped (ADR-08).
     ports:
       - "${RELAY_CLICKHOUSE_HTTP_PORT:-8123}:8123"
       - "${RELAY_CLICKHOUSE_NATIVE_PORT:-9000}:9000"
+    environment:
+      # A NEW USER. This does not lift `default`'s restriction and is not meant to:
+      # the image ships `users.d/default-user.xml` limiting `default` to ::1 and
+      # 127.0.0.1, that file belongs to the image, and anyone "fixing" it is editing
+      # somebody else's file. `relay` answers from the host; `default` still does not.
+      CLICKHOUSE_USER: relay
+      CLICKHOUSE_PASSWORD: relay
+      # CREATES THE DATABASE. It does NOT make it the session's: `SELECT
+      # currentDatabase()` over HTTP as `relay` still answers `default`, so an
+      # unqualified CREATE TABLE builds the schema in `default` while this one sits
+      # empty. Every statement in `analytics/` names `relay_analytics` for that reason.
+      CLICKHOUSE_DB: relay_analytics
     volumes:
       - clickhouse-data:/var/lib/clickhouse
     healthcheck:
-      test: ["CMD", "wget", "-q", "-O", "-", "http://localhost:8123/ping"]
+      # RUNS A QUERY, because the old check ran `/ping` — which neither authenticates
+      # nor is network-restricted. It was green from chapter 1.2 to this one while every
+      # query from outside the container was refused. A check that cannot fail for the
+      # reason you care about is not a check.
+      #
+      # `clickhouse-client` is in the image; `curl` is not.
+      test: ["CMD", "clickhouse-client", "--user", "relay", "--password", "relay",
+             "--query", "SELECT 1"]
       interval: 5s
       timeout: 3s
       retries: 5
       start_period: 15s
 
   mailpit:
clickhouse-1 | ClickHouse Database directory appears to contain a database;
               Skipping initialization

The published DDL does not apply

Received exception from server (version 25.3.14):
Code: 450. DB::Exception: TTL expression result column should have DateTime or
Date type, but has DateTime64(3, 'UTC'). (BAD_TTL_EXPRESSION)

Six divergences, and five of them are the same argument

analytics/0000_message_events.sql
-- SAD 6.2's raw event table, with every divergence from the published DDL commented.
-- The document is amended to match (docs/05-sad.md, and the chapter says so).
CREATE TABLE IF NOT EXISTS relay_analytics.message_events (
    environment_id  UUID,
    channel_id      UUID,
    -- DIVERGENCE 1: SAD publishes UUID. `messages.user_id` is nullable -- a message whose
    -- author was deleted has none -- and a NULL inserted into a non-nullable UUID becomes
    -- the ZERO UUID without failing, inventing one active user per environment.
    -- Nullable makes uniqExact ignore it, exactly as count(DISTINCT user_id) does.
    user_id         Nullable(UUID),
    ts              DateTime64(3, 'UTC'),
    event           LowCardinality(String),   -- created|edited|deleted
    -- DIVERGENCE 2 and 3: both are Nullable because the value is sometimes NOT KNOWN, and
    -- 0 is a different claim. lengthUTF8(NULL) inserts 0 into a non-nullable column, and a
    -- text_length of 0 says a zero-length message was sent. A tombstone preserves no text
    -- and an unedited attachment-only message has none either.
    text_length     Nullable(UInt32),
    attachment_count Nullable(UInt8),
    -- DIVERGENCE 4: SAD publishes UInt32, and NOTHING PRODUCES THIS COLUMN until
    -- FR-ANL-10's chapter (FR-006a). Non-nullable, every row would read 0 ms -- a
    -- measured claim about a delivery nobody timed. An absent producer writes NULL.
    delivery_latency_ms Nullable(UInt32)
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(ts)                     -- DR-07
ORDER BY (environment_id, ts)                 -- tenant-scoped range scans
-- DIVERGENCE 5: SAD publishes `TTL ts + INTERVAL 90 DAY`, which this server refuses:
--   Code: 450. TTL expression result column should have DateTime or Date type,
--   but has DateTime64(3, 'UTC'). (BAD_TTL_EXPRESSION)
TTL toDateTime(ts) + INTERVAL 90 DAY          -- DR-09

A second store needs a ledger of its own

$ node analytics/apply.mjs
database relay_analytics ready
applied 3: 0000_message_events.sql, 0001_daily_usage.sql, 0002_schema_applied.sql
skipped nothing

$ node analytics/apply.mjs
database relay_analytics ready
applied nothing
skipped 3: 0000_message_events.sql, 0001_daily_usage.sql, 0002_schema_applied.sql
apply failed: 0001_daily_usage.sql changed after it was applied
              (ledger 385b775b5df80d89, file e676989c630ad7c9). ClickHouse has no
              ALTER path for most of this; add a new statement file instead
ledger names 1 file(s) no longer on disk: 0003_emoji_events.sql — their tables
are still in relay_analytics and nothing here will drop them

Loading it, and what a store reconstructed from state cannot know

The question, asked again

                    store                  rows       best of 3   days
   4.1   Postgres, ordered by id        1,000,000      585.9 ms     91
   4.2   ClickHouse, (environment_id, ts)  1,241,071    13.22 ms     91
MinMax        Parts 4/4   Granules 154/154     skips nothing
Partition     Parts 4/4   Granules 154/154     skips nothing
PrimaryKey    Parts 4/4   Granules 131/154     skips 23

The rollup, and the one day that can never agree

SELECT day, sum(messages), uniqMerge(active_users_state)
  FROM relay_analytics.daily_usage
 WHERE environment_id = ? AND day >= toDate(now() - INTERVAL 90 DAY)
 GROUP BY day
days compared   91
days agreeing   90
days differing   1     2026-06-15: raw 6,220 · rollup 11,161 · difference 4,941
uniqMerge(active_users_state)   5,000
uniqExact(user_id)              5,000
difference                          0      0.0000%
distinct     uniq          error
  60,000     60,000        0%
  65,000     65,000        0%
  70,000     70,359        0.5129%
 500,000    502,646        0.5292%