Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Miscellaneous API Endpoints & Daemon Root

Crate: kinetic-daemon Stage: 9 Reading time: 20 minutes Depends on: 18_api_server.md, 17_auth_middleware.md

What Is This?

This document covers the kinetic-daemon library root structure. It also covers a collection of miscellaneous but critical HTTP API endpoints. While earlier documents covered the core API lifecycle (routing, middleware, and domain resolution), the daemon API is broader. It also exposes several administrative endpoints. It exposes deep integration endpoints. These endpoints include:

  • Configuration management
  • Node health observation
  • Cross-network bridging configuration
  • Network time delivery
  • Real-time P2P gossip streaming

Specifically, we are examining four distinct API modules and the library root:

1. The Config Module (config.rs) This module contains endpoints for managing the node’s configuration state. It allows checking operational health. It allows retrieving the active network status. It allows querying the names owned by the node’s identity.

2. The Gossip Module (gossip.rs) This module contains endpoints for broadcasting arbitrary JSON payloads to the network. It provides a mechanism to subscribe to live topics. It uses Server-Sent Events (SSE) for this streaming.

3. The Atlas Module (atlas.rs) This is a webhook endpoint. It is designed specifically for the kinetic-atlas bridge. It synchronizes traditional ICANN TLDs and Web3 TLDs dynamically.

4. The Time Module (time.rs) This is an endpoint that serves verified Kinetic network time. The time is derived cryptographically from Drand. It avoids relying on local NTP clocks.

5. The Library Root (lib.rs) This is the structural entry point of the crate. It binds the HTTP API together. It binds the local proxy server together. It binds CA management and background services together into a single cohesive library.

These endpoints form the integration layer of the daemon. They are what allow external tools (like the Kinetic CLI) to function. They allow local frontend applications to interact with the node. They allow external bridging services to integrate with the underlying Kademlia network. They allow querying local storage databases. All of this is possible without having to speak raw P2P protocols or manage raw socket connections.

Why Kinetic Needs This

A decentralized node cannot function purely as an isolated black box. It requires observation. It requires dynamic configuration. It requires extensive integration points for external tooling. Each of these miscellaneous modules solves a specific operational problem for the Kinetic network architecture.

The Necessity of Configuration and Observation (config.rs) Saif, as an operator, you need visibility into your node. You need to know if your node is actually connected to the DHT. You need to know how many peers it currently sees. You need to know whether its underlying storage engine is corrupted or responding normally. The health and network status endpoints provide this exact telemetry. Furthermore, the Kinetic CLI needs a way to query what names the node owns. It needs to dynamically change its network mode. For example, switching from a client to a bootstrap node. Providing HTTP endpoints for these actions ensures that the CLI does not need to directly mutate the YAML configuration files. It ensures the CLI does not need to lock the sled databases. Directly touching files or databases from the CLI would cause contention and race conditions. The API acts as the safe, synchronized gatekeeper for node state.

The Necessity of Real-time Web Integration (gossip.rs) Kinetic relies on Gossipsub for real-time publish/subscribe message propagation. However, standard web browsers cannot natively speak libp2p Gossipsub. Local frontend applications cannot easily compile libp2p stacks. Basic external scripts cannot speak the complex libp2p Gossipsub protocol over TCP/QUIC. To bridge this significant gap, the daemon exposes an HTTP publish endpoint. It also exposes an SSE (Server-Sent Events) subscribe endpoint. This allows a standard, unmodified web application to stream live Kinetic gossip events. It does this using nothing but a standard unidirectional HTTP connection. If this abstraction did not exist, building live, reactive applications on Kinetic would be virtually impossible. You would have to compile massive WASM libp2p stacks directly into the browser.

The Necessity of The ICANN Bridge Configuration (atlas.rs) Kinetic intercepts local DNS traffic to resolve .kin names. But it should only intercept .kin and other registered Web3 TLDs. It needs to dynamically know which TLDs it should handle natively. It needs to know which TLDs it should blindly forward to standard ICANN DNS servers (like 1.1.1.1). The kinetic-atlas bridge is a separate, specialized service. It monitors a smart contract (or remote registry) of supported foreign TLDs. It uses this webhook endpoint to push the latest list of TLDs into the daemon’s runtime memory. Without this dynamic synchronization, the daemon would not know how to dynamically adapt its DNS routing. When new decentralized TLDs are registered on the network, the node must adapt. Failing to adapt would lead to broken DNS resolution for users.

The Necessity of Cryptographically Verified Time (time.rs) Decentralized, distributed systems cannot trust local system clocks. If a user’s local clock is drastically wrong (due to a dead CMOS battery or a malicious NTP spoofing attack), everything breaks. Cryptographic operations like verifying signatures will fail. Validating certificates will fail. Checking block freshness will fail. The time.rs endpoint queries the daemon’s synchronized Drand state. It calculates the exact KineticTime. This gives local applications a trusted, cryptographically verifiable source of time. This time does not rely on NTP. It ensures that the local ecosystem remains synchronized with the global network consensus.

The Necessity of A Unified Library Architecture (lib.rs) While the Kinetic daemon is distributed as a massive, long-running binary executable, structurally it is built as a modular library. By exposing api, ca, proxy, and services as public modules in lib.rs, Kinetic is flexible. It allows other Rust crates (like integration test harnesses) to embed the daemon programmatically. It allows simulators to spawn multiple daemons in a single test process. It allows custom GUI wrappers to bundle the daemon. This prevents the daemon from being locked into a standalone executable format.

How It Works

The Library Structure and Modularity

If you examine kinetic-daemon/src/lib.rs, you will notice it simply exposes four top-level modules. These are api, ca, proxy, and services.

-> See: crates/kinetic-daemon/src/lib.rs — Lines 5 to 13

This intentional modularity ensures that the HTTP server (api) does not tightly couple with the TLS certificate generation logic (ca). It does not tightly couple with the P2P networking loops (services). Instead of passing direct references to each other, they communicate safely. They communicate by passing shared Arc state structures (like ApiState). They communicate using message-passing channels across module boundaries. This separation of concerns is critical for preventing spaghetti code in a concurrent environment.

Managing Configuration and Health Checks

The config.rs module handles several distinct operational queries and mutations:

Active Health Checks When a user or load balancer queries the /health endpoint, the daemon actively tests its internal components. It does not just blindly return a static “OK”. It asks the network channel for its status. It attempts to read a known key (DB_PREFIX_LAST_DRAND) from the sled storage engine. If either of these operations times out or returns an error, it reports that specific subsystem as unresponsive in the JSON payload.

-> See: crates/kinetic-daemon/src/api/config.rs — Lines 88 to 111

Retrieving Owned Names The handle_owned_names endpoint reads the DB_PREFIX_OWNED_NAMES key directly from storage. Because storage interactions can fail, it must be robust. It can yield corrupted byte arrays. Therefore, it uses serde_json::from_slice. Crucially, if the bytes are corrupted, it logs a KIN-IMPL-003 tracing error. This alerts you to the corruption. It then gracefully falls back to returning an empty list rather than panicking the entire API server.

-> See: crates/kinetic-daemon/src/api/config.rs — Lines 33 to 43

Configuration Mutation When a CLI user updates the configuration, the handle_set_config endpoint activates. It loads the current KineticConfig from the disk. It mutates the network mode field. It saves it back to the disk. Notice that it does not attempt to restart the node process itself. It simply returns a JSON message instructing the user to restart the daemon. This maintains absolute simplicity in the API layer and avoids dangerous self-termination routines.

The Server-Sent Events (SSE) Gossip Stream

The handle_gossip_subscribe function is arguably one of the most complex HTTP handlers in the entire API. It utilizes Server-Sent Events (SSE) combined with an asynchronous stream generator macro. When a client hits this endpoint, the Axum framework keeps the HTTP connection open indefinitely. This overrides the standard request-response lifecycle.

Inside the handler, the daemon subscribes to its internal tokio::sync::broadcast channel. This is done via state.gossip_tx.subscribe(). This channel acts as the firehose receiving all Gossipsub messages from the Kademlia network.

-> See: crates/kinetic-daemon/src/api/gossip.rs — Lines 20 to 42

The handler uses the async_stream::stream! macro to create a Rust generator. It enters an infinite loop. It asynchronously awaits messages from the broadcast channel. If the message’s topic matches the requested topic in the URL path, it processes it. It attempts to parse the payload as a UTF-8 string. It then yields it as an SSE Event.

Critically, it handles the RecvError::Lagged(skipped) error variant. Broadcast channels in tokio are bounded. They will forcefully drop messages if the receiver is too slow to process them. If a web client has a slow connection, the daemon will drop their missed messages. It logs a warning indicating how many messages were skipped. It then gracefully continues streaming new messages. This mechanism prevents a single slow HTTP client from causing the entire daemon to run out of memory.

The Atlas Webhook Synchronization

The handle_atlas_sync endpoint receives a JSON AtlasSyncPayload. This contains a raw list of TLD strings from the Atlas bridge. It iterates over these TLDs and sanitizes them. It trims whitespace. It conditionally removes leading dots. It normalizes them to lowercase. It then inserts them into a fresh, allocated HashSet.

-> See: crates/kinetic-daemon/src/api/atlas.rs — Lines 28 to 40

Once this clean set is fully built in memory, the handler attempts to acquire a write lock on state.atlas_tlds. This is an Arc<RwLock<HashSet<String>>>. It replaces the entire old set with the newly generated set in one atomic assignment operation. This pattern is important. The proxy server is constantly acquiring read locks on atlas_tlds for every single DNS request it intercepts. By building the new set outside of the lock, we optimize performance. By only acquiring the write lock to perform the instantaneous swap, the lock is held for mere nanoseconds. This ensures that DNS resolution is never blocked during an Atlas synchronization event.

Cryptographic Time Delivery

The handle_get_time endpoint initializes a fresh DrandClient instance. It passes in a clone of the daemon’s storage layer. It fetches the latest verified drand round (referred to as a kyn). It then utilizes the KineticTime::from_kyn function. It passes in the globally defined KINETIC_GENESIS_DRAND_KYN constant. This calculates the exact network time mathematically.

-> See: crates/kinetic-daemon/src/api/time.rs — Lines 14 to 28

If the node is offline and cannot fetch a kyn from the network or storage, it must fail safely. The daemon chooses to return an INTERNAL_SERVER_ERROR (500). It does this rather than falling back to an unverified mathematical time estimation. As noted in the inline comments, returning a hard error ensures safety. It ensures that consumers (like the CLI or local apps) are aware that the node is unsynchronized. It ensures they know it cannot provide trustworthy time data.

Key Pieces

handle_config / handle_set_config

What it does:

  • Retrieves or mutates the node’s persistent KineticConfig.
  • Specifically targets properties like the network mode.

Where it lives:

  • kinetic-daemon/src/api/config.rs

Why it matters:

  • It provides the CLI with the critical ability to observe the daemon’s underlying configuration state.
  • It provides the ability to modify this state.
  • It does this without needing raw file system access.
  • It enforces API-level JWT access controls.
  • It prevents data races on the YAML files.

handle_health

What it does:

  • Actively verifies the Kademlia network loop is responsive.
  • Actively verifies the sled storage engine is responsive.
  • Returns a comprehensive JSON health report.

Where it lives:

  • kinetic-daemon/src/api/config.rs

Why it matters:

  • This serves as the primary liveness probe.
  • If the node runs in a Docker container, this dictates restart policies.
  • If the node runs in a Kubernetes pod, this endpoint dictates whether the orchestrator should forcefully restart the node process.

handle_gossip_subscribe

What it does:

  • Upgrades an HTTP connection into an SSE stream.
  • Continuously yields live Gossipsub messages for a specific topic.
  • Handles slow consumers gracefully by dropping lagged messages.

Where it lives:

  • kinetic-daemon/src/api/gossip.rs

Why it matters:

  • It acts as the vital bridge between complex P2P network chatter and standard web frontends.
  • It relies on tokio_stream and yield generators.
  • It pushes events continuously over a single, multiplexed HTTP connection.

AtlasSyncPayload and handle_atlas_sync

What it does:

  • Defines the JSON schema for incoming TLD updates.
  • Processes the webhook payload pushed from the external atlas bridge service.
  • Atomically updates the in-memory HashSet used by the DNS resolver.

Where it lives:

  • kinetic-daemon/src/api/atlas.rs

Why it matters:

  • It is the singular mechanism by which the daemon dynamically learns which domain spaces it is responsible for intercepting.
  • The atomic RwLock swap pattern guarantees that DNS resolution remains blazingly fast.
  • It ensures DNS is never blocked during an update.

handle_get_time

What it does:

  • Uses the synchronized Drand state from storage.
  • Cryptographically calculates the KineticTime.
  • Returns the time as a JSON response.

Where it lives:

  • kinetic-daemon/src/api/time.rs

Why it matters:

  • It provides a trustless, cryptographically verified time source for local applications.
  • It allows the Kinetic ecosystem to bypass unreliable local system clocks.
  • It prevents attacks based on spoofed NTP data.

How This Connects to the Rest of Kinetic

Storage Layer Integration

  • Both the config.rs and time.rs endpoints rely on the storage layer.
  • They read owned names and the latest Drand kyn.
  • CROSS-CRATE: Sled Storage — defined and explained in docs/learn/storage/01_overview.md

Network Gossip Facade

  • The gossip.rs module acts as an HTTP facade directly over the Kademlia network’s libp2p publish/subscribe capabilities.
  • It bridges libp2p logic directly to HTTP.
  • CROSS-CRATE: Gossipsub — defined and explained in docs/learn/network/04_gossip.md

Drand Synchronization

  • The time endpoint relies directly on the Drand verification client.
  • It fetches cryptographic proofs of time from this client.
  • CROSS-CRATE: DrandClient — defined and explained in docs/learn/core/04_drand.md

DNS and Proxy Interaction

  • The atlas.rs webhook directly mutates the shared, in-memory state.
  • This state is exactly what the PAC server and DNS resolver use.
  • They use it to deterministically decide if a network request should be intercepted or forwarded to the open internet.

Quick Reference

Config Endpoints:

  • /config (GET/POST): Retrieves or sets node configuration.
  • /health: Checks internal daemon health (storage and network).
  • /status: Gets network metrics.
  • /owned-names: Retrieves domains owned by this node.
  • /peer-id: Returns the node’s Kademlia identity.
  • These generally require the Admin or Publish roles where appropriate, authenticated via JWT.

Gossip Endpoints:

  • /gossip/subscribe/:topic (GET): Establishes an SSE stream.
  • /gossip/publish/:topic (POST): Broadcasts JSON to the Kademlia network.

Atlas Endpoint:

  • /atlas/sync (POST): Synchronizes ICANN TLDs.
  • requires the Atlas or Admin JWT role to prevent unauthorized routing table poisoning.

Time Endpoint:

  • /time (GET): Returns verified network time.
  • Requires a verified, up-to-date Drand state to be present in local storage.
  • Otherwise returns a 500 error.

State Injection:

  • All of these Axum handlers rely on axum::extract::State<ApiState>.
  • They use it to access safely shared resources like broadcast network channels and database connections.

Concurrency Protection:

  • State updates (like the Atlas TLD synchronization) use RwLock.
  • They prioritize fast, concurrent reads.
  • They safely allow infrequent, atomic writes.

Open Questions / Things to Revisit

Configuration Hot-Reloading

  • Currently, /config updates the configuration YAML file on disk.
  • However, it just returns a static message telling the user to manually restart the daemon.
  • In the future, implementing true hot-reloading of the configuration without restarting the process would greatly improve node uptime and user experience.

Gossip Lag Handling

  • In gossip.rs, if an HTTP client lags due to poor network conditions, we log a warning.
  • We then irrevocably skip messages.
  • There is currently no mechanism to retroactively fetch missed messages from the stream.
  • This means SSE clients that experience brief network drops will silently lose data.
  • We may need to implement an ephemeral buffer or cache of recent gossip events to allow for seamless, lossless reconnections.

Time Endpoint Fallback Behavior

  • The time.rs endpoint currently returns a hard 500 error if it cannot fetch the latest Drand kyn.
  • While this accurately reflects that the node is unsynchronized, it might cause brittle behavior.
  • Local frontend apps simply need a roughly accurate timestamp when the node is temporarily partitioned from the network.
  • We should seriously consider returning an estimated mathematical time alongside an is_synced: false boolean flag instead of failing the request entirely.