Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Daemon Runtime & Graceful Shutdown

Crate: kinetic-daemon Stage: 9 Reading time: 60 minutes Depends on: 02_main_1.md, kinetic-network, kinetic-core

What Is This?

This file documents the second half of the main runtime loop inside the kinetic-daemon crate (specifically src/main.rs, lines 336 to 670). Once the basic storage, configuration, and cryptographic identity are loaded and verified (as covered in the previous document), the daemon must actually come alive. It does this by bringing up a series of concurrent background tasks that collectively run the entire Kinetic network node. Specifically, it handles:

  • Bootstrapping the network governance state from external seed nodes over HTTP.
  • Wiring up the asynchronous communication channels required for inter-task coordination.
  • Spawning the network event loop to dial peers and join the distributed swarm.
  • Spawning all the highly-concurrent sub-services (like the gossip processor, Proof-of-Work miner, local proxy server, and DNS resolver).
  • Registering itself dynamically with the kinetic-pac proxy auto-configuration system via the filesystem.
  • Trapping OS-level signals and waiting for a shutdown command to clean up its local state gracefully without stranding the user’s internet connection. This file acts as the central nervous system of the node. Everything that happens continuously in the background is orchestrated, spawned, and monitored by the code documented in this file. It is the critical bridge between static initialization and a living, breathing network participant.

Why Kinetic Needs This

A decentralized network node is not a simple, single-threaded script that runs from top to bottom and then exits. It is a concurrent, long-running engine that must perform a multitude of complex I/O and compute-bound tasks simultaneously. Consider the workload of a fully operational Kinetic node under normal conditions:

  1. Networking: It must continuously listen for incoming libp2p peer connections over the internet.
  2. Consensus: It must constantly mine VDF (Verifiable Delay Function) proofs to participate in network consensus and earn block rewards.
  3. Web Interception: It must act as a local HTTP proxy server for the user’s web browser, intercepting decentralized traffic and serving local responses on the fly.
  4. Domain Resolution: It must answer raw UDP and TCP DNS queries if it is configured to act as a local domain name resolver.
  5. Administration: It must respond instantly to local API calls originating from the command-line interface (e.g., when a user types kinetic status). If these distinct systems were not split into separate, non-blocking background tasks, the node would freeze instantly.

The Problem with Single-Threaded Execution

Imagine if the daemon used a simple loop architecture. If a slow API request from the CLI took five seconds to process, it would block the network layer from gossiping blocks for those entire five seconds. This would cause the node to fall out of sync with the rest of the network, potentially leading to slashed stakes or rejected blocks. Alternatively, if a complex Proof-of-Work hash was calculated on the main thread, it would starve the proxy server. This means the user’s web browser would time out trying to load a Kinetic website simply because the daemon is too busy doing math to answer the HTTP request.

The Tokio Solution

To solve this, Kinetic uses the tokio async runtime. Tokio ensures that all these moving parts make progress simultaneously, multiplexing them efficiently across the available CPU cores. It uses cooperative multitasking, meaning tasks yield control back to the runtime when they are waiting for I/O (like a network response), allowing other tasks to run in the meantime. This allows a single process to handle thousands of concurrent operations without breaking a sweat.

The Necessity of Graceful Shutdown

Furthermore, a long-running system daemon cannot just be killed abruptly without consequence. When the user hits Ctrl+C in their terminal, or issues a stop command through the systemd service manager, the daemon must shut down cleanly. If the Kinetic daemon exits abruptly without de-registering its local proxy configuration, the user’s operating system will continue to blindly route all web traffic to a proxy port that is no longer listening. This effectively breaks the user’s regular internet connection until they figure out what happened and manually reset their network settings. Graceful shutdown prevents this user experience by running essential cleanup logic before the process is permitted to terminate.

How It Works

The startup sequence follows a precise, deliberate order. Dependencies between services dictate this order—for example, you cannot start the block miner before the network layer is active. Similarly, you cannot start the network layer before the cryptographic keys and governance rules are fully loaded. Let’s walk through the exact steps the daemon takes to boot up, breaking down the mechanics of each section.

1. Bootstrapping Governance State

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/main.rs` — Lines 355 to 440
}

Before the network can validate any rules, process any blocks, or even understand what a valid peer looks like, it needs the current Governance State. The Governance State is a critical data structure that dictates network constants, protocol versions, and critical timestamps like genesis. The daemon first looks for the governance.key file on the local disk inside the base configuration directory (typically ~/.kinetic/). If the file is not found, the daemon realizes it is a brand-new node joining an existing network for the very first time. To get the state, it reaches out to the bootstrap nodes or seed domains specified in its configuration over standard HTTP (port 8000). Why HTTP and not the libp2p swarm? Because to join the libp2p swarm and validate peer identities, you already need the governance state. It creates a chicken-and-egg problem. HTTP provides a simple, out-of-band way to pull the initial ruleset. It creates a reqwest::Client with a tight 5-second timeout, ensuring the boot process doesn’t hang forever if a seed node is offline or unresponsive. When it successfully downloads the state bytes, it deserializes them using bincode. Bincode is a compact binary serialization format used in Rust. It is used instead of JSON because it is significantly faster to parse and maps exactly to the Rust struct’s memory layout, saving precious CPU cycles during boot. However, the node does not blindly trust the download. It performs a strict content validation check. It verifies if the genesis_timestamp_sec in the downloaded state matches the compiled KINETIC_GENESIS_TIME constant defined locally in the core crate.

Warning

This is a critical security measure. Without this check, a malicious seed node (or a man-in-the-middle attacker spoofing DNS) could hand the node a valid, cryptographically signed governance state for a different testnet or a private fork. The node would accept it, join the wrong network, and partition itself from reality.

By checking the hardcoded genesis time, the node guarantees it is connecting to the correct timeline. Once the payload is validated, the state is saved to disk so the node won’t have to download it again on the next boot. Finally, it is loaded into the GLOBAL_GOVERNANCE_STATE mutex so the rest of the application can safely read the current network rules from memory at any time.

2. Wiring Up Asynchronous Communication Channels

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/main.rs` — Lines 441 to 447
}

Tasks in Rust need a safe way to talk to each other without sharing memory directly, which can cause deadlocks or data races. We use channels as asynchronous pipes between these isolated tasks. The daemon creates two vital asynchronous channels before spawning any workers:

  1. incoming_tx / incoming_rx: This is a Multi-Producer Single-Consumer (mpsc) channel with a capacity of 32. This channel is used to route inbound proxy requests. Multiple concurrent network connections can produce requests simultaneously (hence Multi-Producer). However, a single local handler task consumes them sequentially and routes them to the local backend (hence Single-Consumer). If the handler gets overwhelmed, the channel fills up to 32 items, and then producers are forced to wait, preventing memory exhaustion.
  2. gossip_tx / gossip_rx: This is a broadcast channel with a capacity of 100. In a broadcast channel, every subscribed consumer gets a perfect copy of every message. The network layer publishes raw gossip messages (like new blocks, transactions, or drand beacons) to this channel. Any interested worker (like the block processor or the mempool manager) can subscribe to this channel and listen independently. If a consumer is too slow to process messages, it will eventually lag by more than 100 messages, at which point tokio will drop the oldest messages for that specific consumer and return a Lagged error. This guarantees that one slow worker cannot halt the entire network pipeline.

3. Spawning the Network Event Loop

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/main.rs` — Lines 448 to 468
}

The node must connect to the outside world. The NetworkEventLoop is the absolute heart of the libp2p implementation. It is initialized with the local cryptographic key, the database storage layer, and the communication channels we just created. Once initialized, it is immediately spawned into a detached background task using tokio::spawn. From this precise moment forward, the node is officially “online” in the libp2p swarm. It will begin dialing peers, discovering the network topology, responding to ping requests, and participating in the Distributed Hash Table (DHT). The daemon also immediately uses the returned network_client to subscribe to the Quicknet Kyn Gossip topic, ensuring it receives the random beacon numbers necessary for block production.

4. Spawning Sub-Services and Workers

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/main.rs` — Lines 469 to 488
}

With the network active, the daemon fires up the domain-specific workers.

  • PoW Miner Loop: Begins hashing and attempting to mine blocks, provided mining is enabled in the node’s configuration. It requires the network_client to broadcast successful blocks to the rest of the network.
  • Gossip Processor: A dedicated background task that listens to the gossip_rx broadcast channel. As the network layer receives gossip from external peers, this processor decodes the messages, validates their cryptographic signatures, and applies them to the local database state.

5. Starting the Certificate Authority and Proxy Servers

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/main.rs` — Lines 490 to 532
}

Kinetic intercepts decentralized domain traffic (e.g., domains ending in .kyn). To do this over HTTPS without triggering alarming browser warnings, it must act as its own local Certificate Authority (CA) on the user’s machine. The daemon calls ca::load_or_create_root_ca. If it is the very first run, it generates a brand new root certificate and private key. It then wraps a leaf certificate cache in an Arc<tokio::sync::Mutex<LeafCertCache>> so multiple proxy threads can reuse generated certificates for specific domains without recreating them on every single request. Why use an Arc<Mutex>?

  • Arc (Atomic Reference Counted) allows the cache memory to be safely shared across multiple spawned tokio tasks. It keeps track of how many tasks hold a reference, and cleans up the memory only when the count drops to zero.
  • Mutex (Mutual Exclusion) ensures that only one task can write a new certificate to the cache at a time. If two tasks try to create a certificate for example.kyn at the exact same millisecond, the Mutex forces one to wait, preventing data corruption or duplicated effort.
  • It specifically uses tokio::sync::Mutex, not std::sync::Mutex, because tasks need to hold the lock across .await yield points while waiting for the cryptographic signing to complete. It then spawns two major proxy tasks into the tokio runtime:
  1. start_proxy_server: The actual forward proxy server that binds to a local port (e.g., 8080) and handles CONNECT requests directly from the user’s web browser.
  2. handle_incoming_proxy_requests: A reverse proxy handler that takes requests originating from the P2P network and routes them to local backend services securely.

6. Starting the API and Republisher

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/main.rs` — Lines 534 to 557
}

The daemon needs an interface for local administration. The API server (running on port 8000 by default) is started. This is the HTTP endpoint that local CLI tools (like kinetic status or kinetic wallet) connect to when you run commands in your terminal. It allows the user to inspect the state of the node without stopping it. Simultaneously, the Republisher loop is started. Decentralized networks rely on nodes announcing what data they have. The republisher periodically iterates over the data stored in the local database and re-announces it to the DHT, ensuring the wider network knows this node is still actively hosting the content.

7. Registering with Kinetic-PAC

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/main.rs` — Lines 559 to 581
}

The daemon must tell the operating system to send .kyn traffic to its specific proxy server port. It does this through a decoupled, file-based registry system. It resolves the local data directory using dirs::data_local_dir(). This function maps to ~/.local/share on Linux, ~/Library/Application Support on macOS, and %APPDATA% on Windows, preventing brittle hardcoded paths. It creates a JSON file in kinetic_global/proxies/<tld>.json. This tiny JSON file simply contains the proxy IP and proxy port. The kinetic-pac utility (which runs independently as a separate system-level service) constantly watches this directory and builds a dynamic Proxy Auto-Configuration (PAC) script for the operating system. By simply dropping this JSON file into the folder, the daemon dynamically registers itself. When the daemon shuts down, it will remove this file, unregistering itself instantly without needing direct IPC (Inter-Process Communication) with kinetic-pac.

8. The DNS Server (Optional)

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/main.rs` — Lines 583 to 618
}

If standard DNS resolution is needed for legacy compatibility. If enable_dns is true in the user’s configuration, the daemon spins up a UDP and TCP listener using the external hickory_server crate on port 53 (or a custom port if overridden). It passes all incoming DNS queries to a custom KineticDnsHandler. This allows the daemon to resolve traditional DNS A/AAAA records for Atlas domains, acting as a transparent bridge between the decentralized network and legacy networking stacks.

9. Graceful Shutdown

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/main.rs` — Lines 620 to 631
}

The orchestration of a clean, safe exit. Everything comes together in a tokio::select! block at the very end of async_main. The tokio::select! macro is a powerful Rust concept that waits for multiple asynchronous futures simultaneously, but only proceeds when the first one completes. The other pending futures are instantly cancelled. It is waiting for two distinct events:

  1. api_future: The API server crashing and returning an error.
  2. shutdown_signal(): A function that listens for a Ctrl+C (SIGINT) or SIGTERM signal from the operating system. If the user presses Ctrl+C, the shutdown_signal() future resolves first. The block immediately executes its cleanup code.

Important

The most critical piece of cleanup is deleting the proxy JSON file from the global proxies directory. If the daemon failed to delete this file before exiting, the OS would keep trying to route web traffic to a proxy port that is dead, breaking the user’s internet connection.

10. The Main Entrypoint

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/main.rs` — Lines 633 to 670
}

The synchronous wrapper around the asynchronous engine. The actual main function is remarkably short. It first installs the rustls crypto provider, which is required for secure connections in modern Rust TLS stacks. Then, it manually builds the tokio multithreaded runtime using Builder::new_multi_thread().enable_all().build(). The enable_all() call is crucial—it turns on the I/O and Timer drivers, which are required for networking sockets and sleep timeouts to function correctly. It then blocks the main thread on the async_main function. async_main parses the command-line arguments using clap (Cli::parse()). Depending on the subcommand, it will either install the daemon as a systemd service (Install), uninstall it (Uninstall), start the background service via systemctl (Start), or run the daemon loop directly in the foreground (Run).

11. Cryptography Initialization

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/main.rs` — Lines 634 to 639
}

Before Tokio is even started, the daemon must prepare its cryptographic primitives. The daemon calls rustls::crypto::ring::default_provider().install_default(). Rustls is a modern, memory-safe TLS library written in Rust. By default, rustls requires a cryptography provider (the actual engine that does the math) to be installed. Here, we install the ring provider, which is optimized and widely audited. If this provider is already installed (for instance, if another dependency initialized it first), it gracefully ignores the error and continues. Without this step, any attempt to establish an HTTPS connection (like the proxy server or the governance bootstrap) would panic immediately.

12. Command Line Interface (CLI) Subcommands

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/main.rs` — Lines 651 to 667
}

The daemon does not just run the network node; it also manages its own installation. Using the clap crate, it parses the command line arguments into an enum called Commands. Based on the matched command, it routes execution to different handler functions:

  • Install: Takes a user and config_dir argument and generates a systemd service file, installing the daemon to run automatically on boot.
  • Uninstall: Removes the systemd service file and disables the daemon.
  • Start: A convenience wrapper that issues a systemctl start kinetic-daemon command to the OS.
  • Stop: A convenience wrapper that issues a systemctl stop kinetic-daemon command.
  • Run (or no command): This is the default path. It actually executes the heavy run_daemon() function, which is the massive asynchronous loop we documented above. This unified binary design makes it easy for users to manage the daemon without writing their own service files.

13. The tokio runtime block_on

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/main.rs` — Lines 641 to 646
}

The main function is synchronous, but async_main is asynchronous. To bridge the gap, main calls block_on(async_main()) on the runtime it just built. This function blocks the OS thread until the entire future provided to it resolves. When async_main finally returns (due to a graceful shutdown or fatal error), block_on completes, the runtime is dropped, all remaining background tasks are cancelled, and the process exits to the operating system.

14. Error Handling and the Result Type

Throughout the daemon’s startup, you will notice many functions return anyhow::Result<()>. This is because starting a network node is dependent on the environment: files must be readable, ports must be available, and the network must be reachable. If a critical step fails (like being unable to bind to the DNS port because another process is using it), the daemon uses the ? operator to bubble the error up to main. This ensures the daemon fails fast and loudly with a descriptive error message, rather than limping along in a broken state.

Key Pieces

Here is a detailed breakdown of the essential concepts and types introduced in this section of the codebase.

tokio::spawn

  • What it does: The primary mechanism for kicking an asynchronous task into the background.
  • Where it lives: Used throughout main.rs, particularly around lines 459, 507, 524, and 609.
  • Why it matters: Every major service (network, proxy, API) is spawned this way so they run concurrently without blocking the main thread. Without it, the node would process one thing at a time and grind to an absolute halt.

reqwest::Client

  • What it does: Used during startup to download the governance.key from bootstrap nodes if it does not exist locally.
  • Where it lives: Lines 364 to 367.
  • Why it matters: Configured with a very short timeout to prevent boot hangs if the seed node is unreachable.

tokio::sync::broadcast

  • What it does: A channel type used for gossip_tx.
  • Where it lives: Line 442.
  • Why it matters: It allows the single network loop to broadcast a message (like a new block) to multiple independent listeners simultaneously. It intentionally drops slow consumers if they lag behind, preventing unbounded memory growth.

tokio::sync::mpsc

  • What it does: A Multi-Producer Single-Consumer channel.
  • Where it lives: Line 441.
  • Why it matters: Used for incoming proxy requests, allowing multiple concurrent network threads to safely funnel requests to a single localized handler without locking overhead.

hickory_server

  • What it does: The external crate used to run the built-in DNS server.
  • Where it lives: Lines 593 to 607.
  • Why it matters: It expertly manages both UDP and TCP socket listeners efficiently, conforming to strict RFC standards for DNS packet handling.

kinetic_global/proxies

  • What it does: The shared filesystem directory where the daemon communicates its proxy port to the system-wide PAC script generator.
  • Where it lives: Lines 560 to 578.
  • Why it matters: This file-drop strategy decouples the daemon from the PAC service, meaning they don’t have to be tightly integrated via complex IPC (Inter-Process Communication).

tokio::select!

  • What it does: The macro that races multiple futures against each other.
  • Where it lives: Lines 620 to 628.
  • Why it matters: It allows the node to wait for either a fatal error in a critical service or a graceful shutdown signal from the operating system, ensuring that an exit path is always available.

dirs::data_local_dir()

  • What it does: A utility function to locate the correct user-specific data directory based on the operating system.
  • Where it lives: Line 560.
  • Why it matters: Hardcoding paths like /etc/ or ~/.local/share breaks on Windows or macOS. This ensures cross-platform compatibility for proxy registration.

tokio::sync::Mutex

  • What it does: An asynchronous lock that guarantees mutually exclusive access to shared data.
  • Where it lives: Line 501.
  • Why it matters: Unlike the standard library’s std::sync::Mutex, Tokio’s Mutex allows a task to hold the lock across .await points without blocking the underlying OS thread, which is vital for high-concurrency environments like the proxy server.

How This Connects to the Rest of Kinetic

This file touches almost every other crate in the Kinetic ecosystem, acting as the final consumer of the various libraries.

  • CROSS-CRATE: GovernanceState — defined and explained in docs/learn/core/04_governance.md
  • CROSS-CRATE: NetworkEventLoop — defined and explained in docs/learn/network/02_event_loop.md
  • CROSS-CRATE: shutdown_signal — defined and explained in docs/learn/core/07_shutdown.md
  • CROSS-CRATE: KineticDnsHandler — defined and explained in docs/learn/dns/01_overview.md

Quick Reference

For fast lookup when returning to this file:

  • Bootstrap Nodes: Queried via HTTP on port 8000 to download governance state on the very first boot.
  • Proxy Registration Path: ~/.local/share/kinetic_global/proxies/<tld>.json (on Linux).
  • DNS Server Ports: Binds to both UDP and TCP on the configured DNS port (default is usually 53, but often overridden).
  • Shutdown Trigger: Ctrl+C (SIGINT) or SIGTERM triggers tokio::select! to unblock and run cleanup.
  • Critical Cleanup Action: Deletes the proxy registration JSON so the OS PAC script stops routing traffic to this node.
  • Async Runtime: Uses Tokio’s multi-threaded scheduler with I/O and timer drivers enabled via enable_all().

Open Questions / Things to Revisit

Several design decisions in this file present potential edge cases that should be reviewed as the network scales:

Seed Node Trust

  • The bootstrap process downloads the governance state via HTTP. While it cleverly validates the genesis_timestamp_sec, an attacker could potentially spoof a node if they know the correct timestamp but provide malicious parameters (like vastly inflated block rewards).
  • Question: Is there a way to verify a cryptographic signature on the governance state from the Genesis block itself to prevent this vector entirely?

API Server Crash Handling

  • If the API server crashes, the tokio::select! block unblocks and the daemon exits.
  • Question: Is this intended behavior? It means an unhandled API error could bring down the entire decentralized node, which seems fragile for a production environment where the network layer could otherwise survive an API failure.

DNS Server Errors

  • The DNS server future is spawned in the background, but if it crashes, it just logs an error using tracing::error! and does not bring down the daemon.
  • Question: This is inconsistent with the API server behavior. Should a DNS crash also trigger a graceful shutdown sequence, or should the API server failure also be handled gracefully without bringing down the entire node?

Proxy State on Crash

  • If the node panics or is killed with SIGKILL (which bypasses the graceful shutdown trap entirely), the proxy JSON file is left behind in the kinetic_global directory.
  • Question: This will break the user’s internet because the OS will keep routing to a dead proxy. Should there be a heartbeat mechanism or stale-file check in the kinetic-pac daemon to handle ungraceful exits automatically?

Channel Capacities

  • The incoming_tx channel has a hardcoded capacity of 32, and gossip_tx has a capacity of 100.
  • Question: Are these numbers empirically derived from network simulation? Under heavy DDoS or a massive gossip storm, will these small buffers drop legitimate traffic, or are they sized to apply backpressure?

Root CA Expiry

  • The ca::load_or_create_root_ca function loads a Root CA, but there is no logic visible here for handling what happens if the Root CA reaches its expiration date while the daemon is actively running.
  • Question: Does the CA automatically rotate, or does the user have to restart the daemon to generate a new valid Root CA?