Event Loop Command Handler

Crate: kinetic-network Stage: 8 Reading time: 30 minutes Depends on: 01_overview.md, 10_event_loop_core.md

What Is This?

The command handler is the critical translation layer for the Kinetic network. It sits between the rest of the application and the libp2p network event loop.

When you build a peer-to-peer application using Rust’s libp2p ecosystem:

• The network state is contained within a Swarm.
• This Swarm manages all TCP and QUIC connections.
• It handles the Kademlia routing tables for the DHT.
• It manages the Gossipsub mesh for pub/sub messaging.
• It tracks the AutoNAT status to determine public reachability.

Because Rust enforces strict ownership rules to prevent data races:

• The Swarm is restricted to running on a single asynchronous task.
• This task is the main event loop.
• The Swarm cannot be safely shared across multiple threads.

However, the rest of the Kinetic daemon runs across multiple concurrent Tokio threads. These components include:

• The HTTP REST API handlers.
• The background block verification workers.
• The consensus engine and mempool.

These components constantly need to interact with the network. They need to:

• Publish domains to the DHT.
• Resolve names from the network.
• Broadcast new blocks and transactions.

The command handler (command_handler.rs) solves this concurrency problem. It implements an Actor Model for network communication. It receives high-level commands from other threads. These commands are sent via an asynchronous message channel (MPSC). The handler then converts these abstract requests into low-level operations. These operations mutate the state of the Kademlia DHT, the Gossipsub mesh, or the custom Content Delivery Network (CDN) layer.

Crucially, it also manages the asynchronous return paths. Networking takes time, so requests cannot be answered instantly. The command handler ensures that requests initiated by the application are properly tracked. It routes them into the peer-to-peer mesh. Finally, it guarantees that the eventual results are routed back exactly to the caller thread that requested them.

Why Kinetic Needs This

To understand why this specific file is so necessary, imagine a scenario where it didn’t exist. Imagine if the REST API tried to share the Swarm using an Arc<Mutex<Swarm>>.

1. A user hits the /resolve endpoint on the REST API.
2. The REST thread locks the mutex.
3. It asks the Swarm to resolve a domain.
4. It waits for the resolution to complete.

But libp2p relies on continuously polling the Swarm. Polling processes background network events like:

• Incoming peer connections.
• Keep-alive pings.
• Protocol negotiations.

If the REST thread holds the lock while waiting for a DHT resolution:

• The resolution could take hundreds of milliseconds or even seconds.
• During this time, the entire network layer is frozen.
• The event loop cannot poll.
• TCP Connections would time out and drop.
• Gossipsub would miss heartbeat intervals.
• The node would rapidly fall out of the mesh network.

Kinetic avoids this by using the NetworkEventLoop as a dedicated, continuously running loop. This loop owns the Swarm completely. Other threads can never touch the Swarm directly. Instead, they communicate by sending messages through a multi-producer, single-consumer (MPSC) channel.

The command_handler.rs file is the instruction manual for the event loop. It defines exactly how the event loop should react when a message arrives on that MPSC channel.

Here is the exact flow of a command:

• A client wants to resolve a domain.
• The client constructs a Command message.
• Attached to that message is a one-time-use return channel (tokio::sync::oneshot::Sender).
• The client sends the command over the MPSC channel.
• The command handler (inside the event loop) unpacks the Command message.
• It triggers the actual libp2p behaviors (like kademlia.get_record).
• It registers the return channel in a state map (like pending_gets).
• The event loop continues polling.
• Later, the network replies with the requested data.
• The event loop looks up the corresponding return channel in the state map.
• It sends the answer back to the exact REST thread that asked for it.

Without this file, the application would be severed from the P2P network capabilities. The REST API would have no way to ask the network to do anything. The background workers would have no way to broadcast their verifications.

How It Works

The core logic lives inside the handle_command function. This function processes a massive match statement over every possible Command variant. Because networking is fundamentally asynchronous, most commands cannot be answered immediately. Instead, they require kicking off a network request and storing a “pending state.”

Let’s break down exactly how the most important commands are processed step by step.

1. Publishing Redundantly to the DHT (Command::PublishRedundant)

When the event loop receives a Command::PublishRedundant, it needs to ensure the data is distributed widely. It must survive node churn (nodes going offline). Kademlia stores data on the nodes whose IDs are closest to the data’s key. If we only stored the data under one key, and the nodes closest to that key went offline, the data would be lost.

Here is the step-by-step process for redundant publishing:

1. Key Derivation

   • It first derives a set of storage keys.
   • It uses the derive_storage_keys function from kinetic-core.
   • This takes the human-readable domain name (like saif.kyn).
   • It generates multiple distinct cryptographic DHT keys.
   • This ensures the record is replicated across different regions of the network’s logical address space.

2. Local Validation

   • Before it ever touches the network, it calls enqueue_dht_puts.
   • This helper function attempts to insert the record into the node’s local Kademlia store first.
   • Why do this locally first? This is a massive optimization and security feature.
   • If the record is invalid, the local store’s validation logic will reject it.
   • Reasons for invalidity include a wrong cryptographic signature, insufficient proof-of-work, or incorrect formatting.
   • If the local store rejects it, every other honest node on the network will reject it too.
   • By failing fast locally, Kinetic prevents the node from wasting outbound bandwidth.
   • It prevents spamming the network with garbage records.
   • It immediately returns an error to the caller without generating any network traffic.

3. Network Dispatch

   • If local validation succeeds, it proceeds to network distribution.
   • It iterates over every derived key.
   • It fires off a put_record request to the kademlia behavior in the Swarm.
   • It requests a Quorum::One.
   • This means it expects at least one successful acknowledgment per key from the network.

4. State Tracking

   • Every put_record call returns a QueryId.
   • The handler records this QueryId in the query_id_to_name map.
   • This allows the event loop to know what operation a future response belongs to.
   • It also registers the expected number of successful responses in the pending_puts map.
   • It stores the client’s oneshot::Sender here as well.
   • As network acknowledgments trickle in over the next few seconds, they are tallied.
   • Once all expected keys are published, the oneshot channel is triggered.
   • The client is notified of success.

2. Resolving Data (Racing Kademlia vs. CDN) (Command::ResolveRedundant)

The Command::ResolveRedundant variant is one of the most critical paths for user experience. When a user requests a domain, the resolution needs to be fast. However, Kademlia DHT lookups are notoriously slow. They require iteratively querying multiple nodes to find the closest peers to a key.

To solve this latency problem, Kinetic implements a sophisticated “racing” mechanism. It queries two different network layers simultaneously:

1. Offline Fallback Check

   • First, it checks the network info for active peers.
   • If the node has zero active peers, it is offline.
   • In this case, it falls back to querying its own local Kademlia store.
   • It derives the keys and checks if it holds the data.
   • This is crucial for local testing or offline development.
   • It also saves nodes that temporarily lose internet connectivity but have recently cached the required domain.

2. CDN Blast (The Fast Path)

   • If online, it immediately attempts a CDN blast.
   • It selects up to 3 currently connected peers from the swarm.
   • It uses the custom cdn behavior to fire off a direct request for the domain to these 3 peers.
   • This skips the iterative, multi-hop routing of Kademlia.
   • If one of these direct peers happens to have the domain cached in their edge storage, they will return it instantly.
   • This cuts resolution time from ~800ms down to ~50ms.

3. DHT Querying (The Reliable Path)

   • At the exact same time as the CDN blast, it queries the DHT.
   • It derives the storage keys for the domain.
   • It dispatches formal Kademlia get_record queries into the DHT via dispatch_dht_queries.
   • This is the slow, fallback method.
   • It is guaranteed to find the data if it exists anywhere on the network, even if the 3 peers from the CDN blast didn’t have it.

4. First to Win

   • The state for the CDN requests is recorded in pending_cdn_requests.
   • The state for the DHT queries is recorded in pending_gets.
   • The event loop will accept the first valid response it receives from either system.
   • Whichever subsystem returns cryptographically valid data first will trigger the oneshot response to the user.
   • This architecture guarantees the speed of a centralized CDN with the decentralized, uncensorable reliability of a DHT.

3. Verifying Quorums (Command::VerifyQuorum)

Sometimes, Kinetic needs to know not just what a record is, but whether the network agrees on it. The VerifyQuorum command is used for consensus checks.

• It derives the storage keys for a domain.
• It fires off Kademlia queries using a special QueryType::Quorum.
• It registers the expected payload in pending_quorums.
• As the network responds, the event loop counts how many distinct nodes returned the exact same payload.
• It then reports this “match count” back to the client.
• This allows the application to prove that a record is widely distributed.
• It ensures the data has not been eclipsed by a malicious actor.

4. Handling Gossipsub Messages and Validation

The libp2p::gossipsub behavior manages the unstructured mesh network. This mesh is used for broadcasting blocks and mempool transactions. The event loop interacts with it via three main commands:

• Subscribing (Command::SubscribeGossip)

  • When the application wants to listen to a new topic (e.g., a specific block height), it sends this command.
  • The handler takes the string topic and converts it into a libp2p::gossipsub::IdentTopic.
  • It calls subscribe on the gossipsub behavior.
  • This tells the libp2p swarm to actively seek out peers who are also interested in this topic.
  • It starts maintaining a local mesh network for it.
  • If successful, the node will begin receiving events for this topic.

• Broadcasting (Command::BroadcastGossip)

  • When the node creates a new block or transaction, it needs to propagate it.
  • This command takes the payload and the topic.
  • It creates the IdentTopic and calls publish.
  • The swarm then forwards the message to a subset of its connected peers in the mesh.
  • Those peers will then forward it to their peers, propagating it globally.

• Validation (Command::ReportGossipValidation)

  • Kinetic uses “strict validation” for Gossipsub.
  • When a message is received, it is not immediately forwarded.
  • Instead, it is offloaded to a background thread to check the VDF proofs and signatures.
  • Once the background thread finishes its heavy cryptographic work, it sends this command back to the event loop.
  • The command contains the MessageId, the PeerId of the sender, and whether the message was valid.
  • The handler translates this into MessageAcceptance::Accept or Reject.
  • It calls report_message_validation_result.
  • If rejected, the peer’s score is penalized.
  • If a peer sends too many invalid messages, they are banned.
  • This is the fundamental mechanism protecting the network from flood attacks.

5. Proxy Requests for Light Clients

Light clients cannot participate fully in the Kademlia DHT or Gossipsub mesh. They lack the bandwidth, CPU, and storage required. Instead, they rely on full nodes to proxy their requests. The command handler facilitates this via the custom proxy behavior (part of the CDN).

• Sending Requests (Command::SendProxyRequest)

  • When the node needs to ask another specific peer for data directly (without routing through the DHT), it sends this command.
  • It provides the target libp2p::PeerId and the ProxyRequest payload.
  • The handler invokes self.swarm.behaviour_mut().proxy.send_request.
  • This returns an OutboundRequestId.
  • The handler then maps this ID to the client’s oneshot::Sender in the pending_proxy_requests map.
  • When the target peer eventually replies, the event loop can look up the ID and route the response.

• Sending Responses (Command::SendProxyResponse)

  • When a remote peer asks this node for data, the request is surfaced to the application.
  • The application processes the request.
  • Once the application has the answer, it sends this command back to the event loop.
  • The command includes the ResponseChannel provided by libp2p during the initial request, along with the data.
  • The handler calls send_response to push the data back over the wire to the requesting peer.

6. Network Status (Command::GetNetworkStatus)

This is a diagnostic command used by the CLI tooling. When triggered, it queries the swarm for its current state and returns JSON containing:

• The number of currently connected peers.
• The node’s local PeerId.
• The multiaddresses it is currently listening on.
• Its NAT status (whether it is reachable from the public internet).
• The size of its local DHT store. This provides users with a real-time dashboard of their node’s health.

Key Pieces

Command Enum

-> See: kinetic-network/src/client/command.rs — Lines 10 to 101

This is the exhaustive list of all instructions the event loop understands. It defines the complete API surface between the network layer and the rest of the node. Every variant typically contains:

1. The required parameters (like a domain name, a payload, or a topic string).
2. A responder: tokio::sync::oneshot::Sender. This sender routes the asynchronous answer back to the exact thread that issued the command. Without this enum, the node’s business logic would have no way to communicate with the P2P mesh.

handle_command Function

-> See: kinetic-network/src/event_loop/command_handler.rs — Lines 89 to 314

This is the massive match statement that consumes Command instances. It dictates exactly how the state of the NetworkEventLoop mutates in response to client requests. This is the main entry point for everything described in the “How It Works” section.

enqueue_dht_puts Method

-> See: kinetic-network/src/event_loop/command_handler.rs — Lines 10 to 70

This method is responsible for taking a payload and validating it against the local store rules to prevent spam. It then queues multiple put_record requests to the Kademlia behavior to ensure redundant network storage. It gracefully handles immediate failures and warns the node operator if all puts fail immediately.

dispatch_dht_queries Method

-> See: kinetic-network/src/event_loop/command_handler.rs — Lines 72 to 87

This method fires off Kademlia get_record queries for a list of derived keys. Crucially, it registers the resulting query_id in the query_id_to_name mapping. This ensures the event loop knows exactly what domain a future, asynchronous network response belongs to.

How This Connects to the Rest of Kinetic

• CROSS-CRATE: The Command variants rely on derive_storage_keys and derive_heartbeat_keys from kinetic_core::types. These map human-readable domain names to the cryptographic Kademlia address space.
• CROSS-CRATE: The commands themselves are constructed and sent by the NetworkClient. This client lives in kinetic-network but provides a clean, async API utilized extensively by the kinetic-rest server and the background workers in kinetic-daemon.
• Internal Integration: This handler directly mutates the pending_gets, pending_puts, and pending_quorums state maps defined in event_loop/core.rs. It provides the setup, while the teardown (completing the request) happens when network events arrive later in the loop.

Quick Reference

• Publishing (PublishRedundant):

  • Validates locally to prevent spam.
  • Puts to the DHT redundantly across multiple keys.
  • Returns success when the network acknowledges.

• Resolving (ResolveRedundant):

  • Checks the offline local cache first.
  • Races direct CDN requests to 3 peers against slow DHT queries.
  • Returns the fastest valid response.

• Quorum Verification (VerifyQuorum):

  • Queries the DHT to ensure a critical mass of nodes agree on the exact same payload.
  • Used for consensus and security checks.

• Gossipsub Management:

  • BroadcastGossip and SubscribeGossip act as simple pass-throughs.
  • They call methods on the underlying libp2p::gossipsub behavior.
  • This allows the node to participate in block and mempool meshes.

• Validation Loopback:

  • ReportGossipValidation allows offloaded CPU-heavy checks to report back.
  • It penalizes malicious peers sending bad VDFs or signatures.

• Network Status:

  • Generates diagnostic JSON summarizing peer connections, local DHT size, and NAT status.
  • Used by the CLI.

Open Questions / Things to Revisit

1. CDN Racing Peer Count

   • The ResolveRedundant command blindly takes the first 3 connected peers for its CDN blast: self.swarm.connected_peers().copied().take(3).
   • Is 3 the optimal number?
   • Should it prioritize peers based on known latency, bandwidth, or past reliability?
   • Just taking the first 3 from the iterator might select slow peers, losing the CDN blast advantage.

2. Offline Fallback Scope

   • The offline fallback for ResolveRedundant currently only checks the local Kademlia store.
   • If a record was previously fetched via the CDN layer but never formally inserted into the local Kademlia store, the offline node will fail to resolve it.
   • Should the CDN cache be queried during offline mode as well to provide better resilience?

3. Local Put Failure Handling

   • In enqueue_dht_puts, if the local put fails, it aborts the network publish.
   • It assumes this is a validation error.
   • This is correct for cryptographic validation failures.
   • But what if the local store fails due to a disk I/O error or full disk?
   • It would silently prevent the node from publishing valid data to the network.
   • This could potentially isolate the node’s outputs.

4. Bootstrap Redialing

   • In the Bootstrap command, it redials the hardcoded bootstrap peers.
   • If those peers change IP addresses, this command might endlessly dial dead endpoints.
   • A DNS re-resolution might be necessary here if they rotate IPs.