Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Event Loop Handlers: Gossip, CDN, and Proxy

Crate: kinetic-network Stage: 8 Reading time: 30 minutes Depends on: 18_network_event_loop.md, 16_kademlia_routing.md

What Is This?

These files (gossipsub.rs, cdn.rs, proxy.rs, and their parent mod.rs) are specialized sub-handlers for the NetworkEventLoop. Rather than packing all the event handling logic for Gossipsub, CDN, and Proxy into one massive, unreadable match block inside the main loop, Kinetic splits them into separate files.

These handlers process the actual decentralized network events that come off the libp2p swarm. They dictate exactly how the Kinetic node reacts to incoming broadcast messages, direct name record requests, and proxy traffic. The main event loop is simply a dispatcher; these handlers hold the actual business logic for how the node behaves when data arrives over the wire.

Think of the NetworkEventLoop as the mail sorting room of the node. It looks at the envelope, determines the protocol, and hands it off. These sub-handlers are the specific departments (Gossip, CDN, Proxy) that actually open the mail, read the payloads, verify the signatures, and route the final data to the application layer. By separating them, Kinetic ensures that the central event loop remains a fast, non-blocking state machine.

Why Kinetic Needs This

A decentralized network is loud, chaotic, and often actively hostile. If the main event loop handled every single message natively in one place, three major architectural failures would occur:

  1. The Loop Stalls (The Async Blocking Problem): Rust’s tokio async runtime relies on cooperative multitasking. If a task takes too long, no other task can run. Cryptographic verification is inherently slow because it requires heavy CPU math. If the main async loop stops to verify a Drand signature or check a Governance root key, it cannot yield back to the executor. As a result, the node cannot respond to other peers’ ping messages or keep TCP connections alive. If the loop stalls, libp2p assumes the node is dead and drops the connections. We must move math off the main thread.

  2. Resource Exhaustion (The DDoS Threat): A malicious peer could flood the node with garbage gossip messages on the governance topic. If the node eagerly tries to process every single one by spawning a task or doing math, it will run out of memory or exhaust its thread pool. There must be an aggressive rate-limit applied before the message is even parsed.

  3. Spaghetti Code and Unmaintainable State: The NetworkEventLoop already manages Kademlia state, Swarm lifecycle events, ping, identify protocols, and local application channels. If it also had to manually construct proxy responses, parse name records, and manage gossip validation logic, the file would be thousands of lines long and impossible to maintain. Separation of concerns is a hard requirement for a maintainable network stack.

We need isolated handlers that can:

  • rate-limit incoming gossip using semaphores to protect the node from malicious spam.
  • Offload CPU-heavy cryptographic work to background thread pools to protect the async runtime from stalling.
  • Process CDN and Proxy requests cleanly, acting as independent micro-services within the broader network layer.
  • Translate low-level libp2p network errors into high-level application errors that the Daemon can understand.

How It Works

1. The Module Structure (mod.rs)

The mod.rs file acts as the organizer and namespace manager. It simply exports the various sub-handlers: -> See: crates/kinetic-network/src/event_loop/handlers/mod.rs — Lines 1 to 4

This structure makes it easy for the NetworkEventLoop to simply call handlers::gossipsub::handle(...) or handlers::cdn::handle(...) without worrying about the internal details of how the event is parsed. It enforces a strict separation of concerns: the loop dispatches, the handler processes. The module pattern ensures that if we add a new protocol later (like a dedicated storage protocol), we just add a new file and a new line to mod.rs.

2. Gossipsub Handler (gossipsub.rs)

Gossipsub is the protocol libp2p uses for broadcast messages (one-to-many). In Kinetic, this is primarily used for global Drand randomness beacons and global Governance votes. Because these messages go to everyone on the network, they are the prime vector for spam.

Phase 1: Rate Limiting (The Semaphore) Before the handler even looks at a message’s content, it tries to get a permit from the event loop’s shared gossip_semaphore. -> See: crates/kinetic-network/src/event_loop/handlers/gossipsub.rs — Lines 16 to 32

It uses try_acquire_owned(). This is a crucial Rust concept: it does not wait in line. try_acquire_owned is non-blocking. If the semaphore is saturated (meaning there are too many gossip messages currently being processed by the background threads), it immediately rejects the message. It logs a warning: “Gossip semaphore saturated — dropping message”. It then sends a CommitGossipValidation { is_valid: false } message back to the main loop and drops the payload entirely. This strict cut-off prevents the node from being DDoS’d by a flood of gossip. It prioritizes the survival of the node over processing every single message.

Phase 2: CPU Offloading (The Verification) If a permit is successfully acquired, the handler needs to verify the message. Because verification blocks the CPU, it spawns a background thread. -> See: crates/kinetic-network/src/event_loop/handlers/gossipsub.rs — Lines 37 to 62

It uses spawn_blocking to move the work off the main async loop into Tokio’s dedicated blocking thread pool. This is the only safe way to do cryptography in an async environment. Without this, the entire swarm would freeze.

  • If the topic is GOSSIP_TOPIC_DRAND, it parses the JSON payload into a RawKyn and calls verify(). Drand signatures use BLS cryptography, which is heavy.
  • If the topic is GOSSIP_TOPIC_GOVERNANCE, it fetches the GLOBAL_GOVERNANCE_STATE, extracts the current root key, and verifies the cryptographic signatures on the governance action. Governance actions can alter network rules, so this check is absolute. Any failure here results in a fast rejection.

Phase 3: Validation Loopback Once the background thread finishes verifying the message, it must inform libp2p whether the message was valid. Gossipsub requires explicit validation before it will forward a message to other peers. This is what prevents bad messages from propagating. -> See: crates/kinetic-network/src/event_loop/handlers/gossipsub.rs — Lines 64 to 71

It sends a CommitGossipValidation command back to the main loop via loopback_tx. The main loop will then tell the libp2p swarm to either propagate the message (if valid) or penalize the peer who sent it (if invalid). If valid, it also forwards the payload to gossip_tx so the rest of the Kinetic node (specifically the application layer and consensus layer) can actually use the data.

3. CDN Handler (cdn.rs)

The CDN (Content Delivery Network) protocol is a custom fast-path, point-to-point request-response system. It is used to fetch Name Records directly, bypassing the slower multi-hop Kademlia DHT walk.

Serving Requests (When a peer asks us for data): -> See: crates/kinetic-network/src/event_loop/handlers/cdn.rs — Lines 15 to 39

When a Message::Request arrives, a peer is asking us for a specific domain. The handler derives the correct storage keys for that domain, and then manually reaches into our local Kademlia RecordStore to see if we have it cached. If we do, we instantly send a CdnResponse back via the provided channel. We do this using send_response on the swarm’s CDN behaviour. We then increment our proxy_cdn_usage counter, which tracks how much bandwidth we are providing to the network. This allows a node to quickly serve data it already possesses without forcing the requester to do a DHT search.

Handling Responses (When a peer replies to our request): -> See: crates/kinetic-network/src/event_loop/handlers/cdn.rs — Lines 41 to 59

When a Message::Response arrives, we check if the request_id exists in our pending_cdn_requests map. If it does, and the peer gave us a valid record, we deserialize it into a NameRecord. We then inject it into our own Kademlia store using handle_record().

If it’s valid, we log “CDN Hit!”. This is a massive performance win. We then check event_loop.pending_gets to see if any local processes were currently waiting for this domain to resolve. If so, we iterate through the responders and instantly send them the data, resolving their lookup much faster than waiting for a full DHT query to complete.

4. Proxy Handler (proxy.rs)

The Proxy handler manages general point-to-point data traffic. This is used when routing a user’s web request through the network via the proxy service. It is essentially a bridge between libp2p’s network layer and Kinetic’s local application layer.

Incoming Requests (Acting as a Proxy Server): -> See: crates/kinetic-network/src/event_loop/handlers/proxy.rs — Lines 14 to 19

When a peer sends us a proxy request, we do not process the HTTP traffic here. Network layer code should not be parsing HTTP headers or managing streams. We simply forward the raw request to incoming_proxy_tx using an async task. We hand it off to the Daemon layer (which holds the actual HTTP proxy logic) to figure out what to do with it.

Outgoing Responses and Failures (Acting as a Proxy Client): -> See: crates/kinetic-network/src/event_loop/handlers/proxy.rs — Lines 21 to 46

When a peer responds to a proxy request we made (meaning they fetched a webpage for us and sent the HTML back), we pop the waiting channel from pending_proxy_requests and send the data through. If the request fails (e.g., the peer went offline, leading to OutboundFailure::Timeout or DialFailure), we map libp2p’s low-level error into our custom ProxyError enum, and send that Err back to the waiting process. This ensures the caller knows exactly why their proxy request failed, allowing them to retry with a different peer or surface an error to the user gracefully.

Detailed Message Flow Examples

Example 1: Receiving a Drand Gossip Message

  1. libp2p fires gossipsub::Event::Message from the swarm.
  2. The main event loop delegates it to gossipsub::handle.
  3. The node attempts to acquire gossip_semaphore. It succeeds.
  4. spawn_blocking is called. The RawKyn is deserialized and verified using BLS cryptography on a background thread.
  5. The thread finishes. The signature is valid.
  6. A CommitGossipValidation is sent to loopback_tx.
  7. The message is sent to gossip_tx to be ingested by the blockchain subsystem.
  8. The NetworkEventLoop reads the loopback and tells libp2p to forward the Drand message to all connected peers.

Example 2: Receiving a CDN Request for “saif.kyn”

  1. libp2p fires request_response::Event::Message::Request on the CDN protocol.
  2. The main event loop delegates it to cdn::handle, which reads the domain "saif.kyn".
  3. It derives the storage keys for this domain and the current NETWORK_ID.
  4. It searches the local Kademlia RecordStore for those keys.
  5. It finds a match! A previously cached NameRecord is loaded from memory.
  6. It sends a CdnResponse containing the record back down the libp2p channel.
  7. It increments proxy_cdn_usage by 1 to track bandwidth contribution.

Example 3: A Proxy Outbound Failure

  1. The local Daemon asks the network to fetch http://example.com via a peer.
  2. The NetworkEventLoop dials the peer, but the peer’s connection drops midway.
  3. libp2p fires request_response::Event::OutboundFailure with OutboundFailure::ConnectionClosed.
  4. The main event loop delegates it to proxy::handle.
  5. It looks up the original request ID in pending_proxy_requests.
  6. It maps ConnectionClosed to ProxyError::ConnectionClosed.
  7. It sends Err(ProxyError::ConnectionClosed) back to the local Daemon through the waiting channel.
  8. The local Daemon receives the error and returns a 502 Bad Gateway to the user’s browser.

Key Pieces

  • mod.rs: The organizational root. Exports the sub-handlers to keep the main event loop clean and modular.
  • gossipsub::handle(): The rate-limiter and verifier. Manages the concurrency semaphore, offloads heavy crypto validation to threads, and orchestrates the loopback validation system to tell libp2p what to propagate.
  • cdn::handle(): The lookup accelerator. Provides a fast-path for name record resolution by querying and updating the local Kademlia store directly upon request or response.
  • proxy::handle(): The data bridge. Simply connects libp2p’s request-response network protocol to Kinetic’s internal asynchronous proxy channels.

How This Connects to the Rest of Kinetic

  • CROSS-CRATE: kinetic_core::drand::RawKyn — defined and explained in docs/learn/core. (Verified by the gossipsub handler when Drand randomness arrives).
  • CROSS-CRATE: kinetic_core::governance::SignedGovernanceMessage — defined and explained in docs/learn/core. (Verified by the gossipsub handler when governance actions arrive).
  • CROSS-CRATE: kinetic_core::types::NameRecord — defined and explained in docs/learn/core. (Parsed and stored by the CDN handler).
  • FORWARD DEPENDENCY: The incoming_proxy_tx channel in proxy.rs connects up to the Daemon crate (Stage 9). The Daemon uses it to expose local HTTP proxying to the user. For now, treat it as an opaque pipe that carries data to the application layer.
  • Main Event Loop: All of these handlers are called directly by the NetworkEventLoop documented in 18_network_event_loop.md.
  • Kademlia Storage: The CDN handler directly interacts with Kademlia’s RecordStore, pulling and pushing records as needed.

Quick Reference

  • gossipsub.rs: Handles network-wide broadcasts. Protects the node from spam using a semaphore. Protects the async runtime from stalling using spawn_blocking.
  • cdn.rs: Handles direct, point-to-point requests for domains. Bypasses Kademlia routing to provide instant responses if the record is cached locally. Acts as a layer-2 cache on top of Kademlia.
  • proxy.rs: Bridges the network’s proxy requests to the user’s local daemon, translating libp2p network errors into application-level proxy errors.
  • Semaphore Rate Limiting: The technique of using try_acquire_owned() to immediately drop messages when the system is under heavy load, ensuring stability over perfect processing.
  • Loopback Validation: The two-step process where a handler verifies a message in the background, then sends a command back to the main loop to officially accept or reject it in the eyes of libp2p. This allows async validation without blocking the main event loop.
  • CPU Offloading: The practice of using spawn_blocking to move cryptographic math away from Tokio’s async executors.

Open Questions / Things to Revisit

  • Semaphore Tuning Under Load: The gossipsub semaphore uses try_acquire_owned(), meaning it drops messages immediately if the background threads are saturated. If the network experiences a legitimate, sudden spike in traffic (e.g., a rapid governance event), we might drop valid, important messages simply because our queue is full. We might need a small bounded wait instead of an immediate reject, or a prioritized queue for governance vs. drand.
  • Blocking Thread Pool Exhaustion: While spawn_blocking prevents the async executor from halting, an attacker could still flood us with garbage governance messages. The semaphore limits concurrency, but we could still tie up all available blocking threads with fake cryptographic checks, starving other components that rely on spawn_blocking (like disk I/O).
  • CDN Trust Model: The CDN handler injects received records straight into the local Kademlia store via handle_record. It relies on Kademlia’s internal verification to ensure the peer didn’t send a valid but functionally incorrect or outdated record. There is a risk of cache poisoning if skip_verify in dev mode leaks into production logic.
  • Proxy Error Mapping Completeness: The proxy error mapping handles Timeout, DialFailure, ConnectionClosed, and UnsupportedProtocols. All other libp2p errors are dumped into a generic ProxyError::Other. As the proxy feature matures, we may need a more granular error mapping to provide better feedback to the user and allow the Daemon to implement better retry logic.

Deep Dive: The Loopback Validation Architecture

One of the most complex interactions in the Gossipsub handler is the “Loopback Validation” system. To understand why this exists, you have to understand how libp2p::gossipsub protects the network from spam.

When a node receives a gossip message, it doesn’t immediately forward it to its peers. If it did, a single malicious node could bring down the entire network by sending one billion fake messages per second. Instead, libp2p puts the message in a “validation queue”. The application (Kinetic) is required to inspect the message and call gossipsub.report_message_validation_result(message_id, ValidationMode::Accept).

However, there’s a catch:

  1. The gossipsub state lives inside the Swarm, which is owned by the NetworkEventLoop.
  2. The cryptographic verification is slow, so we moved it to a background thread using spawn_blocking.
  3. The background thread does not have access to the Swarm. It cannot call report_message_validation_result directly because it doesn’t own the network state.

This creates a paradox: the thread that knows if the message is valid cannot tell libp2p about it.

The solution is the Loopback Channel (loopback_tx). When the background thread finishes its math, it constructs a LoopbackCommand::CommitGossipValidation enum containing the message_id and a boolean is_valid. It sends this enum down the channel. The main NetworkEventLoop (which does own the Swarm) listens on the receiving end of this channel. When it pops the command off the queue, it reaches into the Swarm and finally calls report_message_validation_result.

This architecture allows Kinetic to perform heavy, blocking cryptography asynchronously while still satisfying libp2p’s strict state ownership rules. It is a textbook example of “message passing” over “shared memory” in Rust.

Deep Dive: CDN vs Kademlia

The cdn.rs handler might seem redundant at first glance. Why do we need a custom Request-Response protocol to fetch Name Records when we already have the Kademlia DHT?

The answer is latency.

Kademlia is a Distributed Hash Table. When you look up a record in Kademlia, you don’t usually connect directly to the node that has the data. Instead, you connect to a node that is closer to the data, and ask them who they know. They give you a list of closer nodes. You connect to those nodes, and ask them. This process (the DHT walk) takes multiple network hops. In a global network, each hop might take 100ms. A full Kademlia lookup can easily take 500ms to 2 seconds.

For a web browser (the primary consumer of Name Records in Kinetic), waiting 2 seconds just to resolve a domain name before even starting the HTTP request is unacceptable.

The CDN protocol solves this by acting as a Layer-2 Cache. If a node believes a specific peer might already have the Name Record (perhaps because they are a known heavy-hitter or proxy provider), they can use the CDN protocol to ask them directly, bypassing Kademlia entirely.

When the peer receives this request (handled by cdn.rs), they simply check their local Kademlia cache (kademlia.store_mut().get()). If they have it, they return it instantly in a single network hop.

If it works, the lookup drops from 2 seconds to 50ms. If it fails (the peer doesn’t have it), the node falls back to the slow Kademlia walk. The CDN handler is what makes Kinetic’s domain resolution fast enough for human web browsing.

Core Concepts: Tokio Semaphores and Spawn Blocking

To fully grasp the gossipsub.rs handler, you need to understand two specific tools from the Tokio asynchronous runtime: the Semaphore and the Blocking Pool.

The Tokio Semaphore (try_acquire_owned)

A Semaphore is a concurrency primitive that holds a certain number of “permits”. In Kinetic, the gossip_semaphore is created in the NetworkEventLoop with a fixed number of permits (e.g., 50). When a message arrives, the handler calls try_acquire_owned().

  • If a permit is available: The function returns Ok(permit). The handler proceeds to process the message. When the message is fully processed (or dropped), the permit is automatically returned to the semaphore, allowing a new message to be processed.
  • If zero permits are available: The function returns an Err. This means 50 messages are currently being processed at this exact millisecond. Instead of waiting (which would stall the loop), the handler instantly matches on the Err, logs a warning, and discards the message.

This mechanism is the ultimate shield against network floods. It guarantees that no matter how much data a peer blasts at the node, the node will never attempt to process more than 50 messages simultaneously.

The Blocking Pool (spawn_blocking)

Tokio operates on a small number of worker threads (usually equal to the number of CPU cores). These threads execute async tasks. However, async tasks are expected to be “polite” — they should run quickly and yield control back to the worker thread when they hit an await point (like waiting for a network packet). Cryptographic verification (like verifying a BLS signature in Drand) does not yield. It is a tight loop of pure math. If you run it on a standard Tokio worker thread, that thread is held hostage until the math is done. If you get 8 messages at once on an 8-core machine, your entire network stack freezes.

To solve this, Tokio provides spawn_blocking. This function moves the closure onto a separate thread pool (the blocking pool), which can scale up to 500 threads. This keeps the primary async worker threads free to continue routing network traffic, while the heavy math happens in the background. Once the math finishes, the result is sent back to the async world via a channel (in our case, the loopback_tx).

Deep Dive: Proxy Error Mapping

The proxy.rs handler doesn’t just pass data; it also acts as an error translator. Libp2p generates very low-level network errors. The local application Daemon doesn’t care about libp2p’s internal state; it just wants to know why its proxy request failed.

When an OutboundFailure occurs, the handler inspects it:

  1. OutboundFailure::DialFailure: The peer we tried to proxy through is unreachable. They might be offline, or blocked by a firewall. We map this to ProxyError::Offline.
  2. OutboundFailure::Timeout: The peer is online, but they took too long to fetch the webpage and send it back to us. We map this to ProxyError::Timeout.
  3. OutboundFailure::ConnectionClosed: The peer was online and we were communicating, but the TCP connection unexpectedly dropped. We map this to ProxyError::ConnectionClosed.
  4. OutboundFailure::UnsupportedProtocols: The peer doesn’t actually support the Kinetic Proxy protocol (perhaps they are running an outdated version of the node software). We map this to ProxyError::UnsupportedProtocols.

By translating these errors, the Proxy handler ensures a clean boundary between the network layer (libp2p) and the application layer (the Daemon).

Security Implications: Dev Mode vs Production in CDN

Inside cdn.rs, when a NameRecord is received from a peer, it is injected into the local Kademlia store using the handle_record(&record, skip_verify) function. -> See: crates/kinetic-network/src/event_loop/handlers/cdn.rs — Lines 47 to 48

Notice how the skip_verify flag is determined: kinetic_core::config::is_dev_mode(). This is a critical security boundary.

In Production (when is_dev_mode() is false), skip_verify is false. This means that even though we received this record via the fast-path CDN, we still force Kademlia to verify the cryptographic signature on the record against the domain owner’s public key. If a malicious peer tries to poison our cache by sending a fake Name Record (e.g., redirecting a domain to a phishing IP), the verification will fail, and the record will be silently dropped.

In Development mode, verification is skipped to allow for faster iteration and easier testing with dummy data. This highlights why dev mode must never be exposed to the public internet, as it turns the CDN handler into an open vector for cache poisoning.

Summary of Handler Traits and Performance

When evaluating the performance of a decentralized node, the network handlers are the absolute bottleneck.

  • Latency: The proxy.rs and cdn.rs handlers add virtually zero overhead. They match on the request and instantly toss it over an asynchronous channel. Their latency is measured in microseconds.
  • Throughput: The throughput of the gossipsub.rs handler is hard-capped by the size of the Tokio blocking pool and the number of permits in the gossip_semaphore. If you have a 32-core machine, you can process 32 gossip signatures simultaneously in hardware.
  • Memory Footprint: By dropping messages when the semaphore is saturated, Kinetic ensures that the memory footprint of the gossip queue remains constant, regardless of network spam. A naive implementation that queues messages for processing would see memory usage spike to gigabytes during a network storm. The immediate drop guarantees O(1) memory usage for the gossip ingestion pipeline.

In short, these handlers are designed for one thing: keeping the node alive under the worst possible network conditions, while providing fast-paths for latency-sensitive applications like DNS resolution.

Why try_acquire_owned instead of acquire?

You might wonder why we don’t just use semaphore.acquire().await. If we used .await, the gossip handler would patiently wait its turn. However, .await means the main NetworkEventLoop would pause exactly on that line, waiting for the semaphore. This would defeat the purpose of the semaphore, as it would stall the entire async executor while waiting for background threads to finish. By using try_acquire_owned(), we check the queue synchronously, and if it’s full, we discard the message and move on immediately, keeping the loop spinning at maximum speed.

The Role of proxy_cdn_usage

In the CDN handler, you’ll see event_loop.proxy_cdn_usage.0 += 1. This might look trivial, but it forms the foundation of Kinetic’s incentive structure. By tracking exactly how many times a node successfully serves a Name Record from its cache to a requesting peer, the network can eventually reward nodes that act as high-availability CDN providers. This counters the tragedy of the commons, where nodes might otherwise refuse to serve requests to save bandwidth. Tracking this metric locally is the first step towards a verifiable proof-of-bandwidth system in future network stages.