The Network Event Loop (Core Orchestrator)

Crate: kinetic-network Stage: 8 Reading time: 45 minutes Depends on: docs/learn/network/01_overview.md, docs/learn/core/01_overview.md

What Is This?

This file documents the NetworkEventLoop inside kinetic-network/src/event_loop/core.rs. In Kinetic’s architecture, this is the single central engine that drives all peer-to-peer networking. It operates as an asynchronous actor. It sits in an infinite loop. It constantly polls for network events. It polls for incoming commands. It polls for background task results.

Instead of letting multiple threads freely read and write to network sockets or internal network state, Kinetic centralizes all network mutation into this single thread. Other parts of the Kinetic daemon send messages (Commands) to this loop. This loop executes them against the libp2p::Swarm.

You can think of this loop as:

• The traffic cop routing packets.
• The security guard checking proofs.
• The state manager for the entire P2P layer.

Because Rust enforces strict ownership and borrowing rules, sharing a complex network state across multiple threads safely is notoriously difficult. If Kinetic wrapped the network state in an Arc<Mutex<State>>:

• The resulting lock contention would be massive.
• A high-throughput scenario (like a gossip flood) would immediately bottleneck the node.
• Threads would constantly block waiting for the lock.

By designing NetworkEventLoop as an actor, Kinetic ensures that the network state is exclusively owned by one thread. This thread rapidly processes events in a non-blocking manner. Whenever computationally heavy work is required (like validating a Proof of Work):

• The actor offloads the work to a separate worker thread pool.
• The actor immediately resumes processing the network.
• When the worker finishes, it sends the result back to the actor via a channel.

This guarantees that the node remains responsive to network I/O regardless of CPU load.

The NetworkEventLoop is also the primary point where security policies are enforced. Rather than spreading out security checks across dozens of handler functions, Kinetic centralizes them here.

• When a peer sends bad data, the loopback mechanism reports it here.
• When a peer connects, their Sybil resistance status is determined here. This makes auditing the security model of the network significantly easier.

Why Kinetic Needs This

Networking in Rust is asynchronous. Consider the sheer volume of concurrent events happening in a P2P node:

• Peers connecting and disconnecting randomly.
• Distributed Hash Table (DHT) queries taking seconds to resolve.
• Gossipsub messages arriving continuously from multiple peers.
• Periodic maintenance tasks firing on timers. All of these things happen concurrently and unpredictably.

If Kinetic allowed any thread to directly access the Swarm (the underlying libp2p network manager), you would need massive lock contention. Every time a thread wanted to send a message, it would lock the network. When under DDoS attack, this lock contention would freeze the entire daemon as hundreds of threads try to acquire the lock to report bad actors.

Kinetic solves this using the Actor Model. The NetworkEventLoop takes exclusive ownership of the Swarm. Nothing else can touch it. If the local storage engine needs to broadcast a new block:

• It does not lock the network.
• It simply drops a message into a fast, non-blocking mpsc channel.
• The NetworkEventLoop reads that channel when it is ready.

This design guarantees that the network thread never blocks, ensuring high throughput even when the network is chaotic or under attack.

Furthermore, Kinetic implements heavy cryptographic checks:

• Verifiable Delay Functions (VDFs)
• Proof-of-Work (PoW)

If these checks ran on the main network thread, a single bad actor sending junk data could freeze the network. This event loop delegates heavy cryptography to background threads. It receives the results back via a “loopback” channel. This separation of I/O from computation is critical to Kinetic’s Sybil resistance. If you block the network thread to calculate a hash, your node drops offline from the perspective of the rest of the network.

Beyond just concurrency and cryptography, the event loop provides a centralized location for enforcing network policies. For example, Kinetic limits the number of light nodes that can connect to a daemon. It tracks how many invalid gossip messages a peer has sent. It manages peer bans. If this logic was spread across multiple handlers and callbacks, maintaining security invariants would be nearly impossible. By putting all policy enforcement inside the event loop, Kinetic guarantees that every network action passes through a single, auditable choke point.

The overarching philosophy of this file is: “Never Block, Always Delegate, Enforce.”

How It Works

The NetworkEventLoop is driven by a massive tokio::select! block inside the run method. -> See: kinetic-network/src/event_loop/core.rs — Lines 188 to 280

The tokio::select! macro is the heart of the loop. It waits for multiple asynchronous events simultaneously. It wakes up the loop whenever any one of them is ready. Once an event is handled, the loop repeats.

tokio::select! is designed for cancellation safety. If the Swarm stream yields an event, the other branches (like the command receiver or loopback receiver) are safely paused. They will resume polling on the next iteration without losing data.

Here is an exhaustive breakdown of exactly what the loop multiplexes and how it handles each branch:

1. The Periodic Sled Pruning Timer (prune_delay)

-> See: kinetic-network/src/event_loop/core.rs — Lines 190 to 211

Kinetic stores banned peers and DHT records in the local Sled database (which is managed by kinetic-core). Over time, these records expire. However, scanning the database to delete expired records is a blocking I/O operation. The event loop maintains a futures_timer::Delay that triggers periodically.

To prevent all nodes in the network from waking up and performing heavy database I/O at the exact same second (a thundering herd problem), Kinetic calculates an initial_prune_jitter. This jitter is based on the system clock.

When the timer fires, the event loop does NOT block to prune the database. Instead, it spawns a tokio::task::spawn_blocking closure. This closure takes a clone of the storage Arc. It scans the DB_PREFIX_BANNED_PEER prefix. It reads the 8-byte big-endian expiration timestamp (expire_kyn). It compares it against the current_drand_kyn. If the ban has expired, it deletes the key from Sled. Meanwhile, the main event loop immediately resets the timer with a new jitter and continues processing network traffic.

2. The Aggressive Redial Timer (redial_delay)

-> See: kinetic-network/src/event_loop/core.rs — Lines 212 to 259

Maintaining a healthy connection to the mesh is vital. The event loop checks its active peer count every 15 seconds (again, with jitter). It has two primary behaviors here:

• Zero Peers (The Isolation Case):

  • If the node discovers it has 0 peers, it assumes it has been disconnected from the mesh.
  • It immediately iterates over the hardcoded bootstrap_nodes list.
  • It attempts to dial them all.
  • But it doesn’t stop there. It also utilizes a centralized fallback mechanism: DNS TXT Seed Resolution.
  • The loop spawns a background task that performs a DNS lookup on domains like seed.kinetic.network.
  • This returns a list of fallback multiaddresses.
  • The background task filters out unroutable IPs.
  • It sends the valid multiaddresses back to the event loop via the loopback_tx channel.
  • The event loop receives these in the LoopbackCommand::DialResolvedSeed message and dials them.
  • This layered approach ensures that even if IP addresses of seed nodes change, the network can still bootstrap.

• Many Peers (The Load Balancing Case):

  • If the node discovers it has more than 20 peers, it knows it is safely embedded in the Gossipsub mesh and Kademlia DHT.
  • At this point, staying connected to the foundational bootstrap nodes is actually a detriment to the network.
  • It consumes valuable connection slots on the bootstrap servers.
  • Therefore, the loop iterates over its bootstrap_peers list.
  • It intentionally disconnects from them.
  • This ensures that new nodes joining the network always have available slots on the bootstrap servers.
  • It is a polite, cooperative network behavior.

3. The Drand Tick Receiver (drand_kyn_rx)

-> See: kinetic-network/src/event_loop/core.rs — Lines 260 to 267

Kinetic relies on a global, decentralized clock called Drand. The local daemon watches this clock. The event loop holds a tokio::sync::watch::Receiver<u64> that triggers every time the global clock ticks forward.

When the watch channel signals a change, the event loop borrows the new value. If the new kyn (time unit) is greater than the current_drand_kyn, the loop updates its own state. Crucially, it also updates the current_drand_kyn deep inside the Kademlia store behavior. self.swarm.behaviour_mut().kademlia.store_mut().current_drand_kyn = new_kyn This ensures that the DHT storage engine knows the current network time. This allows it to accurately reject records that attempt to forge timestamps in the future. It also allows it to delete records that have expired in the past.

4. The Swarm Stream (self.swarm.next())

-> See: kinetic-network/src/event_loop/core.rs — Line 268

The libp2p::Swarm implements the Stream trait. This means it produces a continuous flow of network events. When a peer connects, when a DHT query finishes, or when a Gossipsub message arrives, it bubbles up here. The event loop catches these events. It dispatches them to handle_swarm_event(). Because handle_swarm_event() is complex, it is typically broken out into its own file. However, the orchestration happens right here in the main select! block. The loop simply awaits the next event from the Swarm and passes ownership to the handler.

5. The Command Channel (command_receiver)

-> See: kinetic-network/src/event_loop/core.rs — Lines 269 to 275

This is how the rest of the Kinetic daemon talks to the network. The NetworkClient exposes user-friendly async functions (like publish_record or get_domain). These internally construct a Command enum and push it into this mpsc (Multi-Producer, Single-Consumer) channel. The loop wakes up. It pops the command. It translates it into a direct instruction to the Swarm. If the command_receiver returns None, it means all NetworkClient instances have been dropped. This signifies that the node is shutting down. The event loop logs this and breaks out of the infinite loop, terminating the network thread gracefully.

6. The Loopback Channel (loopback_rx)

-> See: kinetic-network/src/event_loop/core.rs — Lines 276 to 278

This is the security architecture in action. When a complex verification is required (e.g., verifying a block’s VDF or checking a connecting peer’s PoW), the event loop does not do the math. Doing the math would block the thread. Instead, it spawns a background blocking task. It passes the task a LoopbackCommand channel sender. When the math is done, the background task sends the result (Valid or Invalid) back to the main loop via this channel. The event loop awaits these results on the loopback_rx receiver. When a result arrives, it calls handle_loopback(). This applies the verdict — either committing the valid data to the DHT or banning the invalid peer.

Specific Execution Scenarios

To better understand the orchestration, here is how the event loop handles common network scenarios:

Scenario A: Processing a Malicious Gossip Block

1. The self.swarm.next() branch in the select! loop receives a raw Gossipsub message event.
2. The event loop passes the message to handle_swarm_event().
3. Recognizing it as unverified data, the loop immediately offloads the signature and VDF checks.
4. It uses a worker thread via tokio::task::spawn_blocking.
5. The network loop continues running, polling other events.
6. The worker thread finishes the math and determines the block is invalid.
7. It constructs a LoopbackCommand::CommitGossipValidation with is_valid: false.
8. The worker pushes the command to the loopback_tx channel.
9. The main event loop wakes up on the loopback_rx.recv() branch.
10. It calls handle_loopback(), which sees the invalid verdict.
11. It calls record_invalid_gossip(source).
12. The peer’s strike count goes from 0 to 1 in the bad_vdf_counts cache.
13. Because is_valid is false, it tells the Gossipsub behavior to Reject the message.
14. This ensures it is not propagated to other peers.
15. If the peer was on their 3rd strike, record_invalid_gossip bans them.
16. handle_loopback actively disconnects the underlying TCP socket.

Scenario B: Handling an Inbound Peer Connection (PoW Check)

1. A new peer connects.
2. handle_swarm_event() fires a connection established event.
3. The event loop reads the peer’s connection metadata and extracts their Proof-of-Work nonce.
4. The event loop offloads the PoW hashing algorithm to a background task.
5. The background task finishes hashing and determines the PoW is invalid (or absent).
6. It sends LoopbackCommand::ConnectionPoWVerified(valid: false) to the loopback channel.
7. The event loop wakes up and enters handle_loopback().
8. It checks if the daemon is at its Light Node capacity (50 nodes).
9. Let’s assume it is currently at 49.
10. It extracts the peer’s IP address.
11. It checks light_node_ips.
12. Let’s assume this IP only has 1 active connection.
13. Since both limits are respected, the event loop inserts the peer into the light_nodes HashSet.
14. It allows the connection to remain open, but their capabilities will be restricted in future interactions.

Key Pieces

This section breaks down the vital data structures and functions that make up the event loop orchestrator.

NetworkEventLoop Struct

-> See: kinetic-network/src/event_loop/core.rs — Lines 51 to 102

This is the massive state object that holds the entire P2P context. It contains:

• swarm: The libp2p network manager itself, holding all the active transports and behaviors.
• command_receiver: The ingest channel for commands from the daemon.
• pending_gets: HashMap tracking ongoing Kademlia Get queries.
• pending_quorums: HashMap tracking ongoing quorum lookups.
• pending_puts: HashMap tracking outbound DHT publishes.
• query_id_to_name: Map linking Kademlia QueryId directly to a QueryType.
• pending_proxy_requests: Tracks outbound Request-Response protocol requests.
• pending_cdn_requests: Tracks outbound CDN domain lookups.
• bad_vdf_counts: An LruCache acting as the network’s strike system.
• current_drand_kyn: The local cached copy of the global clock.
• bootstrap_nodes: State tracking foundational network IP multiaddresses.
• seed_domain: DNS domains to query for fallback seeds.
• bootstrap_peers: State tracking currently connected bootstrap peer IDs.
• banned_peers: An LruCache holding peers temporarily blacklisted.
• pow_semaphore: Concurrency limiter for background PoW hashing.
• gossip_semaphore: Concurrency limiter for background gossip VDF verification.
• light_nodes: State tracking peers that failed the Proof of Work check.
• light_node_ips: State tracking light node connection counts per IP address.

QueryType Enum

-> See: kinetic-network/src/event_loop/core.rs — Lines 18 to 23

A simple internal enum mapping a Kademlia query ID back to the requested action.

• Get: standard DHT lookup.
• Quorum: advanced DHT lookup requiring multiple identical responses to form a consensus.
• Put: outbound DHT publish. This is used to correlate asynchronous DHT responses back to the original request intent.

LoopbackCommand Enum

-> See: kinetic-network/src/event_loop/core.rs — Lines 26 to 44

The messages sent from background cryptographic workers back to the main thread.

• CommitVerifiedRecord:
  • A background task verified a DHT record’s signatures and VDFs.
  • It returns the source peer, the parsed record, and a Result indicating the verdict.
• CommitGossipValidation:
  • A background task verified a gossip block.
  • It returns the message ID, the source peer, and a boolean indicating validity.
• ConnectionPoWVerified:
  • A background task calculated the hashing power of a connecting peer.
  • It returns the peer ID, validity boolean, whether it was a bootstrap node, and the remote IP address.
• DialResolvedSeed:
  • The background DNS resolver successfully found seed IP addresses.

record_invalid_gossip Function

-> See: kinetic-network/src/event_loop/core.rs — Lines 120 to 141

This function manages the network strike system. When a peer sends an invalid message, this function is called. It retrieves the current strike count from the bad_vdf_counts LRU cache.

• If the peer’s last strike was more than 60 seconds ago, their slate is wiped clean, and their count resets to 1.
• If their last strike was within the last 60 seconds, their count increments.
• If their count reaches 3, the function emits a warning log and adds the peer to the banned_peers LRU cache.
• It calculates an expiration timestamp of current_drand_kyn + 28800.
• This typically equates to several hours in Kinetic’s clock system.

This precise 3-strikes-in-60-seconds rule is designed to forgive accidental protocol mismatches or isolated corruption while punishing intentional Sybil or flood attacks.

handle_loopback Function

-> See: kinetic-network/src/event_loop/core.rs — Lines 283 to 422

This function processes the results from the loopback channel. It is the execution arm of Kinetic’s security policy.

Handling CommitVerifiedRecord:

• If the verdict is an error with Severity::Error, it triggers the strike system exactly like invalid gossip.
• 3 strikes in 60 seconds results in a ban and an immediate swarm.disconnect_peer_id(source) call.
• If the verdict is successful, it inserts the verified record into the local Kademlia store.
• Crucially, it then calls swarm.behaviour_mut().kademlia.start_providing(record.key.clone()).
• This implements “Edge Caching” — the node actively advertises that it now has a copy of this record.
• This helps distribute the load across the network and reduces strain on the original publisher.

Handling CommitGossipValidation:

• If valid, it tells the Swarm to Accept the message, which forwards it to the rest of the mesh.
• If invalid, it tells the Swarm to Reject the message (preventing forwarding).
• It then calls record_invalid_gossip.
• If the peer is banned as a result, it disconnects them.

Handling ConnectionPoWVerified: This is the Sybil resistance mechanism for connection slots.

• If a peer passes PoW, they are fully admitted.
• If a peer fails PoW, they are not immediately rejected.
• Instead, Kinetic attempts to classify them as a “Light Node” (a mobile phone or browser that legitimately cannot compute PoW). However, Light Nodes consume connection slots without contributing heavy validation work. Therefore, Kinetic enforces limits:

1. Global Limit: If the daemon already has 50 Light Nodes connected (self.light_nodes.len() >= 50), it rejects the new connection immediately to prevent resource exhaustion.
2. IP Limit: The function extracts the raw IPv4/IPv6 address from the remote_addr. It checks light_node_ips. If 3 Light Nodes are already connected from that exact same IP address, it rejects the connection. This prevents a single attacker from spoofing 50 Light Node identities from a single machine. If both checks pass, the peer is added to the light_nodes set.

How This Connects to the Rest of Kinetic

• Receives from NetworkClient: The NetworkEventLoop is the backend for the NetworkClient. When other modules (like the REST API or the consensus engine) use the client, the commands flow directly into this loop’s command_receiver. The NetworkClient acts as the frontend interface, abstracting away the complexity of the command channels.

• Sends to Sled (Storage): The loop actively prunes the local Kademlia DHT storage backend, communicating directly with the kinetic-core database architecture. The event loop’s background pruning task directly invokes the Sled scan and delete APIs.

• CROSS-CRATE: Uses kinetic_core::constants::TIMEOUTS_NETWORK_PRUNE_INTERVAL_SECONDS to dictate how often the storage engine is cleaned up. This ensures the network pruning frequency is synchronized with the core database configuration.

• CROSS-CRATE: Uses kinetic_core::constants::DB_PREFIX_BANNED_PEER as the Sled key prefix when scanning for expired bans.

• Validates via kinetic-verify: Though the actual calls to kinetic-verify happen in the background tasks, the loopback commands carry the Result enums defined by the verification crate. The event loop inspects the severity() of these errors to determine if a peer made a harmless mistake or committed a malicious offense requiring a ban.

• DNS Resolution: Interacts with crate::dns_tree::resolve_dns_tree to fetch fallback seeds when the primary bootstrap nodes are unreachable.

Quick Reference

• Design Pattern: Asynchronous Actor Model. A single NetworkEventLoop struct holding exclusive ownership of the Swarm, processing events sequentially via message passing.
• Concurrency Control: Background blocking tasks for cryptography, results returned via mpsc unbounded channels (loopback_tx / loopback_rx).
• Strike Policy: Tracked via bad_vdf_counts LRU cache. 3 invalid cryptographic validations within 60 seconds = temporary ban (via Sled) + immediate disconnection.
• Light Node Policy: Peers failing PoW are branded as Light Nodes. Limited to 50 total Light Nodes per daemon to preserve resources. Limited to 3 Light Nodes per IP address to prevent Sybil connection draining. Tracked via light_nodes and light_node_ips.
• Redial Policy: Checks peer count every 15 seconds. At 0 peers, dials DNS seeds. At >20 peers, drops hardcoded bootstrap nodes to save bandwidth for the network.
• Drand Synchronization: Watches a tokio::sync::watch channel for global time updates, propagating the current_drand_kyn down into the Kademlia storage engine.
• Edge Caching: When a DHT record is successfully verified via loopback, the node automatically calls start_providing to advertise its local cached copy.

Open Questions / Things to Revisit

• Unbounded Loopback Channel: The loopback_tx is a tokio::sync::mpsc::UnboundedSender. If background tasks generate loopback commands faster than the event loop can process them, memory could theoretically grow unbounded, leading to an Out-Of-Memory (OOM) crash. In practice, the concurrency of background tasks is limited by the pow_semaphore and gossip_semaphore, which naturally caps the loopback rate. However, relying on external semaphores to protect an unbounded channel is a slight architectural fragility that might warrant an explicit bounded channel in the future.

• Jitter Logic: The random jitter for timers uses SystemTime::now().duration_since(UNIX_EPOCH).as_millis() % 60. While sufficient for desynchronizing nodes and preventing thundering herds, it is not cryptographically uniform or unpredictable. A deterministic adversary could theoretically predict exact pruning times.

• Banned Peer Persistence: Banned peers are stored in Sled. The pruning mechanism deletes expired bans, but the loop itself uses an in-memory LruCache (banned_peers) for fast lookups during active connections. The synchronization between the Sled ban list and the LRU cache might have edge cases on node restart. If the node restarts, does the LRU cache correctly repopulate from Sled, or is it empty until a new offense occurs?

• Light Node IP Spoofing: The limit of 3 Light Nodes per IP address mitigates basic Sybil attacks. However, if an attacker has access to a large botnet or a proxy pool, they could easily bypass this by routing connections through different IPs, exhausting the global limit of 50 Light Nodes. Further analysis may be needed to determine if Light Nodes require an additional challenge/response mechanism beyond simply failing PoW.

• Drand Time Jumps: If the local system clock jumps forward drastically, the drand_kyn_rx will receive a updated time. How gracefully does the Kademlia store handle massive time jumps when evaluating record expirations?