Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Background Services: Heartbeats and Gossip

Crate: kinetic-daemon Stage: 9 Reading time: 20 minutes Depends on: docs/learn/network/01_overview.md, docs/learn/core/01_overview.md

What Is This?

In the kinetic-daemon crate, the src/services/ module contains the continuous background loops that keep a node alive, synchronized, and active within the broader Kinetic network. While the REST API handles direct, synchronous user commands (like “register this name”) and the P2P swarm handles low-level socket connections, these background services act as the autonomous nervous system of the daemon. They run endlessly, driven by timers and event streams, independent of user interaction.

Specifically, we are looking at two critical architectural loops:

  1. The Heartbeat & Drand Loop (heartbeat.rs): A timed, active loop that serves dual purposes. First, it constantly fetches the latest network time (Drand kyns). Second, it uses that time to continually broadcast cryptographic proofs of ownership to the Kademlia DHT, ensuring the names registered by this node do not expire.
  2. The Gossip Processor (gossip.rs): A reactive, listening loop that waits for network-wide pub-sub (publish-subscribe) broadcasts. It monitors channels for critical governance votes and new Drand kyns, validates them cryptographically, seamlessly applies those updates to the daemon’s local memory and disk state.

The mod.rs file simply groups these loops (along with network.rs, which handles mining loops) into a unified services module. These services are what elevate the daemon from a simple passive database into a participating citizen of the decentralized network.

Why Kinetic Needs This

Kinetic is fundamentally designed as a decentralized, living network. It rejects the model of a central database where you can register a name and walk away forever. To prevent the network from accumulating dead data, hijacked names, or stale state, Kinetic requires continuous action from its participants.

Why the Heartbeat Loop is Constitutionally Mandatory: When you register a name in the Kinetic ecosystem, you are not buying it; you are leasing it. To maintain that lease and prove you are a legitimate, active participant, your daemon must repeatedly announce to the Kademlia DHT that you are still online and still in control of the private key for that specific name. You achieve this by signing the current Drand clock tick (the “kyn”). If the daemon did not have an automatic background loop doing this every 30 seconds, your names would rapidly expire, disappearing from the network, and someone else could immediately claim them. Furthermore, the heartbeat loop is the engine responsible for fetching those Drand kyns in the first place. Without Drand, the daemon has no concept of absolute network time. Without time, it cannot sign valid heartbeats, nor can it verify the heartbeats or proofs of other peers. The entire cryptographic validation pipeline would stall.

Why the Gossip Processor is Constitutionally Mandatory: Kinetic uses a pub-sub (publish-subscribe) model for network-wide announcements, utilizing the libp2p::gossipsub protocol. This is different from the Kademlia DHT. The DHT is for point-to-point data retrieval (like looking up an IP address). Gossipsub is for shouting critical information to the entire network simultaneously. When the governance council successfully completes a vote to grant a premium name, or when a peer discovers a new Drand kyn, they shout this payload over the gossip network. If your daemon lacked the gossip.rs background loop, you would be deaf to these shouts. Your local node would never realize a premium name was granted, leading it to reject valid transactions simply because its local state was out of date. The gossip processor is the mechanism that ensures your node’s localized worldview stays synchronized with the global network consensus.

How It Works

Because these are two distinct sub-systems, we will break down their mechanics and code paths separately.

Understanding the Tokio Channels

The background services do not operate in a vacuum; they must communicate their findings to the rest of the daemon. They do this using two very specific types of Tokio channels.

1. The Watch Channel (tokio::sync::watch::Sender<u64>) Used for broadcasting Drand kyns. A watch channel is a single-producer, multi-consumer channel where only the most recent value is kept. If the heartbeat loop discovers kyn 100, then quickly discovers kyn 101 before anyone reads 100, kyn 100 is silently dropped and replaced by 101. This is perfect for timekeeping. When the REST API or the mining loop needs to know the time, they only care about the absolute latest time. They do not want a backlog of old times. Both heartbeat.rs and gossip.rs hold a Sender to this channel, pushing new kyns into it, while the rest of the daemon holds a Receiver that can instantly read the most current value without waiting. If a receiver of a watch channel is too slow, it never receives an error—it just skips to the newest value next time it looks.

2. The Broadcast Channel (tokio::sync::broadcast::Receiver) Used for Gossipsub messages. A broadcast channel is a multi-producer, multi-consumer channel where every consumer sees every message. If 10 governance votes arrive, the channel stores all 10 in a queue. If the gossip.rs loop falls behind (known as “lagging”), it might miss messages. In this case, Tokio intervenes to prevent memory exhaustion by dropping the oldest unread messages and throwing a RecvError::Lagged(_) error to warn the receiver that it missed data. Unlike the watch channel, we cannot simply drop older governance votes. Every vote must be processed in sequence. The broadcast channel ensures that as long as the processor doesn’t fall too far behind, every network shout is systematically ingested.

Part 1: The Heartbeat and Drand Loop (heartbeat.rs)

When the daemon initializes, it spawns start_heartbeat_loop using tokio::spawn. This places the loop onto the Tokio async runtime, allowing it to execute concurrently with the HTTP server and the P2P swarm.

Step 1: The Three-Second Tick and State Tracking The loop relies on an atomic integer to track time across threads safely. It initializes an Arc<AtomicU64> called last_known_live_kyn. -> See: kinetic-daemon/src/services/heartbeat.rs — Lines 20 to 21. It then enters an infinite loop {}. The pace of this loop is controlled by a Tokio interval set to wake up every 3 seconds (tokio::time::interval(Duration::from_secs(3))). Every time interval.tick().await completes, the loop evaluates the state of the network time.

Step 2: Securing the Clock (The Drand Fallback Mechanism) Kinetic’s reliance on Drand means time synchronization is a critical vulnerability. The user can configure the daemon to run in p2p_only mode, which instructs the daemon to avoid centralized HTTP requests and only acquire Drand kyns by listening to other peers via the gossip network. However, heartbeat.rs contains a crucial, hardcoded safety valve. It calculates what the current Drand kyn should be based on your computer’s local wall-clock time (expected_kyn). If the latest kyn in your local cache is more than 5 kyns behind the expected time (a drift of approximately 15 seconds), it triggers a fallback warning. It temporarily overrides the p2p_only restriction and forces an HTTP fetch from a Drand relay. -> See: kinetic-daemon/src/services/heartbeat.rs — Lines 30 to 49. This logic prevents a failure scenario where your node gets disconnected from the gossip mesh, loses track of time, and starts having all its names expire because it cannot sign current heartbeats.

Step 3: Broadcasting Time to the Swarm If the daemon is permitted to fetch (either because p2p_only is false, or the fallback triggered), it calls hb_drand.fetch_latest().await. If successful, and the kyn is new, it immediately broadcasts this kyn payload to the rest of the network over the GOSSIP_TOPIC_DRAND topic. This altruistic behavior is how the P2P mesh helps keep firewalled or p2p_only nodes synchronized. It also updates the rest of the local daemon using the watch channel (drand_kyn_tx_hb). -> See: kinetic-daemon/src/services/heartbeat.rs — Lines 51 to 72.

Step 4: The 30-Second Name Renewal Cycle While the loop ticks every 3 seconds to check the Drand clock, it only performs the heavy lifting of name renewals every 30 seconds. It tracks this using tokio::time::Instant. Before doing any renewals, it checks !kyn.is_usable_for_heartbeat(current_live). If the kyn is technically valid but functionally useless (e.g., it’s a fallback kyn from hours ago), the loop simply skips this cycle via continue. It will not attempt to renew names with an invalid or outdated timestamp, preventing accidental spam. -> See: kinetic-daemon/src/services/heartbeat.rs — Line 88. When the 30-second mark is hit, it queries the local Sled database for the list of names you own under the key DB_PREFIX_OWNED_NAMES. For each name retrieved, it constructs a fresh Heartbeat struct containing the name and the absolute latest Drand kyn. -> See: kinetic-daemon/src/services/heartbeat.rs — Lines 92 to 105.

Step 5: Offloading Cryptography (Avoiding Executor Blocking) To sign the heartbeat, the daemon uses the ML-DSA post-quantum signature scheme. Cryptographic signing is a CPU-intensive, heavy operation. If we ran this math directly inside the async loop, it would “block the executor”—meaning the Tokio thread would freeze, unable to handle incoming network requests until the math finished. To prevent this, the daemon uses tokio::task::spawn_blocking. This macro takes the CPU-heavy closure and moves it to a dedicated background thread pool specifically designed for synchronous blocking work. Once the thread pool returns the generated signature, it is packed into the heartbeat payload. The loop then spawns a tiny, lightweight async task to call hb_network.publish_heartbeat, sending the signed heartbeat directly into the Kademlia DHT for peers to discover. -> See: kinetic-daemon/src/services/heartbeat.rs — Lines 107 to 128.

Part 2: The Gossip Processor (gossip.rs)

Unlike the heartbeat loop, which operates on a rigid timer, the gossip processor is purely reactive. It is a listener.

Step 1: Message Ingestion from the Void The loop blocks on gossip_rx.recv().await. When a message arrives from the network swarm, it is unpacked into four components: the topic, the raw byte payload, a unique message_id, and the propagation_source (the PeerId of the node that handed us the message). -> See: kinetic-daemon/src/services/gossip.rs — Lines 19 to 23. If the channel buffer fills up because the loop is processing too slowly, it receives a Lagged error and simply continues to the next available message.

Step 2: Processing Governance Edicts If the incoming topic matches GOSSIP_TOPIC_GOVERNANCE, the daemon recognizes this as a high-stakes consensus message. It attempts to deserialize the payload into a SignedGovernanceMessage. It then locks the GLOBAL_GOVERNANCE_STATE—a system-wide Mutex holding the current network council votes—and feeds the message into the core process_governance_message function. -> See: kinetic-daemon/src/services/gossip.rs — Lines 24 to 38.

Step 3: Direct Database Injection for Premium Names If the governance message is cryptographically valid and results in a state change—specifically, GovernanceEffect::PremiumNameGranted—the background loop takes immediate, direct action. It manually crafts a NameRecord::Premium struct. It sets the granted_at time to the current UNIX epoch, leaves the signature arrays empty (since governance dictates the validity, not a user signature), formats the database key to match DB_PREFIX_REVEAL + name, and shoves the serialized bytes directly into the local Sled storage engine using storage.put(). -> See: kinetic-daemon/src/services/gossip.rs — Lines 52 to 74. This is a critical architectural shortcut. It bypasses the standard network registration flow because premium names are granted by network-wide governance consensus, not by individual user proof-of-work. By writing the record straight to the Sled database, the premium name becomes instantaneously usable and recognized by the local node. This design decision highlights a core philosophical point: governance is absolute. When the network votes on a state change, the node does not question it or route it through pending transaction queues. It treats it as an immutable fact and writes it to disk. Likewise, if the effect is GovernanceEffect::PremiumNameRevoked, it immediately issues a storage.delete() command to rip the name out of local memory.

Step 4: Processing Drand Gossip If the topic is GOSSIP_TOPIC_DRAND, the loop deserializes the payload into a RawKyn. Before trusting it, it must verify the BLS signature on the Drand kyn. Similar to the heartbeat loop’s signing process, it uses tokio::task::spawn_blocking to push the CPU-heavy verification math onto a background thread pool. If the verification passes, and the kyn is greater than the one currently held in the local cache, it saves the new kyn and alerts the rest of the daemon via the drand_kyn_tx_gossip channel. -> See: kinetic-daemon/src/services/gossip.rs — Lines 102 to 121.

Step 5: Network Feedback and Peer Scoring At the absolute end of processing any message (whether it was Governance or Drand), the loop makes a vital call to network_client.report_gossip_validation(...). This is a strict requirement imposed by the underlying libp2p networking stack. Libp2p maintains a complex trust-scoring system for all connected peers. If a peer sends us invalid governance votes, malformed JSON, or fake Drand kyns, we must report is_valid = false. Libp2p will then lower that peer’s reputation score, eventually disconnecting and banning them if they continue to spam. This feedback loop is how the network organically protects itself from malicious actors. -> See: kinetic-daemon/src/services/gossip.rs — Lines 101 and 122.

Key Pieces

start_heartbeat_loop

  • Location: kinetic-daemon/src/services/heartbeat.rs — Lines 11 to 134
  • What it does: The primary clock driver and survival mechanism. It ensures Drand is fresh and continuously renews the user’s names on the DHT.
  • Why it matters: If this function crashed or stalled, all names owned by the daemon would expire and be swept from the network within minutes, resulting in total data loss for the user.

The last_known_live_kyn Atomic Tracker

  • Location: kinetic-daemon/src/services/heartbeat.rs — Line 20
  • What it does: Uses Arc<AtomicU64> to track the highest valid Drand kyn seen so far.
  • Why it matters: An atomic integer allows thread-safe, lock-free read and write access. This is significantly faster and eliminates the risk of deadlocks that would come from wrapping a simple number in a Mutex. By using Ordering::Relaxed, it checks the time with near-zero overhead. When an updated kyn arrives, it uses lklr.store(kyn.kyn, Ordering::Relaxed) to overwrite the old value. This allows the inner asynchronous spawned tasks inside the loop to reference the exact same state without needing to pass heavy locks back and forth across thread boundaries. It guarantees the loop always references a monotonically increasing time source.

The P2P Drand Fallback Threshold

  • Location: kinetic-daemon/src/services/heartbeat.rs — Lines 39 to 45
  • What it does: If the daemon is in p2p_only mode but observes that its internal clock has fallen 5 kyns (about 15 seconds) behind expected real-world time, it panics and falls back to HTTP polling.
  • Why it matters: Gossip networks can fragment, causing partitions where nodes stop hearing updates. This safety hatch ensures the node doesn’t become paralyzed if it loses its connection to the Drand gossip mesh.

start_gossip_processor

  • Location: kinetic-daemon/src/services/gossip.rs — Lines 4 to 125
  • What it does: The ever-listening ear for network-wide broadcasts. It acts as a filter, applying valid state changes and discarding junk.
  • Why it matters: This is the exclusive pathway for a local daemon to stay up to date with global governance decisions. Without it, the node operates in a permanent blind spot.

tokio::task::spawn_blocking Integration

  • Location: kinetic-daemon/src/services/heartbeat.rs (Line 112) & gossip.rs (Line 108)
  • What it does: Bridges the asynchronous Tokio runtime with synchronous, CPU-bound cryptographic operations.
  • Why it matters: If you attempt to verify a BLS signature directly inside an async task, you block the executor. No other async tasks (like handling incoming HTTP API requests) can run until the math finishes. spawn_blocking solves this by ejecting the work to a separate thread pool.

How This Connects to the Rest of Kinetic

  • CROSS-CRATE: kinetic-core — Both loops are dependent on the core cryptographic and state definitions found in Stage 7. Specifically, they rely on kinetic_core::drand::DrandClient, kinetic_core::types::Heartbeat, and the GLOBAL_GOVERNANCE_STATE.
  • CROSS-CRATE: kinetic-network — The NetworkClient (covered in Stage 8) is injected into these loops. The heartbeat loop relies on it to execute .publish_heartbeat(), and the gossip loop relies on it to execute .report_gossip_validation() and .broadcast_gossip().
  • Storage Subsystem: These background tasks are intimately, perhaps dangerously, tied to the local Sled database. They must read DB_PREFIX_OWNED_NAMES to know what to renew, and the gossip loop directly writes to DB_PREFIX_REVEAL when governance dictates a state change.

Quick Reference

  • Heartbeat Tick Frequency: Wakes up every 3 seconds to ensure Drand is current.
  • DHT Renewal Interval: Actually constructs and publishes name renewals to the DHT every 30 seconds.
  • P2P Fallback Trigger: 5 kyns (approximately 15 seconds) behind expected wall-clock time.
  • Gossip Topics Handled: Listens to kinetic_drand_v1 and kinetic_gov_v1.
  • Watch Channel (watch::Sender): Keeps only the newest value, dropping older ones. Used for Drand time updates.
  • Broadcast Channel (broadcast::Receiver): Keeps all values in a queue. Used for ingesting network chatter.
  • Crypto Threading: Both loops mandate the use of spawn_blocking to ensure ML-DSA signing and BLS verification do not freeze the background runtime.
  • Libp2p Requirement: All processed gossip messages MUST be validated and reported back to the network client to maintain peer scoring and prevent spam.

Open Questions / Things to Revisit

  • Direct Storage Coupling: In gossip.rs, the loop directly formats a raw database key string (DB_PREFIX_REVEAL + name) and shoves it straight into the Sled storage engine. This works, but it breaks encapsulation. If the storage layout or prefix architecture ever changes in kinetic-storage, this background gossip service will break silently. This injection logic ideally belongs inside an abstracted method in the storage layer, not the daemon layer.
  • Hardcoded Fallback Thundering Herd: The 5-kyn threshold in heartbeat.rs is hardcoded across all nodes. In a rare scenario where the Drand network itself goes offline or stutters globally, all daemons on the network might simultaneously decide they are “behind” and launch massive DDoS-style HTTP requests to the Drand relay servers at the exact same second. Adding jitter or a randomized backoff here would be safer.
  • Heartbeat Congestion and Task Explosion: If a user registers and owns 10,000 names, the loop will attempt to sign and publish 10,000 individual heartbeats every 30 seconds. The loop currently spawns a brand new Tokio task for each individual publish operation. This could easily lead to a massive task explosion, exhausting the node’s resources and flooding the Kademlia DHT client. Batching heartbeats into a single payload or pacing the network requests might become necessary as node portfolios grow.