Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

API VDF Registration Layer (Part 1)

Crate: kinetic-daemon Stage: 9 Reading time: 45 minutes Depends on: docs/learn/core/06_vdf_consensus.md, docs/learn/network/04_dht_publish.md

What Is This?

This document provides a comprehensive, line-by-line architectural breakdown of the first half of the kinetic-daemon/src/api/vdf.rs file. Specifically, it covers lines 1 through 367 of the source code. This file serves as the primary HTTP gateway for the Kinetic daemon. It is the exact endpoint hit when a user, an automated client, or the web dashboard attempts to register a brand new Kinetic name. Because Kinetic operates without a centralized naming authority, names cannot simply be purchased or inserted into a database. Instead, they must be cryptographically earned. This earning process is achieved by expending significant computational effort. The effort is quantified and proven using a Verifiable Delay Function (VDF). This file is the specific location in the codebase where that massive computational process is initiated. It acts as a complex background job orchestrator. It receives a fast, synchronous HTTP request from the user. It translates that request into a long-running, asynchronous lifecycle. This lifecycle can take anywhere from a few minutes for long names, to several months for very short, contested names. During this extended lifecycle, the daemon must manage internal state. It must enforce strict concurrency limits to prevent the host machine from crashing. It must interface with external randomness beacons to prove the timeline of the registration. It must perform heavy cryptographic evaluations without locking up the rest of the node. Finally, it must interact with the Distributed Hash Table (DHT) to publish the proofs. This specific document focuses exclusively on the handle_vdf_register function. This function manages the initial parsing, the spawning of the background worker, and the execution of the entire state machine.

Why Kinetic Needs This

In a traditional web architecture, registering a username is trivial. The API receives a request, checks a database for availability, and performs an immediate insert. The entire transaction completes in milliseconds. The HTTP connection remains open, and the user gets instant feedback. Kinetic’s architecture shatters this paradigm. Because Kinetic uses VDFs to enforce a verifiable time delay, registration is an heavy operation. A single name registration request might force a CPU core to run at 100% utilization for weeks on end. If the Kinetic daemon attempted to handle this within the standard HTTP request-response cycle, it would fail catastrophically. The HTTP connection from the client would instantly time out. More importantly, the async executor running the Axum web server would lock up completely. The entire node would become unresponsive to all other network traffic, API requests, and peer communications. Therefore, Kinetic requires a dedicated subsystem to decouple the fast HTTP request from the slow cryptographic work. This file provides that exact decoupling mechanism. It allows the dashboard to say, “Start registering this name.” The API responds immediately with an HTTP 200 Success and a tracking task ID. The user can then navigate away, while the daemon handles the heavy lifting safely in the background.

Furthermore, this file acts as the node’s frontline defense against resource exhaustion attacks. VDF evaluations are intentionally brutal on system resources. If a malicious actor on the local network, or a buggy automated script, sent one hundred simultaneous registration requests, the operating system would try to spawn one hundred concurrent VDF threads. This would instantly starve the host machine of CPU cycles and memory. The node would crash, and it could potentially take down the underlying host operating system. By implementing strict state locks and concurrency semaphores, this code ensures node stability. It guarantees that no matter how many requests are received, the daemon will only ever process one VDF at a time.

Finally, this file is the implementation site of a critical consensus fix known as the C-1 fix. It meticulously orchestrates the exact sequence of cryptographic operations. It dictates exactly when the commitment is generated, when the proof is computed, and when the reveal is broadcast. This precise ordering ensures that legitimate registrations are never rejected by the network due to DHT data pruning windows. Simultaneously, it preserves the mathematical guarantees that prevent front-running attacks by other nodes.

How It Works

The registration process is initiated when a POST request hits the /vdf/register API endpoint. The Axum router immediately passes control to the handle_vdf_register function. The execution then flows through a structured, multi-stage pipeline.

Step 1: Request Validation and Role Verification

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/main.rs` — Lines 43 to 64
}

The function signature uses Axum extractors to parse the incoming JSON body. This body is mapped into a VdfRegisterRequest struct. Before any processing begins, the daemon performs crucial authorization checks. It checks the caller’s role via the injected Extension<Role>.

Important

VDF registration requires elevated privileges. If the caller does not possess the VDF or Admin role, the request is immediately aborted. An HTTP 403 Forbidden is returned. This ensures that unauthorized clients cannot force the node to burn expensive CPU cycles.

Next, the requested name string is sanitized. It is passed through kinetic_core::types::normalize_name. This ensures the name follows the strict lowercase, alphanumeric requirements of the protocol. It is then validated using is_valid_apex_name. This confirms it is a structurally valid top-level identifier. If the name contains invalid characters, an HTTP 400 Bad Request is returned.

The code also inspects the optional iterations parameter. Users can theoretically request a higher iteration count to burn more time intentionally. This makes their registration more secure against attackers with specialized hardware.

Warning

However, the daemon enforces a hard safety limit of 10,000,000 iterations (MAX_USER_ITERATIONS). If a user requests more than this, the request is rejected. This prevents users from accidentally bricking their node by requesting a decade-long computation.

Finally, a unique UUID string is generated using uuid::Uuid::new_v4(). This UUID serves as the primary key to track this specific task instance throughout its lifecycle.

Step 2: Concurrency Control and State Locking

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/api/vdf.rs` — Lines 67 to 104
}

Before the background worker is spawned, the daemon must ensure the environment is safe. It acquires a blocking mutex lock on the vdf_tasks HashMap. This map resides within the shared, global ApiState. This lock is critical to the node’s stability. It prevents race conditions if multiple HTTP requests arrive at the exact same millisecond.

While holding the exclusive lock, the code iterates through all existing tasks. It specifically looks for any task where the progress is less than 100% and no error flag is set.

Important

If it finds even a single active task, it rejects the new request. It returns an HTTP 409 Conflict. This enforces the strict protocol rule: The Kinetic daemon will process exactly one VDF registration at a time.

To prevent the state map from growing infinitely and causing a memory leak, the daemon performs routine housekeeping. It filters out completed or errored tasks to keep the map clean. If the total number of historical tracked tasks exceeds 50, it rejects the new request. It returns an HTTP 429 Too Many Requests. This forces the user or client to wait until older tasks are cleared from the state. Once all safety checks pass, the new task is inserted into the map. Its initial status is set to “Initializing”. Its progress is set to 0%. The mutex lock is then deliberately dropped. The API request is now cleared to spawn the worker without holding up other threads.

Step 3: Spawning the Worker and Fetching External Randomness

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/api/vdf.rs` — Lines 113 to 124
}

The API handler invokes tokio::spawn. This creates a detached, asynchronous background task. The moment this task is spawned, the HTTP response is sent back to the user. The background task continues executing independently of the HTTP connection.

Inside this background worker, the first action is to interface with the Drand network. The code initializes a new DrandClient. It provides the client with a storage_clone so it can cache beacon data if needed. It then calls fetch_latest().await.

Important

This is a non-negotiable requirement of the Kinetic consensus protocol. To prove that a VDF was computed after a specific point in time, the commitment must incorporate unpredictable randomness. This prevents attackers from pre-computing VDFs years in advance.

The background worker updates its status in the shared state map to “Fetching Drand beacon”. It then awaits the network response. If the Drand network is unreachable, the task errors out immediately. It updates its state so the user’s dashboard can display the exact network failure.

Step 4: Generating the Cryptographic Commitment Hash

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/api/vdf.rs` — Lines 136 to 173
}

With the Drand randomness secured, the daemon must generate the commitment hash. This commitment is the mathematical lock that prevents front-running on the network. If the daemon simply broadcasted the finished VDF proof in plaintext, it would be vulnerable. Any malicious node could intercept it, substitute their own public key, and steal the name before the network finalized it.

To prevent this, the daemon generates a unique, opaque, one-way hash locally. First, it loads the node’s local identity keypair from the filesystem at identity.key. It extracts the raw public key bytes from this pair. Next, it uses the operating system’s cryptographic random number generator (getrandom::fill).

Warning

It creates a secure 32-byte salt. If this OS-level RNG fails, the task aborts for security reasons.

The code then instantiates a SHA-256 hasher. It feeds the hasher the following precise ingredients in a very specific order:

  1. The requested FQDN string.
  2. The generated 32-byte secret salt.
  3. The Drand randomness signature bytes.
  4. The node’s raw public key bytes.

The resulting 32-byte hash is wrapped inside a Commitment struct. Because the salt and the public key are kept secret within the daemon’s local memory, this hash is impossible to reverse. It locks the node’s identity to the specific name. Crucially, it accomplishes this lock without revealing what the name actually is to the rest of the network.

Step 5: The Heavy VDF Evaluation and Threading Isolation

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/api/vdf.rs` — Lines 175 to 219
}

This step represents the actual execution of the Proof of Work. The daemon calculates the required number of iterations. This is based on the consensus math scaling rules for the specific length of the name. It takes the maximum of the strict consensus requirement and the user’s requested override. The daemon then instantiates the core ChiaVdfEngine.

Before executing the math, the daemon acquires a permit from the vdf_semaphore. This acts as a secondary, engine-level defense mechanism. Even though the HTTP layer checked for active tasks, the semaphore ensures global safety across the daemon. If any other part of the daemon codebase attempts to spawn a VDF, they will be blocked here. This prevents concurrent executions from ever crashing the entire host system.

Important

The most critical architectural decision in this entire file is the use of tokio::task::spawn_blocking. The Chia VDF evaluation is a tightly optimized, purely CPU-bound mathematical loop. It does not contain any .await yield points. It will never yield control back to the Tokio async runtime voluntarily. If this engine were run directly on an async worker thread, that thread would freeze completely. It could freeze for weeks. All other asynchronous tasks assigned to that thread would stall instantly. This would cause a failure of the node’s networking and API layers. By wrapping the engine execution in spawn_blocking, Tokio moves the heavy computation. It moves it to a dedicated thread pool specifically designed for synchronous, blocking operations.

The main async task then simply .awaits the result from that separate thread pool. Once the proof is successfully returned, the semaphore permit is dropped. This frees the engine for any future requests.

Step 6: The C-1 Consensus Fix and Deferred Broadcast

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/api/vdf.rs` — Lines 221 to 254
}

This section of code implements a massive architectural fix for the Kinetic protocol. In the original protocol design, the commitment hash was broadcast to the DHT before the VDF evaluation started. The original logic seemed sound: announce your intent, do the work, then reveal the proof. However, this ignored the reality of distributed peer-to-peer systems. DHT nodes routinely prune old records to save disk space and maintain performance. If a user attempted to register a short name, the VDF might take two months to compute. By the time the two months elapsed, the original commitment broadcast had been purged from the global DHT entirely. When the user finally published their reveal, the network validators would look for the initial commitment. They would fail to find it. They would then reject the registration as invalid. This systemic flaw was known as the C-1 bug.

Note

The solution implemented here is called Deferred Broadcast. The commitment hash is generated at the very beginning of the pipeline. This ensures it incorporates the correct, early Drand timestamp. However, it is held private in local memory while the VDF runs. Only after the VDF proof is successfully computed does the daemon serialize the commitment. It is then broadcast to the DHT via publish_redundant_payload.

This guarantees that the commitment is fresh on the network when the reveal arrives. However, the consensus rules dictate that a commitment must “mature” before a reveal is accepted. This maturation window ensures it has propagated widely enough to prevent last-second front-running attacks. Therefore, after broadcasting the commitment, the daemon sleeps. It calculates the exact wait time mathematically. It multiplies CONSENSUS_MINIMUM_COMMIT_AGE_KYNS by the DRAND_PERIOD. It then adds a 2-second safety buffer to account for network latency and clock drift. During this sleep, the commitment propagates through the DHT. The commitment hash remains opaque during this time. Therefore, front-running is still impossible during this maturation window.

Step 7: Publishing the Reveal and Finalizing State

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/api/vdf.rs` — Lines 258 to 365
}

Once the maturation sleep concludes, the daemon constructs the final Reveal payload struct. This struct is the ultimate cryptographic proof of registration for the network. It contains the FQDN, the secret salt, the exact Drand timestamp, and the Drand signature used. It also contains the total number of iterations computed. It contains the raw VDF proof bytes. It contains the node’s public key.

Crucially, it also includes a default DnsZone payload embedded inside it. This ensures that the moment the name is registered globally, it has a valid DNS zone attached to it. Even though the zone is empty, it is ready for the user to configure immediately. The entire Reveal struct is then serialized into a standard byte array format. The daemon uses its ML-DSA private key to generate a post-quantum signature over this specific payload. This finalizes the struct and irrevocably proves ownership.

The complete, signed Reveal is serialized to JSON. It is broadcast to the DHT using the exact same redundant publication mechanism as the commitment. If the network accepts the broadcast without throwing errors, the registration is effectively complete from a global consensus perspective.

However, the daemon must still update its own local internal state. It acquires a lock on OWNED_NAMES_LOCK. It opens the local storage database specifically pointing at DB_PREFIX_OWNED_NAMES. It appends the newly registered FQDN string to the list of owned names. This ensures the user’s dashboard will instantly reflect the successful acquisition without needing to query the network. It implements a safety bound here: if the list exceeds 10,000 names, it deliberately truncates the oldest entries. This prevents the local database array from bloating infinitely and causing parsing delays. Finally, the daemon creates the actual physical zone file on the local filesystem. It writes the empty JSON structure to disk inside the configured zones_dir. This allows the user to immediately begin editing their DNS records via the local dashboard interface. The background task status is finally updated to “Complete” at 100% progress. The background thread then gracefully terminates, freeing all resources.

Key Pieces

NameRenewRequest & VdfRegisterRequest

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/api/vdf.rs` — Lines 15 to 30
}

These are the data structures responsible for deserializing the incoming JSON payloads from the HTTP client. They are intentionally minimalist in design to keep the API surface area small. They require only the name string to be operated on. They accept an optional iterations field for advanced users. The iterations field allows developers or secure users to force the VDF to run longer than the consensus minimum. This provides an extra layer of security against attackers with specialized, fast hardware. This field is capped by the daemon’s internal validation logic to prevent self-inflicted denial of service.

handle_vdf_register

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/api/vdf.rs` — Lines 38 to 367
}

This is the master orchestration function for the entire name registration lifecycle. It operates across two distinct execution domains simultaneously. First, it handles the synchronous, immediate HTTP request-response cycle for the API. Second, it manages the asynchronous, long-lived background task domain for the actual cryptographic work. Its primary architectural responsibility is to safely bridge these two isolated worlds. It must accomplish this without ever compromising the stability or responsiveness of the host node.

The Task State Lock (state.vdf_tasks.lock())

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/api/vdf.rs` — Lines 69 to 104
}

This standard library Mutex is the node’s absolute primary defense against state corruption. It is also the primary defense against CPU resource exhaustion. By locking the shared state map before checking the active task count, it ensures absolute execution atomicity. Without this lock, two simultaneous HTTP requests could theoretically both read the active task count as zero simultaneously. Both requests would then proceed to spawn massive VDF background tasks at the same time. This would instantly violate the concurrency limits of the system and crash the node.

tokio::task::spawn_blocking

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/api/vdf.rs` — Lines 197 to 216
}

This is a vital Tokio framework API utilized to protect the asynchronous runtime. The Rust async model relies on cooperative yielding to function efficiently. Tasks must periodically .await to give the CPU back to the executor so it can run other pending tasks. The Chia VDF engine inherently does not yield. It is a pure, unbreakable mathematical loop. If run directly, it would hijack the executor thread and permanently. spawn_blocking forces this rogue, non-yielding computation to a separate thread pool. This preserves the health and responsiveness of the core async system.

The C-1 Fix Sleep Mechanism

#![allow(unused)]
fn main() {
-> See: `kinetic-daemon/src/api/vdf.rs` — Lines 247 to 254
}

This small block of code represents a massive architectural evolution in the Kinetic protocol. By calculating the exact required maturity time (CONSENSUS_MINIMUM_COMMIT_AGE_KYNS * DRAND_PERIOD), the daemon is precise. By forcing an async sleep, the daemon ensures that the locally-held, freshly-broadcast commitment has enough time to propagate globally. It ensures it satisfies the network validators before the reveal is sent. This bypasses the historical issue where long-running VDFs would cause their commitments to expire before the proof could even be computed.

How This Connects to the Rest of Kinetic

The API VDF layer is dependent on multiple other crates within the Kinetic ecosystem. It cannot function in isolation. It acts as the central coordinator, pulling in functionality from across the entire codebase to execute the lifecycle.

  • CROSS-CRATE: The initial string validation relies on kinetic_core::types::normalize_name and kinetic_core::types::is_valid_apex_name. These functions enforce the network’s strict naming conventions. (See Stage 7).
  • CROSS-CRATE: The external randomness required for the cryptographic commitment is fetched using the kinetic_core::drand::DrandClient. (See Stage 7).
  • CROSS-CRATE: The actual time-burning mathematical evaluation is handed off to kinetic_vdf::ChiaVdfEngine. The daemon code here does not understand the math; it simply feeds the engine the challenge hash and the iteration count. (See Stage 6).
  • CROSS-CRATE: All communication with the global network, including broadcasting the Commitment and the final Reveal, is routed through the network_clone.publish_redundant_payload() method. This is provided by the kinetic-network crate. (See Stage 8).
  • CROSS-CRATE: The cryptographic signing of the final Reveal struct utilizes the ml_dsa post-quantum signature schemes. These are deeply integrated into kinetic-core. (See Stage 7).

Quick Reference

For rapid recall of the constraints and workflows defined in this specific file:

  • Role Requirement: The API caller MUST possess either the VDF or Admin role to successfully trigger this endpoint.
  • Iteration Hardcap: The maximum allowed user-requested iterations is capped at 10,000,000. Anything higher is instantly rejected with an HTTP 400.
  • Concurrency Limit: The daemon enforces a strict limit of exactly 1 active VDF registration task at any given moment to protect CPU resources.
  • Task Memory Limit: The in-memory tracking map retains a maximum of 50 task histories. Once this is reached, it begins pruning older entries or rejecting new requests entirely.
  • Workflow Order: Request Validation -> Fetch Drand Randomness -> Generate Secret Hash Locally -> Compute VDF (On a Blocking Thread) -> Broadcast Hash to DHT -> Sleep for Maturity Window -> Broadcast Final Reveal -> Update Local Database.
  • Async Safety: All VDF mathematical evaluation is isolated using Tokio’s spawn_blocking mechanism to prevent executor starvation and node lockup.
  • Local Artifacts: Upon success, the name is appended to DB_PREFIX_OWNED_NAMES in local storage, and a blank .json configuration file is generated in the zones_dir.

Open Questions / Things to Revisit

While this architecture successfully handles the C-1 bug and protects the async runtime, there are several areas that may require future architectural review by the original author (Saif):

  • Lack of Persistent State for Active Tasks: The entire vdf_tasks state tracking mechanism is held purely in volatile memory within the Axum ApiState. If a user is attempting to register a 3-character name that requires two months of VDF computation, and the daemon is restarted for a routine software update on day 59, the entire process is permanently lost. There is currently no checkpointing or serialization mechanism implemented here to resume a partially completed VDF evaluation from disk. This is a significant UX vulnerability for long-running registrations.
  • Redundant Concurrency Checks: The code currently employs a double-layer defense against concurrency. First, it locks the state map and counts active tasks directly (lines 71-82). Second, it acquires a permit from a vdf_semaphore right before mathematical execution (line 189). While defense-in-depth is generally valuable, the semaphore might be functionally redundant for HTTP-initiated tasks. Unless other internal daemon processes are also permitted to bypass the HTTP API and directly request VDF evaluations, the initial state lock should be sufficient to prevent concurrency.
  • Error Granularity in Blocking Tasks: When the tokio::task::spawn_blocking call fails, the error is caught generically as a “Task panic” and written to the status map. The daemon makes no distinction between a mathematical error thrown gracefully by the ChiaVdfEngine and a thread-spawning failure thrown by the underlying operating system. More granular error matching and handling here could significantly improve dashboard diagnostics for the end user when registrations fail unexpectedly.