Engineering a Multi-Exit Flow Router with XDP/eBPF

Manuscript status: This paper presents an anonymized engineering case study reconstructed from development issues, change history, implementation review, and recorded integration-test descriptions. Historical test reports are not presented as independently reproduced experiments. Hardware performance, statistical confidence, and unrestricted concurrency correctness remain subjects for the evaluation specified below.

Abstract

A flow-aware multi-exit router must do more than select an interface quickly. It must interpret incomplete observations, execute configurable policies within kernel verification constraints, and preserve routing intent when accelerated forwarding requires assistance from the conventional network stack. This paper presents the design and engineering evolution of an IPv4 routing prototype implemented with Rust and XDP/eBPF. TCP observations are associated through normalized bidirectional endpoint keys, while discovery type, observed path visibility, and routing decisions remain distinct. Operator expressions are compiled into bounded instructions rather than interpreted as a general-purpose language in the kernel. The development process exposed several failures that were not apparent from host-side tests: verifier-invisible bounds checks, excessive verification state exploration, stale embedded bytecode, logical-to-physical VLAN mismatches, accidental forwarding of locally addressed traffic, and loss of exit policy during neighbor resolution. To address the last problem, the prototype transfers a selected exit identifier through XDP metadata to a TC ingress classifier, which establishes a routing mark for the Linux routing policy database. This enables kernel-assisted neighbor resolution without intentionally recomputing the exit policy. The paper develops a conditional correctness argument, separates documented engineering outcomes from outstanding empirical claims, and specifies experiments covering forwarding, recovery, concurrency, and performance. Its central finding is that accelerated routing requires explicit contracts across observation, verification, device transmission, and slow-path boundaries; correctness at one boundary does not establish correctness at the next.

Keywords: XDP; eBPF; software routing; flow observation; policy routing; neighbor resolution; fast-path/slow-path consistency.

1. Introduction

1.1 Motivation

Consider a Linux router connected to several exits with different latency, capacity, and operational cost. An administrator may wish to reserve a low-latency exit for connections that remain small, while moving sustained transfers to a higher-capacity exit. The challenge is that a connection's eventual size is unknown at its first packet. Information-agnostic scheduling research, including PIAS, demonstrates the value of adapting treatment to service already received rather than assuming advance knowledge of flow size ^[1]. In this paper, that principle motivates a configurable routing policy, not a claim of a new scheduling algorithm.

XDP provides a suitable execution point for early packet processing while retaining integration with Linux. Its original design explicitly combines programmable processing in driver context with selective use of the conventional network stack ^[2]. However, this integration does not automatically preserve application-specific decisions. A packet returned to the kernel carries only the information that the fast path actually preserves.

The motivating failure arose when an accelerated router selected one exit but could not resolve its next-hop MAC address. A bare XDP_PASS handed the packet to ordinary Linux routing without the selected exit. Source-based routing rules configured for the router's own addresses did not match the transit source. Consequently, the packet followed the management route. Because it never traveled toward the selected gateway, it also failed to trigger the neighbor resolution needed by subsequent fast-path packets.

This incident changes the framing of the problem. The relevant question is not merely whether Linux can forward a packet after XDP stops processing it. It is whether Linux will continue the same routing decision.

1.2 Research Questions and Contributions

The study addresses four questions: how to represent TCP behavior under partial observation; how to express operator policies without exceeding verification and resource constraints; how to preserve an exit decision across neighbor-resolution fallback; and what evidence is necessary to distinguish implementation progress from actual forwarding correctness.

The first contribution is an observation model that separates connection identity, discovery, visibility, and route selection. The second is an implementation architecture that makes configuration complexity and verification boundaries explicit. The third is a narrowly scoped, policy-preserving neighbor slow path, together with a statement of its assumptions and remaining failure modes. The fourth is an evaluation methodology derived from failures in which unit tests, successful attachment, or redirect counters provided insufficient evidence.

These are system-design and engineering contributions. Neither byte-threshold demotion, eBPF tail calls, XDP metadata, nor routing marks are claimed as new primitives. The contribution lies in their composition, the failures that motivated that composition, and the precision of the resulting contracts.

1.3 Scope and Evidence

The prototype targets IPv4 and uses stateful policy evaluation for TCP. UDP and IPv4 ICMP can be assigned fixed exits; other recognized non-TCP traffic uses the configured default. Application-layer identification, machine-learning classification, automatic link optimization, NAT, and complete IPv6 support are outside its implemented scope.

The engineering narrative distinguishes an initial requirement from a subsequently merged behavior. In particular, an early all-errors-fail-open design was superseded for neighbor misses, and an early SYN-only tracking model was replaced by observation recovery. A completed issue or merged change establishes development history, not a universal operational guarantee. Section 6 therefore distinguishes recorded integration outcomes, source-level deductions, and measurements still required.

2. Background and Related Work

2.1 Accelerated Processing and Kernel Assistance

XDP supplies an early programmable packet-processing path; TC provides a later hook operating on the skb representation. The kernel documents that TC-BPF can access custom XDP metadata through data_meta after an XDP_PASS transition ^[3]. This provides a communication mechanism, but the producer and consumer must agree on what the metadata means.

Linux also exposes routing and neighbor information through bpf_fib_lookup(). Its result distinguishes successful resolution from conditions requiring additional processing. A direct lookup into a specified table uses BPF_FIB_LOOKUP_DIRECT together with BPF_FIB_LOOKUP_TBID; a routing mark is separately usable by the routing policy database [8,10]. These facilities make selective kernel assistance possible without placing the entire routing policy in TC.

2.2 Dynamic Flow Treatment

Hedera schedules large flows across data-center paths using network-level information ^[4]. PIAS demotes traffic across priority queues according to bytes already transmitted, without requiring the final flow size in advance ^[1:1]. LetFlow uses flowlet switching and its adaptive behavior to balance traffic in asymmetric data-center networks ^[5].

The proposed router operates at a different scope. It applies local administrator-defined predicates and changes a logical exit identifier. It does not implement PIAS queue scheduling, Hedera's global scheduling, or LetFlow's flowlet boundaries. Moreover, asymmetry in this paper primarily concerns missing directional observations, whereas LetFlow studies asymmetry in available network paths. These distinctions prevent superficially similar terminology from implying equivalent algorithms or guarantees.

2.3 Consistency as a Cross-Boundary Property

Reitblatt et al. distinguish per-packet and per-flow consistency during network updates ^[6:1]. Their work shows why correct endpoint configurations are insufficient when intermediate processing states are inconsistent. The present study concerns a different transition: an individual packet moves from XDP to kernel forwarding after a routing decision has already been made.

The proposed mechanism does not establish consistent network-wide updates. It also intentionally permits a connection's exit to change over time. Its narrower objective is that a packet requiring neighbor assistance should not lose the exit already selected for that packet.

3. System Model and Routing Semantics

3.1 Logical Exits and Deployment Assumptions

Let $E = {e_0, \ldots, e_{n-1}}$ be the configured exits. Each exit has a logical identifier, a routing table, an expected logical egress device, and a transmission target. These identities are deliberately separate. A VLAN subinterface may be the correct FIB egress while its underlying physical device is the appropriate XDP transmission target.

One interface role authorizes routing decisions. Additional roles may observe returning traffic. A management path remains available independently of the selected transit exits. Deployments must provide suitable source-address reachability, return routing, MTU configuration, and compatible XDP/TC attachment behavior.

The correctness domain is intentionally restricted to the validated packet forms and deployment assumptions. In particular, unchanged transport endpoint identity is a prerequisite for ordinary TCP continuity across exit migration; this router does not implement a transport extension that changes TCP endpoints ^[7].

3.2 Identity, Discovery, and Visibility

For TCP, the key is a canonical ordering of two address-port endpoints:

endpoint_a = (source IPv4 address, source port)
endpoint_b = (destination IPv4 address, destination port)

key = (min(endpoint_a, endpoint_b), max(endpoint_a, endpoint_b))

The protocol is implicit in the TCP-only map. Canonical ordering associates reverse packets without identifying which endpoint initiated the connection.

Discovery records the first packet observed in the current state epoch: Syn, SynAck, or Midstream. Visibility records whether decision-path traffic, return-path traffic, or both have actually been observed. Neither field is a complete TCP state machine. Observing SYN-ACK is evidence about the packet encountered, not authentication of a peer or proof that the router witnessed a valid handshake. TCP endpoint state and sequence processing are considerably richer than this observer model ^[7:1].

The visibility mask is monotonic within one state epoch: seeing the other role adds information. However, “both roles observed” does not establish that all packets were captured. Conversely, a zero return counter cannot distinguish an absent return stream from an unobserved return path.

3.3 Accounting and Observation Epochs

The prototype's byte counter uses the received frame length visible in the XDP context, not application payload or unique delivered TCP bytes. Retransmitted packets therefore contribute again, and visible encapsulation can affect the counter. The fixed parser prefix is only a decoding buffer; its length must not replace the actual observed packet length in accounting.

age measures time since the current observation state was created. idle measures the interval preceding the current packet. Byte and packet predicates include the packet currently being processed. After expiration, eviction, or restart, the new state begins another observation epoch: its age is not the connection's true lifetime.

The state table uses a bounded LRU hash. Linux documents automatic eviction under capacity pressure ^[8]. For this design, bounded memory therefore comes at the cost of potentially losing routing history. Increasing capacity reduces exposure to eviction but does not prove a bounded classification error or uninterrupted route retention.

3.4 Bootstrap Is Not Generic Fallback

The early design overloaded the default exit with unrelated purposes: ordinary stateless forwarding, storage failures, midstream recovery, and the first response of a reverse-open TCP connection. Separating a TCP bootstrap exit from generic fallback removes this ambiguity.

Persistent state matters. Sending only a first SYN-ACK through the bootstrap exit, while leaving its subsequent payload untracked, would introduce an unintended immediate route change. Conversely, promoting all recovered traffic to the bootstrap exit would reward state loss by moving established bulk connections onto the reserved path.

For a live state, SYN-ACK does not reset the route. A decision-path bare SYN is treated as a tuple-reset signal by the minimal observer; this is a policy heuristic, not a substitute for TCP sequence-number validation.

3.5 Ordered Policies and Conditional Invariants

A rule consists of a predicate and an exit identifier. The first matching predicate determines the route; no match preserves the current route. Return-path processing updates observations without intentionally evaluating policy. Consequently, a return packet that crosses a byte threshold affects the next decision-path packet rather than causing immediate forwarding from the observation hook.

Three invariants guide the design. First, a successful fast-path resolution must correspond to the selected logical exit. Second, an eligible neighbor-miss packet must carry that decision into the kernel path. Third, failure to encode the decision must not silently become an unmarked neighbor fallback.

These are not unrestricted concurrency or availability guarantees. The return-path invariant describes the sequential transition function; shared-map update races are analyzed in Section 7. Similarly, preserving a selected exit does not guarantee delivery when its gateway is unreachable.

4. Architecture and Bounded Policy Compilation

4.1 Separation of Responsibilities

The implementation separates a userspace control plane, shared pure logic, and kernel adapters. Userspace validates configuration, compiles predicates, resolves interfaces, populates maps, installs policy-routing rules, and manages attachments. Shared logic defines observation transitions, instruction evaluation, forwarding-result classification, and fixed-layout data structures. Kernel adapters access packet memory, invoke helpers, persist map values, and perform the XDP/TC handoff.

Only a logical exit identifier crosses from policy evaluation into forwarding. The forwarding component does not inspect rule names, byte thresholds, or application descriptions. Likewise, TC does not repeat flow lookup or rule evaluation: it consumes a completed decision.

This decomposition makes host-side tests useful without treating them as proof that the corresponding kernel path is reachable, accepted, or capable of transmission.

4.2 Configuration as a Compilation Boundary

CEL provides a useful expression syntax and a parse/check/evaluate model ^[9]. The prototype accepts only a restricted subset and lowers it into its own instruction format. It does not run a complete CEL interpreter in the kernel.

Supported predicates compare observed bytes, packets, age, idle time, and current packet addresses or ports. Explicit units distinguish byte thresholds from durations. Boolean operators combine comparisons, and IPv4 prefix membership is a primitive operation. Unsupported constructs and inconsistent units are rejected rather than silently approximated.

A postfix instruction sequence avoids expanding arbitrary Boolean expressions into disjunctive normal form, whose representation can grow rapidly through distributive expansion. It also avoids a recursively traversed kernel expression tree. Boolean evaluation uses a bounded bit stack, while rule descriptors reference windows in a separate instruction array instead of embedding an entire program on the BPF stack.

The reviewed configuration imposes 64 exits, 32 rules, 32 instructions per rule, and 512 instructions across all rules. These are implementation budgets, not universal eBPF limits. The global budget is shared: permitting 32 rules does not imply that all may simultaneously consume the maximum per-rule allocation.

4.3 Parser Exhaustion Before Semantic Validation

A configuration failure occurred before any eBPF program was involved. Deeply nested parentheses exhausted the expression parser's stack. Checking the final abstract syntax tree did not solve the problem because parsing had already happened, and redundant parentheses might not remain in that tree.

The remedy was a source-level nesting check before invoking the parser, followed by separate syntax-tree and instruction budgets. This establishes a sequence of bounded stages rather than relying on a late check to protect an earlier unbounded stage.

Another failure arose at the YAML boundary: a leading unquoted ! can be interpreted as a YAML tag indicator instead of an expression operator ^[10]. Expressions beginning with negation must therefore be quoted. This is a configuration-language interaction, not a defect in Boolean evaluation.

4.4 ABI and Build Integrity

Map values use fixed-width layouts with explicit interpretation of instruction tags. A valid memory size is not sufficient to establish semantic compatibility: replacing the meaning of a field while retaining the same structure size still invalidates old state. The reviewed observation change therefore cannot justify reusing an older pinned map merely because its value size matches.

Embedding the BPF object in one executable reduces deployment mismatch, but development exposed a separate failure: a userspace rebuild could retain stale embedded bytecode when changes to shared or kernel-side sources were not tracked. The fix expanded build-script dependency tracking. Cargo's rerun-if-changed mechanism is the relevant external contract ^[11].

The resulting lesson is that source identity, host executable identity, embedded BPF identity, and loaded program identity must be traceable. A successful build command establishes less than is often assumed.

5. Engineering Difficulties and Design Evolution

5.1 Safe Host Parsing Did Not Produce a Verifiable Packet Parser

The initial parser operated on bounds-checked Rust slices and passed host tests. Its kernel adapter derived a slice length from the difference between XDP packet pointers. The verifier nevertheless rejected packet accesses.

The kernel verifier tracks pointer types and accessible ranges, not only the mathematical plausibility of a source-level calculation ^[12]. In the rejected generated program, checking a derived scalar length did not establish the necessary readable range for the packet pointer.

The revised adapter uses bpf_xdp_load_bytes() to read a fixed prefix into a bounded buffer. The shared parser then accesses known offsets. Variable IPv4 header-length processing was removed from the fast path; packets with IPv4 options take ordinary kernel handling. This was a targeted response to the generated program and tested verifier, not a claim that all variable-offset parsing is fundamentally impossible in eBPF.

The fixed prefix also reduced verification branching. A variable-length slice introduced repeated runtime checks throughout decoding; a constant-size buffer allowed many checks to become compile-time constants. Crucially, this established memory-access properties only. It did not establish complete IPv4 validity or correct treatment of fragments, a remaining distinction discussed in Section 7.

5.2 Bounded Execution Still Caused Verification-State Explosion

Bounding the number of rules was insufficient to make the original monolithic evaluator load. Parsing, field dispatch, comparison operations, Boolean evaluation, state updates, and forwarding created a large combined analysis space. Reducing loop limits did not by itself eliminate rejection.

The implementation evolved toward a policy program reached through a tail call, bounded helper-driven iteration using bpf_loop(), and separate functions for mutually exclusive decision and observation paths. A later change also moved the outer rule scan into helper-driven iteration. These changes altered the shape of the verifier's analysis rather than the operator-visible first-match policy.

A redundant runtime instruction-fuel counter was particularly revealing. Although intended as a safety measure, different remaining-budget values prevented otherwise similar analysis states from converging. Removing that counter did not remove all execution bounds: validated instruction windows and bounded iteration retained the runtime limits.

The kernel documents both verification limits and the need to establish loadability by loading the actual program ^[13]. The engineering record therefore distinguishes emitted BPF instructions, instructions or states explored during verification, and instructions executed by a packet. A reduction in verification work is not a measured forwarding-speed improvement. Likewise, early rejection after few analyzed instructions is not evidence of an efficient implementation.

5.3 Stack Pressure Required a Different Data-Passing Strategy

Once later stages became reachable to verification, combined stack usage became another blocker. Header copies, FIB parameters, tagged and untagged frame forms, and nested function frames exceeded the tested stack budget. The documented BPF stack constraint makes these otherwise small buffers significant ^[13:1].

The implementation moved temporary forwarding storage into per-CPU scratch maps and reused those buffers across phases. Policy handoff state likewise travels through an explicit map rather than relying on a caller's stack surviving a tail call. Per-CPU arrays provide separate storage regions for each CPU ^[14].

This solved a resource-placement problem, not the separate concurrency problem of multiple CPUs updating the same connection. Per-CPU temporary storage and a shared flow table require different correctness arguments.

5.4 A Logical VLAN Interface Was Not Necessarily a Transmission Device

The deployment represented exits as VLAN subinterfaces. A route lookup could identify the expected VLAN interface, and a redirect helper could accept its device slot, yet packets still failed later in transmission.

The solution separated the expected logical FIB device from the physical XDP transmission device. The route remains associated with its VLAN, while DEVMAP points to an appropriate underlying device and the frame carries the required tag. DEVMAP is a redirect target mechanism, not a guarantee that every referenced netdevice implements the required transmit path [11,12].

Moving attachment to the physical trunk introduced a second problem: all VLANs now arrived with the same physical interface index. Interface-index classification could no longer distinguish decision traffic, return traffic, and unrelated management traffic. The router therefore introduced a VLAN-to-role table and verified the actual attached program and XDP mode rather than inferring them from configuration.

The role table itself exposed a subtle default-value bug. BPF array elements exist and are zero-initialized even when userspace has never configured them ^[14:1]. If zero means “decision ingress,” every unconfigured VLAN inherits that role. Reserving zero for “unconfigured” prevents an allocated slot from being mistaken for an authorized traffic class. Priority-tagged VLAN identifier zero must also remain distinct from an untagged compatibility sentinel.

Inline tag visibility was a deployment precondition. The recorded trunk configuration disabled receive-side VLAN stripping because its parser consumed inline tags. Driver-provided VLAN metadata offers another possible mechanism ^[3:1], but it was not an implemented substitute in this path. The capability to read hardware metadata should not be confused with the separate custom-metadata contract used by the slow path.

5.5 Exit-Scoped Routing Accidentally Captured Local Traffic

Restricting lookup to an exit table fixed one policy problem but introduced another. An exit table containing a default route could resolve a packet addressed to the router itself as transit traffic. The accelerated path then sent it outward instead of allowing local delivery.

IPv4 routers distinguish local delivery from forwarding, and Linux normally represents local destinations through its routing policy and local table [19,10]. The revised forwarding path first performs an unscoped preflight lookup, then queries the selected exit table only when that preflight permits forwarding.

The two lookups have deliberately different responsibilities. The preflight uses the ordinary routing-rule path and skips neighbor lookup, so a missing neighbor on the main-table route cannot prevent a usable selected exit. Its packet length is zero for the helper's MTU decision, preventing the main route's MTU from deciding suitability for a different selected exit. The second lookup uses the selected table and the actual IPv4 total length.

A related bug supplied Ethernet-frame length to the FIB MTU check. IPv4 Total Length covers the IP datagram, not the enclosing Ethernet header ^[15]. Using the larger link-layer length caused near-MTU packets to be classified as requiring fragmentation even when their IP datagrams fitted the exit.

Finally, the FIB structure is an input/output object. A successful first lookup can replace interface and destination fields and populate union-backed output fields. The implementation clears and reconstructs the entire request before the second lookup. Reusing the first answer as a partially modified second request would ask the kernel a different routing question from the one intended ^[16].

5.6 Neighbor Refresh Repaired Availability but Not Routing Semantics

After successful loading and attachment, unresolved next hops still prevented accelerated forwarding. Manually sending traffic toward an exit gateway restored operation. An intermediate userspace mechanism periodically refreshed exit neighbors.

This was useful operationally but insufficient semantically. A periodic probe could create the missing state after a delay, yet packets arriving before that probe still entered unmarked fallback. Reducing the refresh interval would reduce some failure windows without removing the incorrect forwarding behavior.

The failure can be reconstructed as a dependency cycle:

XDP selects exit e
    -> selected gateway has no resolved MAC
    -> unmarked XDP_PASS
    -> Linux chooses the management route
    -> no packet reaches the selected gateway's neighbor path
    -> the next XDP packet encounters the same missing neighbor

The revised requirement was therefore not simply “keep neighbors warm.” It was “preserve the selected exit while requesting neighbor assistance.” Refreshing and policy preservation solve different problems and must be separate experimental variables.

5.7 A Minimal Decision Token Connects XDP to TC

For a neighbor miss with a matching expected egress device, XDP reserves metadata and writes a fixed-layout record containing a versioned signature and an exit identifier. If it cannot write that record, it drops and counts the packet rather than passing it unmarked.

TC ingress checks that enough metadata is accessible, validates the signature and identifier, maps the identifier into a reserved mark namespace, and sets the skb mark. It does not reevaluate the flow. The Linux routing policy database then maps that mark to the selected exit table. The metadata-access and mark-routing primitives are established mechanisms [9,10]; their use here carries a completed decision rather than a request to select another route.

selected exit e
    |
    +-- resolved neighbor -> header rewrite -> XDP redirect
    |
    +-- missing neighbor, expected interface matches
            -> write decision metadata
            -> XDP_PASS
            -> TC validates metadata and applies mark(e)
            -> Linux consults table(e)
            -> neighbor resolution and kernel forwarding

The neighbor diversion occurs before forwarding headers are committed to the packet. Linux therefore receives the original forwarding input rather than a packet whose TTL has already been consumed by the fast path. On successful fast forwarding, the router updates TTL, the IPv4 checksum, and link-layer addresses. Incremental checksum handling follows the arithmetic described in RFC 1624 ^[17].

The TC program exposed another verifier-sensitive failure. An earlier scalar metadata-length calculation did not establish the pointer range needed by the eventual load, and compiler optimization removed a redundant-looking pointer comparison. The final check directly proves that the record fits before data. The general pointer-range issue is covered by verifier documentation ^[12:1]; the specific compiler interaction was an implementation failure uncovered during development.

5.8 Terminal Rules and Installation Order Extend the Contract

A per-exit mark rule alone is insufficient. If that rule disappears, lookup can continue to lower-priority rules. The control plane therefore installs a terminal prohibition for its owned mark range after its exit rules. This prevents unmatched owned marks from silently falling through, subject to the host's complete routing-rule order ^[18].

Startup installs routing rules before TC, and TC before the XDP producer. Otherwise, a producer could emit metadata before a consumer or route mapping exists. Cleanup removes owned resources and preserves unrelated attachments. These lifecycle measures address ordinary startup and shutdown; they do not establish an atomic transaction across arbitrary process crashes or privileged external reconfiguration.

The failure contract remains deliberately uneven:

5.9 Return-Path Logic Existed Before Return-Path Observation

An early change implemented return accounting and passed shared-logic tests, but the loader still attached the XDP program only at the decision ingress. The return branch was correct as a function yet unreachable in that deployment.

Later changes attempted observation attachment on distinct configured exits while preserving existing foreign programs. A failed optional attachment produces reduced visibility rather than disabling decision-path forwarding. Diagnostics report the actual role, attachment target, program identity, and mode.

This failure motivated the explicit visibility model: return observations are additional evidence, not a hidden prerequisite. It also yields a general test principle. Testing a transition function does not establish that the operating system can generate the events that invoke it.

5.10 Redirect Counters Were Not Delivery Counters

The project repeatedly encountered apparent success before the last relevant boundary. Linux documents redirect as a staged process: the helper records a target, the driver handles the redirect action, and queued frames are later flushed ^[19]. Failure can therefore occur after the eBPF program has returned its verdict.

The required evidence is correspondingly layered: policy selection, redirect submission, device transmission, and receiver delivery are separate observations. A redirect_success counter cannot stand in for all four. Receiver records and external captures are essential; an ordinary host capture can also miss traffic that bypasses its capture path.

The prototype uses bounded per-CPU counters, with userspace summing cumulative snapshots rather than repeatedly adding each snapshot as a new increment. Tests seed distinct nonzero slot values to reveal swapped mappings that an all-zero endpoint test would miss. Counter scope is explicit: in trunk mode, unrelated VLANs rejected before ownership classification are not part of the configured-traffic denominator.

6. Correctness Analysis and Evaluation Methodology

6.1 What the Development Evidence Supports

The reviewed records describe successful target-kernel loading and namespace/veth integration cases after the corresponding changes. Those reports support a development narrative of failures and remedies. They do not supply an independently audited performance dataset, establish support for every driver, or verify every failure branch.

A test harness maintained outside the distributable source tree is not automatically a publicly reproducible artifact. The evaluation package must separately preserve topology setup, traffic generation, fault injection, packet identities, raw observations, and analysis procedures. Removing identifying addresses from a paper must not remove the causal structure required to reproduce its tests.

6.2 Conditional Slow-Path Argument

Consider a valid transit packet for which policy has selected exit e. Assume that the selected-table lookup reports a neighbor miss on the expected logical device; metadata is written and preserved to the intended TC hook; TC validates the record and applies an injective mark encoding; routing-rule order selects the corresponding exit table; and that table still resolves the destination through the intended exit.

Under these assumptions, TC supplies Linux with a mark identifying e, so the slow path consults the intended table rather than intentionally repeating the original flow policy. If the owned per-exit rule is missing and the packet reaches the terminal owned-mark prohibition, it is rejected rather than proceeding to the main table.

This is a composition argument, not a claim of a mechanically verified implementation. It guarantees neither neighbor reachability nor delivery. It also does not cover a higher-priority external rule, an incorrectly populated exit table, disappearance of metadata, or another forwarding failure class.

The packet-specific token has an important advantage over re-reading the flow at TC: a later packet may already have changed the flow's selected route. The token retains the decision for the packet entering the slow path instead of substituting the flow's newer state.

6.3 Functional and Fault-Injection Experiments

Functional tests must follow the same TCP connection across threshold crossings. A test that creates new SYN packets only cannot establish migration because the bootstrap packet intentionally skips ordinary rule evaluation. Boundary cases should include the packet that reaches the threshold, ordered overlapping rules, no-match retention, and return traffic that affects the next decision packet.

Observation tests should cover SYN-first, SYN-ACK-first, return-only discovery followed by decision traffic, midstream insertion, idle expiration, LRU eviction, restart, and loss of an optional observation attachment. After state loss, the oracle must expect a new observation epoch rather than retained lifetime counters.

Neighbor tests should independently remove each exit's neighbor entry, confirm that background refresh is disabled, and observe the first eligible packet, ARP on the selected interface, kernel-assisted forwarding, and subsequent fast-path traffic. Selecting a non-default exit is necessary to distinguish decision preservation from a hard-coded default fallback.

The management route must remain usable and observable. Installing a blackhole there is an invalid shortcut: the router's global preflight may reject the packet before the selected-exit path is reached. An external management sink makes wrong-exit forwarding detectable without preventing it by construction.

Negative controls include removing one owned mark rule, corrupting metadata through an isolated test variant, changing an exit table to another interface, removing a DEVMAP target, and causing tail-call handoff failure. Runtime-injected outcomes must be distinguished from branches evaluated only in host tests.

6.4 Performance Baselines and Ablations

A fixed Linux policy-routing baseline should first be compared with a fixed-exit accelerated configuration using the same effective routes. This isolates forwarding cost. A separate comparison evaluates the dynamic policy against fixed exits or a semantically comparable policy engine; static Linux routing is not an equivalent implementation of byte-threshold migration.

The workload matrix varies packet size, concurrent flow count, offered rate, rule count, predicate complexity, and rule-hit position. First-rule hits, last-rule hits, and complete misses exercise different scans. Flow-table measurements must include eviction pressure, not only an empty or comfortably sized table.

Cold-neighbor and warm-neighbor conditions are separate experiments. A two-factor ablation varies both policy-preserving fallback and userspace neighbor refresh. This distinguishes correctness supplied by the slow path from fewer neighbor misses supplied by periodic probes.

Reported metrics should include delivered throughput, packets per second, loss, CPU usage, map memory, and latency distributions. RFC 2544 provides an established laboratory benchmarking framework ^[20]. Saturation testing must remain isolated from production networks, as clarified by RFC 6815 ^[21].

For dynamic routing, the important outcomes are short- and long-flow completion time, delivered goodput, utilization of the constrained exit, and migration-associated retransmission and reordering. A successful interface change alone is not a user benefit. An unfavorable or statistically inconclusive workload result remains a valid result and must not be replaced with an assumed improvement.

6.5 Statistical Reporting and Packet-Level Evidence

Assign each injected test packet an unambiguous identifier. Define $R_{wrong exit}$ as the number of distinct identifiers observed on an unintended exit divided by the number injected. Independently report intended-receiver delivery and duplicates. A packet may reach both an unintended exit and an intended receiver, so these categories are not automatically complementary.

Zero observed wrong-exit packets must be reported as $0/N$ over a stated duration and failure schedule. Under an explicitly assumed independent Bernoulli model, the one-sided 95% upper bound is $1 - 0.05^{1/N}$, approximately $3/N$. Bursty and correlated failures may invalidate that model; the formula is not a universal non-leakage guarantee.

Predefine warm-up, measurement duration, packet-generation method, and repetition units. Five independent runs may be an initial allocation, not an assurance of adequate statistical power. Report dispersion and confidence intervals at the level of independent runs. Tail latency must come from sufficient packet or request samples rather than percentiles of a few run averages.

For one-way delay, document clock synchronization, timestamp placement, and uncertainty. Otherwise use an appropriate round-trip measurement or a shared clock domain. RFC 7679 explicitly treats clock and timestamp uncertainty as part of the delay metric ^[22].

6.6 Reproducibility Requirements

The experiment identity comprises the compiler and linker, dependency lockfile, embedded BPF object, userspace executable, kernel, driver, firmware, XDP mode, and network configuration. Record RSS and IRQ placement, CPU frequency settings, VLAN offloads, MTUs, and any veth/NAPI prerequisites. Archive exact configurations and artifacts rather than assuming a mutable branch or release label is sufficient.

Verification experiments must preserve both acceptance and rejection logs. An earlier parser rejection can conceal later map, stack, or policy failures. Each architectural variant therefore needs a clearly identified first failure, and runtime comparisons should include only variants that actually load and execute the intended path.

7. Discussion and Remaining Limitations

7.1 Partial Observation Is Not Merely a Smaller Counter

Missing return packets can delay byte-based migration, but the effects are broader than undercounting. Return observations also influence last_seen, idle intervals, and expiration. Enabling an observation hook may therefore change decisions even when decision-path traffic is unchanged.

Observation visibility is consequently part of the policy's measurement environment. A rule calibrated in a fully observed topology is not automatically transferable to a decision-only topology. Nor does a short observed flow establish an application category or benign intent.

7.2 Shared-Map Replacement Does Not Serialize a Flow

Implementation review exposes a concurrency risk separate from the documented deployment failures. The flow adapter copies a map value, modifies the copy, and replaces the stored value. Linux documents atomic element replacement and concurrent access, but atomic replacement does not make a sequence of lookup, computation, and replacement a transaction ^[8:1].

A possible interleaving is:

CPU A, decision path: read state with route r0
CPU B, return path:   read state with route r0
CPU A:               evaluate policy and store route r1
CPU B:               store its updated observation containing stale route r0

No policy ran on CPU B, yet the final stored route can revert. Counters and visibility updates may likewise be lost. The issue also applies to policy state held across a handoff while another packet updates the shared entry.

This is a source-level counterexample to an unconditional concurrency claim, not a measured incident frequency. The current case study must therefore qualify route-preservation statements about return updates as sequential semantics. Stronger guarantees require an appropriate synchronization or ownership design, followed by cross-CPU tests. Changing only temporary buffers to per-CPU storage does not solve this problem.

7.3 Memory Safety Does Not Establish Protocol Validity

The reviewed parser verifies accessible fixed offsets and explicitly rejects IPv4 options, but does not establish a complete IPv4 validity contract. In particular, it does not explicitly reject non-initial fragments before interpreting fixed-position bytes as TCP or UDP fields. Such bytes may be fragment payload rather than a transport header. IPv4 fragmentation and header-length semantics make this a protocol issue distinct from bounds checking ^[15:1].

The evaluation domain should therefore be restricted to validated, unfragmented packets until fragment handling is hardened. Length consistency, malformed transport headers, and checksum-invalid traffic also require explicit treatment before claiming safe behavior for arbitrary IPv4 inputs. Unrecognized ethertypes being passed to Linux is not evidence that every unsupported IPv4 form is detected and passed correctly.

7.4 Metadata Ownership and Failure Scope

A valid signature and route identifier are an internal format contract, not cryptographic authentication. The TC consumer also treats absent metadata as ordinary traffic, so it cannot detect every case in which metadata intended by XDP disappears before TC. Conversely, rejecting unrelated nonempty metadata may conflict with another application using the same area. Deployment must establish compatible ownership of metadata, marks, and hook ordering [9,10].

The terminal rule protects only packets that reach it with an owned mark. It cannot repair an earlier external rule that captures the packet or an exit table whose apparently valid route is semantically wrong. Other FIB and helper failure classes still include unmarked fallback. Therefore the prototype is not a general no-leakage firewall or a universal fail-closed routing system.

7.5 Exit Migration Is Not Seamless Transport Migration

Threshold crossing may redirect a packet immediately, without a flowlet gap, hysteresis, or a congestion-feedback loop. The current design therefore does not inherit the ordering properties or adaptive mechanisms of flowlet-based systems ^[5:2].

Different exit delays can produce reordering; different MTUs or stateful middleboxes can produce additional failures. If exit-specific NAT changes the externally visible endpoint, ordinary TCP continuity may be lost ^[7:2]. Source-address stability is necessary but not sufficient: reverse-path reachability, NAT state consistency, and path behavior must also be validated.

These restrictions do not invalidate configurable migration. They define the environments in which its benefits must be demonstrated rather than presumed.

7.6 The Trade-Off Between Scope and Guarantees

Moving unsupported traffic to Linux reduces fast-path complexity but does not preserve every custom policy. Increasing rule expressiveness expands both implementation complexity and the space of verifier behaviors. Bounding state memory makes resource use predictable while making observation history expendable.

The strongest next steps are therefore not additional features alone. They are an explicit fragment policy, a concurrency-safe state-update design, broader policy-preserving failure handling, and reproducible evidence for the driver and workload combinations actually supported.

8. Conclusion

This paper examined the engineering evolution of an observation-aware, multi-exit XDP router. Its main difficulties occurred where one subsystem's guarantees were incorrectly assumed to extend into another.

Host-safe parsing did not automatically establish verifier-visible packet ranges. Finite rule counts did not prevent excessive verification-state exploration. A fresh userspace build did not necessarily contain fresh BPF bytecode. A resolved logical VLAN did not necessarily support physical XDP transmission. An exit-table route did not determine whether a packet belonged to local delivery. A return-accounting function did not ensure return traffic could reach its hook. Most importantly, ordinary kernel fallback did not preserve the exit already selected by XDP.

The resulting architecture separates observation from endpoint identity, compilation from kernel verification, logical exits from transmission devices, and routing decisions from forwarding assistance. Its neighbor slow path carries a packet-specific exit token through XDP metadata and TC marking so that Linux can resolve the intended next hop without intentionally selecting another exit.

The resulting guarantee remains conditional. Concurrency, packet-validation coverage, metadata transport, external policy interference, and remaining fail-open classes limit stronger claims. The evaluation framework makes those boundaries explicit and separates functional evidence from performance evidence.

The general lesson is that a fast path is not a complete routing system. A routing system becomes trustworthy only when the observations behind its decisions, the transitions between its execution paths, and the evidence of actual delivery are specified together.

Acknowledgements

I would like to express my sincere gratitude to my girlfriend, Jingyi Wang, for her patience, encouragement, and unwavering support throughout this research. During the most difficult stages of the project—particularly when I encountered technical setbacks, uncertain design choices, and the frustration that inevitably comes with experimental systems work—her support helped me remain focused and continue moving forward. Her understanding and encouragement were an important source of motivation throughout the development and writing of this work, and I am deeply grateful to have had her by my side.

References