Automated Edge Failover with BGP Anycast and Health Probes

In a globally distributed Anycast network, multiple edge Points of Presence (PoPs) advertise identical IP prefixes to upstream Tier-1 transit providers and Internet Exchange Points (IXPs) using Border Gateway Protocol (BGP). Global internet routers steer user traffic to whichever PoP presents the shortest AS-path or lowest BGP metric.

However, Anycast by itself is network-layer routing—it has no intrinsic knowledge of application-layer health. If a web server daemon or reverse proxy crashes inside a specific PoP while the border routers continue announcing the prefix, incoming user traffic will sinkhole into that degraded location. Automated edge failover bridges this gap between layer 7 application status and layer 3 BGP announcements.

The Decoupling of BGP Daemon and Application State

To prevent edge sinkholing, edge PoP nodes run an integrated routing supervisor (such as ExaBGP or BIRD) that dynamically conditions BGP route announcements upon continuous application health probes.

[ Upstream Transit Provider (AS 64512) ]
              ▲
              │ BGP Announcement (Prefix 198.51.100.0/24)
              ▼
    [ BGP Route Daemon (BIRD) ]
              ▲
              │ Health State Feed (/run/srrrs/health.sock)
              ▼
  [ Edge Health Checker Agent ]
    ├── TCP / HTTPS Probe -> Local Caddy / Edge Proxy
    ├── Synthetic L7 Health Probe -> Identity Engine
    └── System Resource Limits -> Memory, CPU, FD exhaustion

Three-Tiered Probing Architecture

A resilient probing engine evaluates service availability across three cascading dimensions:

1. Process Liveness: Verifies that ingress daemons (Caddy, Envoy, internal tunnel agents) are alive and accepting socket connections on local interfaces.
2. End-to-End Synthetic Latency: Initiates synthetic HTTP GET requests that exercise the complete pipeline—terminating TLS, querying control plane certificates, and resolving static asset paths.
3. Upstream Reachability: Checks whether the local PoP maintains uninterrupted tunnel backhaul to persistent data regions. If transit from a regional PoP to the primary control cluster is severed, serving traffic locally would result in state errors.

Graceful BGP Drain vs. Hard Withdrawal

When a maintenance event or localized failure is detected, immediate hard withdrawal of a BGP route can cause transient packet loss while intermediate autonomous systems recalculate path convergence (BGP convergence delays can range from seconds to several minutes).

Modern edge failover employs a two-phase drain sequence:

Phase 1: BGP Prepending and Local Preference Deprecation

The health supervisor signals the local BGP daemon to append the local Autonomous System Number multiple times (e.g., AS-path prepending) and attach BGP community tags that depress local preference at upstream transit providers:

# BIRD configuration for graceful route poisoning on degraded health
filter anycast_export {
  if !health_check_ok() then {
    bgp_path.prepend(65001);
    bgp_path.prepend(65001);
    bgp_path.prepend(65001);
    bgp_community.add((65001, 666)); # Upstream blackhole / low-pref community
  }
  accept;
}

This artificially inflates the metric seen by global routers, causing transit networks to smoothly shift incoming traffic to adjacent healthy Anycast PoPs without abruptly breaking active TCP streams.

Phase 2: Full Route Withdrawal

Once ingress traffic drains below a safe threshold, the edge daemon sends a BGP WITHDRAW message for the prefix. The degraded PoP is completely excised from global routing tables until health probes recover and sustain consecutive passing states for a predefined stabilization window (typically 300 seconds), protecting the private perimeter against route flap dampening penalties.