BGP traffic engineering with flow data: communities, local-pref, AS-path prepending

Traffic engineering on a multi-homed network is two separate problems that people keep treating as one. Outbound traffic leaves by a path you choose, so you control it completely. Inbound traffic arrives by a path that thousands of other networks choose, so you can only influence it — and every lever you have is a request that a remote network is free to ignore.

The group's own network is a useful reference point for this: AS203136 announces 185.143.176.0/22 and is multi-homed to three upstreams — Caucasus Online, System Net and Silknet. Everything below is the reasoning that applies to a topology of that shape.

The best-path order decides which lever wins#

Before touching anything, remember the order BGP applies its tie-breakers, because a lever only matters if the comparison reaches it:

1. Highest weight (Cisco-local, never leaves the router)
2. Highest local-preference
3. Locally originated routes
4. Shortest AS path
5. Lowest origin type
6. Lowest MED (only between routes from the same neighbouring AS, by default)
7. eBGP over iBGP
8. Lowest IGP metric to the next hop
9. Oldest route, then lowest router-id, then lowest peer address

The single most important consequence: local-preference is compared before AS path length. If a remote network sets local-pref on routes from its cheap transit, your prepending never gets evaluated. Prepending only breaks ties among paths that are otherwise equal from the remote network's point of view.

Outbound: local-pref is the lever you fully own#

Local-preference is a well-known discretionary attribute: mandatory inside your AS, never sent to an eBGP peer — which is precisely why it is reliable. You set it on import, and every router in your AS then agrees on which exit to use.

BIRD 2:

filter import_upstream_a {
    bgp_local_pref = 150;
    accept;
}

protocol bgp upstream_a {
    local as 203136;
    neighbor 192.0.2.1 as 65001;
    ipv4 {
        import filter import_upstream_a;
        export filter { if net = 185.143.176.0/22 then accept; else reject; };
        import limit 1000000 action restart;
    };
}

FRR:

route-map FROM-UPSTREAM-A permit 10
 set local-preference 150
!
router bgp 203136
 neighbor 192.0.2.1 remote-as 65001
 address-family ipv4 unicast
  neighbor 192.0.2.1 route-map FROM-UPSTREAM-A in
  neighbor 192.0.2.1 route-map TO-UPSTREAM-A out
  neighbor 192.0.2.1 maximum-prefix 1000000 restart 15
 exit-address-family

Set local-pref selectively rather than globally. A blanket "prefer upstream A" moves all traffic and usually recreates the congestion somewhere else; matching a community your upstream tags, or a prefix list of the destinations you actually want to move, changes only what you meant to change.

Inbound: four levers, in ascending order of desperation#

Communities offered by your upstream#

This is the first thing to try and the one most people skip. Most transit providers publish a community list that lets you say "prepend three times to peer AS X", "do not announce to AS Y" or "set local-pref 80 inside my network". These act at the upstream's edge, closer to the decision point, so they are precise and reversible.

! example: apply a provider-published TE community on announcement
route-map TO-UPSTREAM-A permit 10
 match ip address prefix-list OUR-PREFIXES
 set community 65001:1234 additive

RFC 8092 large communities matter here: with 4-byte ASNs, an ASN:function:parameter triplet is the only encoding that fits, so a modern provider's TE documentation will list them in that form.

Two standard communities are worth knowing regardless of provider: no-export (65535:65281) keeps a prefix inside the neighbouring AS, and graceful shutdown (RFC 8326, 65535:0) tells a peer to deprioritise a session you are about to take down for maintenance — far cleaner than simply clearing the session.

AS-path prepending#

Prepending is blunt but universal. It works only where local-pref does not already decide, which in practice means it steers the traffic that reaches you via networks with no strong policy of their own.

Junos:

policy-options {
    policy-statement to-upstream-a {
        term announce {
            from {
                route-filter 185.143.176.0/22 exact;
            }
            then {
                as-path-prepend "203136 203136";
                accept;
            }
        }
        then reject;
    }
}

Cisco IOS:

route-map TO-UPSTREAM-A permit 10
 match ip address prefix-list OUR-PREFIXES
 set as-path prepend 203136 203136

Start with one prepend and measure before adding another. Long prepends do not shift proportionally more traffic; past a few, the remaining traffic is arriving because of local-pref decisions elsewhere and no amount of prepending will move it. Very long AS paths also attract filtering from networks that treat them as a leak signature.

MED#

MED is only useful when you have several sessions with the same neighbouring AS — two circuits to the same upstream, for instance. It is compared per neighbouring AS by default, and turning on always-compare-med to change that is a well-known way to create non-deterministic best-path selection. Leave it off.

More-specifics, and the ROA that governs them#

Announcing a more-specific is the lever that works when nothing else does, because longest-prefix-match beats every policy — within limits: IPv4 prefixes longer than /24 and IPv6 longer than /48 are filtered by most of the DFZ, so a /22 gives you exactly two useful steps and a /24 gives you none. It is also the lever with real costs: it adds an entry to the global table for every network that carries a full view, and it is sticky — the more-specific stays in the DFZ until you withdraw it and the withdrawal has propagated.

There is a harder constraint. The group's ROA for 185.143.176.0/22 is issued with maxLength /22. That means any /23 or /24 carved out of it, announced from AS203136, is RPKI Invalid — not merely unusual, but actively dropped by every network doing Route Origin Validation. Deaggregating for traffic engineering therefore requires updating the ROA first, and accepting that a loose maxLength is exactly what makes forged more-specific hijacks possible.

Using flow data to decide what to move#

Levers are cheap; knowing what to pull them for is the hard part. This is where a flow pipeline pays for itself. The workflow:

1. Measure the split. Group traffic by upstream interface and destination AS over a representative window — a business-day peak, not a weekend average. Typically a small number of destination networks account for the bulk of outbound volume.
2. Find the movable share. Traffic to a destination reachable through more than one of your upstreams is movable. Traffic that only one upstream carries is not, no matter what you configure.
3. Pick the smallest change that fixes the problem. Moving one destination AS with a community is preferable to prepending your whole announcement, because the blast radius is one AS instead of the whole internet.
4. Model the result before applying. If you move that AS, does the receiving uplink have headroom at its peak, not at the current moment?

The SQL for step one is in Building a NetFlow/IPFIX pipeline with Akvorado and ClickHouse; with BMP enrichment enabled you get the real AS path and communities attached to each flow, which is what makes step two answerable.

Making the change safely#

• One variable at a time. Change one policy, on one session, and watch. Two simultaneous changes produce a result you cannot attribute.
• Filters before policy. A prefix-list on export and a maximum-prefix limit on import are not traffic engineering, but they are what stops a policy mistake from becoming an incident.
• Change window and rollback text. Write the exact rollback commands before you apply the change, not after it goes wrong.
• Soft reconfiguration. Use inbound soft reconfiguration or route refresh so you can re-evaluate policy without tearing down sessions.

Verifying that it worked#

Local state first — what you are actually sending:

# Junos
show route advertising-protocol bgp 192.0.2.1 185.143.176.0/22 detail

# FRR
vtysh -c "show bgp ipv4 unicast neighbors 192.0.2.1 advertised-routes"

# BIRD
birdc "show route export upstream_a all"

Then the outside view, which is the only one that matters for inbound traffic. RIPE RIS and public looking glasses show what remote networks see, including how many prepends survived and which communities are still attached. Give propagation time before judging: convergence across the global table is not instantaneous, and a measurement taken one minute after the change tells you nothing.

Finally, confirm the traffic actually moved by re-running the same flow query you used to plan the change. Routing tables showing a new path and interface counters showing a new split are two different claims, and only the second one is the goal.

Mistakes we see most#

• Prepending five times, seeing no change, and prepending more — instead of asking whether local-pref upstream is the real decision.
• Setting local-pref globally on one upstream and moving traffic that was fine where it was.
• Announcing a more-specific without checking the ROA maxLength first.
• Communities that are silently dropped because the upstream strips inbound communities from customers, which is common and rarely read about in advance.
• No maximum-prefix limit, so a full-table leak from a customer becomes your outage.
• Treating a maintenance shutdown as a policy change instead of using RFC 8326 graceful shutdown, and taking a hard reconvergence hit for no reason.

The registry side of this — ROAs, route objects and the as-set your upstreams filter you with — is covered in RIPE NCC resources end to end. If the change is large enough to be risky, rehearse it first: Testing a migration in EVE-NG.