Testing a migration in EVE-NG before touching production (network digital twin)

A change window is short, and the cost of discovering a mistake inside it is measured in the length of the outage plus the length of the rollback. A digital twin moves that discovery to a place where mistakes are free: a lab that runs the same software, the same configuration and the same topology as production, where you can break things deliberately and repeatedly.

EVE-NG is the usual platform for this in the networking world. It is not a simulator — it runs the vendors' real images under KVM — which is exactly why the results transfer, and also exactly where its limits are.

What a twin is for, and what it is not#

A lab reproduces the control plane faithfully: routing protocol behaviour, policy evaluation, convergence order, configuration syntax, the exact command sequence of a migration. That covers most of what goes wrong during a change.

It does not reproduce the forwarding plane. There is no ASIC, no hardware queueing, no real line-rate behaviour, no vendor-specific control-plane policing, no optics. If your question is "will this policy select the right path", the lab answers it. If your question is "will this platform forward 40 Gbit/s of small packets with this feature enabled", the lab cannot, and pretending otherwise is worse than not testing at all.

Decide which of the two questions you are asking before building anything.

Sizing the host#

Virtual routers are memory-hungry and modestly CPU-hungry. Before anything else, confirm the host can nest virtualisation, because a KVM guest that falls back to emulation is unusably slow:

# CPU supports hardware virtualisation?
grep -cE '(vmx|svm)' /proc/cpuinfo

# nested virtualisation enabled? (Intel; use kvm_amd on AMD)
cat /sys/module/kvm_intel/parameters/nested

# KVM modules actually loaded
lsmod | grep -E '^kvm'

If EVE-NG itself runs inside a hypervisor, nested virtualisation must be enabled on the outer layer too, and the guest CPU must be passed through rather than emulated.

Plan resources per node from the vendor's minimum for the image you use, not from an average. A full-featured virtual router asks for several vCPU and several gigabytes of RAM; a container-based routing daemon asks for a fraction of that. This drives a practical design choice: build the parts of the topology you are testing with the real vendor images, and build the surrounding internet or campus with lightweight nodes. Twenty FRR containers announcing routes make a perfectly good "rest of the world" for a BGP policy test.

Getting images in and named right#

EVE-NG is strict about directory naming — the platform template is derived from the directory name, and a wrong name produces a node that will not boot with no useful error.

# qcow2 images live under a per-image directory
mkdir -p /opt/unetlab/addons/qemu/vmx-14.1R4/
# copy the disk in with the file name the template expects, then:
/opt/unetlab/wrappers/unl_wrapper -a fixpermissions

Two practical points. Licensing is your problem: vendor images generally require an entitlement, and the free tiers of some virtual platforms cap throughput to a level that is fine for control-plane work and useless for anything else — which is acceptable, because control-plane work is what the lab is for. And keep an image inventory with the exact software versions running in production. Testing a migration against a different release than the one deployed produces confidence that is not transferable.

Importing production configuration honestly#

The temptation is to hand-write a "representative" configuration. Don't — the bugs live in the parts nobody would think to write by hand.

Pull the real running configuration, then adapt exactly three categories of thing:

1. Secrets. Strip or replace every credential, SNMP community, BGP MD5 key, RADIUS secret and certificate. A lab is not a secure environment and its snapshots get copied around.
2. Interface names. Virtual platforms rarely expose the same naming as hardware. Map ge-0/0/0 to what the image actually offers, and keep the mapping table with the lab — future you will need it to read the diffs.
3. Management and out-of-band. Rewrite the management VRF, the NTP and syslog targets and any addresses that would make a lab node talk to production systems. This is the one that bites: a lab router that successfully reaches the production RADIUS server, or worse, forms a real BGP session, has stopped being a lab.

Everything else — policies, prefix lists, route maps, communities, IGP metrics — copies verbatim. That is the point.

Wiring the lab to the outside#

You usually need two kinds of external connection: management access for automation, and traffic sources for validation. EVE-NG exposes bridged Cloud interfaces for both. Put management on one bridge and any traffic-generating hosts on another so a test flood cannot interfere with your own access to the lab.

For traffic, small Linux nodes are enough for the questions a control-plane lab can answer:

# path MTU behaviour: 1472 payload + 28 bytes headers = 1500
ping -M do -s 1472 10.20.30.1

# throughput and loss between two lab hosts
iperf3 -c 10.20.30.1 -u -b 100M -t 30

# is the policy actually dropping what it should?
hping3 -S -p 443 -c 5 10.20.30.1

Build the test plan before the change#

The lab is only useful if you know what "correct" looks like. Capture state before, apply the change, capture state after, and diff. Do it with a script, not by eye — a human comparing two 400-line BGP summaries reliably misses the one line that matters.

# capture the same command set before and after a change
import re

from netmiko import ConnectHandler

COMMANDS = [
    "show bgp summary",
    "show route summary",
    "show interfaces terse",
    "show configuration | display set",
]

def snapshot(host, tag):
    conn = ConnectHandler(device_type="juniper_junos", host=host,
                          username="lab", password="lab")
    for cmd in COMMANDS:
        out = conn.send_command(cmd)
        slug = re.sub(r"[^a-z0-9]+", "_", cmd.lower()).strip("_")
        with open(f"{tag}-{host}-{slug}.txt", "w") as fh:
            fh.write(out)
    conn.disconnect()

Then compare them pairwise and read every line of the result:

for f in before-*.txt; do diff -u "$f" "after-${f#before-}"; done

Useful invariants to assert explicitly:

• Neighbour count and session state per device, before and after.
• Prefix counts received and advertised per session — a large drop is a filter mistake, a large rise is a leak.
• A full ping and traceroute matrix between the edges you care about.
• Route counts per protocol, so a redistribution mistake shows up as a number rather than as a customer complaint.

Inject the failures you are afraid of#

A migration that works only on the happy path is not tested. In the lab, breaking things costs nothing:

• Shut a link mid-change and confirm the traffic path and the timing of reconvergence.
• Kill a node hard (not a graceful shutdown) and see what the neighbours do with their timers.
• Simulate a route leak from a "customer" node announcing a full table, and verify maximum-prefix fires the way you configured it.
• Introduce an MTU mismatch on one hop and watch which protocols survive it and which silently do not.
• Roll back mid-way, from a half-applied state — this is the scenario nobody rehearses and everybody eventually meets.

Rehearse the change, then write the runbook#

The final pass is a full dress rehearsal: apply the migration exactly as it will be applied in production, from the same script or the same command list, timing each step. What comes out of it is the deliverable:

• The exact command sequence, in order, with the expected output after each step.
• Time per step, so the change window is planned from measurement rather than optimism.
• The rollback sequence, tested from the same half-applied state you rehearsed above.
• The verification commands and what their correct output looks like — so the engineer at 03:00 is comparing against something concrete.

Keep the lab afterwards#

The common waste is deleting the topology once the migration succeeds. Export it instead. A twin that already matches production is the cheapest possible environment for the next change, for reproducing an incident safely, and for onboarding — an engineer can break a copy of the real network without consequence, which is a far better introduction than reading a diagram.

The changes most worth rehearsing this way are the ones with wide blast radius: routing policy edits, which are covered in BGP traffic engineering with flow data, and anything that touches the origin of your prefixes — see RIPE NCC resources end to end.