Hands-on IS-IS operations

IS-IS Watcher bootcamp runbook

Make one controlled change at a time, predict its effect, then verify the same fact from the Watcher event through Topolograph.

Start the runbook

Useful links: IS-IS Watcher - repository and deploy guide · Topolograph

1. How this runbook works

For each exercise: make one change, predict the IS-IS result, observe the Watcher event, then find the same fact in Topolograph Monitoring, in the SDK, and - for the correlation exercise - in the agent's answer. Restore the lab before the next independent exercise.

isis01 is a six-router FRR domain in one area (49.0001, AS 65100). router2-router3 runs Level-1-2; router3 to router6 and router3's leg to the router4/router5 LAN are Level-2-only. The Watcher rides a Level-2 adjacency on router1 and resolves each System ID to its hostname through the IS-IS dynamic-hostname TLV, so events name router2, router3, router6 rather than 0100.1001.000x.

  1. The Watcher CSV line is the deterministic source event.
  2. Monitoring and the SDK prove the event was ingested and is queryable.
  3. The agent is used only to correlate several events into one incident, never as a replacement for the source line.

2. Bringing up the six-router lab

Run the public GRE-based isis01 topology from the IS-IS Watcher repository. prepare.sh also creates the isis-br-dr bridge and loads the MPLS kernel modules that IS-IS TE needs.

First check whether a lab is already running with sudo clab inspect --all. If a stale isis01 is listed, tear it down with sudo clab destroy --topo isis01.clab.yml --cleanup from containerlab/isis01/ before deploying again - name the topology file so no other lab is touched.

Command

cd containerlab/isis01
sudo clab inspect --all
sudo clab destroy --topo isis01.clab.yml --cleanup   # only if a stale isis01 is listed
sudo ./prepare.sh
sudo clab deploy --topo isis01.clab.yml
sudo docker logs clab-isis01-isis-watcher
sudo tail -f watcher/logs/watcher1.isis.log
Required observations Facts to confirm before continuing
  • Six router containers plus the watcher are up: docker ps --filter name=clab-isis01 lists clab-isis01-router1..6 and clab-isis01-isis-watcher, all State=Up.
  • The watcher has the LSDB and is sniffing: docker logs clab-isis01-isis-watcher shows ISIS LSDB has been received and Sniffing packets on interface: eth1.
  • Adjacencies are Up: docker exec clab-isis01-router1 vtysh -c 'show isis neighbor' lists router3 in the Up state; the first Topolograph graph then contains six routers.

Rollback: When you finish the runbook, remove the lab with sudo clab destroy --topo isis01.clab.yml --cleanup from containerlab/isis01/.

3. Network event record format

The runbook uses three event families: host, metric, and network. Read event_object as the changed object, event_status as the transition, and event_detected_by as the advertising or detecting router. graph_time is the watcher's own label for the run and selects the Topolograph graph.

IS-IS lines carry a level field (1 or 2) that OSPF lines do not - it is the third field, right after watcher_name. Read the metric line above as one sentence: at 2026-09-07T06:55:14Z the watcher lab-isis01 saw router3 re-advertise its Level-1 link toward router2 with the metric changed from 10 to -1 (adjacency lost), on the interface addressed 192.168.23.2 in area 49.0001 / AS 65100. The identity behind router2 is its NET / System ID 49.0001.0100.1001.0002.00; the watcher prints the hostname because every router advertises the dynamic-hostname TLV.

A plain host or network up/down line carries no cost fields, so Fluent Bit forwards only the paired changed line. Topolograph reads an adjacency going down from the metric line whose new_cost is -1, and coming back from the one whose old_cost is -1.

  1. Fields in order: watcher_time, watcher_name, level, event_name, event_object, event_status, [cost fields], event_detected_by, graph_time, area_num, asn, [local_ip, remote_ip | subnet_type, int_ext_subtype], sesid, srcid.
  2. area_num 49.0001 and asn 65100 identify the routing domain; sesid is the watcher session, srcid the source router System ID (router1, 0100.1001.0001).

Watcher CSV

host:    2026-09-07T06:55:14.754Z,lab-isis01,1,host,router2,down,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,192.168.23.2,192.168.23.1,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
metric:  2026-09-07T06:55:14.756Z,lab-isis01,1,metric,router2,changed,old_cost:10,new_cost:-1,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,192.168.23.2,192.168.23.1,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
network: 2026-09-07T06:55:14.761Z,lab-isis01,1,network,192.168.23.0/24,changed,old_cost:10,new_cost:-1,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,internal,0,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001

4. Where events come from

One IS-IS change produces one Watcher CSV line per level. Fluent Bit forwards it to Topolograph, which stores it and exposes it through the Monitoring page, the event API, and the SDK. Each exercise below checks the same fact at every level available to you.

  1. isis01 Watcher log -> Fluent Bit CSV parser -> Topolograph ingestion -> Monitoring page + event API -> Topolograph SDK
  2. Monitoring page: OSPF/IS-IS Real-Time Monitoring. Pick the graph by its timestamp in Choose the graph, set the From/To window in UTC, turn on the L1 and L2 toggles, click Find logs; the New/Old Subnets, Up/Down Links and Changed metric toggles filter what is listed.
  3. The graph selector lists every topology snapshot the watcher reported - that is the topology the watcher sent.
Topolograph OSPF/IS-IS Real-Time Monitoring controls for graph 07Sep2026_06h49m43s_6_hosts: the graph selector, the From/To time window, the L1 and L2 level toggles both on, the New/Old Subnets, Up/Down Links and Changed metric toggles, and Find logs. The Watchers Status panel reads "No watchers registered yet" because the containerlab watcher posts topology but not heartbeats.

SDK request

from topolograph import Topolograph

topo = Topolograph(url="http://<your-topolograph>:8080",
                   username="<email>", password="<password>")
graph = topo.graphs.get(latest=True)
print(graph.graph_time, graph.protocol, graph.hosts)

SDK output

07Sep2026_06h49m43s_6_hosts isis {'count': 6}

5. Cost change on a point-to-point link

Change router2 eth1 toward router3. IS-IS metrics are directed, so the reverse router3-to-router2 metric must not be described as the same scalar. router2 eth1 has no explicit isis metric, so it starts at the wide-metric default of 10. router2-router3 is a Level-1-2 circuit, so the change is advertised at both levels.

Command

sudo docker exec clab-isis01-router2 vtysh \
  -c 'conf t' -c 'interface eth1' -c 'isis metric 222'

Verify the change. Open each source available in your environment and check it against the examples below.

Watcher log The metric and network lines for the change and its rollback, at L1 and L2 L1 + L2

Watcher CSV

# router2: interface eth1 / isis metric 222
2026-09-07T06:52:15.035Z,lab-isis01,1,metric,router3,changed,old_cost:10,new_cost:222,router2,07Sep2026_06h49m43s_6_hosts,49.0001,65100,,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:52:15.037Z,lab-isis01,1,network,192.168.23.0/24,changed,old_cost:10,new_cost:222,router2,07Sep2026_06h49m43s_6_hosts,49.0001,65100,internal,0,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:52:15.045Z,lab-isis01,2,metric,router3,changed,old_cost:10,new_cost:222,router2,07Sep2026_06h49m43s_6_hosts,49.0001,65100,,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
# rollback (isis metric 10):
2026-09-07T06:52:37.514Z,lab-isis01,1,metric,router3,changed,old_cost:222,new_cost:10,router2,07Sep2026_06h49m43s_6_hosts,49.0001,65100,,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:52:37.526Z,lab-isis01,2,metric,router3,changed,old_cost:222,new_cost:10,router2,07Sep2026_06h49m43s_6_hosts,49.0001,65100,,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
Monitoring The 10 -> 222 metric change on the event feed

On OSPF/IS-IS Real-Time Monitoring, pick 07Sep2026_06h49m43s_6_hosts in Choose the graph, set the From/To window around 06:52 UTC, turn on L1 and L2, and click Find logs. With Changed metric on, the feed shows object router3, detected by router2, 10 -> 222 - the router2 -> router3 direction only, once for L1 and once for L2. The reverse direction is not listed.

SDK verification get_adjacency_events and get_network_events return the same object and costs 2 events

SDK request

adj = graph.events.get_adjacency_events(
    start_time="2026-09-07T06:52:14Z", end_time="2026-09-07T06:52:20Z")
net = graph.events.get_network_events(
    start_time="2026-09-07T06:52:14Z", end_time="2026-09-07T06:52:20Z")
for e in adj["adjacency_cost_change_events"]:
    print(e.event_object, e.event_detected_by, e.old_cost, "->", e.new_cost, "L" + str(e.level_number))
for e in net["network_cost_change_events"]:
    print(e.event_object, e.old_cost, "->", e.new_cost, "L" + str(e.level_number))

SDK output

router3 router2 10 -> 222 L1
router3 router2 10 -> 222 L2
3ffe::192:168:23:2/127 10 -> 222 L1
192.168.23.0/24 10 -> 222 L1
3ffe::192:168:23:2/127 10 -> 222 L2
192.168.23.0/24 10 -> 222 L2

The metric event names the router2 -> router3 direction (event_object router3, event_detected_by router2), 10 -> 222, at both levels; the IPv4 and IPv6 connected subnets carry the same move. The reverse direction is absent.

Ask the agent A generic incident question about router2

Prompt: What happened with router2 in the last 10 minutes?

A passing answer names the router2 -> router3 direction and both metric values (10 and 222) without the question naming cost or metric, and does not claim the reverse router3 -> router2 direction changed.

Rollback: Run isis metric 10 (or no isis metric) on router2 eth1 and confirm the inverse 222 -> 10 metric and network events.

6. Detecting internal and external prefix events

Run each prefix experiment independently. Unlike OSPF, IS-IS advertises a loopback with the mask it is configured with and costs it at the interface metric - a /24 stays a /24 at cost 10, it is not collapsed to a /32 host route. router6 is Level-2-only, so its prefixes appear at L2 alone.

  1. 6a. On router2, interface lo / ip address 192.168.123.1/24; observe 192.168.123.0/24 up and cost -1 -> 10 at L1 and L2.
  2. 6b. On router6, interface lo / ip address 10.10.36.6/24; observe 10.10.36.0/24 up and cost -1 -> 10 at L2.
  3. 6c. On router6, no ip route 6.6.6.6/32 192.168.36.3; observe 6.6.6.6/32 down and cost 11 -> -1 at L2. FRR redistributes it into IS-IS without the external bit, so the watcher tags it internal.

Command

sudo docker exec clab-isis01-router2 vtysh \
  -c 'conf t' -c 'interface lo' -c 'ip address 192.168.123.1/24'

Verify the change. Open each source available in your environment and check it against the examples below.

Watcher log One up/down and one changed line per sub-exercise 3 prefixes

Watcher CSV

# 6a router2: interface lo / ip address 192.168.123.1/24
2026-09-07T06:52:52.074Z,lab-isis01,1,network,192.168.123.0/24,up,router2,07Sep2026_06h49m43s_6_hosts,49.0001,65100,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:52:52.075Z,lab-isis01,1,network,192.168.123.0/24,changed,old_cost:-1,new_cost:10,router2,07Sep2026_06h49m43s_6_hosts,49.0001,65100,internal,0,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
# 6b router6: interface lo / ip address 10.10.36.6/24
2026-09-07T06:53:39.099Z,lab-isis01,2,network,10.10.36.0/24,up,router6,07Sep2026_06h49m43s_6_hosts,49.0001,65100,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:53:39.099Z,lab-isis01,2,network,10.10.36.0/24,changed,old_cost:-1,new_cost:10,router6,07Sep2026_06h49m43s_6_hosts,49.0001,65100,internal,0,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
# 6c router6: no ip route 6.6.6.6/32 192.168.36.3
2026-09-07T06:54:14.533Z,lab-isis01,2,network,6.6.6.6/32,down,router6,07Sep2026_06h49m43s_6_hosts,49.0001,65100,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:54:14.533Z,lab-isis01,2,network,6.6.6.6/32,changed,old_cost:11,new_cost:-1,router6,07Sep2026_06h49m43s_6_hosts,49.0001,65100,internal,0,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
Monitoring The new and withdrawn prefixes on the feed

On OSPF/IS-IS Real-Time Monitoring, pick this graph, set the window around 06:52-06:55 UTC, turn on L1 and L2, and click Find logs. With New/Old Subnets on, 192.168.123.0/24 and 10.10.36.0/24 appear as added and 6.6.6.6/32 as withdrawn.

SDK verification get_network_events, one changed event per sub-exercise 3 events

SDK request

for start, end in [("2026-09-07T06:52:50Z", "2026-09-07T06:52:55Z"),
                   ("2026-09-07T06:53:37Z", "2026-09-07T06:53:42Z"),
                   ("2026-09-07T06:54:12Z", "2026-09-07T06:54:17Z")]:
    net = graph.events.get_network_events(start_time=start, end_time=end)
    for e in net["network_up_down_events"]:
        print(e.event_object, e.event_status, e.event_detected_by,
              f"{e.old_cost} -> {e.new_cost}", "L" + str(e.level_number), e.subnet_type)

SDK output

192.168.123.0/24 changed router2 -1 -> 10 L1 internal
192.168.123.0/24 changed router2 -1 -> 10 L2 internal
10.10.36.0/24 changed router6 -1 -> 10 L2 internal
6.6.6.6/32 changed router6 11 -> -1 L2 internal

The /24 keeps its mask - no /32 collapse. Availability is an up/down line the forwarder drops; the cost move is the changed event the SDK returns. router2's loopback is seen at L1 and L2, router6's only at L2, and the redistributed 6.6.6.6/32 is tagged internal.

Rollback: 6a: no ip address 192.168.123.1/24 on router2 interface lo. 6b: no ip address 10.10.36.6/24 on router6 interface lo. 6c: ip route 6.6.6.6/32 192.168.36.3 on router6. Confirm the inverse event after each.

7. Detecting a connectivity loss and its recovery

Shut router2 eth1, inspect the correlated event set, then restore the interface with no shutdown. Because router2-router3 is Level-1-2, every line appears once for L1 and once for L2.

Command

# down
sudo docker exec clab-isis01-router2 vtysh \
  -c 'conf t' -c 'interface eth1' -c 'shutdown'
# recovery
sudo docker exec clab-isis01-router2 vtysh \
  -c 'conf t' -c 'interface eth1' -c 'no shutdown'

Verify the change. Open each source available in your environment and check it against the examples below.

Watcher log The L1 down set and the L1 recovery set for the router2-router3 link down + up

Watcher CSV

# router2: interface eth1 / shutdown
2026-09-07T06:55:14.754Z,lab-isis01,1,host,router2,down,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,192.168.23.2,192.168.23.1,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:55:14.756Z,lab-isis01,1,metric,router2,changed,old_cost:10,new_cost:-1,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,192.168.23.2,192.168.23.1,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:55:14.760Z,lab-isis01,1,metric,router3,changed,old_cost:10,new_cost:-1,router2,07Sep2026_06h49m43s_6_hosts,49.0001,65100,,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:55:14.761Z,lab-isis01,1,network,192.168.23.0/24,changed,old_cost:10,new_cost:-1,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,internal,0,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
# router2: interface eth1 / no shutdown
2026-09-07T06:55:39.545Z,lab-isis01,1,metric,router3,changed,old_cost:-1,new_cost:10,router2,07Sep2026_06h49m43s_6_hosts,49.0001,65100,,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:55:39.607Z,lab-isis01,1,metric,router2,changed,old_cost:-1,new_cost:10,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,192.168.23.2,192.168.23.1,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
Monitoring The outage as one wave on the feed

On OSPF/IS-IS Real-Time Monitoring, pick this graph, set the window around 06:55 UTC, turn on L1 and L2, and click Find logs. With Up/Down Links on, router2 and router3 each show their side of the link going to -1 and back, and 192.168.23.0/24 goes unreachable and returns - twice, once per level.

SDK verification get_adjacency_events returns the paired down/up moves 4 moves

SDK request

down = graph.events.get_adjacency_events(
    start_time="2026-09-07T06:55:14Z", end_time="2026-09-07T06:55:16Z")
for e in down["all_host_up_down_events"]:
    print(e.event_object, "detected_by", e.event_detected_by,
          f"{e.old_cost} -> {e.new_cost}", "L" + str(e.level_number))

SDK output

router2 detected_by router3 10 -> -1 L1
router3 detected_by router2 10 -> -1 L1
router2 detected_by router3 10 -> -1 L2
router3 detected_by router2 10 -> -1 L2

host router2 down detected by router3, both directed metrics to -1, 192.168.23.0/24 down - at L1 and L2; then the mirror recovery. On recovery the watcher also logs a node attr:attached flap on router2, which Topolograph does not yet ingest.

Ask the agent A generic incident question about the network

Prompt: What happened in the network in the last 30 minutes?

Captured answer (Qwen; wording varies between runs)

In the last 30 minutes the IS-IS domain (area 49.0001, AS 65100) had 7 event waves detected by watcher lab-isis01, starting at 06:55:15 UTC. Every device that went down recovered within the window.

06:55:15 - router2 and router3 each lost the adjacency between them, at L1 and L2 (router2 detected by router3, router3 detected by router2).

06:55:40 - both routers restored that adjacency at L1 and L2, followed by a few repeated up notifications through 06:56:33.

06:57:23 - router6 lost its L2 adjacency to router3; 06:57:56 - router6 and router3 restored it.

In short: the router2-router3 link dropped at 06:55:15 and came back at 06:55:40; the router3-router6 link dropped at 06:57:23 and recovered at 06:57:56 - both fully converged.

The answer names the failed adjacency (router2 - router3), that router3 detected router2's loss and vice versa, both metrics at -1, and the recovery - the same facts as the CSV lines above, from a question that never says "adjacency" or "failure". The router6 lines are the transit exercise below, caught by the same 30-minute window.

Rollback: no shutdown on router2 eth1 restores the interface; wait for the host, network and metric recovery events at both levels.

8. Reporting on a broadcast transit segment

router6 eth1 faces router3 on a broadcast (LAN) circuit. router6 carries isis priority 100 against router3's 64, so router6 is the DIS and originates the pseudonode LSP. The circuit is Level-2-only, so every line is L2. router6 eth1 has no explicit isis metric, so the baseline is the default 10.

Command

# cost
sudo docker exec clab-isis01-router6 vtysh \
  -c 'conf t' -c 'interface eth1' -c 'isis metric 66'
# then, separately: shutdown, then no shutdown
sudo docker exec clab-isis01-router6 vtysh \
  -c 'conf t' -c 'interface eth1' -c 'shutdown'
sudo docker exec clab-isis01-router6 vtysh \
  -c 'conf t' -c 'interface eth1' -c 'no shutdown'

Verify the change. Open each source available in your environment and check it against the examples below.

Watcher log The L2 cost change with its network side-effects, plus the shutdown/recovery pair cost + up/down

Watcher CSV

# router6: interface eth1 / isis metric 66   (rollback: isis metric 10)
2026-09-07T06:56:47.139Z,lab-isis01,2,network,192.168.36.0/24,changed,old_cost:10,new_cost:66,router6,07Sep2026_06h49m43s_6_hosts,49.0001,65100,internal,0,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:56:47.140Z,lab-isis01,2,metric,router3,changed,old_cost:10,new_cost:66,router6,07Sep2026_06h49m43s_6_hosts,49.0001,65100,,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:57:04.561Z,lab-isis01,2,metric,router3,changed,old_cost:66,new_cost:10,router6,07Sep2026_06h49m43s_6_hosts,49.0001,65100,,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
# router6: interface eth1 / shutdown
2026-09-07T06:57:22.037Z,lab-isis01,2,network,192.168.36.0/24,changed,old_cost:10,new_cost:-1,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,internal,0,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:57:22.055Z,lab-isis01,2,host,router6,down,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,192.168.36.3,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:57:22.055Z,lab-isis01,2,metric,router6,changed,old_cost:10,new_cost:-1,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,192.168.36.3,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
# router6: interface eth1 / no shutdown
2026-09-07T06:57:55.962Z,lab-isis01,2,host,router6,up,router3,07Sep2026_06h49m43s_6_hosts,49.0001,65100,192.168.36.3,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
2026-09-07T06:57:55.973Z,lab-isis01,2,metric,router3,changed,old_cost:-1,new_cost:66,router6,07Sep2026_06h49m43s_6_hosts,49.0001,65100,,,e50b9d98-aa86-11f1-92b0-46ed252a638b,0100.1001.0001
Monitoring The transit cost change and the router6 outage, L2 only

On OSPF/IS-IS Real-Time Monitoring, pick this graph, set the window around 06:56-06:58 UTC, turn on L2, and click Find logs. The metric event object is router3 with event_detected_by router6, and the change is L2 only - the circuit is level-2-only. On shutdown the feed shows host router6 down and 192.168.36.0/24 unreachable, then the recovery.

SDK verification get_adjacency_events for the transit cost and the shutdown/recovery L2 only

SDK request

cost = graph.events.get_adjacency_events(
    start_time="2026-09-07T06:56:46Z", end_time="2026-09-07T06:56:49Z")
for e in cost["adjacency_cost_change_events"]:
    print("cost", e.event_object, e.event_detected_by, f"{e.old_cost} -> {e.new_cost}", "L" + str(e.level_number))
flap = graph.events.get_adjacency_events(
    start_time="2026-09-07T06:57:21Z", end_time="2026-09-07T06:58:00Z")
for e in flap["all_host_up_down_events"]:
    print("updown", e.event_object, e.event_detected_by, f"{e.old_cost} -> {e.new_cost}", "L" + str(e.level_number))

SDK output

cost router3 router6 10 -> 66 L2
updown router6 router3 10 -> -1 L2
updown router6 router3 -1 -> 10 L2
updown router3 router6 -1 -> 10 L2

On the broadcast segment the metric event still names the neighbour (router3), detected by router6, and only at L2; the metric change also moves 192.168.36.0/24 and the two IPv6 /127s. The shutdown is host router6 down at L2, then recovery.

Ask the agent A generic incident question about router6

Prompt: What happened on router6 in the last 20 minutes?

A passing answer names the router6 -> router3 transit metric moving 10 -> 66 and back, then the router6 attachment going down (metric -1 at L2) and recovering, without the question naming cost, DIS, or shutdown.

Rollback: isis metric 10 (or no isis metric) on router6 eth1 restores the baseline 10; no shutdown restores the adjacency. Confirm the inverse metric events.

9. Changing a link's TE attributes

router2 eth1 toward router3 already carries RFC 5305 TE sub-TLVs: mpls-te on plus per-interface link-params advertise Administrative Group, Maximum (Reservable) Bandwidth, and per-priority Unreserved Bandwidth alongside the ordinary metric. Changing one of those without touching the IGP metric still produces its own event family, temetric, distinct from metric -- and it fires once per level on this Level-1-2 circuit, same as metric does.

admin-grp is the field IS-IS/MPLS-TE uses for CSPF constraint-based routing ("avoid links outside the gold class"). 0x647a0001 sets bits 0,17,19,20,21,22,26,29,30; the change below clears bit 0 and sets bit 1, leaving every other bit untouched.

Command

sudo docker exec clab-isis01-router2 vtysh \
  -c 'conf t' -c 'interface eth1' -c 'link-params' \
  -c 'admin-grp 0x647a0002'

Verify the change. Open each source available in your environment and check it against the examples below.

Watcher log The temetric events for the change and its rollback, at L1 and L2 L1 + L2

Watcher CSV

# router2: interface eth1 / link-params / admin-grp 0x647a0002
2026-09-12T19:26:19.170Z,lab-isis01,1,temetric,router3,changed,1_17_19_20_21_22_26_29_30,1410065408,1410065408,9600000_9680000_9760000_9840000_9920000_10000000_1410065408_1410065408,23,router2,12Sep2026_19h04m47s_6_hosts,49.0001,65100,192.168.23.1,,dabf99b6-aedc-11f1-a144-46ed252a638b,0100.1001.0001,
2026-09-12T19:26:19.179Z,lab-isis01,2,temetric,router3,changed,1_17_19_20_21_22_26_29_30,1410065408,1410065408,9600000_9680000_9760000_9840000_9920000_10000000_1410065408_1410065408,23,router2,12Sep2026_19h04m47s_6_hosts,49.0001,65100,192.168.23.1,,dabf99b6-aedc-11f1-a144-46ed252a638b,0100.1001.0001,
# rollback (admin-grp 0x647a0001):
2026-09-12T19:30:54.362Z,lab-isis01,1,temetric,router3,changed,0_17_19_20_21_22_26_29_30,1410065408,1410065408,9600000_9680000_9760000_9840000_9920000_10000000_1410065408_1410065408,23,router2,12Sep2026_19h04m47s_6_hosts,49.0001,65100,192.168.23.1,,dabf99b6-aedc-11f1-a144-46ed252a638b,0100.1001.0001,
2026-09-12T19:30:54.376Z,lab-isis01,2,temetric,router3,changed,0_17_19_20_21_22_26_29_30,1410065408,1410065408,9600000_9680000_9760000_9840000_9920000_10000000_1410065408_1410065408,23,router2,12Sep2026_19h04m47s_6_hosts,49.0001,65100,192.168.23.1,,dabf99b6-aedc-11f1-a144-46ed252a638b,0100.1001.0001,
Monitoring The temetric row on the event feed

On OSPF/IS-IS Real-Time Monitoring, pick 12Sep2026_19h04m47s_6_hosts in Choose the graph, set the From/To window around 19:26 UTC, turn on L1 and L2, and click Find logs. The change appears as a temetric row for router3 detected by router2, at both levels.

SDK verification get_adjacency_events returns the temetric change, correlated to the link and timestamp L1 + L2

SDK request

adj = graph.events.get_adjacency_events(
    start_time="2026-09-12T19:26:18Z", end_time="2026-09-12T19:26:21Z")
for e in adj["adjacency_cost_change_events"]:
    print(e.event_name, e.event_object, e.event_detected_by, "L" + str(e.level_number))

SDK output

temetric router3 router2 L1
temetric router3 router2 L2

The SDK confirms a temetric change on this link at both levels, correlated to the same timestamp as the Watcher lines above -- but get_adjacency_events does not yet surface the changed admin-group, bandwidth or TE-metric values themselves; read those from the Watcher CSV or the temetric_change collection directly.

Rollback: Run admin-grp 0x647a0001 under link-params on router2 eth1 and confirm the inverse temetric events at L1 and L2.

10. Reading TE attributes and filtering edges by IS-IS level

Every parsed link carries its metric and RFC 5305 TE sub-TLVs (admin group, bandwidth pools, TE metric) alongside isis_level -- 1 or 2 for a link advertised at one level only, 3 (L1L2) for one advertised at both. router3-router6 is Level-2-only, so it is the one edge on this graph tagged isis_level: 2; every other link here is 3.

This graph (18Sep2026_08h50m15s_6_hosts) came from a full LSDB parse (POST /graphs with router1's show isis database detail). A Watcher-sourced graph stores per-level TE snapshots the same way, so it answers a level-aware request with real data too.

The Web edge tooltip for router2-router3, now split into two side-by-side panels -- 0100.1001.0003 to 0100.1001.0002 (Metric 10) on the left and 0100.1001.0002 to 0100.1001.0003 (Metric 50) on the right -- each with its own Local/Remote IP, TE Metric, Bandwidth and Unreserved BW badges, plus the L1/L2 breakdown further down. Before the router2 mpls-te router-address fix (Point-to-point link section rollback note), router2's directed edges landed on two different vertices and this same link showed a single column with one direction missing.

Command

curl -u "<email>:<password>" "http://<your-topolograph>:8080/api/graph/18Sep2026_08h50m15s_6_hosts/edges?isis_level=2"
curl -u "<email>:<password>" "http://<your-topolograph>:8080/api/graph/18Sep2026_08h50m15s_6_hosts/edges?admin_group=0x647a0001"
curl -u "<email>:<password>" "http://<your-topolograph>:8080/api/graph/18Sep2026_08h50m15s_6_hosts/edges?temetric__gt=25"

SDK request

edges = graph.edges_list(isis_level=2)
for e in edges["items"]:
    print(e["src"], "->", e["dst"], e["isis_level"], e["cost"])

SDK output

10.10.10.3 -> 10.10.10.6 2 10
10.10.10.6 -> 10.10.10.3 2 10
Required observations Facts to confirm before continuing
  • isis_level=2 lists exactly one edge per direction on the Level-2-only circuit: 10.10.10.3 -> 10.10.10.6 and 10.10.10.6 -> 10.10.10.3, both isis_level: 2 -- no other link in this six-router graph matches.
  • admin_group=0x647a0001 lists 2 edges: 10.10.10.1 -> 10.10.10.3 (cost 10) and 10.10.10.2 -> 10.10.10.3 (cost 50) -- the two directions router1/router2 advertise toward router3 with that group, each carrying its own admin_group, temetric and unreserved_bw_0..unreserved_bw_7 fields.
  • temetric__gt=25 lists 2 edges: 10.10.10.1 -> 10.10.10.3 (temetric: 111111) and 10.10.10.3 -> 10.10.10.2 (temetric: 32) -- the range operator reads the same TE-metric field the Web tooltip's TE Metric badge shows.

11. Computing CSPF for one IS-IS level

GET /graph/{graph_time}/cspf-path/{node_a}/{node_b}?level=1|2 answers a question the combined/default calculation cannot: which path, if any, exists using strictly that level's own advertised metrics and TE data. router3-router6 is Level-2-only, so router6 is not even a member of the Level-1 topology -- a Level-1 request between router1 and router6 is not a strict-constraint failure, it is src/dst not found.

Requesting an explicit level against a graph that predates per-level storage -- such as one uploaded before per-level snapshots were introduced -- fails closed with 422 and code: isis_level_calculation_unavailable rather than silently answering from the merged view.

Command

curl -u "<email>:<password>" "http://<your-topolograph>:8080/api/graph/18Sep2026_08h50m15s_6_hosts/cspf-path/10.10.10.1/10.10.10.6"
curl -u "<email>:<password>" "http://<your-topolograph>:8080/api/graph/18Sep2026_08h50m15s_6_hosts/cspf-path/10.10.10.1/10.10.10.6?level=1"
curl -u "<email>:<password>" "http://<your-topolograph>:8080/api/graph/18Sep2026_08h50m15s_6_hosts/cspf-path/10.10.10.1/10.10.10.6?level=2"

SDK request

for level in (None, 1, 2):
    r = graph.cspf_path("10.10.10.1", "10.10.10.6", level=level)
    print(level, r)

SDK output

None {'cost': 20, 'path': ['10.10.10.1', '10.10.10.3', '10.10.10.6'], 'reason': ''}
1 {'cost': None, 'path': [], 'reason': 'src/dst not found'}
2 {'cost': 20, 'path': ['10.10.10.1', '10.10.10.3', '10.10.10.6'], 'reason': ''}
Required observations Facts to confirm before continuing
  • No level (combined/default view): "path": ["10.10.10.1", "10.10.10.3", "10.10.10.6"], "cost": 20.
  • level=1: "path": [], "reason": "src/dst not found" -- router6 has no Level-1 membership at all, on this or any graph, since its only circuit is Level-2-only.
  • level=2: identical to the combined view here ("cost": 20) -- this graph's one Level-1-2 hop, router1-router3, also advertises at L2, so the L2-only path matches the default one.
  • The same request against the pre-per-level graph 12Sep2026_19h04m47s_6_hosts (?level=1) returns 422 with "code": "isis_level_calculation_unavailable", "action": "reupload_graph" -- re-run the LSDB upload under the current Topolograph version to get a real per-level answer instead.

12. Load the 13-router IS-IS lab

The six-router lab is useful for inspecting Watcher events and TE fields, but it does not provide enough alternative routes for meaningful CSPF exercises. Use the 13-hosts-demo-isis lab: deploy it yourself, or skip the deployment and download the ready LSDB demo_isis_LSDB.txt.

Upload the LSDB to Topolograph as an FRR IS-IS file: your capture from r30, or the downloaded demo_isis_LSDB.txt. The resulting graph has 13 nodes and 52 directed edges: 12 at Level 1 and 40 at Level 2. Fifty edges carry TE attributes; the two directions of r110-r111 do not.

Command

cd containerlab/13-hosts-demo-isis
sudo clab deploy --topo 13-hosts-demo-isis.clab.yml
sudo docker exec clab-13-hosts-demo-isis-r30 vtysh -c 'show isis database detail'

SDK request

from topolograph import Topolograph

topo = Topolograph(url="http://<your-topolograph>:8080", username="<email>", password="<password>")
graph = topo.graphs.upload(open("demo_isis_LSDB.txt").read(), vendor="FRR", protocol="isis")
print(graph.graph_time, graph.hosts)
print(graph.edges_list(per_page=200)["pagination"]["total"])
print(graph.edges_list(is_te_link=True, per_page=200)["pagination"]["total"])
print(graph.edges_list(is_te_link=False, per_page=200)["pagination"]["total"])

SDK output

<new graph time> {'count': 13}
52
50
2
Required observations Facts to confirm before continuing
  • The uploaded graph reports {'count': 13} hosts and 52 edges.
  • edges_list(is_te_link=True) returns 50 edges; edges_list(is_te_link=False) returns the two r110-r111 directions.

13. Read TE values before computing a path

Inspect the graph's TE attributes before applying a constraint. Identify the gold and red admin groups, high TE metrics, the small priority-7 bandwidth pool, SRLG-bearing links, and links that are not TE-enabled; a plain shortest-path calculation does not show whether a link is suitable for a tunnel.

On this graph, gold (0x00000002) matches 10 directed edges, red (0x00000004) matches 2, temetric__gt=30 matches 4, unreserved_bw_7__lt=1000000 matches 2, and is_te_link=False returns the two directions of r110-r111.

Command

# Run from the SDK against the graph uploaded in step 1

SDK request

for query in (
    {'admin_group': '0x00000002'},
    {'admin_group': '0x00000004'},
    {'temetric__gt': 30},
    {'unreserved_bw_7__lt': 1_000_000},
    {'is_te_link': False},
):
    result = graph.edges_list(per_page=200, **query)
    print(query, result['pagination']['total'])

SDK output

{'admin_group': '0x00000002'} 10
{'admin_group': '0x00000004'} 2
{'temetric__gt': 30} 4
{'unreserved_bw_7__lt': 1000000} 2
{'is_te_link': False} 2
Required observations Facts to confirm before continuing
  • The 8 edges on r10-r100 and r100-r110/r100-r111 carry SRLG data; the two parallel r10-r100 links keep their separate groups.
  • Passing is_te_link='yes' is rejected with HTTP 400: Wrong type, expected 'boolean'.

14. Record the unconstrained route

Calculate the unconstrained path from r10 to r14 and keep it as the baseline for the remaining exercises. Each following constraint can then be compared with a known route.

The reference path is r10 r100 r110 r14, with cost 30. It uses the eth2 parallel link between r10 and r100.

Command

# No router change: calculate the reference CSPF path

SDK request

print(graph.cspf_path('r10', 'r14'))

SDK output

{'cost': 30, 'path': ['r10', 'r100', 'r110', 'r14'], 'reason': ''}
Required observations Facts to confirm before continuing
  • The baseline uses IGP metrics. The next four steps deliberately move it with one constraint at a time.

15. Avoid shared-risk groups

Use srlg_exclude to route around a link or shared duct under maintenance. The reference path r10 r100 r110 r14 costs 30: its first hop belongs to SRLG 300, while both parallel first hops share SRLG 100.

Excluding 300 keeps the same routers but selects eth1, raising the cost to 35. Excluding 100 gives r10 r11 r100 r110 r14 at cost 40; excluding 200 gives r10 r100 r101 r111 r14 at cost 45; excluding both 100 and 200 gives r10 r11 r101 r111 r14 at cost 50.

Command

# No router change: change the excluded risk group in each request

SDK request

for groups in ([300], [100], [200], [100, 200]):
    print(groups, graph.cspf_path('r10', 'r14', srlg_exclude=groups))

SDK output

[300] {'cost': 35, 'path': ['r10', 'r100', 'r110', 'r14'], 'reason': ''}
[100] {'cost': 40, 'path': ['r10', 'r11', 'r100', 'r110', 'r14'], 'reason': ''}
[200] {'cost': 45, 'path': ['r10', 'r100', 'r101', 'r111', 'r14'], 'reason': ''}
[100, 200] {'cost': 50, 'path': ['r10', 'r11', 'r101', 'r111', 'r14'], 'reason': ''}
Required observations Facts to confirm before continuing
  • The route changes because the constraint removes links, not because any IGP metric changed.

16. Select a route by bandwidth and setup priority

Request an 8 Mbit tunnel at different setup priorities. The reference route r10 r100 r110 r14 costs 30, but r100-r110 advertises only 4 Mbit in its priority-7 pool and 10 Mbit at priorities 0-3.

At priority 7, CSPF moves the tunnel to r10 r100 r111 r14, cost 40. At priority 0, it keeps r10 r100 r110 r14, cost 30. A 20 Mbit request fits on no available path and returns a constraint refusal.

Command

# No router change: vary bandwidth and setup_priority

SDK request

for priority in (7, 0):
    print(priority, graph.cspf_path('r10', 'r14', bandwidth='8M', setup_priority=priority))
print(graph.cspf_path('r10', 'r14', bandwidth='20M'))

SDK output

7 {'cost': 40, 'path': ['r10', 'r100', 'r111', 'r14'], 'reason': ''}
0 {'cost': 30, 'path': ['r10', 'r100', 'r110', 'r14'], 'reason': ''}
{'cost': None, 'path': [], 'reason': 'no path satisfies the requested constraints'}
Required observations Facts to confirm before continuing
  • Setup priority selects the advertised unreserved-bandwidth pool used for placement; it does not alter the route's IGP cost.

17. Compute with the TE metric

Recalculate the path with metric_type='te' after checking that every candidate link carries a TE metric. The IGP path r10 r100 r110 r14 costs 30, but r100-r110 has TE metric 100 while r100-r111 has TE metric 10.

CSPF selects r10 r100 r111 r14 with TE cost 30. The same path has IGP cost 40, which makes the effect of the selected metric visible.

Command

# No router change: select TE cost instead of IGP cost

SDK request

print(graph.cspf_path('r10', 'r14', metric_type='te'))

SDK output

{'cost': 30, 'path': ['r10', 'r100', 'r111', 'r14'], 'reason': ''}
Required observations Facts to confirm before continuing
  • If a candidate link has no TE metric, CSPF falls back to IGP for that calculation; inspect the edge attributes first.

18. Include or exclude admin-group bits

Use admin-group bits to require or avoid link classes. Bit 1 is gold and bit 2 is red. The unconstrained path from r13 to r15 is r13 r100 r110 r15, cost 30.

Requiring gold with admin_include_all=['1'] changes that path to r13 r101 r111 r15, cost 45; from r10 to r14, it gives r10 r101 r111 r14, cost 55. Avoiding red with admin_exclude_any=['2'] gives r13 r100 r111 r15, cost 40.

Command

# No router change: admin groups are bit numbers, not hexadecimal mask values

SDK request

print(graph.cspf_path('r13', 'r15', admin_include_all=['1']))
print(graph.cspf_path('r10', 'r14', admin_include_all=['1']))
print(graph.cspf_path('r13', 'r15', admin_exclude_any=['2']))

SDK output

{'cost': 45, 'path': ['r13', 'r101', 'r111', 'r15'], 'reason': ''}
{'cost': 55, 'path': ['r10', 'r101', 'r111', 'r14'], 'reason': ''}
{'cost': 40, 'path': ['r13', 'r100', 'r111', 'r15'], 'reason': ''}
Required observations Facts to confirm before continuing
  • Bit 1 means mask 0x00000002; bit 2 means mask 0x00000004. Bit 0 is the least significant bit.

19. Restrict CSPF to one IS-IS level

Set level=1 or level=2 to calculate inside one IS-IS topology and verify that both endpoints belong to it. Between r30 and r31, the direct Level-1 path costs 10, while the Level-2 path r30 r10 r11 r31 costs 40.

The r130-r131 path costs 10 at Level 1 but returns src/dst not found at Level 2. The combined topology has an r130-r15 path of cost 50, but neither individual level has that path because it crosses the level boundary.

Command

# No router change: constrain the request to Level 1 or Level 2

SDK request

for level in (None, 1, 2):
    print(level, graph.cspf_path('r30', 'r31', level=level))
print(graph.cspf_path('r130', 'r131', level=2))

SDK output

None {'cost': 10, 'path': ['r30', 'r31'], 'reason': ''}
1 {'cost': 10, 'path': ['r30', 'r31'], 'reason': ''}
2 {'cost': 40, 'path': ['r30', 'r10', 'r11', 'r31'], 'reason': ''}
{'cost': None, 'path': [], 'reason': 'src/dst not found'}
Required observations Facts to confirm before continuing
  • An explicit level request on a graph stored before per-level data returns HTTP 422 with code: isis_level_calculation_unavailable; it must be reuploaded.

20. Distinguish an unsatisfied constraint from a missing endpoint

Read reason whenever CSPF returns an empty path. The 20 Mbit request from step 5 has valid endpoints but no path with enough bandwidth, while a Level-1 request from r130 to r15 has no single-level topology containing both endpoints.

The bandwidth request returns no path satisfies the requested constraints; the wrong-level request returns src/dst not found. These failures require different corrective actions.

Command

# Repeat the failed requests from steps 5 and 8

SDK request

print(graph.cspf_path('r10', 'r14', bandwidth='20M'))
print(graph.cspf_path('r130', 'r15', level=1))

SDK output

{'cost': None, 'path': [], 'reason': 'no path satisfies the requested constraints'}
{'cost': None, 'path': [], 'reason': 'src/dst not found'}
Required observations Facts to confirm before continuing
  • The first reason calls for relaxing bandwidth, affinity, or SRLG constraints; the second means the requested topology does not contain both endpoints.

21. Edit one edge and recompute

Model maintenance on r100-r110 without changing the routers. On a disposable graph, use update_edge to add SRLG 999 to the Level-2 edge, recalculate while excluding that group, and then clear the SRLG.

Excluding 999 changes the path from r10 r100 r110 r14, cost 30, to r10 r100 r111 r14, cost 40; clearing the SRLG restores the reference path. A Level-2-only edge rejects isis_level=1. Because replace_edge removes per-level data, an explicit-level CSPF request returns HTTP 422 after a full replacement.

Command

# Run this on a disposable graph: the edit is persistent
# Find the edge id first with edges_list(include=['edge_key'])

SDK request

edge = next(e for e in graph.edges_list(include=['edge_key'], per_page=200)['items']
             if e['src'] == 'r100' and e['dst'] == 'r110')
print(graph.update_edge(edge['id'], isis_level=2, srlg=[999]))
print(graph.cspf_path('r10', 'r14', srlg_exclude=[999]))
print(graph.update_edge(edge['id'], isis_level=2, srlg=[]))
print(graph.cspf_path('r10', 'r14'))

SDK output

... srlg: [999] ...
{'cost': 40, 'path': ['r10', 'r100', 'r111', 'r14'], 'reason': ''}
... srlg: [] ...
{'cost': 30, 'path': ['r10', 'r100', 'r110', 'r14'], 'reason': ''}
Required observations Facts to confirm before continuing
  • update_edge is partial; replace_edge is a full rewrite. Do not use either on the reference graph you want to keep unchanged.

22. Place tunnels and inspect residual bandwidth

Add four 4 Mbit tunnels from r10 to r14, one at a time, and inspect the residual bandwidth and selected path after each placement. Before the first placement, the 4 Mbit priority-7 pool on the reference path r10 r100 r110 r14 is available.

T1 uses r10 r100 r110 r14, cost 30; T2 uses r10 r100 r111 r14, cost 40; T3 uses the other r10-r100 link, cost 45; and T4 uses r10 r11 r101 r110 r14, cost 55. Delete all test LSPs when the exercise is complete.

Command

# No router change: create four 4 Mbit RSVP-TE tunnels through the SDK

SDK request

for name in ('T1', 'T2', 'T3', 'T4'):
    lsp = graph.add_lsp({
        'name': name, 'src': 'r10', 'dst': 'r14', 'bandwidth': '4M',
        'paths': {'primary': {'role': 'primary', 'bandwidth': '4M'}},
    })
    primary = lsp['paths']['primary']
    print(name, primary['path'], primary['cost'])
print(graph.edges_list(include=['lsp_left_bw', 'lsps'], per_page=200)['items'])
graph.delete_lsps()

SDK output

T1 ['r10', 'r100', 'r110', 'r14'] 30
T2 ['r10', 'r100', 'r111', 'r14'] 40
T3 ['r10', 'r100', 'r110', 'r14'] 45
T4 ['r10', 'r11', 'r101', 'r110', 'r14'] 55
... lsp_left_bw_7 and lsps for each edge ...
{'deleted': 4}
Required observations Facts to confirm before continuing
  • Use edges_list(include=['lsp_left_bw', 'lsps']) after each placement to see which pool caused the next route move.
  • Always call delete_lsps() in cleanup; LSP placement changes the stored graph.
Topolograph 2.72.1 📣 Join to the community!