It's the network.
Show me the evidence.

01 / Connectivity / OutgoingPorts

Only odd numbered ports?

At one point I ran into a network where outbound connections appeared to work only when the destination port was odd numbered. That's weird enough that I wanted a way to prove exactly what was happening instead of guessing at it.

"The internet doesn't work" is basically useless as a troubleshooting statement. If TCP/443 works, TCP/444 fails, TCP/445 works, and TCP/446 fails, now we actually have something interesting.

Build a test for the exact problem

I ended up building OutgoingPorts so I could test arbitrary outbound TCP ports and see whether a three-way handshake actually completed. The remote side also records the source and destination port it sees, which makes the tool useful when NAT is involved.

That last part matters. You might think you're testing one source port locally while a firewall is translating it into something completely different before the packet reaches the server.

Instead of trying random websites or doing fifty different telnet tests, I could produce a repeatable pattern and then compare that pattern against the firewall, NAT, and packet captures.

02 / DHCP / QinQ / MPLS

The OFFER left. Then it disappeared.

A phone boots, sends DHCP, and never gets an address. Easy enough to blame DHCP. Except the DHCP server was answering.

We captured the DHCP transaction on the server side and could clearly see the DHCP OFFER leaving the firewall. The problem was that the same OFFER never appeared on the far side of the transport network.

That immediately changes the troubleshooting scope. If the server generated the OFFER and I can physically see that packet leave its interface, I'm done trying random DHCP server settings until somebody proves the packet actually arrives at the client side.

And only one VLAN was broken

The path crossed a managed transport network using VLAN encapsulation and QinQ. Other traffic was working. The failure appeared to be isolated to VLAN 116, which made the problem even more interesting.

We also rebooted a phone at another remote site and reproduced the same behavior. The phone dropped off the network and could not obtain a new lease.

Shortly afterward another phone at that location went offline. For a minute, that looked like the problem was spreading.

It wasn't.

Somebody had moved the original non-working phone onto the port where the working phone had been connected. The second phone lost power because it had been unplugged. Completely unrelated to DHCP, but exactly the kind of thing that can send troubleshooting in the wrong direction if the physical event isn't accounted for.

Packet boundaries matter

Capturing at only one point would have left this as "DHCP isn't working." Capturing on both sides turned it into something much more specific: the OFFER exists here and does not exist there.

03 / VoIP / SIP / RTP

RTP was fine. The call still dropped.

Calls were dropping a few minutes after being established. The first assumption with VoIP is usually packet loss, jitter, switching, QoS, or something else in the media path.

So I captured it.

RTP between the phones was flowing directly between the endpoints. The captures showed bidirectional media and I wasn't seeing the packet-loss event that would explain the disconnect.

Then the SIP side gave away what was actually happening.

The network didn't kill the call

At roughly the same point in the failed calls, the call-control server sent a SIP BYE. In one capture it happened at about 235 seconds.

That's a very different problem from RTP falling apart.

If the PBX tells both endpoints to tear the call down, the phones are doing exactly what they were instructed to do. The next questions become session timers, endpoint state, registration, call-control logic, keepalives, resource events, and server logs.

Capture the right server

Another useful discovery came later when we realized one of the phones was registered to a different call-control appliance than the appliance being captured.

That's an easy mistake to make in a distributed voice environment. You can have a beautiful packet capture of completely irrelevant traffic.

We expanded the test to compare captures around the phones, the switching path, and the actual call-control system. We also looked at the periodic control and keepalive traffic instead of focusing exclusively on RTP.

04 / Switching / SPBm / VOSSLoopTrace

Making a noisy VOSS trace useful

Sometimes the switch already has the answer. The problem is buried inside five thousand lines of output.

I've spent a lot of time working with Extreme Fabric Engine/VOSS and SPBm. When something starts hammering the control plane, one of the useful tools is the switch's trace output.

The downside is that trace output gets noisy very quickly.

Find the things that keep showing up

VOSSLoopTrace came from wanting to automate the boring part. It can process an existing trace or collect one directly from a switch, then start identifying ports, VLANs, and MAC addresses that repeatedly appear in traffic hitting the control plane.

That doesn't mean every MAC it reports is guilty. It means those are the things I want to look at first.

I've dealt with environments where loops, port flapping, inconsistent tagging, missing fabric services, and other Layer 2 problems were all possible suspects. Being able to reduce a huge trace into a short list of interfaces and sources makes the next packet capture considerably easier.

05 / Routing / BGP / BFD / OSPF

The BGP problem wasn't BGP

BGP was repeatedly dropping. BFD was involved. There were also fabric problems nearby, so there were plenty of places to point a finger.

This is where troubleshooting by protocol name gets dangerous. The alarm says BGP went down, so everybody starts staring at BGP.

BGP was the victim.

Another routing protocol was changing the path

OSPF was also enabled on the firewall. The interaction between the routing protocols was changing reachability underneath the BFD/BGP session and the adjacency would flap.

Once OSPF was removed from that path, BFD and BGP stabilized.

There were other things worth cleaning up in the environment too, including fabric-service inconsistencies, MLT tagging differences, and interfaces that had been flapping. But those shouldn't be combined into one giant root cause simply because they existed at the same time.

Symptoms aren't ownership

A BGP-down alarm tells me which protocol noticed the failure. It doesn't necessarily tell me which protocol caused it.

06 / Packet experiments / IPv4 / IPv6 / PMTU

The 33,000-byte question

I wanted to see exactly what happened when I shoved a roughly 33,000-byte ping through a network that obviously couldn't carry it as one Ethernet frame.

This turned into one of those experiments where Wireshark is more useful than the ping result itself.

One packet becomes a pile of packets

The capture showed the oversized IP packet being carried as a series of fragments. In one of the tests I was looking at roughly 22 fragments for the original packet.

That also forced me to be precise about what "packet size" meant. Application payload, ICMP size, IP packet size, and what appears on the wire are not interchangeable numbers.

I also deliberately experimented with Path MTU Discovery and different Don't Fragment behavior because a successful ping by itself doesn't tell you whether PMTUD is actually working correctly.

IPv6 makes this more interesting

IPv6 moved fragmentation responsibility away from intermediate routers. If a packet is too large, the network is supposed to return an ICMPv6 Packet Too Big message and let the source adjust.

That's great until ICMPv6 gets filtered somewhere and everything smaller works while larger traffic mysteriously dies.

Those failures are why "I can ping it" has never been enough evidence for me that a path is healthy.

07 / IPv6 / BGP / FRR / FortiGate

Why is my next hop link-local?

Getting my own ASN and IPv6 space meant I could finally break BGP in ways that were entirely my own fault.

One of the more interesting lab problems involved IPv6 routes being received correctly while the next hop being advertised was a link-local address.

That's perfectly normal in plenty of IPv6 scenarios. It becomes less helpful when your BGP peer is several routed hops away through an IPv4 SIT tunnel.

The route existed. The next hop didn't.

I was peering FRRouting and FortiGate across 6in4 tunnels while announcing my IPv6 allocation. BGP could exchange the NLRI, but a route is useless if the receiving router cannot actually resolve the next hop.

That sent me through route maps, global IPv6 addresses, static routes, next-hop-self behavior, EBGP multihop, and the difference between seeing a prefix in the BGP table and having a usable route in the forwarding table.

I also hit another classic BGP issue during this work: inbound updates being discarded because an inbound policy wasn't present. The peer was established, routes were being advertised, and they still weren't being installed.

Established doesn't mean working

Seeing Established is only the beginning. I want to know what I advertised, what I received, what next hop was attached to it, whether that next hop resolves, and whether the route actually made it into the FIB.

08 / Multicast / PIM-SM / IGMP

Stop flooding it everywhere

Multicast is one of those things that can appear to work while doing something absolutely terrible to the network underneath it.

I worked through an environment where multicast traffic was contributing to alerts and generally behaving more like traffic that was being sprayed around than intentionally routed.

Give multicast an actual topology

I implemented PIM Sparse Mode through the routed network and used a loopback as the static Rendezvous Point. The other routers were configured to use the same RP and participate in the sparse-mode topology.

Afterward, the amount of multicast-related protection noise dropped significantly.

That makes sense. The point of multicast isn't "send this everywhere and hope somebody wanted it." IGMP tells the local network who is interested. PIM gives the routed network a method to build distribution trees toward those receivers.

Working isn't the same as correct

Broadcast-style flooding can make an application appear functional while creating completely unnecessary traffic elsewhere. The fact that the stream is visible at the receiver doesn't mean multicast routing is designed correctly.

09 / Private LTE / RANPI / OpenCellSurvey

It connects. How good is it?

I had built a Private LTE network with Open5GS, srsRAN, an SDR, and a SIM I provisioned myself. My phone connected. Data worked. That answered exactly one question.

I still had no idea how good the RF coverage actually was.

I could walk around the property while the UE stayed attached, but "still connected" is a terrible RF measurement.

Get the measurements out of the radio

I started digging into the srsRAN PHY layer and built a measurement utility that could expose values like RSSI, RSRP, and RSRQ as structured data.

That gave me numbers, but a JSON object saying RSRP is -93 dBm isn't very useful six hours later if I don't know where I was standing when it happened.

Add location

OpenCellSurvey came from combining those measurements with GPS. The software records the survey path, associates measurements with locations, displays live signal information, and can generate a report afterward.

I eventually added live RSRP graphs, survey sessions, map overlays, different LTE bands, CBRS support, indoor floorplan support, and report exports.

RANPI grew around the same idea: take the Private LTE lab and turn it into something portable enough that I can actually walk around with it and collect useful information.

10 / SDR / LibreFi / Low-bandwidth networking

From barely moving data to streaming audio

LibreFi started with a pretty simple question: how much useful IP traffic can I actually move over a radio link I built myself?

"Useful" became the important word.

A modem moving a few packets in a test isn't the same thing as having enough consistent throughput and latency control to support an actual application.

Six kilobits was already a fight

Early on, I could get a 6 kbps Opus audio stream through for a few seconds at a time before playback would stall or fall apart.

That gave me something much better than a synthetic throughput number. Audio is extremely obvious when the link isn't delivering packets consistently.

I kept working through the modem, buffering, packet handling, timing, and the rest of the path until the link could sustain the 6 kbps stream.

Then 12 kbps worked

The next milestone was being able to effectively stream the 12 kbps Opus version of the same audio.

Doubling from 6 kbps to 12 kbps sounds almost comically small compared with normal Ethernet or Wi-Fi speeds. On a custom SDR modem where I control the physical link, modulation, framing, and network interface, it was a pretty big jump.

This is also why I like testing networks with real applications. An iperf number can tell me throughput. Listening to a song tells me immediately whether the application can actually tolerate the way those packets are arriving.