RouterOS, the XB8, and the upgrade that knocked me offline

A RouterOS upgrade on my router took down my Comcast connection, and I spent way too long blaming Xfinity for it.

This was supposed to be boring maintenance. My MikroTik is the core router, the XB8 is the Comcast gateway sitting on the primary internet connection, and nothing about the situation felt like it needed a blog post.

So I upgraded RouterOS from 7.22.1 to 7.23. The router came back. The internet did not.

TL;DR: If RouterOS 7.23 or newer kills DHCP behind an XB8, Rogers, Spectrum, or similar cable gateway, check IPv6 Neighbor Discovery. RouterOS may be sending IPv6 Router Advertisements out your WAN interface. Disable the default IPv6 ND entry or bind it only to your LAN bridge or LAN VLAN interfaces.

The setup

My MikroTik CCR2004 is not just doing NAT for Netflix. It is the core router for my home network, and it is also the physical foundation that a lot of AS215855 depends on.

The XB8 sits on the Comcast connection. The MikroTik connects to it on the WAN side, pulls a DHCP lease, and handles the rest of the network behind it. I also have Starry available as a secondary path, but Comcast is the main connection.

Earlier that day, neighbors in my building had been complaining in the community chat about internet issues. I had none and everything was working fine on my end, so I did not think much of it. I did the RouterOS upgrade that evening and went to bed with no internet.

Now, when half the building is already complaining about Xfinity, and then your own internet dies right after a router reboot, the conclusion basically writes itself. Especially when your trust in Xfinity is... not high.

The Xfinity rabbit hole

So I opened the Xfinity chat at around 9:53 PM. The bot immediately told me there were no outages in my area and that everything looked healthy. Then it walked me through the usual script before concluding that the issue was not related to Xfinity Internet service and suggesting I contact the equipment manufacturer.

I then spent about five minutes trying to get a human out of it.

Eventually I got through to a live agent named Rohit. I explained the DHCP lease failures, that my ARP table was showing the default gateway as incomplete at L2, and that multiple neighbors had been reporting issues since the day before. He ran a diagnostic, confirmed signals looked fine, and then mentioned something interesting: a new firmware had recently been pushed to the modem. d11_v_cgm4981comims_ultrafastgenlld_c02.cm.

That felt significant.

Rohit ran a reprovisioning. After rebooting the modem I got connectivity for about three pings, then it died again. Then it came back. Then it died. Classic flapping, exactly what my neighbors had been describing for a day and a half.

I asked about rolling back the firmware and pinning it. Rohit said yes, sure, then went quiet for a bit, then came back and told me he actually could not do it. My building is on a bulk account, and only the bulk team has access. He escalated the case, gave me their direct number, and said to expect a callback within an hour.

At that point I was pretty convinced this was Comcast.

Spoiler: it was not Comcast.

The test that changed the direction

The first thing I wanted to rule out was the physical path. The CCR2004 was connected to the XB8 through an SFP+ copper module, so I moved the WAN connection from the SFP+ module to one of the onboard ethernet ports.

Same result.

So it was not the module.

Then I grabbed a Linux laptop, plugged it straight into the XB8, and got a DHCP lease immediately. Like nothing was wrong. The XB8 was fine.

That made the situation much more annoying.

Next I set up the Linux machine as a DHCP server on the ISP-facing interface of the MikroTik and let the router pull a lease from that instead. That worked too. The MikroTik got a lease and was perfectly happy.

So now I had two facts that did not fit the Xfinity theory:

• A Linux laptop could pull DHCP from the XB8.
• The MikroTik could pull DHCP from a Linux DHCP server.

The cable was fine. The port was fine. The XB8 could hand out leases. The MikroTik DHCP client could receive leases.

But the MikroTik and the XB8 together did not work after RouterOS 7.23.

That is when this stopped looking like a normal DHCP problem.

Update: 7.23.1 has the same problem

A few weeks later I upgraded to 7.23.1, hoping the patch release might have addressed whatever changed in 7.23. Same result. The DHCP client on the Comcast-facing interface sat at status=searching... indefinitely.

The XB8 was fine, Starry was fine, the interface had link. Just no lease.

So this was not a one-off regression in 7.23. Whatever changed between 7.22.1 and 7.23 was still present in 7.23.1.

I rolled back to 7.22.1 again. The lease came up immediately.

The root cause: IPv6 Router Advertisements

After some digging, it turns out this had nothing to do with the DHCP client.

The RouterOS 7.23 changelog includes this entry:

ipv6,ra - fixed sending only DNS or MTU when prefix is set to 'none'

That sounds harmless. It is not a headline feature. It does not look like the kind of thing that should take down a WAN connection.

But in 7.22.1 and earlier, when the IPv6 ND prefix was set to none, the router would not send Router Advertisement packets containing DNS or MTU information. RouterOS 7.23 changed this so that RA packets are now sent even when the prefix is none.

My default IPv6 ND entry was not pinned to a LAN interface. In practice, that meant RouterOS could treat it as applying broadly. After the 7.23 RA change, the router started emitting Router Advertisements where it previously stayed quiet, including toward the WAN interface facing the XB8.

The practical result is that the XB8 appears to stop passing traffic for the customer device. From the MikroTik side, it looks like DHCP simply never succeeds.

This explains every symptom perfectly. The DHCP client was not broken. The XB8 was not having an outage. The MikroTik was advertising itself as an IPv6 router on a segment where the cable gateway does not tolerate that behavior.

A 14-page thread on the MikroTik forum describes the same issue across Rogers in Canada, Spectrum, and Xfinity with various MikroTik models. As of 7.23.1 and 7.24beta, MikroTik had not changed the default ND behavior.

How to confirm this is your issue

If you hit this after upgrading RouterOS, do not only stare at the DHCP client. Check whether your router is sending IPv6 Router Advertisements out the WAN interface.

Start by looking at your IPv6 ND config:

/ipv6 nd print
/ipv6 nd prefix print

If the default ND entry is enabled and not constrained to your LAN, that is suspicious.

Then sniff for ICMPv6 Router Advertisements on the WAN interface. Replace <wan-interface> with the interface facing your cable modem:

/tool/sniffer quick interface=<wan-interface> ip-protocol=icmpv6

If you see Router Advertisement packets leaving the WAN interface, that is very likely the problem. Your customer router should not be advertising itself as an IPv6 router toward the cable gateway.

The fix

The fix is to stop the default ND entry from sending Router Advertisements out WAN interfaces.

Do this carefully. If you disable the default ND entry without adding LAN-specific entries, IPv6 SLAAC on your LAN may stop working.

If you are using VLAN interfaces and no single LAN bridge, disable the default entry and add explicit entries for each LAN interface:

/ipv6 nd set [find default=yes] disabled=yes
/ipv6 nd add interface=<LAN-interface-or-VLAN> advertise-dns=yes

In my case, that looked more like this:

/ipv6 nd set [find default=yes] disabled=yes
/ipv6 nd add interface=Home advertise-dns=yes
/ipv6 nd add interface=IoT advertise-dns=yes
/ipv6 nd add interface=Mgmt advertise-dns=yes

If you have a single LAN bridge, you can bind the default entry to that bridge instead:

/ipv6 nd set [find default=yes] interface=bridge

After applying this, RouterOS 7.23+ can run without sending Router Advertisements toward the cable modem.

The downgrade gotcha: extra packages

One more thing worth mentioning. When I needed to roll back from 7.23.1 to 7.22.1, I uploaded the routeros-7.22.1-arm64.npk to the router, ran /system/package/downgrade, and it rebooted.

Still on 7.23.1.

Tried again.

Same thing.

No error, no explanation, just silently refused to downgrade.

The problem turned out to be the zerotier extra package. It was installed at version 7.23.1, and there was no matching 7.22.1 zerotier .npk on the router. RouterOS will not downgrade the main package if any extra package would be left at a higher version, and it does not clearly tell you that this is why it failed.

The fix was to uninstall zerotier first:

/system/package/uninstall zerotier

Then I ran the downgrade again. It worked on the first try after that.

If you are trying to roll back RouterOS and it keeps silently rebooting into the same version, check your extra packages. Remove them or provide matching .npk files for all of them before downgrading.

The annoying part

The annoying part is that the failure mode looked exactly like an ISP problem.

DHCP stuck searching. ARP weirdness. Neighbors complaining. Fresh modem firmware. A support agent confirming everything looked fine on their side. All the evidence pointed at Comcast.

But the actual fix was one RouterOS IPv6 ND entry.

If you run MikroTik behind a cable gateway, especially with IPv6 enabled, make sure Router Advertisements are only going where you actually want them. RouterOS 7.23 did not break DHCP for me. It changed when IPv6 Router Advertisements were sent, and my cable gateway reacted badly enough that DHCP was the thing that appeared broken.

That distinction matters, because it changes where you look for the fix.