Rendered at 04:52:15 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
linsomniac 45 minutes ago [-]
A buddy was asking me if this story "holds water" earlier today (because I'm more into networking). I eventually decided it was plausible, but only through at least one, and probably multiple "rookie mistakes".
As described it was in the course of switching out copper TDM lines with fiber. On it's face that could be straightforward, but it's easy to imagine the router(s) terminating these lines didn't support the required line cards, so router upgrade(s) were required. So theoretically straightforward could have become more involved.
However, when doing far less critical updates, it's common to set up a test environment, do a dry run of the work, test the roll-back, test the monitoring and alerting, that sort of thing. Doesn't guarantee no outages, and I'm playing armchair QB here, but it seems like a botched job.
It's also telling the Transportation Secretary said things like "we aren't sure they if they aren't doing a great work for us". He also said "this is something businesses have been doing for decades and the FAA is the last to do it"; you'd assume that this is a process that someone (maybe not those doing the work) have developed deep experience with.
I'm not familiar with this L3HARRIS, I gather they are a defense contractor with billion dollar revenue.
Lots of things in this definitely raised my eyebrows.
kevin_nisbet 26 minutes ago [-]
I don't have the full story because it happened in another business unit, but one of the worst outages I ever worked the rumour was it was caused by someone doing prep work for a later cutover to new hardware. Whatever they were doing to prep for the upcoming cutover, the router rebooted. They didn't know it rebooted, so they did the same thing to the secondary and tertiary a few minutes later. Also happened to be on a Friday.
So sometimes it's easy to rehearse all the critical stuff, and get a bit complacent with the stuff that is just expected to work, plugging something in that's not configured and getting it ready for the big migration. And it's hard to get a 100% perfect replica in a lab. So I think odds are you're right, but I think there is also a longer tail of possibilities as well. And in my experience, there is always tension between chasing the 9's of availability, and getting some "safe" work done when convenient.
kjellsbells 14 minutes ago [-]
The explanation could hold water, but that only demonstrates plausibility, not truth.
Seems that a router that previously terminated copper interfaces was switched to terminate fiber. That alone doesn't really tell us much without knowing whether this is really all that happened or if it's shorthand for a bigger move at a higher level in the protocol stack. For example you might "just" have changed copper Cat6 ports to fiber/SFP but still be doing GigEthernet and I rather doubt that a move that simple could make a router misbehave.
Moving up the stack this get more interesting. For example they might have moved from a copper star topology to a fiber ring, and rings need care or else chaos ensues. Similarly Ethernet gets pissy if there are loops but the spanning tree protocols need very careful usage. I assume since the word router was used that Ethernet was fine and we are in layer 3. An interface switch could have triggered recalculation or reannouncement of routes, for example by erroneously configured routing protocols. eg the Arkansas router announces it is the best hop for traffic to reach Chicago, gets swamped with traffic from Dallas, NY, etc, and falls over. (Yes, the tools provide guardrails. Who knows if they were enabled?)
However, and esp because Duffy is pointing at L3Harris, I suspect they installed (or perhaps removed) a Juniper TDM to Ethernet converter module (eg DS3 SmartSFP from that vendor) and something went bad. These modules are supposed to encap TDM into Ethernet but they don't (iirc) have any semantic awareness of a telephone call, so if they were to be hit by, say, a storm of TDM circuit messages (call setups, say) they might just pass them on, not rate limit them like a true telco packet switch would.
It might be pure coincidence that an AI system was hooked up to the FAA system the week before.
Bottom line: could be bad luck, could be basic screw up, we won't know for years.
I was the Sr. Internet Mail Administrator at AOL at the time, and this article was what The Register wrote based on what I had submitted to them. They made a few changes. I know they were trying to protect my identity, but I think they also wanted to make it a better story.
For us, everything started with a planned downtime to upgrade the main core router for the entire service. We had the best network engineers in the business, but even they can make a mistake. And they did. And then there was a whole domino effect of events that followed.
"See this router here? If I switch it off, the ENTIRE US AIRSPACE GOES DOWN."
"Sure thing buddy, I believe you."
bparsons 47 minutes ago [-]
For the network engineers on here: does this explanation make sense to you?
pigggg 15 minutes ago [-]
Not enough information.
But I recall years ago the ATC system made heavy use of multicast in ways that scared me. I dunno if that stuff is still out there but it was some real I wouldn't do that shit if I were you stuff. And yeah mcast can sure fuck a network fast.
As for in general there's plenty of ways a single box can melt a network. Routing protocols let you do bad things fast and pull in more traffic than you can eat.
horacemorace 34 minutes ago [-]
I mean, if rookies are in charge of the plan then sure. It’s the network engineering equivalent of a pilot ignoring all standard safety checks and a flight plan.
linsomniac 43 minutes ago [-]
TL;DR of my top-level comment: Yes, but only if the work was done by clowns. ;-)
Do these routers not have rate limiting on them, and if not why in the world not? Sounds like a route loop that then spread out to the rest of the network.
morkalork 1 hours ago [-]
The irony of not being able to read the article because archive.is giving a 500 error for the page
zzzeek 31 minutes ago [-]
so everything we were told was correct:
1. it was L3Harris (CBS news): check
2. it was a "newer system overloading the legacy system" (Duffy): check
3. it was "old copper wires" (Duffy): check
4. it was not related whatsoever to the SMART AI system: check
5. the process of upgrading the legacy systems is part of the problem: check
6. the combination of a "newer" system with an "older" system, something that likely should have been planned for if the contractors weren't being rushed, was involved [1]: check
As described it was in the course of switching out copper TDM lines with fiber. On it's face that could be straightforward, but it's easy to imagine the router(s) terminating these lines didn't support the required line cards, so router upgrade(s) were required. So theoretically straightforward could have become more involved.
However, when doing far less critical updates, it's common to set up a test environment, do a dry run of the work, test the roll-back, test the monitoring and alerting, that sort of thing. Doesn't guarantee no outages, and I'm playing armchair QB here, but it seems like a botched job.
It's also telling the Transportation Secretary said things like "we aren't sure they if they aren't doing a great work for us". He also said "this is something businesses have been doing for decades and the FAA is the last to do it"; you'd assume that this is a process that someone (maybe not those doing the work) have developed deep experience with.
I'm not familiar with this L3HARRIS, I gather they are a defense contractor with billion dollar revenue.
Lots of things in this definitely raised my eyebrows.
So sometimes it's easy to rehearse all the critical stuff, and get a bit complacent with the stuff that is just expected to work, plugging something in that's not configured and getting it ready for the big migration. And it's hard to get a 100% perfect replica in a lab. So I think odds are you're right, but I think there is also a longer tail of possibilities as well. And in my experience, there is always tension between chasing the 9's of availability, and getting some "safe" work done when convenient.
Seems that a router that previously terminated copper interfaces was switched to terminate fiber. That alone doesn't really tell us much without knowing whether this is really all that happened or if it's shorthand for a bigger move at a higher level in the protocol stack. For example you might "just" have changed copper Cat6 ports to fiber/SFP but still be doing GigEthernet and I rather doubt that a move that simple could make a router misbehave.
Moving up the stack this get more interesting. For example they might have moved from a copper star topology to a fiber ring, and rings need care or else chaos ensues. Similarly Ethernet gets pissy if there are loops but the spanning tree protocols need very careful usage. I assume since the word router was used that Ethernet was fine and we are in layer 3. An interface switch could have triggered recalculation or reannouncement of routes, for example by erroneously configured routing protocols. eg the Arkansas router announces it is the best hop for traffic to reach Chicago, gets swamped with traffic from Dallas, NY, etc, and falls over. (Yes, the tools provide guardrails. Who knows if they were enabled?)
However, and esp because Duffy is pointing at L3Harris, I suspect they installed (or perhaps removed) a Juniper TDM to Ethernet converter module (eg DS3 SmartSFP from that vendor) and something went bad. These modules are supposed to encap TDM into Ethernet but they don't (iirc) have any semantic awareness of a telephone call, so if they were to be hit by, say, a storm of TDM circuit messages (call setups, say) they might just pass them on, not rate limit them like a true telco packet switch would.
It might be pure coincidence that an AI system was hooked up to the FAA system the week before.
Bottom line: could be bad luck, could be basic screw up, we won't know for years.
I was the Sr. Internet Mail Administrator at AOL at the time, and this article was what The Register wrote based on what I had submitted to them. They made a few changes. I know they were trying to protect my identity, but I think they also wanted to make it a better story.
For us, everything started with a planned downtime to upgrade the main core router for the entire service. We had the best network engineers in the business, but even they can make a mistake. And they did. And then there was a whole domino effect of events that followed.
L3Harris statement on the Nov. 13 ground stop
https://news.ycombinator.com/item?id=50501844
"Sure thing buddy, I believe you."
But I recall years ago the ATC system made heavy use of multicast in ways that scared me. I dunno if that stuff is still out there but it was some real I wouldn't do that shit if I were you stuff. And yeah mcast can sure fuck a network fast.
As for in general there's plenty of ways a single box can melt a network. Routing protocols let you do bad things fast and pull in more traffic than you can eat.
Do these routers not have rate limiting on them, and if not why in the world not? Sounds like a route loop that then spread out to the rest of the network.
1. it was L3Harris (CBS news): check
2. it was a "newer system overloading the legacy system" (Duffy): check
3. it was "old copper wires" (Duffy): check
4. it was not related whatsoever to the SMART AI system: check
5. the process of upgrading the legacy systems is part of the problem: check
6. the combination of a "newer" system with an "older" system, something that likely should have been planned for if the contractors weren't being rushed, was involved [1]: check
[1] https://www.flyingpenguin.com/faa-outage-why-duffy-is-spelle... - this is a much better article and no paywall