Level(3) ex-twtelecom midwest packet loss (4323)

[mel at beckman.org (Mel Beckman)]

Blake,

There's no call to be blatantly offensive like that. 

-mel 

> On Aug 28, 2015, at 8:35 AM, Blake Dunlap <ikiris at gmail.com> wrote:
> 
> I'll just leave this here....
> 
> https://honestnetworker.wordpress.com/2013/11/04/the-true-meaning-behind-most-rfos/
> 
>> On Fri, Aug 28, 2015 at 8:26 AM, Jason Canady <jason at unlimitednet.us> wrote:
>> Mike, I would take it to mean someone screwed something up and they don't
>> want to admit to it.  :-)  That's just a guess.
>> 
>> --
>> 
>> Jason Canady
>> Unlimited Net, LLC
>> Responsive, Reliable, Secure
>> 
>> www.unlimitednet.us
>> jason at unlimitednet.us
>> twitter: @unlimitednet
>> 
>> 
>>> On 8/28/15 12:00 AM, Mike Hammett wrote:
>>> 
>>> 08/28/2015 3:08 AM GMT
>>> Event Conclusion Summary
>>> 
>>> Start: August 27, 2015 13:20 GMT
>>> Stop: August 28, 2015 00:00 GMT
>>> 
>>> Root Cause: A protocol issue impacted IP services in multiple markets.
>>> Fix Action: Adjustments were made to clear the errors.
>>> 
>>> Summary:
>>> The IP NOC began investigating the root cause with Tier III Technical
>>> Support. It was reported that the issue was causing packet loss for
>>> customers. Operations Engineering teams were engaged, and Field Services
>>> were dispatched to a site in Chicago, IL to assist with investigations.
>>> Troubleshooting identified a protocol issue, and Operations Engineering
>>> worked with Tier III Technical Support to perform adjustments on the links.
>>> It was confirmed that the errors cleared. The traffic load was also lowered
>>> on cards in Chicago to alleviate any further issues. Should any additional
>>> impact be experienced, please contact the Level 3 Technical Service Center.
>>> 
>>> What the hell is a "protocol issue"?
>>> 
>>> I'm not an idiot, you can tell me specifically what happened...
>>> 
>>> 
>>> 
>>> 
>>> -----
>>> Mike Hammett
>>> Intelligent Computing Solutions
>>> http://www.ics-il.com
>>> 
>>> ----- Original Message -----
>>> 
>>> From: "Ryan Gelobter" <ryan.g at atwgpc.net>
>>> To: "Mel Beckman" <mel at beckman.org>
>>> Cc: "<nanog at nanog.org>" <nanog at nanog.org>
>>> Sent: Thursday, August 27, 2015 3:14:59 PM
>>> Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323)
>>> 
>>> If you have access to the Level3 portal you should see ticket #9639047
>>> under Network Events now.
>>> 
>>> Event Summary:IP Network Event ~ Multiple Markets
>>> 
>>> 08/27/2015 8:05 PM GMT
>>> 
>>> Level 3 Tier III and Operations Engineering teams have identified Internet
>>> Protocols dropping, affecting customer services. Restoration efforts are
>>> in progress, but an estimated time of restoral is not available at this
>>> time.
>>> 
>>> 08/27/2015 6:36 PM GMT
>>> 
>>> IP and Transport Tier III, Operations Engineering and Field Services
>>> continue collaboratively working the issue.
>>> 
>>> 
>>> 08/27/2015 4:59 PM GMT
>>> 
>>> Operations Engineering is engaged and Field Services is on site in
>>> Chicago,
>>> IL investigating the issue.
>>> 
>>> 
>>> 08/27/2015 4:38 PM GMT
>>> 
>>> The engineers are currently migrating traffic in efforts of restoring
>>> services while troubleshooting continues. Field Services is being
>>> dispatched to a Chicago, IL site to assist.
>>> 
>>> 08/27/2015 4:21 PM GMT
>>> 
>>> IP services are affected across multiple markets and the root cause is
>>> currently under investigation. The IP NOC and IP and Transport Tier III
>>> are
>>> actively troubleshooting and working to isolate the cause. The engineers
>>> have detected peering issues which are resulting in packet loss for
>>> customers. Please be advised that updates will be provided at minimum of
>>> hourly unless otherwise noted.
>>