Dismiss Notice
It can take 24-48 hours for the hosting/Teamspeak applications to be reviewed. Opening a thread before 48 hours, asking about the application timeline will result in your account and application being deleted permanently.

Before signing up for an account, please see our Forbidden Countries List (https://www.instafree.com/forbidden_countries.php). If you are on that list, please do not attempt to sign up, as you will not be given a hosting account. Using a proxy to circumvent that list is a violation of our TOS and will result in immediate deletion of your account.

OUTAGE: RESOLVED 6/17/18 Outage - Dallas

Discussion in 'Network and Server Status' started by Bryan, Jun 17, 2018.

  1. Bryan

    Bryan Administrator

    Messages:
    6,720
    Likes Received:
    1,429
    Currently working on what appears to be a routing issue to several servers within the Dallas datacenter.

    More info shortly.
     
    Jase Wolf likes this.
  2. Bryan

    Bryan Administrator

    Messages:
    6,720
    Likes Received:
    1,429
    Most sites appear to be back up for me, but seem to be taking alternate routes and are much slower than normal.

    Can anyone else confirm?
     
    Jase Wolf likes this.
  3. Bryan

    Bryan Administrator

    Messages:
    6,720
    Likes Received:
    1,429
    A handful of our external monitors are still showing some route issues. Still working with the datacenter on re-optimizing the routes and making sure our edge routers are behaving.
     
    Jase Wolf likes this.
  4. Bryan

    Bryan Administrator

    Messages:
    6,720
    Likes Received:
    1,429
    Minor packet loss as well on the new routes.
     
    Jase Wolf likes this.
  5. Bryan

    Bryan Administrator

    Messages:
    6,720
    Likes Received:
    1,429
    It looks like things have settled down and are again working as they should, or close to it.

    For a rundown of what occurred:

    Telia (one of the major bandwidth providers in the datacenter) started experiencing some minor issues and packet loss early this evening. According to reports from clients here, it was somewhere around 6:30 pm Pacific Time. At that point, none of our internal or external monitors were triggered by the issue, nor were any monitors in the datacenter.

    At 8:28 pm, an internal datacenter monitor was triggered by apparent packet loss on Telia. External monitors were not triggered, indicating minimal loss to the world. The packet loss lasted around 9.5 minutes before recovering.

    Down 00:9:26 Jun/16/2018 8:28:47 PM - Jun/16/2018 8:38:13 PM

    Additional clients reported issues around 8:45 pm (presumably from the 8:28 packet loss) but noted that their sites had recovered within a matter of minutes.

    Shortly after 12 am, an additional monitor was triggered internally, and shortly thereafter, all internal and external monitors were triggered, indicating a complete loss of traffic through the server. Internal out-of-band access to the server indicated that the server itself was up and functioning correctly, however, the server's network cards were rebooted around 12:10 am followed by the server itself, for troubleshooting purposes. Server successfully rebooted gracefully, and we proceeded with an examination of the network components and core routers.

    Down 00:5:59 Jun/17/2018 12:03:14 AM - Jun/17/2018 12:09:13 AM

    It was discovered that Telia appeared to have a complete loss of connectivity to our edge router, and for reasons yet to be determined, the other bandwidth blends were not being utilized. We are working on that issue now.

    At 12:45 am, GTT (another major bandwidth provider) was switched on exclusively, which resulted in longer routes and higher ping times along certain paths, particularly overseas.

    As of 3:25 am, we are slowly introducing Telia and the other providers into the mix, small percentages at a time. The routes are still set to favor GTT, and this is still resulting in potentially longer routes and higher ping times in some instances, particularly overseas.

    All sites should be 100% operational, however, since around 12:45 am PDT. If your site is not working, please open a ticket or PM me. In that ticket, please provide a complete traceroute (including your IP address).


    Monitor showing outbound server traffic decreasing significantly shortly after 5 pm PDT. Around 12 am inbound and outbound traffic cease entirely.
    graph.png



    Additional monitor showing traffic on the router itself. Correlates findings with Monitor #1.
    showgraph.png
     
    Jase Wolf and Dr. Pit like this.
  6. sander k

    sander k Hosting Client

    Messages:
    99
    Likes Received:
    34
    Location:
    Netherlands
    It’s back online. Thanks mate.
     
    Jase Wolf and Bryan like this.
  7. Fedora

    Fedora Premium Hosting Client VPS Client

    Messages:
    2,153
    Likes Received:
    2,259
    Everything is ok now. Thank you! :D:D:D
     
    Jase Wolf and Bryan like this.
  8. Konstantin

    Konstantin Premium Hosting Client VPS Client

    Messages:
    1,411
    Likes Received:
    805
    So sorry I didn't see the "Can anyone else confirm" note. I was looking over this thread but realized I forgot to subscribe to it.
     
    Jase Wolf and Bryan like this.
  9. David Gregoire

    David Gregoire Premium Hosting Client Hosting Client

    Messages:
    178
    Likes Received:
    302
    Location:
    Maine
    I know this is very late, but I've been going through older forum posts as time allows.

    I wanted to say that I'm pretty impressed with the quality and frequency with which you update people concerning the inner workings and issues related to the servers and site. It has been my experience that it's the lack of disclosure that turns people away or upsets them. They may not fully understand the issues, but they like to know that folks are working on it, that there's a solution in sight, and that they're considered important enough to be told what's going on.

    Which is a very long-winded way of saying thank you. I figured the most recent thread would be a good spot to do that, even though it's a bit dated.
     
    Bryan, Fedora and Jase Wolf like this.
  10. Bryan

    Bryan Administrator

    Messages:
    6,720
    Likes Received:
    1,429
    You're welcome! We very seldom have network or server issues, but when we do, it only makes sense to keep everyone informed. :)
     
    Jase Wolf, Fedora and David Gregoire like this.
  11. Jase Wolf

    Jase Wolf Premium Hosting Client VPS Client

    Messages:
    989
    Likes Received:
    907
    Location:
    UK
    Dallas will you behave
     
  12. Konstantin

    Konstantin Premium Hosting Client VPS Client

    Messages:
    1,411
    Likes Received:
    805
    Yep. Went offline :-(. I’m sure @Bryan is on it
     
    Bryan and Jase Wolf like this.
  13. Jase Wolf

    Jase Wolf Premium Hosting Client VPS Client

    Messages:
    989
    Likes Received:
    907
    Location:
    UK
    Meanwhile Jasja is still alive and well in Las Vegas xD.
     
    Bryan likes this.
  14. Jase Wolf

    Jase Wolf Premium Hosting Client VPS Client

    Messages:
    989
    Likes Received:
    907
    Location:
    UK
    Though I think Jasja got the Las Vegas datacentre drunk yesterday ain't that right @Bryan xD
     
    Bryan likes this.
  15. Jase Wolf

    Jase Wolf Premium Hosting Client VPS Client

    Messages:
    989
    Likes Received:
    907
    Location:
    UK
    Thank you very much Dallas
     
    Bryan likes this.
  16. Bryan

    Bryan Administrator

    Messages:
    6,720
    Likes Received:
    1,429
    Looked like another minor routing issue somewhere. I again did not experience any issues, but one of our external monitors caught it. It fixed itself pretty quickly. Still waiting to hear back from the datacenter to see if they were showing any routing issues. Think it might have been outside of the DC.
     
    David Gregoire and Jase Wolf like this.
  17. David Gregoire

    David Gregoire Premium Hosting Client Hosting Client

    Messages:
    178
    Likes Received:
    302
    Location:
    Maine
    My monitor measured 17 minutes of downtime - a first since I've been here. I didn't even notice. ;-)

    I don't have it configured to email me or send me an SMS or anything. I don't actually do anything that important!
     
  18. David Gregoire

    David Gregoire Premium Hosting Client Hosting Client

    Messages:
    178
    Likes Received:
    302
    Location:
    Maine
    I take that back! It was apparently the second outage. I didn't notice either of 'em. Maybe I should enable email notifications? LOL

    It's not like they happen often enough to be bothersome emails. I'm gonna enable notifications. I ain't scared! I ain't never scared!
     
    Jase Wolf likes this.
  19. Jase Wolf

    Jase Wolf Premium Hosting Client VPS Client

    Messages:
    989
    Likes Received:
    907
    Location:
    UK
    I only noticed it because I was playing about with my CSS xD. happened just after I saved what I was doing and was trying to see my changes xD.
     
    Bryan and David Gregoire like this.
  20. Fedora

    Fedora Premium Hosting Client VPS Client

    Messages:
    2,153
    Likes Received:
    2,259
    I noticed it because I received a message from Jetpack:

    Only for that. XD!
     
    Bryan, David Gregoire and Jase Wolf like this.

Share This Page