OLD Server / Website Packet Loss Issues May 28th - 29th!

Discussion in 'Minecraft' started by Kane Hart, May 28, 2015.

Thread Status:
Not open for further replies.
  1. Kane Hart

    Kane Hart Administrator

    • Administrator
    • Administrator
    Joined:
    Aug 3, 2013
    Messages:
    3,029
    Likes Received:
    712
    Update: Servers are back and functioning 100% again. Looks like the install of the new cable was successful and no packet loss.



    Right now the fiber cables have been cut. Last night, a car hit one of the poles along the BHS/MTL via the North route, damaging the optical fiber cable: 2 of the 3 pairs of cables are down. The 3rd one, to Level 3, is still UP.
    We’ve identified where the incident happened and the maintenance teams are currently working to feed a new 96 FO cable at 300m. This will take 12 hours.

    They had now disabled the former connection and were just waiting for the new line to be put in and connected.

    No ETA and expect major packet loss.
     
    #1
    Last edited: May 28, 2015
    Chris likes this.
  2. Kane Hart

    Kane Hart Administrator

    • Administrator
    • Administrator
    Joined:
    Aug 3, 2013
    Messages:
    3,029
    Likes Received:
    712
    Progression Update:

    Thursday, 28 May 2015, 03:30AM
    It's an incident with the cable but not a clear cut. Several fibers are affected but not the whole cable. The problem could be caused by a rodent.
    Since there are available pairs, the repair of the cable is not expected immediately. The maintainer will transfer the affected links to other pairs in good health. It's expected therefore a recovery of the links BHS <> Montreal in a few hours. No accurate ETA can be provided in the current situation.


    Thursday, 28 May 2015, 04:30AM
    The technicians are on site and looking for the damaged fiber.


    Thursday, 28 May 2015, 06:37AM
    Ovh network need an urgent repair, a 30 minute service interruption is necessary during this time the service will be degraded. (no ETA on when) We are working closely with our technicians on the ground to resolve this issue.


    Thursday, 28 May 2015, 11:07AM
    Hello,

    There is currently a very delicate situation at our datacentre in BHS. We have 4 optical fiber cables that connect the datacenter to the world.

    1) BHS to Montreal via the North. The link is up and we have 3 pairs of optical fiber cables: to Montreal, New York (600G capacity) and to level 3 (which provides 200G capacity to New York)
    2) 2) BHS to Montreal via the South. The link is not yet up. Construction of the connection is delayed in the Indian reserve. We will have 3 pairs of optical fiber cables by July 2015
    3) BHS to New York via the South. This link is not yet up either. It takes a particularly long time to get approval to perform work on the Canada/USA border: 3 years. The go-live date: September 2015
    4) A backup of the MTL/BHS backup is UP, but with low capacity (10G).

    Last night, a car hit one of the poles along the BHS/MTL via the North route, damaging the optical fiber cable: 2 of the 3 pairs of cables are down. The 3rd one, to Level 3, is still UP.
    We’ve identified where the incident happened and the maintenance teams are currently working to feed a new 96 FO cable at 300m. This will take 12 hours.

    Around 7-8am local time (1-2pm GMT+2), we will begin maintenance on the cable: we will cut off the optical fiber cables that are still up to weld them on to the new cable. To reduce the welding time, we have 2 teams on site at both ends of the cable. We estimate that it will take 30 minutes to cut off and remount our tube of optical fiber cables, in which we have 3 pairs of optical fiber cables. During this intervention, the datacenter will remain connected, but only via the backup of the backup, and therefore the connection will be seriously downgraded.

    We will post more information before the work begins.


    Thursday, 28 May 2015, 11:13AM
    We've started to install the new cable. The welders are about to insert a new optical fiber cable into the connection box. We will then prepare the cable. We estimate that the cut-off will happen around 5:00AM (11.00 GMT+2)


    Thursday, 28 May 2015, 12:53PM
    The fibers have been cut.
     
    #2
    Chris likes this.
  3. Kane Hart

    Kane Hart Administrator

    • Administrator
    • Administrator
    Joined:
    Aug 3, 2013
    Messages:
    3,029
    Likes Received:
    712
    Thursday, 28 May 2015, 14:01PM
    Work is continuing. We're welding, we're welding ...


    Thursday, 28 May 2015, 14:02PM
    Work should be complete in a few minutes.


    Thursday, 28 May 2015, 14:03PM
    During the work, the connection seriously downgraded :( We're getting packet loss

    Packets Pings
    Host Loss% Snt Last Avg Best Wrst StDev
    1. p19-55-6k.fr.eu 0.5% 392 2.1 19.7 0.2 279.1 51.0
    2. p19-2-6k.fr.eu 0.0% 392 1.8 14.4 0.2 219.2 38.7
    3. th2-g1-a9.fr.eu 0.0% 392 1.0 1.0 0.6 34.0 1.8
    4. rbx-g1-a9.fr.eu 0.0% 392 4.5 4.9 4.2 11.3 0.9
    5. ldn-1-6.uk.eu 99.7% 392 104.6 104.6 104.6 104.6 0.0
    6. nwk-1-6k.nj.us 0.0% 392 76.7 76.7 76.5 83.9 0.6
    7. mtl-1-6k.qc.ca 0.0% 392 84.0 93.5 83.8 265.8 31.1
    8. bhs-g1-6k.qc.ca 7.2% 392 138.9 139.8 138.6 155.6 1.4
    9. bhs-3a-a9.qc.ca 4.9% 391 140.7 141.0 139.0 163.1 2.3



    Looks like the new Pipe is up and running. We have 0 Packet loss now!
     
    #3
    Chris likes this.
  4. Kane Hart

    Kane Hart Administrator

    • Administrator
    • Administrator
    Joined:
    Aug 3, 2013
    Messages:
    3,029
    Likes Received:
    712
    Thursday, 28 May 2015, 14:32PM
    The welding is bad quality. We need to redo it. We will cut everything off, we will breathe and start again.

    Welding is in progress. One of the fibers in our tube is up. We're receiving signal from BHS to MTL. We're waiting to receive it in the other direction.


    Thursday, 28 May 2015, 14:33PM
    Our BHS/MTL 6x100G link is up.


    Thursday, 28 May 2015, 14:33PM
    MTL/BHS 1x100G is UP.


    Thursday, 28 May 2015, 14:34PM
    MTL/NWK 20x10G Level3 is up
     
    #4
    Chris likes this.
  5. dronf

    dronf Active Member

    Joined:
    Aug 22, 2013
    Messages:
    150
    Likes Received:
    36
    Kane are you on OVH? I've got one of my servers there, pretty great prices.
     
    #5
Thread Status:
Not open for further replies.

Share This Page