IMOC Website - Possible short outages / slow page access 30/10

[BenF]

Just FYI - looks like our hosting provider has a problem with their network this evening.

Impact is that very rarely case you may not be able to access the page you request - a 'reload' / 'refresh' should sort it out.

For the last few hours we have been experiencing intermittent packet loss on one of our primary network links, which may be affecting your server(s).

The network engineers are working on the problem, but the exact cause is proving elusive, as is often the way with partial faults. An earlier attempt to fix the issue resulted in a brief (20 second) losses of connectivity, so for now we have reverted to the previous state (with occasional slight packet loss).

For the most part services should be unaffected; at worst operating a fraction more slowly than normal. We are continuing to work on this issue, and it is possible that the work will cause further very brief service interruptions, but obviously we intend to avoid / minimize those.

If you experience difficulty accessing your server(s) please wait for a few minutes and try again rather than immediately issuing an emergency request / email.

Many apologies for this inconvenience. We will be sure to let you know when the problem is resolved, and will provide you with a full report within a few days to let you know what happened and what we intend to do to prevent recurrence.

Kate.

--
Kate Craig-Wood
Managing Director
Memset Ltd.
http://www.memset.com/

[BenF]

Now Fixed - I don't think this would have impacted us anyway :

Towards the end of last week we emailed you regarding a networking issue which was affecting some customers. This exhibited itself as packet loss leading to slightly degraded website performance. We are pleased to say that this networking issue has been resolved. The corrective work was completed on Sunday 1st November, and we have been monitoring this carefully before notifying customers of the resolution.

The network problem we experienced affected a connection between our
2
Reading datacenters. Our network engineers and suppliers of this 10Gbs link identified a problem at one end of the link, so investigations started on the switching equipment and cabling. As a precaution the cabling connecting this switch to our network was replaced as well as the interface converter. This did not resolve the issues with the packet loss so our link provider had to spend some time configuring a new switch to install into our core network to carry this intersite link.
Once this preparation was complete we were able to transfer the intersite link to the new switch to resolve the packet loss issues.

As a result of this problem we will be working with our link provider to ensure we have preconfigured standby equipment to ensure any problems can be resolved much more rapidly. We are also going to carry out some re-routing within our network to decrease traffic over this intersite link, and enable fallback routing should there be a problem with this link in future.

Please accept our apologies for any inconvenience this issue caused, especially because of the time taken to fix it completely.

Regards,
Memset Operations Team.

--
Memset Ltd.