Saturday, 6 August 2011

Outbound email issues - update

Hi,

We believe we have located the issue with sending of outbound emails (this does not effect email reception)

We will update shortly once the issue is corrected.





Some users seeing an ADSL connection issue.

This is just an update regarding some of our ADSL users seeing an issue when trying to connect to the internet, the issue is outside of our network and BT are attempting to rectify it, this does not affect all customers but stops some users getting a PPP session.

The next update from BT will be 11 am, we will update this news post as soon as we get any information.

Once again we apologise for the service disruption we are pushing as hard as possible to get a resolution.

Regards

The support team.

Outbound email issues

Good morning,

A number of our customers are experiencing issues sending emails this morning.

We are currently investigating this issue and will post an update shortly.

Thank you for your patience.



Wednesday, 27 July 2011

Urgent VM work finished

Hi,

This is just an update on the urgent work we started at 11am This was completed by 3pm today.

This is also to confirm this work ONLY affected customers that have a VPS with us that generally ends in IP address 71.xx and did not affect email etc.

Thank you for your patience

The So Internet Support Team



Urgent VPS maintenance 11am 27th July 2011

Dear all,

As some of you will be aware we have been experiencing a few ongoing issues with stability on our VPS platform.

We have a solution to this problem, but to perform this emergency maintenance we need to take each VM (Virtual Machine) offline to move it.

We understand this is an inconvenience but important that we do, to get stability back as quickly as possible.

This work will start at 11am Wednesday 27th July 2011

Thank you for your understanding.

Paul


Friday, 10 June 2011

FIXED - PEM Email, Hosted Exchange and MS SQL

Dear Customer,

As you will be aware we suffered a serious failure of our mass storage device. This device holds the email accounts and databases for many of our customers. It is built across two separate pieces of hardware and two different controllers to provide a redundant system.

At around 8.50pm on Wednesday evening the primary controller failed, and the secondary controller took over as planned. During the later part of the evening we noticed that the mail storage was not accessible even though the SAN was operational for other services. It transpired that the email storage had not transitioned across from the primary controller to the secondary. All attempts to make this change failed and at 11.15am on Thursday morning a decision was made to reboot the entire SAN to see if it would clear the issue.

What we did not realise was that after the full reboot the second controller would not come back online correctly either. This failure took our Hosted Microsoft Exchange and MS SQL service offline.

Within a couple of hours the decision was made to start making alternative arrangements by putting our disaster recovery program in place. This was first put in to action for MS SQL and customer data was started to be restored as requested and required. At the same time we started a build of an alternative email system to get email flowing as quickly as possible.

We have been working closely with SUN/Oracle, the vendor of the hardware, who have had an engineer working on our system.

At around 11.50am on Friday, all Hosted Exchange and SQL services were functional and working correctly.

The SUN/Oracle engineer continued to work on our storage device and by 2pm had managed to re-establish connection to the email store. Email is now flowing correctly, but as you can imagine service is slightly slower than normal due to the huge amount of email traveling through the system.

During the problems we have had, we continued to update our support news site at - http://sointernetnews.blogspot.com and for those using Twitter - @sointernet

I would like to apologise for the problem and inconvenience that has occurred, which has been the first of this size in 5 years. We believed we had a redundant system that could cope with a failure like this. We will be reviewing our implementation fully over the coming days and adjust our configuration as required.

Thank you again for your patience.

Hosted Exchange - FIXED but at RISK

Hi,

Just to let you know we have now restored our Hosted Exchange service without the need to do a disaster recovery.

We are still deeming this service as at risk as SUN/Oracle engineers are still working on the SAN and so it may be reset at any point while final configuration updates are made.

Paul