Network issues resolved again!

Dear all,

We have now stabilised the network and it has been running well for the last few hours.

We are continuing to analyse the events of today and a full response will come over the next few days.

We thank you again for your patience and do apologise for the inconvenience this has caused.

Continued network issues

We are currently aware of continuing problems that we are working to resolve.

As soon as more information is available we will update you.

We apologise again for the inconvenience this is causing.

Network Issues - Resolved

Dear all,

We have now identified the network issue that we experienced earlier. The network is now stable and service is running correctly.

We would like to apologise for the outage but have we been able to identify the network segment the problem occurred on and have put steps in place to protect against it happening again.

Thank you again for your patience.

Network Issues Update

There is a network problem which our network team is investigating now.

Your service will be intermittent over the next 30 mins while our network team resolves the problem.

We'll provide you further updates from our network team shortly.

Apologies for the inconvenience caused.

Network issues

Dear all,

We are currently experiencing some network issues across our network.

This could be affecting Internet access for broadband users as well as problems for people with services located with us.

We are currently working on this issue and service should be restored shortly.

Thank you for your patience.

ADSL packet loss update

Hi,

This is just to update that the engineers have successfully rebuilt the interface to pur upstream provider, the ADSL network is stable and no packet loss has been seen.

If you have any further issues please email support@sointernet.com

Thanks for your patience.

Regards

The support team.

Packet loss for certain ADSL users

Hi,

This is just an update that we have had a small number of customers experiencing packet loss on their ADSL connection, the issue has been identified and the team are working to rectify it.

Issue:

We have seen packet loss from the tunnel provide by our upstream provider which is affecting our network.

Solution:

We are currently working with the upstream provider to re-route the traffic to our equipment to negate the packet loss.

We will continue to provide further updates as we know more.

Regards

The support team.

Core Router Upgrade - 11th October 2011 01:00am - 02:00am

Dear customer,

We would like to inform you that we planned a maintenance on one of our core routers in Manchester.

This maintenance is service affecting and will impact all services delivered through that router – this includes IP transit, MPLS and Layer 2 Transport.

Full details of the maintenance window are listed below:

- Date: 11 october 2011
- Time: 01:00 am (GMT 0) – 02:00 am (GMT 0) – 1 hour
- Device: bb1.man4.uk
- Type of maintenance: software upgrade
- Impact: all IP and layer2 services (VLANs, EoMPLS, IP transit)
- Description: A number of bugs have become apparent over recent weeks, which we need to address through a software upgrade.

Dear All,

We are aware of a network outage for a number of our broadband customers over night.

We are not 100% sure of the root cause of this problem, but it looks to have been an interconnect (how we collect the actual broadband traffic from the BT Exchanges) failure.

We are rolling out an upgrade to help reduce the risk of this happening again and we would like to thank you for your patience and apologise for any inconvenience this may have caused.

Paul

Hosted Exchange - FIXED

Hi,

We have now completed a repair on the affected Hosted Exchange mail store.

If you have a hosted Microsoft Exchange mailbox with us that was affected by this issue, then email is now flowing correctly and has been delivered to your account.

Thank you for your patience and if you continue to experience problems, please do contact our support team on 0161 615 1275.

Exchange Email Issue - Update

Hi,

The repair process is going well. There are now less than 10 customers affected by this problem.

We will update further in another hour or so as the process gets further through.

Exchange Email Issue - Update

Hi,

We are now running the database fixup tool, which has been running for a number of minutes already. We are not exactly sure how long this is going to take, but we know it only has around 36GB of data left to check through.

We will update shortly as we progress through resolving this issue.

Microsoft Hosted Exchange Issue

Dear All,

For those customers that have hosted Microsoft Exchange accounts with us, we have noticed a file corruption issue in one of the main mail stores.

We have currently taken this mailstore offline and are performing a backup of the data before running a repair. We are expecting this operation to take around 2 hours to complete.

During this time you will not be able to access your mailbox, if it is affected by this issue. If your Outlook client is connecting currently with no problems then your account is hosted on an unaffected mailstore.

Further updates will follow.

Email Issues - Fixed

Hi,

At around 8.55 all services were restored and email is flowing again correctly.

We continue to investigate the problem that has been highlighted and will perform further maintenance in the near future to address this.

Mail should not have been lost during this time and should have been queued by the sending mail servers. This email will start flowing through the system over the next couple of hours.

If you experience any further problems, please do feel free to contact the support team on 0161 615 1275

Thank you again for your patience.

Email issue update

Hi,

Engineers are continuing to work on the problem and a resolution is expected very shortly.

Thank you for your patience.

Email issue update

Hi All,

We have now located the fault and engineers are working to correct the problem.

We are expecting the issue with email to be corrected within the next 30 minutes and further updates will be posted shortly.

Issues with picking up email

Good morning.

We have had a few customers this morning saying they are unable to pick up email.

We are currently investigating this issue and will update shortly.





We

Maintenance works for tonight

Hello,

We are looking to carry out maintenance works tonight to bolster our San, we are upgrading the firmware and checking the controllers, this will be carried out at 12 midnight and should last for about 2 hours, this may have an impact on Mail as this upgrade goes through, this will not impact our other services.

Thanks in advance for your patience regarding this.

Regards

The support team.

Isolated broadband outage 4.10 - 4.50pm

Dear All,

Between 4.10pm and 4.50pm our ADSL network experienced an isolated outage that affected around 10% of users.

This issue was traced to a faulty cable connected to one of our routers.

Onsite engineers working with the network team quickly corrected the problem.

Thank you for your patience.

If you are still unable to connect and have rebooted your router then please do contact our support team.

Some users seeing an ADSL connection issue.

This is just an update, it seems that BT has fixed the issue, we are monitoring our Network and once we have a full explanation we will keep you updated, but it seems there was an issue with some legacy migrations affecting their internal network.

If you are still seeing an issue with your connection please email support@sointernet.com or contact 0161 615 1275 and select option 2

Thanks for your continued patience.

Regards

The Support Team.

Outbound email fixed

Hi,

We have located the issue stopping outbound email processing correctly.

This has now been fixed and all services are now running smoothly.

Thank you again for your patience.

Outbound email issues - update

Hi,

We believe we have located the issue with sending of outbound emails (this does not effect email reception)

We will update shortly once the issue is corrected.

Some users seeing an ADSL connection issue.

This is just an update regarding some of our ADSL users seeing an issue when trying to connect to the internet, the issue is outside of our network and BT are attempting to rectify it, this does not affect all customers but stops some users getting a PPP session.

The next update from BT will be 11 am, we will update this news post as soon as we get any information.

Once again we apologise for the service disruption we are pushing as hard as possible to get a resolution.

Regards

The support team.

Outbound email issues

Good morning,

A number of our customers are experiencing issues sending emails this morning.

We are currently investigating this issue and will post an update shortly.

Thank you for your patience.

Urgent VM work finished

Hi,

This is just an update on the urgent work we started at 11am This was completed by 3pm today.

This is also to confirm this work ONLY affected customers that have a VPS with us that generally ends in IP address 71.xx and did not affect email etc.

Thank you for your patience

The So Internet Support Team

Urgent VPS maintenance 11am 27th July 2011

Dear all,

As some of you will be aware we have been experiencing a few ongoing issues with stability on our VPS platform.

We have a solution to this problem, but to perform this emergency maintenance we need to take each VM (Virtual Machine) offline to move it.

We understand this is an inconvenience but important that we do, to get stability back as quickly as possible.

This work will start at 11am Wednesday 27th July 2011

Thank you for your understanding.

Paul

FIXED - PEM Email, Hosted Exchange and MS SQL

Dear Customer,

As you will be aware we suffered a serious failure of our mass storage device. This device holds the email accounts and databases for many of our customers. It is built across two separate pieces of hardware and two different controllers to provide a redundant system.

At around 8.50pm on Wednesday evening the primary controller failed, and the secondary controller took over as planned. During the later part of the evening we noticed that the mail storage was not accessible even though the SAN was operational for other services. It transpired that the email storage had not transitioned across from the primary controller to the secondary. All attempts to make this change failed and at 11.15am on Thursday morning a decision was made to reboot the entire SAN to see if it would clear the issue.

What we did not realise was that after the full reboot the second controller would not come back online correctly either. This failure took our Hosted Microsoft Exchange and MS SQL service offline.

Within a couple of hours the decision was made to start making alternative arrangements by putting our disaster recovery program in place. This was first put in to action for MS SQL and customer data was started to be restored as requested and required. At the same time we started a build of an alternative email system to get email flowing as quickly as possible.

We have been working closely with SUN/Oracle, the vendor of the hardware, who have had an engineer working on our system.

At around 11.50am on Friday, all Hosted Exchange and SQL services were functional and working correctly.

The SUN/Oracle engineer continued to work on our storage device and by 2pm had managed to re-establish connection to the email store. Email is now flowing correctly, but as you can imagine service is slightly slower than normal due to the huge amount of email traveling through the system.

During the problems we have had, we continued to update our support news site at - http://sointernetnews.blogspot.com and for those using Twitter - @sointernet

I would like to apologise for the problem and inconvenience that has occurred, which has been the first of this size in 5 years. We believed we had a redundant system that could cope with a failure like this. We will be reviewing our implementation fully over the coming days and adjust our configuration as required.

Thank you again for your patience.

Hosted Exchange - FIXED but at RISK

Hi,

Just to let you know we have now restored our Hosted Exchange service without the need to do a disaster recovery.

We are still deeming this service as at risk as SUN/Oracle engineers are still working on the SAN and so it may be reset at any point while final configuration updates are made.

Paul

PEM Email, Hosted Exchange and MS SQL - Further Update

Hi,

We are making good progress with SUN/Oracle now and just awaiting details of an engineer visit to site to fix the failed equipment.

They are currently running additional diagnostics and we should have an update shortly.

We have just around 10 MS SQL databases to restore and then that service is fully functional.

PEM Hosted Email - We are still working to bring the mail back online and will update as soon as we have a revised time.

Thank you again for your patience.

Paul

PEM Email, Hosted Exchange and MS SQL - Update

Hi All,

We are progressing well on this issue, although it is still taking a long time.

We are still awaiting on spare parts from SUN/Oracle and hope to have a delivery schedule shortly. In the meantime, here is an update on each problem.

Shared MS SQL - This has now been fixed and we are just restoring the databases, which should be finished over the next couple of hours.

Hosted Exchange - We are currently assessing the situation with this and should have a way forward shortly.

PEM Email - This is our major task as it effects a very large number of customers. We are hoping that we will be able to get a solution running again in the next few hours to allow incoming email.

Paul

PEM Email, Hosted Exchange and MS SQL

Hi,

We are continuing to work on the major issue we are experiencing. We are awaiting spare parts from SUN/Oracle and at this time can't move much further forward until we have this new equipment. We are hoping these parts will arrive by the middle of tomorrow.

In the meantime we are rebuilding the MS SQL setup and restoring data from this morning's backups and this should be fixed by first thing in the morning.

On the PEM email side of things, we are currently building a replacement solution and hope that within the first few hours of the working day on Friday we have the ability restored for customers to receive incoming email.

Thank you again for your patience and I apologize for the inconvenience this will have caused you.

Paul

Continued PEM email and Hosted Echange/MS SQL

Hi,

We are working with Oracle to find a solution to the SAN issue that we are having.

We are going to try another couple of resets to see if this will clear the issue.

I will update again shortly.

Paul

SAN Reboot Update

Hi,

The SAN reboot has not helped the problem at the moment, but has identified the cable/controller issue.

We continue to work on this and will update you shortly.

Paul

Emergency SAN reboot

Hi All,

If you receive Hosted Exchange, or shared MS SQL services from us, we will be resetting our SAN at 11.15 this morning.

This reset should only take around 10 minutes and will hopefully clear the mail issue we are having at the same time.

Thank you again.

Paul

PEM Inbound email problems - Update

Hi,

We have now located the issue with the SAN storage.

The problem lies either with one of the fibre cables or one of the controllers.

We are very close to fixing this issue and I will update you shortly.

Thank you again for your patience.

Paul

PEM Inbound email problems

Hi All,

If your email is hosted in our shared system called PEM, then we are experiencing issues.

We currently have a problem with the way our mail system talks to our storage disks. For some people this is resulting in them not finding any email in their mailboxes.

All new emails are being queued for delivery as soon as we have fixed the issue.

Engineers have been alerted to the problem and we will have a resolution over the coming hours, but this should be at the latest 11am, 9th June 2011

Thank you for your patience.

Paul

Outgoing Mail issue update

Hi all,

We have received a number of calls regarding outgoing mail issues.

We recently injected a new outbound server in to the network to help assist with the increase in outgoing emails. We have noticed an issue with the configuration of this server not accepting incoming mail on port 588. We have now rectified this issue and all services are now running correctly.

Once again thanks for your patience.

Minor ADSL issue 7th June 2011

Dear all,

We were alerted to an issue a couple of hours ago. This problem stopped some users being able to reconnect to our broadband service.

We have identified the cause of this block,which affected just one of our 3 pieces of termination equipment.

We have just run a script to reset and clean up the user connections and the service should be running correctly.

Please do contact support if you continue to experience issues.

Paul

Advance Notice - Network Maintenance - 26th April 2011 05:00 - 06:30am

Dear All,

This is an advance notice of maintenance on our network on 26th April 2011, between 05:00 and 06:30am

This should not affect any of our services, but we just wanted you to be aware of the important work that is being carried out.

The technical notice is as follow:-

Network Maintenance - Router Software Upgrades

The routers BG1 and BG2 in our Manchester datacentre will have an IOS software upgrade during the maintenance window above. The expected downtime as a result of the reload is 5 minutes per router, during which, customers who are served solely by either of these units will suffer a small period of downtime. The upgrades will be performed on one device at a time. Other customers should not be affected, although there is a chance of some routing instability for a few minutes after the reload.

We apologise for any inconvenience this may cause, but it is important to ensure our core routing equipment is running the latest software revisions which the manufacturer advises will address a number of recently discovered bugs and vulnerabilities.

ADSL Issues Resolved

Dear All,

Thank you for your patience.

We have been able to track the routing problem, that was affecting a number of our customers, down to an issue between 2 of our core routers.

We have now been able to correct this issue which seems to have been a legacy configuration from a number of years ago, conflicting with resent updates.

It should not have been necessary to reboot your ADSL router to gain internet access, but please do feel free to contact us if you are still having problems - support@sointernet.com or 0161 615 1275

Paul

Reported ADSL routing issues

Dear All,

We are currently receiving a number of calls relating to customers not being able to get to some parts of the internet.

We are currently investigating and I will update shortly.

Paul

Short ADSL Issues 7/4/11 - 15:45

Dear All,

At around 15:45 today we experienced a short routing issue with traffic leaving our network. We are still investigating the issue, but service was reinstated within a few minutes.

Thank you for your patience.

FTP Issues resolved from ADSL Connections

Dear All,

As some of you will be aware, if you have have broadband and combined hosting with us, you may have experienced problems uploading via FTP to our hosting servers.

Maintenance carried out last night was able to correct this issue and FTP access should now be enabled again.

Thank you for your patience.

Paul

2 Minute ADSL outage - Today around 4pm

Dear All,

Today close to 4pm, we were alerted to a brief issue where we dropped connectivity on part of our network in London. This lasted for a very short period of time (1 - 2 minutes) and only affected a small number of our broadband customer base.

A majority of routers reconnected immediately without any assistance. If you are still having an issue, please turn off your router for 10 minutes and then power back on. This should clear any stale sessions.

If you do still experience issues, please do call our support team on 0161 615 1275

Paul

ADSL Routing Fixed

Hi All,

We have now located the routing issue causing problems for some of our ADSL customers.

If your router has not yet reconnected, please do reboot it and this should correct the problem.

ADSL Routing Issues

Hi All,

We are currently experiencing a few issues with a small number of customers that have multiple IP allocations from us.

We are currently working on this issue, and should have service restored within the next 30 minutes.

Thank you again for your patience.

An update will follow again shortly.

DNS Issues - Fixed

Hi All,

Well, that was a day and a half! I know that you all felt it and I really appreciate your support and patience while we corrected the issue.

As many of you know, we use a main hosting domain of gohost.it This domain was due for renewal on the 16th of February. The domain is registered through one of the largest domain registrars - OpenSRS/Tucows.

Although the renewal request was processed, OpenSRS suffered a coding issue that meant the request was not passed to the correct registry in Italy. We had been told they were working on a solution, but there would be 10 days grace even if the domain expired.

At 6.34am this morning the domain expired and was not renewed. 3 hours later services were lost to anything on our network that made use of that domain. We immediately contacted OpenSRS who tried many different options to get their coding to work.

Around 4.17pm they were able to make an attempt that kicked all systems back in to action and DNS started to work again.

Everything is looking to be coming back to normal now, but we are continuing to monitor the situation.

I hope this gives a full update of the issue from today.

All I can do is apologise, for this issue that was caused by circumstances outside of our control.

Paul

DNS Issue Update - 3

Hi,

The initial fix that we thought had been put in place by our Registrar - OpenSRS/Tucows, did not work.

We are continuing to communicate with them and are hoping for a useful update very shortly, so that we can return normal service.

While writing this I have just received an update from Tucows. They believe they have corrected the issue. I am just waiting to see this update actually happen. As soon as it does, we should quickly start to see services return.

Thank you again for your patience.

DNS Issue Update

Hi All,

We have made some progress with the domain registrar who are trying to force the updates through to fix the issue we have had with our gohost.it domain.

The domain updates have now gone through and we are awaiting the general internet to start catching up.

We would hope that service will start to restore over the next 1 hour.

Thank you again for your patience with the issues.

DNS issue

update on this issue, the registrant has informed us that their DEV issue will be rectified in the next hour.

Again we apologise for the disruption this has caused and will keep you updated.

DNS issue

This is an update as we have an issue affecting a majority of our services, our gohost.it domain is down (we have this on an auto renew) but it seems that the registrant has had a coding issue and the domain has not been renewed.

We are pushing the registrant to get this sorted asap and are awaiting information from them.

We apologise for this issue and will keep you updated as to our progress.

Issues on dedicated server and hosting network

Dear All,

During the course of today we have noticed some issues on M247 network that may have affected some of our hosting customers.

As a precaution we have closed down one of our London links which has helped stabilise the network.

The engineering team will release full details of the issue after further in-depth investigation has taken place, but currently the network is running with no issues

Thank you for your patience and an update will follow in the early part of the week.

Paul

Current ADSL issues

Hi,

We have had a number of customers noticing issues specifically with upload speeds.

We have identified where we believe the issue is and are currently working on a solution.

We are hoping to have restored service within the next few minutes.

Thank you for your patience.

All ADSL issues fixed

Hi,

Just to confirm that we have found the issue that has caused this outage and all circuits are now working.

Please do reboot your router if you can't get online. If this does not correct the issue for you, please do call our support line on 0161 615 1275.

Paul

Further ADSL issues

Hi,

After fixing the first issue we have now found a further which is stopping the customers router log in to the network.

We are currently working on this and will update you shortly.

Thank you again for your patience.

Paul

Current broadband issues

Update - just as I finished writing this the issue was fixed. You may need to reboot your router.


Hi All,

We currently have an issue in our London datacentre that is stopping the correct routing of our broadband connections.

We do have an engineer on site that is working on this and we hope to have it fixed very shortly.

Thank you for your patience.

Paul