Our web site, and hosted customer web sites are not responding. We are looking in to this.

This issue is the same as the one affecting email, and is related to a disk server.

We are working on disk server now

We sites now working.

We've just had a blip causing ADSL lines to drop of and reconnect, this also affected routing for about a minute.

More info to follow shortly.

The issue affected both BGP links and LNS traffic, cauing lines to reconnect. Not all lines and services were affected. The lines have reconnected quickly, and the problem seems to have cleared without any need for any intervention from staff.

We are investigating the cause - this looks to be something external that has triggered this. We are trying to find exactly how this has happened so we can avoid the issue in future.

We have lost access to everything in HEX 6/7 which means L2TP only customers are off line. This also affects some of our general servers. This is being investigated now.

The issue is a result of a port/link failure earlier. We are re-configuring access via an alternative route now.

Access has been restored

Our main web pages and customer web pages are currently inaccessable or very slow due to a denial of service attack. We are taking steps to address this. Sorry for any inconvenience.

This appears to be an attach on one web site, which is being moved.

Load is starting to move to the backup server now.

We have moved a lot of the load over, and adjusted TCP settings to try and ensure we sites are working albeit slightly slow.

Web access is being much more responsive now.

This was a simple planned change - it seems however we have been caught out by bind refusing to reload if one zone has one syntax error. Seems one of our customer zones had a typo in it which our tools had not managed to check and that caused it to refuse to load.

Unfortunately it took quite a few minutes to find what was wrong as this was nothing to do with the changes we actually made.

Most things are OK.

Both authorative serves are serving the aaisp.net.uk and aa.net.uk zones correctly.

There are still problems with the aaisp.net.uk domain - we're working on this.

VoIP is currently affected too, this is being worked on now.

Whilst most things are working, at least from customer lines, there was an issue which caught us out, but was way to far in to the process to sensibly back up. The top level delegation for aaisp.net.uk was going to a special DNS server, which meant when we moved everything it stopped working properly.

Some incoming VoIP is not working, we are still working on this.

Still lots of progress mopping up things. Mostly non customer affecting.

Just to clarify - you should be able to use aaisp.net.uk or aa.net.uk as they interchangable. However, right now, some places are having some issues seeing some of the aaisp.net.uk sub domains.

The preferred version is now aa.net.uk and we should be quoting that everywhere now.

Using aa.net.uk is also a work around for the issues some people are seeing right now, which are down to DNS caches.

At the moment we still have some services affected, these are: some incoming VoIP and accessing our services from outside our network (ie customer using DNS resolvers ither than ours)

We really think this should be sorted now - but monitoring carefully.

Mor details on http://aa.net.uk/news-2012-02-dns.html

We think all is OK now, so closing this incident.

It looks like we have a major issue with mail and web services.

This is being investigated now.

This looks like an issue with the main storage array used by web and email. Being worked on now.

Looks like we have it working again - web pages are fine - email being a bit sluggish catching up.

The disk server has been restarted to clear the problem. The underlying cause of the problem is being investigated.

Routing blipped briefly - we're investigating.

Routing is back now.

Total downtime was from about 11:45 to 11:53

We're investigating the cause of this problem.

We are seeing problems with equipment located in our HEX datacentre, most likely power related.

This will be affecting A&A webspace, customer L2TP connections, beta tester data SIM connections and some customer equipment which is hosted there.

We have been in contact with the data centre staff, who are investigating it.

The problem doesn't look like power, the core routers we have there appear to be rebooting.

Power cycling has restored the service to one of the routers, so access to servers hosted there is working again now. However the second router remains offline and is still causing problems with data SIM testers.

We are working to resolve that problem too.

The main issues have been resolved now - it does not look like power, but trying to find why two separate routers developed problems at the same time in HEX. Data SIMs will be affected still.

We have re-routed the data SIMs and set up backup for future use anyway.

Both routers working now.

Investigating now - if this is the same as we had at the weekend we should be able to sort it quite quickly.

We hope to have this sorted in a few minutes.

This is impacting some VoIP services but not all.

There will be a slight blip on broadband while we sort this.

This looks like some issue with routing through LINX. We may take down the route collector peering until we are happy we have identified the cause.

Still seeing some issues.

Equipment reboot worked briefly and then the problem re-occured. It seems clear this is a routing issue with a peer that is causing a black hole. We do not understand exactly where or how yet and this is being addressed.

We have taken town LINX route server peering and things are looking a lot better - checking things now.

It may be worth explaining this a little. We have dual redundent equipment to allow for failures. If something fails completely, or can be turned off, then the systems re-route to use other equipment. Depending on where such issues are this can mean no outage, a few seconds or a few minutes.

However, if there is a partial failure, such as a single black-hole route for the link to Maidenhead, then this is not an equipment failure. The other routers get that route and expect it to be valid. This can create complex problems that are hard to diagnose, also and mean we have to use various alternative means to access systems which causes delays.

I would stress, just because taking down the LINX route server seems to have addressed the issue does not mean there is an issue with LINX. This could be something odd with our routers, or the LINX router server or a peer via that route server feeding something odd to us as a route. We're trying to identify what has happened but for now we'll leave the route server shutdown until we know.

There is a general issue with packet loss at the moment.

It looks like it might be related to transit, and we're investigating.

Still investigating. Looks like a problem with peering.

Something's recovering - traffic levels are returning to normal.

Looks like a power failure in Telehouse North, which would have affected a lot of peering, and some transit.

It looks like everything is back to normal for us.

We have been trying to resolve this properly and run in to a further snag which means again we have no routing to servers in HEX. Broadband unaffected as before but many services like control pages, hosted servers, our web site, accounts, and so on are all off-line while we reoslve this.

We are waiting for someone on-site at present.

Ok, working again.

Trying to find the exact problem for a permanent fix.

We have a major issue with a firewall/router in HEX at present which will affect access to web pages, control pages, accounts system, and variosu systems. We hope to have this resolved ASAP.

Routing has been reworked - some smaller issues remain.

We seem to have lost access to machines in Harbour Exchange Square.

This includes one of our key machines for RADIUS authentication for broadband lines (clueless) and our accounts server (priceless), customer web servers (Limitless) and a few hosted machines.

We are trying to identify the cause of the problem now.

We have managed to confirm it is not a power issue

We are gettign someone to check the rack in HEX now.

The backup RADIUS and DNS servers are working as they should.

This appears to be an issue with our main firewall in HEX. It is firewalling rather too well all of a sudden.

We are just waiting on a reboot now. We believe we have found the cause of the issue though so can take some preventative measures once the reboot is complete.

Reboot complete - all working

One slight side effect - session tracking timed out as the RADIUS server could not see the LNS. This meant PPP restarts for most lines at some point during the night and a few may do so during the day (on the hour). Until then usage is not being meter for those lines.

We will be applying an update to the firewall shortly