Andrews & Arnold Incidents Feed

[PEW] Broadband: LNS swap over night

2012-02-22T07:10:25Z

We are expecting to move lines between D and C LNS over night. This should be over night at your preferred time and minimal disruption. The RADIUS changes should mean you do not need any issues with the status shown on the control pages either.

Update 22 Feb 2012 01:11:46

Seems to be going to plan...

[PEW] Broadband: New DNS Resolvers (Open)

2012-02-21T12:01:23Z

We will be bringing online our new DNS resolvers (These are the servers you'd know as 217.169.20.20, 217.169.20.21, 2001:8b0::2020 and 2001:8b0::2021).

These are new servers in our new rack and are part of the work we've been doing with new BT links etc.

The first time we brought these online there were problems. We do believe these have been solved and our testing as shown them to be working as expected.

Note: 217.169.20.20, 217.169.20.21, 2001:8b0::2020 and 2001:8b0::2021 are 'anycast' - which means that customer connections and servers in Maidenhead use the same IPs, but actually reach different machines - this work will affect the servers used by ADSL/FTTC customers.

The IPs are not changing, customers do not need to make any changes.

Update 21 Feb 2012 12:02:51

This is happening now.

Update 21 Feb 2012 12:08:20

The new servers are now live, we'll monitor them closely.

[Info] General: SSL Certificates Updated

2012-02-20T11:23:05Z

We've updated SSL certificates for our customer facing servers this morning (email, control pages).

Users who don't have the CAcert root certificate installed may see errors. Details on http://aa.net.uk/cacert.html

[PEW] General: At Risk - Telecity (London) Hosted Services

2012-02-20T09:25:22Z

As this is one of our original racks it affects very few customers:

Those with hosted FireBricks in London for line bonding
Customers we host servers for in London
(non-wholesale) customers with L2TP connections in to us
Data SIMs
DSL/Ethernet/FTTx connections are not routed through this datacentre.

TelecityGroup will be carrying out essential maintenance works to the electrical power supply infrastructure at Harbour Exchange 6/7during an extended maintenance window on 18th - 19th February 2012 between 08:00 - 18:00 (GMT).

The work will involve carrying out a routine generator load test to allow maintenance of our critical electrical infrastructure. Power to customer equipment will be provided by generator supply for the duration of the maintenance window under controlled conditions. There will be a reduced level of resilience to your equipment during this period, however we do not anticipate any impact to your service as the generators are capable of supporting the site load. TelecityGroup’s engineering team will be monitoring the power and generator systems at all times whilst this work is carried out.

[Info] Broadband: RADIUS accounting delays

2012-02-18T11:14:55Z

Over the last few weeks we have seen a number of cases where RADIUS accounting is not working correctly. This is usually when we do an LNS switch or BT have a blip or some other case where there is a lot of traffic for connecting and disconnecting circuits.

We are going to be working on this to make the accounting much more robust, but it will take some time to track down the exact cause and may mean some changes on the LNSs (which originate the RADIUS accounting) as well.

The impact is that it can mean a line appears to be off line on the control pages when it is not, or shows the line status in an odd colour. In most cases it has the side affect of not metering usage and so giving customers free bandwidth.

We'll post more details when we have them.

Update 18 Feb 2012 15:36:24

We have made some changes today and they seem promising. It looks like we have managed to ensure RADIUS accounting is must more reliable. We'll be monitoring during the day.

[PEW] Broadband: LNS change over

2012-02-18T09:37:26Z

In light of one of the new BT links having a 2 minute outage today we are switching lines between LNSs - half of our customers will be affected by the move which should happen over night according to your preferred time for LNS clear.

This will ensure the line is free for BT to do any necessary tests on the affected circuit.

[MSO] Broadband: Blip on ADSL/FTTC Lines

2012-02-17T15:10:32Z

We've just had a blip on some BT ADSL/FTTC lines, most are back online already. We'll investigate the cause.

Update 17 Feb 2012 12:11:53

BT lines on just one of the LNS's blipped - lines on C.Gormless blipped, lines on D.Gormless did not.

Lines are almost all back online now.

Update 17 Feb 2012 12:14:52

This was affecting BT lines. BE lines were not affected.

Update 17 Feb 2012 12:24:32

The network switch port connected to one of our BT fibres was offline between 12:02:36 and 12:04:23.

We're investigating further.

[Minor] Broadband: DNS Problems Today

2012-02-17T15:06:45Z

There have been reports of DNS problems for customers using 217.169.20.20 and 217.169.20.21. Customers may have seen error messages like 'Unknown host' in their programs.

For the past week we've been using new DNS resolvers in our new rack, but for the time being we've taken these offline and are using the original DNS servers whilst we investigate further.

[Minor] VoIP: Sending of SMS not Working

2012-02-16T12:54:57Z

Sending of SMSs is not working at the moment, either from our notification systems (eg when a line drops) or from customers sending via our website or 'curl'.

Update 14 Feb 2012 20:42:34

We believe we have fixed this, and are monitoring the situation now.

[Minor] Broadband: NORTH SHIELDS Exchange outage

2012-02-16T10:37:11Z

Since 16:59:44 yesterday all lines at the NORTH SHIELDS Exchange have been down.
BT are currently investigating and have an engineer on the way to the exchange.

Update 16 Feb 2012 09:45:13

Engineer is now on site and is currently investigating.

Update 16 Feb 2012 10:37:11

service was restored at 10:10.

[Minor] Broadband: BT HALIFAX exchange outage

2012-02-15T15:00:39Z

All BT lines at the halifax exchange are down.
BT have been informed and are investigating.

Engineers are on their way to site to investigate further.

Update 15 Feb 2012 14:12:39

Engineer now on site, BT and Vendor support have been engaged.

Update 15 Feb 2012 15:00:39

Fixed

[Minor] Broadband: BT Congestion at the CHIPPING SODBURY exchange

2012-02-15T09:21:25Z

Since 2012-01-26 we have been seeing Peak latency approx 2am to 10pm on all BT CHIPPING SODBURY exchange lines.

Update 15 Feb 2012 09:21:25

It appears the congestion issue has been resolved.

[Minor] Broadband: 21CN Congestion at the GARFORTH Exchange

2012-02-15T09:20:20Z

We are seeing increased peak and minimum latency on all lines at the GARFORTH Exchange between 8pm till 10pm.

Update 17 Jan 2012 10:08:41

BT are investigating this at the moment.

Update 17 Jan 2012 11:09:54

BT Operate are investiagting.

Update 18 Jan 2012 13:54:29

BT think the problem is being caused by a traffic shaper their side. They have emailed the relevent department to get this sorted.

Update 15 Feb 2012 09:20:20

Looks like the congestion issue has been resolved.

[Minor] Broadband: 21CN Congestion at the DEPTFORD Exchange

2012-02-15T08:59:17Z

We are seeing packet loss between 8pm - 11pm on all 21CN lines at the DEPTFORD exchange

Update 17 Jan 2012 10:08:58

BT are investigating this at the moment.

Update 17 Jan 2012 10:52:00 BT operate are looking into possible causes for this now.

Update 15 Feb 2012 08:59:17

It appears the congestion issues are resolved.

[Minor] Broadband: 21CN Congestion on BRIXTON Exchange

2012-02-15T08:57:26Z

We are seeing packet loss between 8pm to 11pm on all 21CN lines at the BRIXTON exchange.

Update 17 Jan 2012 10:09:38

BT are looking into this at the moment.

Update 15 Feb 2012 08:57:26

It appears the congestion issue is resolved.

[Minor] Broadband: 21CN congestion on the KINGSLAND GREEN Exchange

2012-02-15T08:55:37Z

We are seeing packet loss between 8pm to 11pm on 21CN lines at the KINGSLAND GREEN exchange

Update 17 Jan 2012 10:09:19 BT are investigating this at the moment.

Update 15 Feb 2012 08:55:37

It appears the congestion has been resolved.

[Minor] Broadband: WELWYN GARDEN Exchange Outage

2012-02-15T08:54:27Z

ADSL lines on Welyn Garden City exchange are affected by a cable failure within BT.

It looks like it may be sometime (this evening/tomorrow) before service will be resumed.

Update 10 Feb 2012 13:45:45

Malicious damage to five cables located and cable repair is progressing.

However, BT say the next update will be 8pm.

Update 10 Feb 2012 14:34:53

It's made the news!

http://www.ispreview.co.uk/story/2012/02/10/malicious-damage-to-five-bt-cables-causes-broadband-outage-in-welwyn.html

Update 10 Feb 2012 14:37:40

BT currently have the anticipated fix time of noon Saturday. (This is an estimate though, and service may well be restored earlier or later)

[Minor] Broadband: 21CN-BRAS-RED4-L-WAT BRAS issue

2012-02-15T08:48:06Z

Looks like the BRAS is not handeling the traffic between the end users and itself.
BT are investigating.

Update 15 Feb 2012 08:48:06

Service has been restored as of 17:30

[MSO] General: DNS Problem with aa.net.uk / aaisp.net.uk

2012-02-14T13:06:34Z

We currently have a DNS problem with our main domains aa.net.uk and aaisp.net.uk - this will cause a problem with various services that we run.

Update 13 Feb 2012 12:45:36

This was a simple planned change - it seems however we have been caught out by bind refusing to reload if one zone has one syntax error. Seems one of our customer zones had a typo in it which our tools had not managed to check and that caused it to refuse to load.

Unfortunately it took quite a few minutes to find what was wrong as this was nothing to do with the changes we actually made.

Update 13 Feb 2012 12:55:05

Most things are OK.

Update 13 Feb 2012 13:13:38

Both authorative serves are serving the aaisp.net.uk and aa.net.uk zones correctly.

Update 13 Feb 2012 13:39:09

There are still problems with the aaisp.net.uk domain - we're working on this.

VoIP is currently affected too, this is being worked on now.

Update 13 Feb 2012 13:56:00 We are running in to slightly unexpected errors as well, and working through them. Some things that we have not touched are not working, which kind of makes no sense.

Update 13 Feb 2012 14:43:27

Whilst most things are working, at least from customer lines, there was an issue which caught us out, but was way to far in to the process to sensibly back up. The top level delegation for aaisp.net.uk was going to a special DNS server, which meant when we moved everything it stopped working properly.

Update 13 Feb 2012 14:46:17

Some incoming VoIP is not working, we are still working on this.

Update 13 Feb 2012 15:09:53

Still lots of progress mopping up things. Mostly non customer affecting.

Just to clarify - you should be able to use aaisp.net.uk or aa.net.uk as they interchangable. However, right now, some places are having some issues seeing some of the aaisp.net.uk sub domains.

The preferred version is now aa.net.uk and we should be quoting that everywhere now.

Using aa.net.uk is also a work around for the issues some people are seeing right now, which are down to DNS caches.

Update 13 Feb 2012 16:18:42

At the moment we still have some services affected, these are: some incoming VoIP and accessing our services from outside our network (ie customer using DNS resolvers ither than ours)

Update 13 Feb 2012 17:38:56

We really think this should be sorted now - but monitoring carefully.

Update 13 Feb 2012 18:25:30

Mor details on http://aa.net.uk/news-2012-02-dns.html

Update 14 Feb 2012 13:06:34

We think all is OK now, so closing this incident.

[Minor] VoIP: Phones timing out on registration

2012-02-14T13:05:34Z

Phones registered to our 'C' server are losing registration once their registration times out. We're investigating.

Update 14 Feb 2012 13:05:34

Now fixed.

[PEW] Broadband: At risk - BGP changes

2012-02-14T10:14:05Z

We have a few very minor BGP changes to make which should have no impact at all.

Update 13 Feb 2012 11:55:24

OK, why are we making a change that has no impact at all?

The impact is "what happens if a link breaks". There is a risk that without this change some specific link failures might not automatically fall back as they should.

I hope that makese sense...

[PEW] General: At risk - aa.net.uk domain (Open)

2012-02-13T10:51:15Z

We have had the aa.net.uk domain for a while now, and been publishing it as the main domain we use rather than aaisp.net.uk. In most cases you can use either of these.

However, reverse DNS for our machines is still currently in aaisp.net.uk.

We are planning to change this around, probably later today, so that reverse DNS for all servers will be in aa.net.uk

We do not expect any issues to affect customers.

[MSO] Broadband: All BE lines down

2012-02-10T15:06:28Z

It looks like around half od the BE lines went off around 02:43 and the rest at 02:58.

We were not aware of planned work this morning, though various work has been planned this week.

Update 10 Feb 2012 03:27:35

We have tried routing traffic via our old rack as well, and no joy.

Update 10 Feb 2012 03:30:16

Obviously customers using BE+BT lines as a fallback arrangement are fine, using BT.

Obviously customers using 3G backup are working.

If you only have a BE line you may want to consider some of the fallback arrangements we can offer. Though it is rare for it to be BE that is the carrier that has failed, it is sensible to consider contingencies.

That said, we would hope they have this fixed during the night.

Update 10 Feb 2012 03:39:19

There were unplanned outtages on 14 exchanges at approximately those times, we're still waiting to find out why this occured.

Update 10 Feb 2012 05:30:00

O2 have confirmed that changes made by them to the network overnight resulted in loss of service to large parts of the network, including all wholesale services.

Their engineers are currently rebuilding configurations on all aggregation nodes on their network to reverse the changes. We are beginning to see sessions establish again, about 20% have reconnected in the past few minutes.

Update 10 Feb 2012 08:03:43

It appears that O2 engineers have now finished restoring configurations, as we can see all ISAMs on our monitoring again and we are seeing the majority of customers back online.

Update 10 Feb 2012 06:30:00

O2 engineers have restored configuration on about 40% of core devices. We are seeing roughly that % of sessions back up.

[Minor] Broadband: DNS Oddities - m.facebook.com

2012-02-10T14:10:51Z

There have been a few odd things with DNS resolving this evening - eg m.facebook.com. We've put in temporary work around and will investigate further in the morning.

Update 9 Feb 2012 12:24:15

This does look to be down to facebook's DNS servers - they time out on AAAA record lookups and our DNS resolves assume that they are unavailable.

This may be an intermittent problem, as m.facebook.com is resolving properly at the moment.

[MSO] Broadband: Routing problem affecting ADSL/FTTC lines

2012-02-10T03:38:59Z

There was a routing problem affecting ADSL lines.

We initially thought this was just our office (hence the previous post)

Routing is now restored and we're investigating the cause.

[PEW] Maidenhead Data Centre: UPS Work (No user impact)

2012-02-09T15:03:44Z

We have been informed by the Maidenhead datacentre that they have some planned work on their UPS systems.

They will be carrying out a replacement exercise on end-of-life capacitors in the UPS.

The unit will be taken off-line for the duration of the work however due to the configuration of the installation there will be no impact to the electrical supply or resilience to equipment

The work will be carried out on Monday 16th January, starting at 21:00 and should last 5 hours.

This work is factory advised and forms part of their robust Planned and Preventative maintenance regime.

[Minor] Maidenhead Data Centre: Possible Routing Problem

2012-02-09T15:03:13Z

We're investigating reports of strange routing problems involvng access to/from our Maidenhead data centre

[PEW] Broadband: At Risk - Transit Work

2012-02-08T16:06:29Z

We have multiple transit links - these link us (and you!) to to the rest of the Internet. We'll be moving one of our transit links over to our new rack in telehouse.

This should not affect customers, as the link wil be taken down cleanly, ports swapped, and the link brought back up. Whilst the link is down, traffic will use our other transit links.

Start date: 8th Feb
Start time: 14:30
Finish time: 15:30

Update 8 Feb 2012 14:42:02

This is happening now.

Update 8 Feb 2012 15:00:37

Traffic is now being passed over the new ports/cables.

[PEW] Broadband: BE Planned Outage on 3 exchanges (Open)

2012-02-08T09:43:40Z

We have received this notification of planned work from BE:

Our broadband service will soon be disrupted while maintenance work is
carried out at Cumbernauld (WSCUM), Larbert (ESLAR), Falkirk (ESFAL)
our exchanges. 
Details below: 
Start of outage 16th of February 2012 at 23:00 PM 
End of outage: 17th of February 2012 at 04:00 AM 
Duration: up to 5 hours 

Apologies for any inconvenience caused.

[PEW] Broadband: BE Planned Outage on 13 exchanges (Open)

2012-02-08T09:40:29Z

We have received this notification of planned work from BE:

Our broadband service will soon be disrupted while maintenance work is
carried out at our Titchfield (SDTTCHF), Stubbington (SDSTBBN),
Petersfield (SDPTRSF), Hayling Island (SDHYLNG), Emsworth (SDMSWRT),
Horndean (SDHRNDN), Havant (SDHVNT), Waterlooville (SDWTRLV), Fareham
(SDFRHM), Portsmouth North (SDPNRTH), Cosham (SDCSHM), Gosport
(SDGSPRT), Portsmouth (SDPCNTC) exchanges. 

Details below: 
Start of outage 11th of February 2012 at 22:00 PM 
End of outage: 12th of February 2012 at 07:00 AM 
Duration: up to 9 hours 

Apologies for any inconvenience caused.

[Info] General: Terms and Conditions

2012-02-07T10:20:36Z

We have made some minor changes to clarify our terms and conditions in a few places. See http://aa.net.uk/legal.html

The main change is the Customer Complaints Code which details how to make complaints and how to take then on to ADR or court. The change includes a list of things we need from you when making a claim.

We don't think this is an onerous list - basically it is detailing exactly how much you are claiming and why. It does ask what we have done wrong and how we have breached our terms.

We thing this will help people focus on exactly what they are claiming and make it easier for us to address any complaints.

It is worth noting that we get very few formal complaints.

[Info] Broadband: Progress report

2012-02-06T17:17:07Z

See my blog...

http://revk.www.me.uk/2012/02/silly-set-backs-today.html

[Minor] Email: Authenticated Outgoing Email Problem

2012-02-06T08:53:45Z

There was a problem on one of the outgoing email servers this morning. This has been resolved and we're looking in to the cause.

[PEW] Broadband: At risk - LINX peering move

2012-02-05T16:34:43Z

We are moving LINX peering to the new rack today - this is very low risk, but means traffic routing will change while we do this. It should not have any impact on services.

Update 5 Feb 2012 16:34:43

Look longer than expected, but all working.

[Info] Broadband: LNSs in use

2012-02-05T01:20:58Z

As part of our major upgrade work, customers may find they are on any of the four LNSs currently in use, A/B (in the old rack) and C/D (in the new rack). If you find you have bonded lines with some one A/B and some on C/D then you can ppp kill the A/B lines to catch up. We are working on tools to monitor split bonded lines anyway though this should not normally happen.

We will probably zap all lines still on A/B tomorrow morning so that we are fully operating on the new host links in to BT.

Wholesale customers will not be affected.

P.S. Seeing as I am up anyway (grrrr) I am running through all non wholesale lines that have said a pre 2am is preferred and moving over to new LNSs now.

[PEW] Broadband: BE UPS Work - 5 Minute Outage 8th Feb (Open)

2012-02-02T09:21:27Z

BE will be doing some UPS work which will mean an outage of up to 5 minutes.

Date: 8th Feb 2012
Time Slot: 23:00 - 02:00

This will only affect BE circuits.

[Info] Broadband: Upgrade progress

2012-02-02T04:25:50Z

Well, we have managed to integrate the new rack in to the network now, without any nasty blips. Things are looking good.

We even have one of the staff FTTC lines on the new BT host link working. A whole gigabit host link and 10 FB6000 series gigabit routers being being used by just one line - nice one Paul!

This means we are ready to move some of our interconnect links over, like LINX and LONAP.

At the weekend we plan to switch all customer traffic to the new host link - as a normal LNS switch, over night.

Thank you all for your patience and understanding during this major upgrade.

More on my blog http://revk.www.me.uk/2012/02/well-someone-has-to-test-it.html

[PEW] Broadband: At risk - more router work (Open)

2012-02-01T15:40:30Z

We are reviewing the logic by which the new rack is brought on-line tomorrow. We have a good idea what has gone wrong today and expect to have this fully resolved. When we bring the new routers on line there should be no issues. Please do let us know ASAP if you have any routing problems.

We are doing the work during the day deliberately as (a) we expect no issues and (b) customers are much better at reporting issues. Today we had some odd, and isolated issues mostly relating to IPb6 which would not have been seen if this was done during the night.

The new racks double our immediate capacity, pushing us beyond the gigabit in terms of broadband services. We do apologise if there are any issues resulting from this work.

Update 1 Feb 2012 15:40:20

Work today has gone well and we have proved the new BT host links.

BGP work has progressed well, step by step.

We have hit one minor unexplained error message and so have stopped work for today. This is before connecting everything back to the existing racks. We will review this in detail and hopefully continue tomorrow.

Update 1 Feb 2012 16:31:12

The good news is the error message was very minor and we should be able to continue work. We are, obviously, taking things very slowly.

Update 1 Feb 2012 18:07:24

Well, all the BGP linkages worked as planned - phew!

Tomorrow we are moving some of the peering points to the new rack. That should be fun but equally non-service affecting.

Thank you for your patience.

Update 2 Feb 2012 09:41:58

We are doing some more router upgrades this morning. Should be seamless

Update 2 Feb 2012 17:00:21

We are going to allow lines that reconnect this evening to go on to the new host link to confirm all is well. If you have line bonding and one blips you can ppp-kill the other to retain bonding, but otherwise it will still work, just not downlink bonded. We will put these back tomorrow. Any problems contact me (RevK) on irc, or if all else fails send an MSO text.

[Minor] Broadband: Lines dropped: 15% 20CN and 100% 20CN-REGION-EDINBURGH and 100% 21CN-REGION-FK

2012-02-01T14:43:58Z

Lines: 15% 20CN and 100% 20CN-REGION-EDINBURGH and 100% 21CN-REGION-FK dropped at 2012-01-26 17:21:37
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 1 Feb 2012 14:43:58 This was a BT MSO. Which was fixed at 17:44

[Minor] Broadband: Lines dropped: 100% 21CN-BRAS-RED6-L-STE

2012-02-01T14:42:34Z

Lines: 100% 21CN-BRAS-RED6-L-STE dropped at 2012-01-29 17:36:51
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 1 Feb 2012 14:42:34

BT have been unable to give us an explaination for this outage.

[Minor] Broadband: Lines dropped: 100% 21CN-BRAS-RED2-FK

2012-02-01T14:40:13Z

Lines: 100% 21CN-BRAS-RED2-FK dropped at 2012-01-27 23:54:07
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 27 Jan 2012 23:56:02 Lines: 100% 21CN-BRAS-RED2-FK dropped again at 2012-01-27 23:54:07.

Update 1 Feb 2012 14:40:13

BT have been unable to give us an explaination for this outage.

[Minor] Broadband: Lines dropped: 100% 20CN-REGION-EDINBURGH

2012-02-01T14:36:01Z

Lines: 100% 20CN-REGION-EDINBURGH dropped at 2012-01-26 17:22:08
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 1 Feb 2012 14:36:01 This was a BT MSO. Which was fixed at 17:44.

[Minor] Broadband: Lines dropped: 85% 20CN-REGION-EDINBURGH and 89% 21CN-REGION-FK and 100% 20CN-BRAS-ERX16-EDINBURGH3 and 100% 20CN-BRAS-ERX6-EDINBURGH2 and 100% 20CN-BRAS-ESR4-EDINBURGH5 and 100% 21CN-BRAS-RED2-FK and

2012-02-01T14:35:29Z

Lines: 85% 20CN-REGION-EDINBURGH and 89% 21CN-REGION-FK and 100% 20CN-BRAS-ERX16-EDINBURGH3 and 100% 20CN-BRAS-ERX6-EDINBURGH2 and 100% 20CN-BRAS-ESR4-EDINBURGH5 and 100% 21CN-BRAS-RED2-FK and 100% 21CN-BRAS-RED3-FK dropped at 2012-01-26 17:21:37
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 1 Feb 2012 14:35:29

This was a BT MSO. Which was fixed at 17:44.

[Info] Broadband: Its alive!

2012-02-01T12:24:39Z

OK, the mad scientist style cackling is probably not appropriate, but yes - the new BT host links are up and working. This is one of the key steps in the upgrade plans.

Over the afternoon we are stitiching the network together one step at a time and testing carefully at each step. We may leave the final linking in of the existing rack until after 5pm. More to follow.

We expect to do an LNS switch to the new LNSs on the new host links later in the week.

[MSO] Broadband: Another blip around 2am

2012-02-01T10:01:12Z

There was, quite separate to any other issues, a problem at 02:06 which resulted in the main LNS resetting. All customers quickly reconnected with no problems.

We have detailed logs of the actual problem, but it is baffling us slightly at present. We will be investigating this later in the day. This does, however, appear to be completely unrelated to everything else that has been happening.

We are rebalancing some lines between LNSs over night after this incident.

[PEW] Broadband: At risk - core router upgrades

2012-02-01T10:00:43Z

Over the next few days we will be upgrading several core routers - experience tells us this should have no impact on services - but there is a risk in this.

Update 1 Feb 2012 02:24:14

The core router upgrades actually went seamlessly, and were not part of the issues that were seen.

[MSO] Broadband: Service outage

2012-02-01T02:25:37Z

Investigating

Update 31 Jan 2012 19:12:55

Something major with our LNS - investigating

Update 31 Jan 2012 19:16:09

No clear on cause but looks like our fault not BT this time.

Update 31 Jan 2012 19:16:32

Graphs lost, sorry

Update 31 Jan 2012 19:16:59

Looks like people coming back on line now

Update 31 Jan 2012 19:18:42

Not good - lost graphs - and cause unclear...

[PEW] Broadband: At risk - router work today

2012-01-31T17:28:04Z

Various work is being done today to link in the new rack to the existing rack. This is all low risk, but some backup equipment is being upgraded in the process.

We do not expect any impact on services.

Update 31 Jan 2012 17:28:04

There were some reported routing issues during the work and all changes have been backed out - we can see from usage stats that any issues were fairly isolated.

[PEW] Broadband: At risk - Router uprade

2012-01-28T17:57:54Z

There is a router restart expected this evening but it should have no impact on services.

Update 28 Jan 2012 17:57:54

All to plan, no blips

[Info] Mobile: Data SIM roaming

2012-01-27T16:47:51Z

Thanks to those that have helped with testing the data SIM roaming. We expect at present that there will not be any extra billing for this.

In theory data roaming should work everywhere three have roaming as, unlike voice romaing, it is always sent back through the home network.

Unfortunately it seems that getting the billing systems in place over all of the links involved is proving a challenge, and so at present data SIM roaming is disabled on all SIMs.

Once we have the billing systems, and price lists, this will be offered as an option.

Thanks again for help testing.

[Info] General: 80Mb/s FTTC Trial - Initial Customer Online

2012-01-26T15:13:48Z

2012-01-12 13:35:00, updated 2012-01-26 15:13:14

It is impressive what you can do on a short bit of copper :-)

We have a number of customers on the FTTC 80Mb/s trial now. One has 79.7Mb/s and another has 57Mb/s. This is pretty impressive. The uplink is around 20Mb/s as well...

The trial is not going badly - the lines are syncing up at nice high rates, and the rates are correctly coming through to us.

There are a few minor teething troubles getting the profiles right in the middle somewhere - but this is a technical trial, so not at all surprising.

This part of the trial is expected to last until the end of the month, where the second phase of the trial starts. The second part is expected to be open to the whole country (where FTTC is currently available)

Update 26 Jan 2012 15:12:49

Here is a speed test result.

https://p.twimg.com/AkF64vvCIAAVMQF.png

-a number of speed tests were carried out, this was the highest download result, the results did vary though, and the upload isn't as high as we'd expect - should be nearer 20Mb/s.

The end user will be trying a new router to compare.

[MSO] Maidenhead Data Centre: Problem in Maidenhead - affecting VoIp/ethernet/email/etc

2012-01-26T15:10:49Z

We are seeing a major issue in Midenhead - high levels of transit packet loss that will be affecting VoIP, email, and Ethernet customers (as well out our offices).

Update 24 Jan 2012 21:47:11

Looking like some sort of denial of service attack.

Update 24 Jan 2012 22:34:04

Sorry for the delay posting more details. This affects our links directly and making it difficult. The problem appears to be a huge denial of service attack invovling tens of thousands of sessions and filling gigabit links.

We have identified the target and disconnected it, black holed the target and even tried to divert traffic but to no avail as yet.

We are still working on this.

Update 24 Jan 2012 23:03:38

Just an update to say that this is still being worked on...

Update 24 Jan 2012 23:23:30

Still working on this

Update 24 Jan 2012 23:28:57

The problem is now only affeting Ethernet customers on the same block as the address being DDOS'd. We are still working on the issue. Most other services will be working fine now.

Update 24 Jan 2012 23:57:55

Some side effects on other services from Maidenhead, but we are still working on narrowing down the issue.

Update 25 Jan 2012 00:01:48

DOS attacks are, thankfully, rare. This has to be the biggest we have seen.

We will, of course, be talking to the customer who is being DOSed to fine what could have provoked such a major attack. There is usually a reason.

Update 25 Jan 2012 13:20:48

The blackhole for the target machine was removed at 1pm today, however the traffic was still being sent and affected VoIP, email and Ethernet services.

The block is now in place again, and we'll continue to investigate.

[Minor] Broadband: Lines dropped: 100% 21CN-REGION-PE

2012-01-26T14:45:52Z

Lines: 100% 21CN-REGION-PE dropped at 2012-01-26 14:19:19
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 26 Jan 2012 14:21:02 Lines: 100% 21CN-REGION-PE dropped again at 2012-01-26 14:19:19.

Update 26 Jan 2012 14:24:30

Half the lines are now back.

Update 26 Jan 2012 14:45:52

All lines are now back.
Proactive are going to ask BT operate to investigate what caused the incident.

[MSO] Maidenhead Data Centre: Maidenhead datacentre problems again

2012-01-25T13:16:49Z

Similar to last night, access to servers and services in Maidenhead have high packet loss.

This will affect email, voip and Ethernet services.

Update to follow shortly.

Update 25 Jan 2012 13:13:19

Datacentre staff are working to blackhole the IP address that is the target of this attack.

Update 25 Jan 2012 13:14:43

The target IP address has been blackhole'd and service has been restored.

[PEW] Broadband: Overnight LNS switches

2012-01-25T07:15:36Z

We appreciate that doing an LNS switch in the evening is not ideal, and even though it should only be a few seconds blip in service it can cause inconvenience (especiall for those using NAT).

We have a major network upgrade going on, and we will be doing a number of moves between LNSs over the next two weeks as part of this.

However, the good news is that we believe the overnight LNS switchover scripts are ready to test. This means the LNS switch over is done during the night at a time to suit you. The control pages allow you to pick an hour during the night and we aim to follow your request on that.

However, this does, of course, need testing, so we plan to run this overnight tonight.

See more at http://aa.net.uk/kb-broadband-duallns.html

[PEW] Broadband: LNS switch this evening

2012-01-23T17:31:04Z

We have to do an urgent LNS switch over this evening. Apologies for any inconvenience.

Technical details: We have identified a latent issue in the current code which puts it at risk of problems if there was a failure of both of our RADIUS servers (somewhat unlikely).

Update 23 Jan 2012 17:31:04

The switch over went fine, and allowed testing of the new overnight scripts for this. We hope next time to run an LNS switch over night, and we may try this later in the week.