Andrews & Arnold Incidents Feed

[Minor] Mobile: Faulty Mobile Gateway (GGSN) Within Three Affecting Some SIMS (Open)

2013-05-20T11:20:24+01:00

Started: 2013-05-20 08:00:00

We are currently seeing connection problems to one of Three's GGSNs. Any SIM trying to connect across that GGSN is timing out as a result.Connections across the other GGSN are completing as normal, and each SIM will eventually try to use that GGSN for routing.

Unfortunately this is outside of our control, as the SIM's routing is randomly picked each time. As a result, you may be seeing problems with SIMs connecting. Three are currently investigating the fault, and more information will be provided as soon as it becomes available. Apologies for any inconvenience caused.

Update 20 May 2013 12:55:29

An update from Three

Three have advised that they are currently experiencing a major network fault between two of their sites, resulting in some SIMs not being able to stabilise on a particular route. This is being treated as a priority 1 issue by Three's NOC and we should be receiving an update within the hour.

Update 20 May 2013 16:33:52

Update:

Traffic has been approaching normal levels for the last few hours, and we have been advised by Three that they have identified a fault on one of their routers and are working to resolve.

[Minor] Mobile: New Data SIMs Needing APN

2013-05-20T11:17:09+01:00

Started: 2013-05-07 16:00:00

We've noticed that some new DATA SIMs are requiring the APN to be set to 'm2m.aql.net' as otherwise they will only get an internet connection in to Three, and not in to us.

Normally an APN is not required, and only needed if roaming.

This seems to only be affecting SIMs activated since 1st May 2013.

Our carriers are looking in to the reason for this and will update us shortly.

Update 8 May 2013 16:30:14 Our carrier have confirmed this afternoon that last nights changes was actually stage 2 of 3. We've been advised they will be concluding the roll back tonight and so we should see SIMs recover once they've finished their work.

Update 8 May 2013 08:56:17

This looks to have been caused by a change in the Three network, however Three partly reverted their change. They will finish this on the night of 8th May.

Update 9 May 2013 12:55:38

The carrier have made changed to try to fix this but we are still seeing SIMs onnecting in to Three rather than being relayed in to us. We are chasing this.

Update 10 May 2013 11:41:35

The carrier completed further work on this over night, and it appears that affected SIMs are now working without the need for the APN. We've tested an affected SIM succsessfully in the office.

[MSO] VoIP: Registration timeouts

2013-05-19T09:40:24+01:00

Started: 2013-05-19 08:12:10

A database issue this morning has affected incoming calls due to registration timeouts. This has been corrected and registrations are now working correctly.

[Minor] Broadband: BE Outages at Shepherds Bush, Kensington Gardens and Acton Exchanges

2013-05-17T09:23:23+01:00

Started: 2013-05-16 15:23:05

02 have informed us that there is currently an outage at the Shepherds Bush (LWSHE), Kensington Gardens (WRKGDN) and Acton (LWACT) exchanges.

Engineers are currently investigating and 02 expect to have an update within the next 2 hours.

Update 17 May 2013 09:23:23

This is now resolved, we have not been given any information as to the cause or fix for this outage.

Update 17 May 2013 09:23:29

This is now resolved, we have not been given any information as to the cause or fix for this outage.

[Minor] Broadband: 1/3rd of lines PPP restarted

2013-05-15T14:01:02+01:00

Started: 2013-05-15 13:45:00

Due to human error, we have managed to clear PPP on around a 1/3 of lines (B.gormless) today.

We are looking in to how procedures can be tightened up to avoid this in future.

Update 15 May 2013 13:55:50

This was a controlled restart of the LNS not a crash, and as such the PPP restarts were even quicker as devices did not have to wait for LCP timeout to restart.

And yes, we are seriously looking at the best way to avoid this sort of error happening again.

And yes, the human concerned was me - RevK - so I owe a few people a pint...

Resolution

The answer is css - the control pages of the various machines currently all look identical apart from the host name. Using some css we can make all "live" boxes stand out very clearly.

[Minor] Broadband: Lines dropped: 100% 21CN-REGION-21CN-BRAS-RED10-L-STE and 100% 21CN-REGION-21CN-BRAS-RED13-L-STE and 99% 21CN-REGION-L-STE

2013-05-14T09:35:21+01:00

Started: 2013-05-14 02:19:57
Lines: 100% 21CN-REGION-21CN-BRAS-RED10-L-STE and 100% 21CN-REGION-21CN-BRAS-RED13-L-STE and 99% 21CN-REGION-L-STE dropped at 2013-05-14 02:19:57
We have advised BT
This is likely to have affected multiple internet providers using BT

Update 14 May 2013 09:35:21

Ater chasing BT on the cause for this outage, it would appear that this was caused by a Planned Works at London Stepney Green.

The aim of these planned works was to increase capacity.

Update 14 May 2013 09:35:28

Ater chasing BT on the cause for this outage, it would appear that this was caused by a Planned Works at London Stepney Green.

The aim of these planned works was to increase capacity.

[PEW] Broadband: LNS switch

2013-05-13T23:01:14+01:00

Started: 2013-05-13 16:00:00

Following an incident with one of the LNSs at 4pm today, we will be clearing sessions off C.gormless (one of the LNSs) around 23:30, which moves lines to D.gormless where they should be. Then there is a scheduled LNS switch over tonight at the preferred time. This addresses the underlying cause of the problem today.

There will be overnight LNS switch overs over the rest of this week as part of a general update designed to improve stability fiurther. At this stage we have no further major developments for the LNSs in the pipeline but we will address any issues that arrise, obviously.

The good news is that the issue at 4pm allowed us to see a real life test of a 1/3rd of customers all connecting at once. The LNS and RADIUS servers performed perfectly, possibly better than ever before with some customers reconnecting in litterally a few seconds and others as fast as their routers tried connecting.

Update 14 May 2013 01:16:15

The LNS from which lines are being switched again had an issue just after 1am. We can confirm this is the same problem that affected it at 4pm yesterday and is fixed by the update. It simply means the LNS switch over has been somewhat more aggressive than would otherwise have happened.

[Info] VoIP: 111 and 101

2013-05-11T21:31:24+01:00

Started: 2013-05-11 21:28:48

We have changed centrex dialling such that calls to 101 and 111 will no longer be recognised as centrex dialling.

I.e. if you have three digit local (centrex) dialling, and extensions 111 and 101, we suggest you reassign these internal numbers.

101 and 111 now route to a carrier for non emergency police and HNS direct.

As you know, you cannot use extensions 112 or 999 for internall calling anyway (emergency services numbers).

Sorry for any inconvenience caused.

[PEW] Broadband: 16/05/2013 BE Planned Maintenance Outage (Open)

2013-05-10T13:47:34+01:00

Started: 2013-05-16 00:01:00

We have been notified of the below planned maintenance window affecting some BE DSL connections:

DOWNEND

FISHPONDS

WINTERBOURNE

FROME

WARMINSTER

TROWBRIDGE

MELKSHAM

KINGSMEAD

COMBE DOWN

KINGSWOOD

BITTON

NORTH

WESTBURY-ON-TRYM

AVONMOUTH

CLEVEDON

NAILSEA

PORTISHEAD

CALDICOT

CHEPSTOW

CHIPPING SODBURY

THORNBURY

WORLE

EASTON

EASTVILLE

FILTON

HENBURY

WEST

STOKE BISHOP

SOUTH

BISHOPSWORTH

MIDSOMER NORTON RMCUX

RADSTOCK

STROUD

HARDWICKE

CHELTENHAM

BISHOPS CLEEVE

BARNWOOD

CHURCHDOWN

[PEW] Broadband: BE Planned Maintenance Outage (Open)

2013-05-09T16:27:21+01:00

Started: 2013-05-15 00:01:00

We have been notified of the below planned maintenance window affecting some BE DSL connections:

NEW SOUTHGATE

MUSWELL HILL

NORTH FINCHLEY

POPLAR

TOTTENHAM

DEPTFORD

EUSTON

KENTISH TOWN

KINGS CROSSCLLOW

HENDON

HOUNSLOW

BLOOMSBURY AKA HOWLAND ST

HAMPSTEAD

LORDS

MAIDA VALE

MARYLEBONE

PRIMROSE HILL

SOHO AKA GERRARD STREET

SOUTHBANK

[Minor] Broadband: Issues with c.gormless

2013-05-08T12:41:39+01:00

Started: 2013-04-30 19:17:18

We are trying to get one of our LNSs working on latest code, and this is proving a struggle as there are some major upgrades. We hope to finish this as per the planned work during this evening. This may mean loss of graphs and some PPP restarts for some customers.

Update 30 Apr 2013 21:32:21

This is being an issue, and we will be working on this over night we expect.

Update 1 May 2013 00:46:20

We think working over night on this may have paid off - we're monitoring now, but we hope we have cracked the issue at last.

Update 1 May 2013 01:03:25

c.gormless is now running at normal load (1/3 of customers) and showing no leaked RADIUS slots or stuck sessions or processes. This is good news.

Update 1 May 2013 06:30:12 Monitoring is showing the LNS appears to be stable, but we have seen some issues with stats being recorded on our core RADIUS still. This is somewhat easier to address, and being looked at still.

Update 1 May 2013 17:17:34

It looks like the work last night as paid off. This evening we have the "D" LNS working on the newest code, and are letting lines trickle over to it when they reconnect for any reason. We plan to switch over more lines over night.

But so far it is looking like it is working as planned. We are checking the performance of the new code as well to make sure this was all worth it :-)

Thank you all for your patience on this.

Update 2 May 2013 02:09:03

We have switched traffic to d.gormless over night as planned.

The LNSs are looking stable - we are still seeing a few sessions from the previous few days which have not had proper accounting (so free usage) which are being cleared up.

We are also seeing the odd BT issue from time to time, and it is nice to say that a blip is not, in fact, my fault - more "situation normal".

We're still looking in to why people are getting the the odd up/down emails and tweets as it is all related. Now that the LNSs are being stable we can investigate this somewhat more easily.

Update 3 May 2013 04:57:05

The scheduled LNS switch overs to move on to the new code base seem to be working as planned. Tonight's has gone through without any apparent issues. We will be rolling out the change over the next couple of nights to cover all LNSs.

Once again, thank you all for your patience and understanding.

[Minor] Broadband: Packet loss on BE lines

2013-05-08T12:38:49+01:00

Started: 2013-05-07 12:18:00

There is packet loss on BE lines thoughtout the country, we have reported this BE and they are going to investigate.

Update 7 May 2013 14:54:25

We've had reports that BE Retail customers are also seeing a similar problem, so this seems like a general BE network/backhaul problem.

We have reported this to BE.

Update 7 May 2013 15:58:26

BE are aware of a possible core network problem and they are investigating.

Update 7 May 2013 16:12:51

BE have said that this is due to a technical fault on their network which they are investigating.

Update 8 May 2013 08:38:02

This was resolved just before 9pm - repairs to fibres were made within the BE network.

[Minor] Broadband: Pink status and odd emails

2013-05-02T02:14:55+01:00

Started: 2013-04-18 17:00:00

It looks like we have a blip in RADIUS around 5pm which has resulted in lines showing "pink" or "salmon" on the management pages and some people getting odd status update emails.

Services are working - this is an accounting issue.

We're not sure what happened exactly, but this is a scenario that the systems are designed to cope with relatively sensibly. Some usage may not be metered for the next few hours and some lines will be PPP restarted over night.

We are also doing an LNS switch over tonight, but there are issues with that which means it is likely to all happen later than usual, i.e. around 7am.

As it happens we are working on some updates to RADIUS which is hoped to be phased in over the next few weeks. Incidents like this, whilst minor, and not service affecting, are a nuisance, and they are being incorporated in to the new design to make such issues less likely.

Update 18 Apr 2013 18:58:48

We're clearing the pink lines now, ppp restart.

Update 28 Apr 2013 21:12:03

Some customers on C or D gormless will have gaps in their graphs this afternoon and this evening. The lines have been connected through these gaps but have has some PPP restarts.

[Minor] Email: 'H' server problems

2013-05-01T08:40:42+01:00

We're looking into problems with our 'H' mail server.

IMAP and POP3 are unaffected but this will be affecting some webmail access.

Update 1 May 2013 08:40:42

A reboot of the server fixed it.

[Minor] Email: Mail server down

2013-04-30T11:40:32+01:00

One of our mail servers is down. This will be affecting webmail (specifically RoundCube).

We're working on it.

Update 23 Apr 2013 11:42:40

Back up. Checking that services are running as expected.

Update 23 Apr 2013 11:46:59

All looks normal now.

[PEW] Broadband: BE PEW's 09/05/2013 (Open)

2013-04-30T11:10:56+01:00

Started: 2013-05-09 00:01:00

We have been notified of the below planned maintenance window affecting some O2 DSL connections:

Date - 09/05/2013

Exchanges affected:

Woodford (LNWFD)
Walthamstow (LNWTH)
Eastleigh (STEASTL)
Bishops Waltham (STBISHW)
air Oak (STFAIRO)
Woolston (STWLSTN)
Bursledon (STBURSN)
Fawly (STFAWLY)
Hamble (STHAMBL)
Hythe (STHYTHE)
Moorhill (STMRHLL)
Chandlers Ford (STCFORD)
Leagrave (SMLEA)
Flitwick (SMFK)
Dunstable (SMDB)
Houghton Regis (SMHUR)
Braintree (EABNT)
STRATFORD (LNSTF)
WANSTEAD (LNWSD)
WOOLWICH (LSWOO)
EAST GRINSTEAD (NDEGR)
COPTHORNE (NDCOP)
SHIRLEY (STSHRLY)
ROWNHAMS (STRWNMS)
HAMPTON (STHMPTN)
BASSETT (STBSETT)
HORSHAM CARFAX (SDHRSHM)
SOUTHWATER (SDSTHWT)
AYLESBURY (SMAY)
PRINCES RISBOROUGH (SMPRB)
WENDOVER (SMWE)
GREENWICH (LSGRNW)
TOTTON (STTOTTN)
ROMSEY (STROMSY)

[PEW] Broadband: BE PEW's 10/05/2013 (Open)

2013-04-30T09:52:55+01:00

Started: 2013-05-10 00:01:00

We have been notified of the below planned maintenance window affecting some O2 DSL connections:

Date - 10/05/2013

Exchanges affected:

SMBF - BEDFORD
SMAP - AMPTHILL
SMKT - KEMPSTON
SMSA - SANDY
SMBB - BANBURY
LWRAD - RADLETT
LWGAR - GARSTON
SMAM - AMERSHAM
SMGM - GREAT MISSENDEN
SMBA - BRADWELL ABBEY
LWBUS - BUSHLEY HEATH
SMBK - BERKHAMSTED
THCW - CROWTHORNE
THY - YATELEY
THRG - READING CENTRAL
LNSTB - ST ALBANS
THC - CAMBERLEY
THWM - WOKINGHAM

[PEW] Broadband: RADIUS work (Open)

2013-04-29T11:33:52+01:00

Started: 2013-04-29 18:00:00

We have had a few issues with RADIUS of late, and we have some changes in LNS code and RADIUS server code which should help to resolve these issues, and also resolve the cause of the last time we had an LNS crash.

We are planning to introduce the changes gradually during this week so that the impact of any problems will be reduced, and only effect some customers. We are not really expecting problems, just being cautious. Today we are testing changes on our test LNS which does have a handful of customers connected.

I do apologise in advance for any issues. Mostly RADIUS issues result in free (unmetered) usage, and in some cases extra PPP restarts.

In general we'll be making changes evening or early morning rather than during the day.

Update 29 Apr 2013 21:29:07

Thinks did not go quite as planned this evening when one of the test boxes upset RADIUS slightly and resulted in some PPP restarts. We think we have it sorted now so we can start to roll this out over the next few days.

[Minor] Broadband: Session restarts

2013-04-29T08:31:52+01:00

Started: 2013-04-29 08:20:00

Some customers will have had PPP restarts around 08:30 and possibly moving from the D to the C LNS.

Sorry for any inconvenience.

[Minor] Broadband: RADIUS issues

2013-04-28T18:09:43+01:00

Started: 2013-04-28 08:00:00

We seem to have an issue with RADIUS accounting this morning. We have restarted things and we are investigating the cause.

There have been some odd tweets and emails about the issue, but it lists the previous "AdminReset" from a previous LNS switch over from earlier in the week. This seems to be an odd side effect of the problem which we also need to investigate.

Update 28 Apr 2013 12:33:22

P.S. As usual, this also means some free usage, and will mean some PPP restarts later today and over night.

Update 28 Apr 2013 12:33:58

It also means the lines show "salmon pink" on the control pages. But the lines are working, of course.

Update 28 Apr 2013 16:34:57

We are manually clearing some of the remaining sessions for which there is no accounting. This means a brief PPP restart for some customers.

Update 28 Apr 2013 17:26:29

There is definitely something up with RADIUS which we are stil chasing.

Two thirds of lines are all sorted and accounting cleanly now.

One third is still being working on.

Resolution

We think it is all sorted for now

[PEW] Broadband: BE PEW 02/05/2013 (Open)

2013-04-26T16:43:42+01:00

Started: 2013-05-02 00:01:00

We have been notified of the below planned maintenance window affecting some DSL connections, the exchanges affected: are:

MITCHAM
ORPINGTON
PURLEY

[Info] General: Noise in the Office!

2013-04-26T09:40:38+01:00

Started: 2013-04-23 11:22:09

We apologise in advance to customers calling in and experiencing what sounds like loud drilling. We'll do our best to cope, but may need to call you back at times.

In other news, we'll having air-conditioning installed and working by the end of the week.

[Minor] General: Office Telephones Not Working

2013-04-26T09:40:11+01:00

Started: 2013-04-26 09:20:00

We have a problem with our office telephones that is being investigated at the moment. We do apologise.

Resolution

This has now been fixed, phones are rining again!