We'll be rebooting our core VoIP server this evening. It shouldn't be down long, although it'll break calls in progress when it happens.

We will try and make sure that there are no calls in progress when we do this, but if not possible we'll do it around 22:30.

This was completed successfully.

In light of last night's problems with VoIP, we have made changes to our SIP platform to make debugging easier in future.

The changes require a full restart to go live, so we will be carrying out this restart out of hours on Thursday evening at 23:00.

This affects all of our call routing too, so will drop calls made through the 'A' server and via IAX2 as well as SIP. Phones will need to re-register.

The outage is not expected to last longer than a couple of minutes, and if anything goes wrong we will roll back the changes.

Sorry for the short notice.

Restart is going to be a little later than anticipated. Sorry, won't be long.

All done. Calls testing and working.

Sorry for any disruption.

We have a list of minor niggles that has built up and we are planning to work on these this weekend. Most of the work will, of course, be on out test platform, but as we complete each stage we will be loading on the live VoIP system for final customer testing and confirmation.

This should not cause any problems, obviously. But there is a risk of issues arrising.

In general, the way the new VoIP system works, if there are problems then calls in progress continue but don't get charged.

Part of this work is testing IPv6 VoIP phones.

Work has been done today on the test and backup server. We plan to apply these changes to the live server this evening.

The main changes have been down to slight quirks in the operation. e.g. voicemail not working unless call recoridng email also set; calls from some types of gigaset not gving right ring tone; Quirks when two simultaneous voicemails apply on one call; and such things.

Tomorrows work is going to be related to IPv6 interworking.

OK minor change to the syslog caller display format.

We were unabkle to complete the IPv6. We will be testing some IPv6 work on the test server during the week.

We have had a few snags with recordings and voicemails over the last couple of days as reported. Mostly in delayed sending of recordings, but in some cases calls affected or recordings not sent at all.

We can't reproduce this on the test system, and it looks to be related to load in some way.

We have a few changes to deploy this evening, which may mean some calls in progress being cleared. The first of the changes to be made is to find why we seem to be unable to load a new version without killing the call recordings in progress, as that is something that used to work! Once that is fixed things should not be disruptive.

Sorry for the inconvenience, but I am sure everyone appreciates we need the system working flawlessly.

We have managed to fix the graceful restart issue, allowing fixed to be applied without killing calls. We have also manged to find a major cause of the delays and problems we have had.

Apologies for a couple of calls aborted later in the evening.

As we get closer to handling mobile services we will be adding new features to our call server. These will be added step by step and normally out of hours.

We have already added code to route inbound calls to the mobiles. We have to handle outbound calls. Calls to/from customer call servers and mobiles. Texts in and out and to/from customer text gateways. So quite a bit to do.

The upgrades should not cause problems with active calls. There is always a risk. The most likely risk is calls failing to be made for a few seconds, but this is not that likely as the upgrades are designed to be applied on the fly with calls in progress.

We are doing more work integrating the existing and new call server over the weekend. It should not affect calls. We hope to switch some of the incoming call handling to the new server. Please get me on irc if any issues.

We are continuing the work integrating the new call server.

The existing server has had minor changes to work with a new database. This should cause no problems, but if you have any issues please contact support.

We have switched the code back due to issues calling our support number.

Looks like it was actually only calls to us that were affected!

We have gone back to new databases now and monitoring closely.

We have posted several reports on the new call server and the work we are doing - but I am not sure it has been as clear as it could be.

The plan is that we will switch all calls, in and out, to the new call server. This will take time. Sme changes should be simply a matter of changes to IP addresses our end and it just work. Some people will need individual config changes. The change last night affected DNS and IPs and should have been seamless.

Some things will remain on the old call server for a while, such as IAX handling, voicemail, and IVR. However, in time even these will be moved or retired.

We are moving away from IAX and concentrating on SIP. Staff are able to advise on asterisk config changes for this individually with the few customers using IAX. We'll contact each customer for this.

Correctly set up SIP phones will have no problem with this and will need no changes on your part. We have over 65 different makes and versions of SIP equipment working fine on the new call server now.

While we are doing the work customers can register a SIP phone on the old or new server and it should just work.

We'll post details of more specific work as it happens. Sorry for any concern caused.

The call server can handle calls that come in from untrusted sources, i.e. VoIP from the internet, and as such we have to date sent the CLI as a ? (question mark) in front of what was sent.

However, we are hitting some snags with this and the old call server and some customers, possibly because (unescaped) ? is a URI delimiter in SIP or maybe other reasons.

We are planning a change over the weekend to change the logic so untrusted CLIs simply get a # prefix. # is logically a "number" and this should also translate to mobile calls which could not handle the ? anyway.

Sorry for short notice, but we know that most people are not affected by this change anyway. We don't expect to drop any calls.

We are seeing an issue with one carrier where some incoming calls placed on hold are not coming off hold, and this is affecting call transfer. The carrier insists they have not changed anything, so we are planning to try and resolve the issue this evening. We have managed to pin down the exact circumstances now. We don't expect much disruption, but there is a risk of call set-up delays at some point during the work when we update the live call server.

All looking well now

We are seeing some issues with call transfers. It all seems related to the issues today. The plan is to try and get to the bottom of them outside office hours (this evening). We have a second call server we can carry out some of the testing one as well. There may be some disruption of calls this evening. As usual, calls in progress are likely to be unaffected, and if we do make changes then they will not be charged.

Today has been fun(!) - so techie details to follow...

1. Some minor changes at the weekend to make us actually follow the SIP spec more closely, at the request of a carrier, were nearly right, and caused the other carrier to break, and we worked around that. Fixing the changes broke things horribly, and we have backed them out as the first carrier has worked around anyway and the changes were optional parts of the spec anyway. Arrrg

2. We found that if for any reason things did in fact crash, the system to automatically restart was slightly flawed. It is a long standing tool hat handles "respawn" of applications fgor any reason that they stop, with suitable logging and delays to avoid problems, but it had issues which were only ow apparent. The app is like 5 years old or more, and only now needed a tweak. Changes made mean if the call server does crash it recovers by itself and quickly now.

3. We have discovered an interesting quirk in the call server which means that certain messages for an ongoing call (putting it on hold) when the call cannot be found, and when the call was via a specific carrier, caused a crash. The issue is that if we restart or there is a crash, all in-progress calls are lost, so the chance of such an in-call message where we cannot find the call is much higher causing another crash soon afterwards... Chain reaction. All fixed now.

So, cannot be sure all is well, and it is crazy we have found several cascading issues all at once like this, but it is looking good and we are keeping a close eye on things.

We will be doing more testing on the new call servers during this week.

We hope to be able to move over to the new call server pretty seamlessly, probably early one morning this week when we are happy that we have the new servers behaving exactly as we want them.

There is no change to configuration on customer equipment or IP addresses. This is simply a move to new hardware and setting up fallback hardware.

The legacy call server ("A" server, running asterisk) is not being touched.

This was done as an emergency this evening (Tuesday)

Planned for: Sunday 15th August

To increase resilience and reliability of our VoIP service we will be moving C.Speechless on to new hardware.

The changeover should happen very quickly. We expect there to be just a few minutes where calls will not be able to be placed.

We'll update this post as the work is carried out

Further information: We have 2 VoIP servers, the A server and the C server. The C server is our new server. It's the C server that we will be moving. The new hardware is a pair of servers to enable fall-back etc.

This may be delayed until late in the day.

This may be in the morning, apologies for delay.

We'll probably be doing this some time Monday evening - it should be pretty seemless. Sorry for the delay

We are rescheduling this.

Last weekend we changed the last of the incoming call routes to go via the new call server. This had very few side effects but sadly a couple of customers needed to make slight config changes for the change of IP address.

This weekend we plan to change outgoing calls for people still using the old call server. This is mostly people using NAT or IAX. The change should really be seamless as the call still goes to the old call server. It is what happens once it gets there that we are changing - the call will be directed to the new call server.

Once completed it means all calls will use the new call server for all call routing decisions. The opens the way for newer features, and we hope to launch the new control pages for numbering soon.

The change also means that all call records and billing will be on the new server. As the bills are in arrears, the next bills will have old and new billing (separate pages) but from the bill after that we will only have new billing format.

The change also affects voicemail and call recordings which will use the new call server. The email format is slightly different on the new call server. Voicemails start after the outgoing message rather than including it, and are mono not stereo.

We are planning to retain the old call server for the time being to handle NAT and IAX, but both will be deprecated for any new customers. Eventually we may create special modules for SIP NAT and IAX handling rather than continuing to use asterisk, but this is much longer term.

We also have two new machines for the new call server which will be deployed soon. The change over should be seamless and the IP address for the new server is not changing. This will mean the new server has the power backup that it needs and hot standby.

Any problems with these changes, please do contact support.

This work needs a bit more of a re-think to allow for some complex cases (customers with blocks of numbers and calling line identify control within that block). We expect to do more testing during the week. Sorry for the delay.

We are planning to do this work next weekend.

Changes done and tested - any issues please contact support.

We have done a lot of work investigating NAT and SIP.

We are maintaining some customers on the old server for now as the issues with NAT are just horrendous. I'll probably do a web page on the subject some time. It is mental.

However, we are progressing some of the back-end call routing which means some small changes. These changes were done for one carrier in one go some weeks ago with no apparent effects. We are now working on the other carrier and plan to make changes at the weekend. Given the lack of apparent issues with the first carrier, we do not expect any issues with the second carrier.

This will mean call bills will move to the new format for incoming calls. Which will help make the bills more consistent.

There will be more steps in future.

Not sure if we will be doing this today or not.

The work was completed later than planned.

There was also a slightly un-planned 18 second outage around 15:30 today where calls would not be established but existing (non recorded) calls would have been fine. Apologies for that.

The new call server is having voicemail added so that we are no longer using asterisk for voicemail. This means the voicemail emails will have a different subject line. The recording itself will be mono, not stereo, and will not record the initial outgoing message.

We plan to switch to this some time this week. It may result in slight issues with calls that somehow come in via the old call server where the old voicemail is still in place. We are trying to work out a neat solution for this before the switch over.

I'll post a follow up when we think we have the best solution to ensure all voicemail is on the new call server.

The plan is that any voicemail access (leaving message or changing outgoing message) will be directed to the new server. This will allow one store of outgoing messages regardless of which server you are on.

We are looking at making the change some time this evening (Monday).

Seems to have all gone to plan.

Any comments on new voicemail, please let us know.

The subject on the email now says Voicemail...

We are making some slight config changes this evening which should not have any impact on calls in progress. This is to ensure proper handling of pre-answer audio from one of our carriers.

We are making minor change to the config on the new call server thsi evening. This is largely to make it more firewall friendly for customers. SIp traffic will still come from machines in 81.187.30.110-119 but should come from UDP port 5060 in all cases following the change.

Minor glitch - not quite to plan...

Hmmm, unexpected glitch there - being addressed.

Calls were not makable for a few seconds then - and recorded calls still playing up.

Still issues with recorded calls.

Sorry about the delay.

All is working, but not 100% sure we are handling IPv6 at present.

Sadly some of the work as been reversed as the SNOM's get upset if the calls come from a different IP. You can't win...

Anyway, all is well.

We will be carrying out some maintenace work on our VoIP server a.speechless tomorrow morning.

This is important maintenance, so sorry for the short notice and sorry for any inconvenience this may cause.

We plan to start this work at 07:30, and do not expect it to take more than a few minutes.

This work has now been completed.

If you have any questions, please contact support.