Flooding at Telewest's super-headend site in Knowsley was the main cause of a massive television and Internet service outage that started on New Year's Eve and went through to early Friday.
In a memo to the company's customers posted on the company's Blueyonder website, Telewest's Group Ops Director for Networks, Technology and IT, Colin Whitbread, explained:
"On New Year’s Eve there was severe service disruption to a number of our
services including the loss of TV services nationally.
"We would like to apologise for the inconvenience this caused our
customers, and thank them for the patience and understanding they gave
our staff whilst we worked to restore services.
"A full investigation is taking place. In the meantime, I wanted to give
further details of how the issue was managed.
"At approx. 22:00 on New Year’s Eve, severe weather caused flooding in
our Digital Control Centre in Knowsley, this in turn triggered automatic
alarms to the fire brigade.
"The fire brigade arrived on site by 22:10, and ordered a complete
evacuation of the site and the immediate shutdown of all electrical
power and water supplies. This was done to ensure the safety of staff
and fire personnel dealing with the issue.
"The removal of power at the site disrupted digital TV and some analogue
TV services nationwide. Additionally, Internet services (broadband and
dialup) and telephony in the Liverpool region were also disrupted, as
was access to Internet services dependent on servers in the Centre; such
as e-mail, PWP, and the blueyonder and Telewest websites, including the
status page.
"Due to the loss of access, we regret that our technical support team
were unable to update the status page, mailing lists or the announce
newsgroups. Given the scale of disruption the number of calls coming in
to our technical support teams caused congestion problems, which
prevented some customers getting through.
"By 22:30, after a full assessment of the situation by senior management,
a decision was made to invoke our contingency plan. This involved
routing as many services as possible to a designated recovery centre at
Hayes.
"All on call engineers nationwide were en route by 23:00, despite being
hampered by traffic congestion due to New Year’s celebrations.
"By 23:30 the fire service had contained the flood, and some power was
allowed to be restored from back-up systems. By 01:00 on New Year’s Day,
we restored e-mail and PWP services, but were still unable to restore
full power due to health & safety and power equipment integrity
considerations.
"Re-routing of traffic from Hayes was completed at 04:00, resulting in
the restoration of the majority of TV services nationally, although in
some areas customers we experiencing some residual freezing pictures issues.
"At 06:00 we were allowed to restore all power in Knowsley by the fire
brigade, and we subsequently re-routed all traffic from Hayes back to
Knowsley. This restored all services except in the Liverpool area. By
08:00, all services were fully restored nationally.
"Again, please accept our sincerest apologies for any inconvenience this
incident has caused you and please be assured all involved in Telewest
concentrated efforts to minimise the disruption caused by this serious
problem."
In a memo to the company's customers posted on the company's Blueyonder website, Telewest's Group Ops Director for Networks, Technology and IT, Colin Whitbread, explained:
"On New Year’s Eve there was severe service disruption to a number of our
services including the loss of TV services nationally.
"We would like to apologise for the inconvenience this caused our
customers, and thank them for the patience and understanding they gave
our staff whilst we worked to restore services.
"A full investigation is taking place. In the meantime, I wanted to give
further details of how the issue was managed.
"At approx. 22:00 on New Year’s Eve, severe weather caused flooding in
our Digital Control Centre in Knowsley, this in turn triggered automatic
alarms to the fire brigade.
"The fire brigade arrived on site by 22:10, and ordered a complete
evacuation of the site and the immediate shutdown of all electrical
power and water supplies. This was done to ensure the safety of staff
and fire personnel dealing with the issue.
"The removal of power at the site disrupted digital TV and some analogue
TV services nationwide. Additionally, Internet services (broadband and
dialup) and telephony in the Liverpool region were also disrupted, as
was access to Internet services dependent on servers in the Centre; such
as e-mail, PWP, and the blueyonder and Telewest websites, including the
status page.
"Due to the loss of access, we regret that our technical support team
were unable to update the status page, mailing lists or the announce
newsgroups. Given the scale of disruption the number of calls coming in
to our technical support teams caused congestion problems, which
prevented some customers getting through.
"By 22:30, after a full assessment of the situation by senior management,
a decision was made to invoke our contingency plan. This involved
routing as many services as possible to a designated recovery centre at
Hayes.
"All on call engineers nationwide were en route by 23:00, despite being
hampered by traffic congestion due to New Year’s celebrations.
"By 23:30 the fire service had contained the flood, and some power was
allowed to be restored from back-up systems. By 01:00 on New Year’s Day,
we restored e-mail and PWP services, but were still unable to restore
full power due to health & safety and power equipment integrity
considerations.
"Re-routing of traffic from Hayes was completed at 04:00, resulting in
the restoration of the majority of TV services nationally, although in
some areas customers we experiencing some residual freezing pictures issues.
"At 06:00 we were allowed to restore all power in Knowsley by the fire
brigade, and we subsequently re-routed all traffic from Hayes back to
Knowsley. This restored all services except in the Liverpool area. By
08:00, all services were fully restored nationally.
"Again, please accept our sincerest apologies for any inconvenience this
incident has caused you and please be assured all involved in Telewest
concentrated efforts to minimise the disruption caused by this serious
problem."
