Internetworkers: http://www.ibiblio.org/internetworkers/

[Simon Spero <ses AT unc.edu>]

... where this happens to you? 
Nooooooooooooooooooooooooooooo!


http://www.theregister.co.uk/2004/11/26/dwp_network_outage/

DWP kills 60k+ PCs in Windows XP upgrade lash-up

By John Lettice

Published Friday 26th November 2004 09:30 GMT

Updated Most of the desktop computers in the UK's Department for Work and Pensions were paralysed for four days on Monday, when a failed upgrade took them offline. The outage, covering 75-80 per cent of the DWP's 80,000 PCs, is one of the largest in the UK Government's not entirely impressive IT history.

And possibly one of the most costly. According to staff reports the outage occurred on Monday afternoon, disconnecting staff email, benefits processing and Internet and intranet connectivity. According to one, a limited network upgrade from Windows 2000 to Windows XP was taking place, but instead of this taking place on only a small number of the target machines, all the clients connected to the network received a partial, but fatal, 'upgrade.'

Another source says that the DWP was trialing Windows XP on a small number ("about seven") of machines. "EDS were going to apply a patch to these, unfortunately the request was made to apply it live and it was rolled out across the estate, which hit around 80 per cent of the Win2k desktops. This patch caused the desktops to BSOD and made recovery rather tricky as they couldn't boot to pick any further patches or recalls. I gather that MS consultants have been flown in from the US to clear up the mess." EDS is also thought to be flying in fire brigades.

If these claims are true the DWP could face grave difficulties in rolling all of its machines back to their previous, working state. Staff from Microsoft and EDS are reported to have been working around the clock to dig the Department out of the pit, while speaking on the Today programme this morning a spokeswoman amusingly insisted that the Department's systems had not in fact fallen over. They were working, it was merely the case that "80 per cent of desktop computers are not connecting through to the mainframe systems."

So that's cleared that up then. She added that the emergency payments system was "working perfectly." The emergency system appears to have kicked in on Wednesday, and the Department was preparing a press release on the matter yesterday. No sign of it this morning, yet.

Reports coming in this morning however suggest that at least some of the DWP's systems are coming back online.