IT: That Cyber Attack at the NHS

That Cyber Attack

Follow me here on Twitter 

This may come across as a bit of a rant but this is a serious issue regarding IT in general not just the NHS but also Education and after talking to other IT professionals it’s a real problem.

 

As an IT worker in education we also get hit with these kind of issues, when it fails it’s end of the world when it’s working we often get asked “What do we do all day?”.  The later part is important because we are planning ahead, trying to upgrade equipment that you don’t see and generally working on development.  All under the radar.  No one sees it.

 

I remember moving onwards from Windows XP to Windows 7.  The first thing we did was create an imaging system which allowed us to mass upgrade a few hundred machines in a week.  Instead of manually building one at a time and installing every piece of software (average of 50) by hand taking 10 hours.  That system would take some months to develop, test and make sure it was perfect.  So once that was completed we would need to find the time to upgrade those hundreds of machines.  The good thing is doing student machines is easy during half terms.  It’s not often students are on site so we can easily run that system and walk away in each IT suite.

 

Staff PCs are a very different matter.. 

 

Remember those PCs are in use Monday to Friday 8-5 which is exactly our working hours (can’t be done remotely either because we didn’t have the capability).  We don’t work weekends and most public services will not pay IT over time (unless it was crucial like this Cyber Attack – if it was us in Education we would make up the hours – NOT paid overtime).  We also have to take in to consideration how many people do not follow standard procedures and save correctly.  I can’t tell you how many times I’ve argued with senior leaders in Schools over saving on their network area and NOT the local computer.  What would happen if their PC is infected, dies or a fire occurs – it’s gone.  So we have to manually take that data to avoid being shouted at – delaying the job.  The member of staff would have freed up a single lesson for us which would not be long enough.  We warn them, scream and beg for them not to save locally but they continue to do so.  When senior leaders are not telling them to do so – instead it’s our fault if it fails and they lose the data….

 

So again we tend to wait until half terms.. which the NHS does not have – how lucky are we?  There are times though that staff are on site during the holidays which means we have to work around them or we get another argument with that staff member.   What if they have staff laptops? Which again over complicates the issue because we need it with us when the staff member insists they can’t survive without it.

 

I’ve often argued with staff who have had their laptops stolen, dropped and severely smashed (they denied the later one too till we spot the damage).  It’s our fault that their data is not saved on the N Drive.  One particular member of staff had it happen 3 times!  Till the Head Teacher had to have words.  That’s what we’re dealing with every day.  I’ve often had arguments about staff not saving to the N Drive – any excuse they can use they will use it.

 

This sums up one of the many reasons why this Cyber Attack has occurred – no one listens.  Why listen to IT? We know nothing.  We don’t do our best to predict the worst outcomes for IT, we don’t worry about backups and servers that die.  The only time YOU worry is when something DOES happen.

 

No one Listens to us and everyone knows better

The sad thing is when it comes to simple tasks people want us holding their hands.  When it comes to the bigger projects, the bigger fish and generally the top end of tasks – everyone ignores the IT experts.  You only have to look at twitter regarding sales reps, training seminars and other professionals to see this.  I’ve often covered these in my blog posts.  They hate us because we do know better.  Who knows a car best? The mechanic and engineer not the racing driver.  Who knows a ship (boat) better? It’s not the captain, it’s not the sailor but it’s the chief engineer.   Update:  As I was typing this I already had an IT consultant try and tell me it’s easy to make everything up-to-date completely forgetting the points I’m making.  We don’t control what budgets we get, we don’t control the amount of staff we have and we don’t control the hours we are able to do any type of maintenance.  I even had someone compare painting walls and furniture – you mean when that entire room is closed and not even IT are allowed in? That if those contractors are given 5 days to complete their work they will have it done 1 day early so IT can perform their task?  Again it shows that severe lack of understanding.  We’re IT not site managers or responsible for buildings.  Whoever planned for those contractors will not even tell IT that it’s happening even after we constantly whine about not knowing….

 

This is quite standard in our profession.

 

We can be quite annoying but we are because:

• We like to anticipate the worst outcomes, the what if scenarios and end of the world
• We like backups, to cover the system and generally make sure it’s working 99.9999% of the time
• We like official procedures and doing things properly
• We try our best to predict your needs before you even know them
• We are held back by time, money and staffing levels

 

The NHS Suffered heavy IT Cuts

Even if the Government gave the NHS more money I’d bet my body parts that we would still be in this situation.  Why spend money on IT when it’s currently working?

• Employ more nurses
• Employ more doctors
• Build more rooms
• Purchase more beds
• Hire more cleaners and caretakers
• Hire contractors to re-do rooms

 

There’s a million things the NHS would rather spend money on instead of worrying about an IT system which before the Cyber Attack was working fine.  Everyone does this.  In education IT is always one of the first areas to be hit with budget cuts and potential staff redundancies.  It’s also why Education goes through this nightmare of good periods to terrible periods – rarely in the middle.  You will have 3 years of good budgets, good staffing levels to revert back to poor budgets and too few staff.  It’s sad but it’s true.

 

“Patches can be deployed by SCCM” – something the IT Consultant mentioned to me.  Which costs money, which takes time to develop (anyone that knows SCCM will understand it’s fantastic but not a five minute installation).  Patching a PC also requires that machine to be on at a set time (we actually use WSUS here – something the consultant didn’t mention).  BUT – the machines must be on but when they automatically turn off late at night (we don’t use wakeup LAN – it’s in progress) it’s another milestone.  Systems like these take an age but what if you don’t have the staff? And systems like WSUS can’t be setup by a simple technician who knows how to trouble shoot.

 

We are told those PCs must not be impacted during working hours so we’ve got a period of 2 hours where those updates can deploy.  Updates are also not 100% reliable, we can at times encounter one update has killed a PC with the popular blue screen of death (although it’s no longer a BSD).  We also find that each month we don’t get 1 patch, it’s more like around 10 which must be deployed to almost 1000 PCs.  At the time when we was heavily removing XP from our site, it was a nightmare.  Fighting staff who don’t like change (we don’t make policy) fighting staff who don’t want you to interfere with their work hours (our working hours are the exact same during term time) and finally half terms are the only times we can do such large scale work.  Even those half terms get affected by students/staff on site – which we may not fully know of till last minute.  We also have the occasional building work meaning we can’t even access certain areas of the School or have power outages due to cabling works…  We currently have an entire building undergoing heavy amount of work.  Those contractors know they got till September to get their job done but what about the rest of us? Do we have to wait till staff return to actually get in to the rooms to setup IT? We’re talking dozens of PCs, boards and projectors in the space of a day.  It’s only through the sheer luck that we work well with the site staff who will demand those contractors are completed 1 week ahead of schedule so we can do our job.

 

This is what it’s like.  You are faced with the staff workers, the senior staff and contractors.  While also balancing your own holiday periods (my staff are only allowed time off during half terms).  With a team of 3 this entire thing becomes a balancing act.  We still get it done though.  IT consultants they don’t see that.  They see cooperative staff who listen to IT.  A bit like parents who see their little darling angels going to School to be nice to the teachers yet most teachers will say they are not angels and the parents don’t believe it because they don’t see it.

 

A few years ago this wouldn’t be the case and I can’t stress enough about the potential risk this School would be at if we didn’t have several changes occur.  Now this is a perfect example of what stops IT:

Lack of Time – Through endless amount of fire fighting (please see other posts about BYOD).  If there’s no time for anything to be developed what can we do?

Lack of Budget No server expenditure, no networking improvements and general IT aged to 10+ years old – To develop things like SCCM you need a  £ S E R V E R £ – usually Virtualised

Lack of Staff (2 and a bit man power employed for a very large system with heavy IT reliance)

Doing other tasks – Something especially outside of education no one understands. 

 

We’ve been expected to be:

Administration workers (printing, creating documents, dealing with letters)

Teaching assistants (supporting classrooms with general student support – not IT faults)

Media and film technician (supporting media, sound, creativity and film)

Second main reception (we’ve had a lot of parents calling us up directly)

And being teachers by showing students how to do tasks. 

 

All the above is why at times IT have issues.  It’s like saying it’s easy for a mechanic to maintain 100  cars a week while he deals with being a receptionist, finance officer and dealing with everyone’s complaints and phone conversations.  It all adds up.  Anyone that says otherwise has led a very much easy job with very little pressure  - I’ve often fallen out with IT people because they don’t realise how it is at other places.  I count myself very lucky to be where we are now at my work place.

 

Interesting thing is one of the very people responsible for my team doing the above is one of those who would complain if IT failed in any single way.  At the time we was not spending any time (not even 1 hour a week) developing.

 

Now where are we?

My team perform IT work around 95% of the year – it’s still not perfect but better.

Our servers are reliable to the point they work 98% of the year – still not perfect but we’re getting there.

Our network is 96% 1GB capacity but again still not perfect.

Our average age of equipment is now 6 years old with future ideas to help cut down costs – cost cutting still exists!

 

That’s on top of having a budget cut and staff cut.  What do we do though?  The HT has to make cost savings site wide.  Sooner or later I can see us losing another team member due to the Government costs.

 

We’ve got too many Enemies

Again it’s sad but true.  The SLT will often look at us with £££ on our heads, a resource that they can survive without.  You also have the Heads of Music and ICT who will hate us because we hold them back in their eyes.  We stop them from having what they want over what’s best for the School.  We stop them from having fancy gizmos and gadgets because we would rather spend money upgrading Windows XP to Windows 7.  We would rather spend the money replacing 10 year old equipment instead of that fancy pretty toy.  The Governors will often never see us, the kids only look at us like a stranger to reset a password and the staff… they see us as an annoyance.

 

How many will be expected to turn up Monday morning and want everything to be fixed?  You know by those staff working 9-5 mon- fri? The ones who shouldn’t be working weekends or are being paid peanuts to do extremely high pressured work?

 

You’ve also got to consider massive amount of changes that take endless amount of hours to resolve.  How many Schools go down the route of outsourcing IT, redundancies and replacement staff?  All of that takes time to resolve.  It’s like you’re trying to do your job while on a train – you’ve got no choice but to ride it out.

 

We’ve actually gone through BYOD schemes that take a lot of work out of our week and every year it’s changed in some way – taking more time.  There’s no consistency in IT especially in education.  What do people need the most? Consistency.  When you consider how much IT itself changes when other things change you’ve turned a small task to a bigger one.  The BYOD scheme occurred completely outside of the advice from us in IT.  The mess it caused was unbelievable but who cleans up that mess? We do.

 

We are NOT perfect

I once worked at a School that got heavily infected over a decade ago.  I warned those above that it could happen because we didn’t update and generally had the arrogance that it “won’t hit us”….

 

It did..  luckily enough our servers were protected because I took it out of my own hours to update them.  Our best IT suites were left untouched.  The current person in charge of IT got slaughtered by SLT (rightly so) because he only cared about his own wage package and ambition.  The sad part is.. they actually listened to him quite a lot before that day.  He knew best because he was the socialite.  He always fixed those who had the biggest voices in the School while neglecting everyone else.  Under the radar I was.

 

Then you have to think about who Schools employ – usually fresh ex-students.  They can’t manage servers, they can’t manage systems, they can’t manage budgets and they certainly can’t plan ahead.  What they can do is the simple tasks that anyone can do – reset passwords.  Not nice but it’s true.  Why do they employ ex-students? Money saving and an arrogance in to thinking IT is easy when you know the basics.  Just because someone can change a car tyre it doesn’t mean they can MOT, Service and perform maintenance on a car.  Let’s face it to fully hire a decent IT staff level you’re talking at least £80,000 when you include pensions – that’s 2-3 teachers.

 

Centralised IT

This is the biggest downside about centralising IT, resorting to agency workers and outsourced IT support.  You place all your eggs in one basket and when that has problems – you lose everything.  Centralised IT means you have more servers, less technicians and less expenditure on every day equipment.  All about one thing – cost cutting – certainly nothing more and nothing less.

 

Schools are heading down this route with MATs (Multi Academy Trusts) all for one thing – cost cutting.  They will want to link up their IT systems across the Schools, remove the need to replace equipment and cut down the staff; all for one thing – cost cutting.  They will spend money on Virtualised Servers which are fantastic but it puts them all under a single roof.

 

We will be back here again one day like everyone.  Every few years people dump a lot of money on IT, cut it all back and then revert back to day one.  The vicious IT circle I call it.  The weird thing is the very people complaining about this could be the very people who treat IT people poorly.  Whom often ask – what do they do all day.  IT is like a child where you constantly need to be vigilant.

 

I’m not surprised at all at what’s happened in the NHS and globally.  Schools often go down BYOD/Mobile technology routes because it’s pretty instead of spending money upgrading IT.  They often purchase Macs because they look pretty and you have to remember if the world was Mac – they’d be hit.  I’ve seen an infected Mac but why create virus’s and malware for systems that are rare.  Even Smartphones are being targeted far more than Macs because everyone has a Smartphone.

 

Why is IT expensive? It doesn’t have to be but you get too many fish ponds involved.  You will have the main boss, who asks a senior manager and that person talks to the consultant who then reverts to the actual IT Manager.  It takes far too many steps to get to the truth.  To the person who really is clued up.

 

While typing this blog I’ve already argued with a few who are completely ignoring these points about budget cuts.  It impacts an incredible amount.  Everyone is more bothered about throwing complaints at Hunt who let’s face it won’t understand anything or even care.  No managers at the top really deal with IT unless it’s their own iPad.  They will pass it on to someone else and down the chain.  This will all boil down to the fact that NHS leaders decided to cut back IT like most public services do.

 

Schools are not lucky – Thank the IT Staff who tend to ignore those above

Not many Schools have been hit by this mess.  It’s not thanks to good budgets or good leadership of the Schools – in fact it’s down to IT.  Those who work extra hours for no pay, those who perform remote work from home all because they care.  Many of us see the IT system as our family member.  We want the best of that IT system, we want everything to work regardless.  I can’t stress the amount of unpaid hours I’ve worked in the last 12 months let alone 4 years or almost two decades of working.  All because I care.  We’ve often seen malware infections but only on a per PC basis caused by a single staff member (had three PCs in 4 years).  Staff training really means nothing, you’re fighting staff who really don’t think about what they click on.  They save work from 10 years ago and copy to keep it.  Another 10 years they’ve built up work they didn’t know they have.

 

The thing is none of these ‘attacks’ are new.  Malware has been around for many years I think my first case of malware was around 10 years ago.  These attacks can impact education and who will be the ones blamed – IT.  While those at the top who have made the cutbacks and held back IT will be blameless.  Not many Schools can afford qualified and experienced staff – most leave education to go to business.  Many of us can earn £10K more outside of education but we enjoy the challenge it brings (plus the not so stressful half terms with not many on site).

 

So before you moan about people at the top think about those IT who actually have made big differences for peanuts (also earning less than those working in the NHS IT departments).

 

How many of us have developed systems outside of normal hours because we’re firefighting all day.  Spending our days resetting passwords, in classrooms fixing PCs and projectors while attempting to keep old outdated servers running.

 

At the time I returned to work on Monday I had already gotten over a dozen emails of staff who attempted to advise or ask are we okay.  My response to everyone was we’re safe (touch wood).  We removed XP, we keep our system updated and I made it very clear that this was the result of all the hard work we had done.

Remember us.

 

The rant

Do apologise for this seeming like a huge rant but I’ve personally been on the end of something very similar.  You are being held back by budget, time and staffing yet it’s still your fault.  You have everyone phoning you up while your still trying to fix the problem – each interruption delays you fixing it.  You have everyone telling you how to do it better; when?  Yes set it to go automatic even though you’ve not got the time or money to get that system setup in the first place.  A good little example – while I type this I’m remotely installing a piece of software.  In two whole days this is the only 20 minute window I have.  The member of staff does the same hours as me and this can’t be done remotely because of the type of software it is.  Any issues with this – it’s scrapped and I have to wait another three whole days for a chance to do it again.

 

I even remember having to stay back several evenings with a severe server failure.  One night I left on time to remotely carry on to have one member staff question to someone else why am I not staying till late to get it working.  This is the state of people that don’t fully understand IT and working with day to day people (especially large amounts with SLT who don’t want downtime).  If you tell me Downtime must happen regardless – you’ve not working in Education.

 

We’re going to be installing SSDs to every PC next year – can this be done automatically? No.  Each one has to be done by hand.  We also don’t have the means to build up those SSDs beforehand (we don’t even have them yet till budget renewal in September – meaning can’t do it during the Summer break).  This will not only cause downtime of the staff PCs but require manual installation of SIMs (won’t image) and the staff member to save all data to their network drive (not going to happen no matter how much I beg/scream).

 

People have little to no idea what it’s really like to work in an environment held back by so much, while everyone expects the world.  No one cares till it breaks, when it’s working people think we’re playing games.

 

You look at this cyber-attack, it was really caused by some one click a link on an email after already being warned not to.  That person will not be in trouble, will not get a talking to or anything.  Yet no one cares about this no one cares that this was truly caused by the everyday user.