That Cyber Attack
This may come across as a bit of a rant but this is a
serious issue regarding IT in general not just the NHS but also Education and
after talking to other IT professionals it’s a real problem.
As an IT worker in education we also get hit with these kind
of issues, when it fails it’s end of the world when it’s working we often get
asked “What do we do all day?”. The
later part is important because we are planning ahead, trying to upgrade
equipment that you don’t see and generally working on development. All under the radar. No one sees it.
I remember moving onwards from Windows XP to Windows 7. The first thing we did was create an imaging
system which allowed us to mass upgrade a few hundred machines in a week. Instead of manually building one at a time
and installing every piece of software (average of 50) by hand taking 10
hours. That system would take some
months to develop, test and make sure it was perfect. So once that was completed we would need to
find the time to upgrade those hundreds of machines. The good thing is doing student machines is
easy during half terms. It’s not often
students are on site so we can easily run that system and walk away in each IT
suite.
Staff PCs are a very
different matter..
Remember those PCs are in use Monday to Friday 8-5 which is
exactly our working hours (can’t be done remotely either because we didn’t have
the capability). We don’t work weekends
and most public services will not pay IT over time (unless it was crucial like
this Cyber Attack – if it was us in Education we would make up the hours – NOT
paid overtime). We also have to take in
to consideration how many people do not follow standard procedures and save
correctly. I can’t tell you how many
times I’ve argued with senior leaders in Schools over saving on their network
area and NOT the local computer. What
would happen if their PC is infected, dies or a fire occurs – it’s gone. So we have to manually take that data to
avoid being shouted at – delaying the job.
The member of staff would have freed up a single lesson for us which
would not be long enough. We warn them,
scream and beg for them not to save locally but they continue to do so. When senior leaders are not telling them to
do so – instead it’s our fault if it fails and they lose the data….
So again we tend to wait until half terms.. which the NHS
does not have – how lucky are we? There
are times though that staff are on site during the holidays which means we have
to work around them or we get another argument with that staff member. What if they have staff laptops? Which again
over complicates the issue because we need it with us when the staff member
insists they can’t survive without it.
I’ve often argued with staff who have had their laptops
stolen, dropped and severely smashed (they denied the later one too till we
spot the damage). It’s our fault that
their data is not saved on the N Drive.
One particular member of staff had it happen 3 times! Till the Head Teacher had to have words. That’s what we’re dealing with every day. I’ve often had arguments about staff not
saving to the N Drive – any excuse they can use they will use it.
This sums up one of the many reasons why this Cyber Attack
has occurred – no one listens. Why
listen to IT? We know nothing. We don’t
do our best to predict the worst outcomes for IT, we don’t worry about backups
and servers that die. The only time YOU
worry is when something DOES happen.
No one Listens to us
and everyone knows better
The sad thing is when it comes to simple tasks people want
us holding their hands. When it comes to
the bigger projects, the bigger fish and generally the top end of tasks –
everyone ignores the IT experts. You
only have to look at twitter regarding sales reps, training seminars and other
professionals to see this. I’ve often
covered these in my blog posts. They
hate us because we do know better. Who
knows a car best? The mechanic and engineer not the racing driver. Who knows a ship (boat) better? It’s not the
captain, it’s not the sailor but it’s the chief engineer. Update:
As I was typing this I already had an IT consultant try and tell me it’s
easy to make everything up-to-date completely forgetting the points I’m making. We don’t control what budgets we get, we
don’t control the amount of staff we have and we don’t control the hours we are
able to do any type of maintenance. I
even had someone compare painting walls and furniture – you mean when that
entire room is closed and not even IT are allowed in? That if those contractors
are given 5 days to complete their work they will have it done 1 day early so
IT can perform their task? Again it
shows that severe lack of understanding.
We’re IT not site managers or responsible for buildings. Whoever planned for those contractors will
not even tell IT that it’s happening even after we constantly whine about not
knowing….
This is quite
standard in our profession.
We can be quite annoying but we are because:
We like to anticipate the worst outcomes, the
what if scenarios and end of the world
We like backups, to cover the system and
generally make sure it’s working 99.9999% of the time
We like official procedures and doing things
properly
We try our best to predict your needs before you
even know them
We are held back by time, money and staffing
levels
The NHS Suffered
heavy IT Cuts
Even if the Government gave the NHS more money I’d bet my
body parts that we would still be in this situation. Why spend money on IT when it’s currently
working?
There’s a million things the NHS would rather spend money on
instead of worrying about an IT system which before the Cyber Attack was
working fine. Everyone does this. In education IT is always one of the first
areas to be hit with budget cuts and potential staff redundancies. It’s also why Education goes through this
nightmare of good periods to terrible periods – rarely in the middle. You will have 3 years of good budgets, good
staffing levels to revert back to poor budgets and too few staff. It’s sad but it’s true.
“Patches can be deployed by SCCM” – something the IT
Consultant mentioned to me. Which costs
money, which takes time to develop (anyone that knows SCCM will understand it’s
fantastic but not a five minute installation).
Patching a PC also requires that machine to be on at a set time (we
actually use WSUS here – something the consultant didn’t mention). BUT – the machines must be on but when they
automatically turn off late at night (we don’t use wakeup LAN – it’s in
progress) it’s another milestone.
Systems like these take an age but what if you don’t have the staff? And
systems like WSUS can’t be setup by a simple technician who knows how to
trouble shoot.
We are told those PCs must not be impacted during working
hours so we’ve got a period of 2 hours where those updates can deploy. Updates are also not 100% reliable, we can at
times encounter one update has killed a PC with the popular blue screen of death
(although it’s no longer a BSD). We also
find that each month we don’t get 1 patch, it’s more like around 10 which must
be deployed to almost 1000 PCs. At the
time when we was heavily removing XP from our site, it was a nightmare. Fighting
staff who don’t like change (we don’t make policy) fighting staff who
don’t want you to interfere with their work hours (our working hours are the
exact same during term time) and finally half terms are the only times we can
do such large scale work. Even those half
terms get affected by students/staff on site – which we may not fully know of
till last minute. We also have the
occasional building work meaning we can’t even access certain areas of the
School or have power outages due to cabling works… We currently have an entire building
undergoing heavy amount of work. Those
contractors know they got till September to get their job done but what about
the rest of us? Do we have to wait till staff return to actually get in to the
rooms to setup IT? We’re talking dozens of PCs, boards and projectors in the
space of a day. It’s only through the
sheer luck that we work well with the site staff who will demand those
contractors are completed 1 week ahead of schedule so we can do our job.
This is what it’s like.
You are faced with the staff workers, the senior staff and
contractors. While also balancing your
own holiday periods (my staff are only allowed time off during half
terms). With a team of 3 this entire
thing becomes a balancing act. We still
get it done though. IT consultants they
don’t see that. They see cooperative
staff who listen to IT. A bit like
parents who see their little darling angels going to School to be nice to the
teachers yet most teachers will say they are not angels and the parents don’t believe
it because they don’t see it.
A few years ago this wouldn’t be the case and I can’t stress
enough about the potential risk this School would be at if we didn’t have
several changes occur. Now this is a
perfect example of what stops IT:
Lack of Time –
Through endless amount of fire fighting (please see other posts about
BYOD). If there’s no time for anything
to be developed what can we do?
Lack of Budget No
server expenditure, no networking improvements and general IT aged to 10+ years
old – To develop things like SCCM you need a
£ S E R V E R £ – usually Virtualised
Lack of Staff (2
and a bit man power employed for a very large system with heavy IT reliance)
Doing other tasks
– Something especially outside of education no one understands.
We’ve been expected to be:
Administration workers (printing, creating documents,
dealing with letters)
Teaching assistants (supporting classrooms with general
student support – not IT faults)
Media and film technician (supporting media, sound,
creativity and film)
Second main reception (we’ve had a lot of parents calling us
up directly)
And being teachers by showing students how to do tasks.
All the above is why at times IT have issues. It’s like saying it’s easy for a mechanic to
maintain 100 cars a week while he deals
with being a receptionist, finance officer and dealing with everyone’s
complaints and phone conversations. It
all adds up. Anyone that says otherwise
has led a very much easy job with very little pressure - I’ve often fallen out with IT people
because they don’t realise how it is at other places. I count myself very lucky to be where we are
now at my work place.
Interesting thing is one of the very people responsible for
my team doing the above is one of those who would complain if IT failed in any
single way. At the time we was not
spending any time (not even 1 hour a week) developing.
Now where are we?
My team perform IT work around 95% of the year – it’s still
not perfect but better.
Our servers are reliable to the point they work 98% of the
year – still not perfect but we’re getting there.
Our network is 96% 1GB capacity but again still not perfect.
Our average age of equipment is now 6 years old with future
ideas to help cut down costs – cost cutting still exists!
That’s on top of having a budget cut and staff cut. What do we do though? The HT has to make cost savings site
wide. Sooner or later I can see us
losing another team member due to the Government costs.
We’ve got too many
Enemies
Again it’s sad but true.
The SLT will often look at us with £££ on our heads, a resource that
they can survive without. You also have
the Heads of Music and ICT who will hate us because we hold them back in their
eyes. We stop them from having what they
want over what’s best for the School. We
stop them from having fancy gizmos and gadgets because we would rather spend
money upgrading Windows XP to Windows 7.
We would rather spend the money replacing 10 year old equipment instead
of that fancy pretty toy. The Governors
will often never see us, the kids only look at us like a stranger to reset a
password and the staff… they see us as an annoyance.
How many will be expected to turn up Monday morning and want
everything to be fixed? You know by
those staff working 9-5 mon- fri? The ones who shouldn’t be working weekends or
are being paid peanuts to do extremely high pressured work?
You’ve also got to consider massive amount of changes that
take endless amount of hours to resolve.
How many Schools go down the route of outsourcing IT, redundancies and
replacement staff? All of that takes
time to resolve. It’s like you’re trying
to do your job while on a train – you’ve got no choice but to ride it out.
We’ve actually gone through BYOD schemes that take a lot of
work out of our week and every year it’s changed in some way – taking more
time. There’s no consistency in IT
especially in education. What do people
need the most? Consistency. When you
consider how much IT itself changes when other things change you’ve turned a
small task to a bigger one. The BYOD
scheme occurred completely outside of the advice from us in IT. The mess it caused was unbelievable but who
cleans up that mess? We do.
We are NOT perfect
I once worked at a School that got heavily infected over a
decade ago. I warned those above that it
could happen because we didn’t update and generally had the arrogance that it
“won’t hit us”….
It did.. luckily
enough our servers were protected because I took it out of my own hours to
update them. Our best IT suites were
left untouched. The current person in
charge of IT got slaughtered by SLT (rightly so) because he only cared about
his own wage package and ambition. The
sad part is.. they actually listened to him quite a lot before that day. He knew best because he was the
socialite. He always fixed those who had
the biggest voices in the School while neglecting everyone else. Under the radar I was.
Then you have to think about who Schools employ – usually
fresh ex-students. They can’t manage
servers, they can’t manage systems, they can’t manage budgets and they
certainly can’t plan ahead. What they
can do is the simple tasks that anyone can do – reset passwords. Not nice but it’s true. Why do they employ ex-students? Money saving
and an arrogance in to thinking IT is easy when you know the basics. Just because someone can change a car tyre it
doesn’t mean they can MOT, Service and perform maintenance on a car. Let’s face it to fully hire a decent IT staff
level you’re talking at least £80,000 when you include pensions – that’s 2-3
teachers.
Centralised IT
This is the biggest downside about centralising IT,
resorting to agency workers and outsourced IT support. You place all your eggs in one basket and
when that has problems – you lose everything.
Centralised IT means you have more servers, less technicians and less
expenditure on every day equipment. All
about one thing – cost cutting – certainly nothing more and nothing less.
Schools are heading down this route with MATs (Multi Academy
Trusts) all for one thing – cost cutting.
They will want to link up their IT systems across the Schools, remove
the need to replace equipment and cut down the staff; all for one thing – cost
cutting. They will spend money on
Virtualised Servers which are fantastic but it puts them all under a single
roof.
We will be back here again one day like everyone. Every few years people dump a lot of money on
IT, cut it all back and then revert back to day one. The vicious IT circle I call it. The weird thing is the very people
complaining about this could be the very people who treat IT people
poorly. Whom often ask – what do they do
all day. IT is like a child where you
constantly need to be vigilant.
I’m not surprised at all at what’s happened in the NHS and
globally. Schools often go down
BYOD/Mobile technology routes because it’s pretty instead of spending money
upgrading IT. They often purchase Macs
because they look pretty and you have to remember if the world was Mac – they’d
be hit. I’ve seen an infected Mac but
why create virus’s and malware for systems that are rare. Even Smartphones are being targeted far more
than Macs because everyone has a Smartphone.
Why is IT expensive? It doesn’t have to be but you get too
many fish ponds involved. You will have
the main boss, who asks a senior manager and that person talks to the
consultant who then reverts to the actual IT Manager. It takes far too many steps to get to the
truth. To the person who really is clued
up.
While typing this blog I’ve already argued with a few who
are completely ignoring these points about budget cuts. It impacts an incredible amount. Everyone is more bothered about throwing
complaints at Hunt who let’s face it won’t understand anything or even
care. No managers at the top really deal
with IT unless it’s their own iPad. They
will pass it on to someone else and down the chain. This will all boil down to the fact that NHS
leaders decided to cut back IT like most public services do.
Schools are not lucky
– Thank the IT Staff who tend to ignore those above
Not many Schools have been hit by this mess. It’s not thanks to good budgets or good
leadership of the Schools – in fact it’s down to IT. Those who work extra hours for no pay, those
who perform remote work from home all because they care. Many of us see the IT system as our family
member. We want the best of that IT
system, we want everything to work regardless.
I can’t stress the amount of unpaid hours I’ve worked in the last 12
months let alone 4 years or almost two decades of working. All because I care. We’ve often seen malware infections but only
on a per PC basis caused by a single staff member (had three PCs in 4
years). Staff training really means
nothing, you’re fighting staff who really don’t think about what they click
on. They save work from 10 years ago and
copy to keep it. Another 10 years
they’ve built up work they didn’t know they have.
The thing is none of these ‘attacks’ are new. Malware has been around for many years I
think my first case of malware was around 10 years ago. These attacks can impact education and who
will be the ones blamed – IT. While
those at the top who have made the cutbacks and held back IT will be
blameless. Not many Schools can afford
qualified and experienced staff – most leave education to go to business. Many of us can earn £10K more outside of
education but we enjoy the challenge it brings (plus the not so stressful half
terms with not many on site).
So before you moan about people at the top think about those
IT who actually have made big differences for peanuts (also earning less than
those working in the NHS IT departments).
How many of us have developed systems outside of normal
hours because we’re firefighting all day.
Spending our days resetting passwords, in classrooms fixing PCs and
projectors while attempting to keep old outdated servers running.
At the time I returned to work on Monday I had already
gotten over a dozen emails of staff who attempted to advise or ask are we
okay. My response to everyone was we’re
safe (touch wood). We removed XP, we
keep our system updated and I made it very clear that this was the result of
all the hard work we had done.
Remember us.
The rant
Do apologise for this seeming like a huge rant but I’ve
personally been on the end of something very similar. You are being held back by budget, time and
staffing yet it’s still your fault. You
have everyone phoning you up while your still trying to fix the problem – each
interruption delays you fixing it. You
have everyone telling you how to do it better; when? Yes set it to go automatic even though you’ve
not got the time or money to get that system setup in the first place. A good little example – while I type this I’m
remotely installing a piece of software.
In two whole days this is the only 20 minute window I have. The member of staff does the same hours as me
and this can’t be done remotely because of the type of software it is. Any issues with this – it’s scrapped and I
have to wait another three whole days for a chance to do it again.
I even remember having to stay back several evenings with a
severe server failure. One night I left
on time to remotely carry on to have one member staff question to someone else
why am I not staying till late to get it working. This is the state of people that don’t fully
understand IT and working with day to day people (especially large amounts with
SLT who don’t want downtime). If you
tell me Downtime must happen regardless – you’ve not working in Education.
We’re going to be installing SSDs to every PC next year –
can this be done automatically? No. Each
one has to be done by hand. We also don’t
have the means to build up those SSDs beforehand (we don’t even have them yet
till budget renewal in September – meaning can’t do it during the Summer
break). This will not only cause
downtime of the staff PCs but require manual installation of SIMs (won’t image)
and the staff member to save all data to their network drive (not going to
happen no matter how much I beg/scream).
People have little to no idea what it’s really like to work
in an environment held back by so much, while everyone expects the world. No one cares till it breaks, when it’s
working people think we’re playing games.
You look at this cyber-attack, it was really caused by some
one click a link on an email after already being warned not to. That person will not be in trouble, will not
get a talking to or anything. Yet no one
cares about this no one cares that this was truly caused by the everyday user.
