Sten Vesterli's Blog

Making IT Live up to its Promise

Tag: Risk

Who is in Charge of Outside-the-Box Thinking?

IT Leadership, Technology That Fits, ,

Everyone can track your license plate – not just the cops. A Belgian security researcher noticed that most parking apps do not validate that you actually own the license plate you add to your app. That means a stalker can add his victim’s license plate to his app and immediately be notified whenever that person parks anywhere…

This is another example of the inside-the-box thinking that developers are prone to. The developers of the Kryptonite bike lock had made it out of extra reinforced steel. Too bad a weakness in the lock allowed a hacker to open it with half of a ballpoint pen.

Finding holes in a system is not just securing the login and checking the encryption. It involves examining the system and its environment and users. That is a skill most developers lack. You need a “red team” who can find the holes before you roll out something embarrassingly insecure.

Do You Know Where Your Data Is?

IT Leadership, ,

Facebook has no idea where they store your data. In a hearing, two senior Facebook employees admitted that they couldn’t say where user data was stored, much less ensure that it was all turned over to the authorities or deleted if required. The investigator said, “surely someone must have a diagram?” The engineers replied, “no, the code is its own documentation.”

The second law of thermodynamics applies to IT systems just like it applies to the rest of the world. It says that the amount of entropy, or disorder, inexorably increases unless someone spends energy actively trying to diminish it.

That becomes a problem when nobody spends time refactoring or cleaning up but lots of time adding new features, integrations, and dependencies. More than half of all organizations are where Facebook is: They don’t have and cannot establish the full picture of how their systems work. That places them at risk of catastrophic and irrecoverable failure. Can you establish a complete overview of your systems?

What is Your Time to Recover?

IT Leadership, Technology That Fits, , ,

It wouldn’t take you three to four weeks to rebuild a critical system, would it? But that’s how they do things in the National Health Service in the UK. Doctors and hospitals have been advised that the central patient record system is offline due to a ransomware attack and will not be back until sometime in September. In the meantime, doctors will have no access to their patients’ medical histories and will have to keep notes on paper or in Microsoft Word on their laptops.

As a national health monopoly, the NHS will not be going out of business. But a private company that lost its manufacturing, logistics, or service management system for a month would be finished.

You do everything you can to prevent bad things from happening. But have you also planned contingent action in case something terrible does happen? The NHS hadn’t.

Documentation is Unnecessary Until You Need It

IT Leadership,

If you have a fire in your server room, your insurance pays out. Insurance is expensive, but a necessary part of your risk management strategy. For many risks, there is a way to get almost free insurance. Yet few people take it. I am talking about documentation.

A chocolate factory in Belgium didn’t follow its own processes and did not document its production. When kids started falling sick with salmonella all over Europe, suspicion fell on the Kinder egg factory in Arlon. The authorities asked for the production documentation. Because the factory couldn’t provide it, the whole plant was shut down. If they had had documentation, they would have been insured against this risk. They could have shut down just one production line instead of the whole plant.

So the reason you might not be able to get chocolate eggs this Easter is bad documentation.

Are You Too Cautious?

Spiritual Programmer

Do you always play it safe? We all have our personal risk profiles. Some people climb mountains without safety ropes, while others won’t climb more than two steps up a ladder. Being very careful to follow all the recommendations might be a good strategy in a pandemic, but being over-cautious also means you miss out on opportunities.

Researchers in the UK have found that teaching children chess made them more willing to take prudent risks. In chess, you need to be able to take prudent risks and sacrifice a piece to gain a decisive advantage. Chess was a safe environment for the children to experiment with risk – the worst thing that could happen was that they lost the game.

If you are being over-cautious in your life, find some place where you can practice taking small risks. You might even take up chess.

Risk Aversion

Technology That Fits,

The U.S. has stopped distributing the Johnson & Johnson vaccine. It has been given to more than 7 million people, and there have been six reported cases of blood clotting. Here in Denmark, we have stopped giving the Astra Zeneca vaccine because of one similar case. That is not risk management, that is risk aversion.

Risk management is one of the basic leadership tasks. The leader has to decide if the benefit of a certain decision is worth the risk of something bad happening. If we could calculate the exact probability and the exact impact, risk management would be a purely mathematical exercise. But since both probability and impact are only vaguely known, the leader has to use his or her experience, evaluate contrasting opinions, and make the call.

There is a classic short story by Stephen Leacock called “The Man in Asbestos.” It is from the time where fire-resistant asbestos was considered one of the miracle materials of the future. The narrator travels to the future to find a drab and risk-averse society where aging has been eliminated together with all disease. Machines produce everything anybody needs. Since everybody will live forever, barring accidents, railroads and cars are outlawed as too dangerous. Nobody needs to go anywhere, and nobody does. In this future, everybody has everything they need and lives forever, but the narrator is appalled at consequent stagnation.

That story was written in 1911 but was very prescient. We have since eliminated many risks and have increased our standard of living immeasurably. And we are less and less willing to accept any risk.

A leader accepts the risk and reaps the benefit. But our decisions are increasingly influenced by experts who point out the dangers. If you have dedicated your life to immunology, you know what the risks are. From the viewpoint of the immunologist, it is safest to lock everybody down until everyone is vaccinated. A political leader takes that input together with input from economists and other experts about the costs of lockdown and makes a leadership decision.

In organizations, the equivalent to the immunologist resides in legal, compliance, QA, risk management, or validation departments. They point out all the risks – children might swallow our product, we might get sued, we might have our operating license revoked. The larger the organization, the more of these departments of innovation prevention you will have. It takes courageous leadership to overrule the objects of the naysayers. The reason smaller organizations are able to out-innovate larger ones is that they can spend their leadership time on innovation and growth and instead of on fighting organizational units dedicated to preserving the status quo.

As an IT leader, it is your job to make sure your organization doesn’t get paralyzed by risk aversion.

Risk Aversion

Beneficial Intelligence, ,

In this episode of Beneficial Intelligence, I discuss risk aversion. The U.S. has stopped distributing the Johnson & Johnson vaccine. It has been given to more than 7 million people, and there have been six reported cases of blood clotting. That is not risk management, that is risk aversion.

There is a classic short story from 1911 by Stephen Leacock called “The Man in Asbestos.” In it, the narrator travels to the future to find a drab and risk-averse society where aging has been eliminated together with all disease. People can only die from accidents, which is why everybody wears fire-resistant asbestos clothes, railroads and cars are outlawed, and society becomes completely stagnant.

We are moving in that direction. Large organizations have departments of innovation prevention, often called compliance, risk management, or QA. It takes leadership to look at the larger benefit and overrule their objections Smaller organizations can instead spend their leadership time on innovation and growth.

As an IT leader, it is your job to make sure your organization doesn’t get paralyzed by risk aversion.

Contingency Plans

Beneficial Intelligence,

Last week’s episode of my podcast Beneficial Intelligence was about contingency plans. Texas was not prepared for the cold, and millions lost power. The disaster could have been avoided, had the suggestions from previous outages been implemented. But because rarely gets very cold in Texas, everybody decided to save money by not preparing their gear for winter. At the same time, Texans have decided to go it alone and not connect their grid to any neighbors.

In all systems, including your IT systems, you can handle risks in two ways: You can reduce the probability of the event occurring, or you can reduce the impact when it occurs. For IT systems, we reduce the probability with redundancy, but we run into Texas-style problems when we believe the claims of vendors and fail to prepare for the scenario when our redundant systems do fail. 

Texas did not reduce the probability, and was not prepared for the impact. Don’t be like Texas.

Contingency Plans

Beneficial Intelligence,

This week’s episode of my podcast Beneficial Intelligence is about contingency plans. Texas was not prepared for the cold, and millions lost power. Amid furious finger-pointing, it turns out that none of the recommendations from the report after the last power outage have been implemented, and suggestions from the report after the outage in 1989 were not implemented either.

As millions of Texas turned up the heat in their uninsulated homes, demand surged. At the same time, wind turbines froze. Then the natural gas wells and pipelines froze. Then the rivers where the nuclear power plants take cooling water from froze. And finally the generators on the coal-powered plants froze. They could burn coal, but not generate electricity. You can built wind turbines that will run in the cold, and you can winterize other equipment with insulation and special winter-capable lubricants. But that is more expensive, and Texas decided to save that money.

The problem could have been solved if Texas could get energy from its neighbors, but it can’t. The US power grid is divided into three parts: Eastern, Western, and Texas. They decided to go it alone but apparently decided to ignore the risk.

In all systems, including your IT systems, you can handle risks in two ways: You can reduce the probability of the event occurring, or you can reduce the impact when it occurs. For IT systems, we reduce the probability with redundancy. We have multiple power supplies, multiple internet connections, multiple servers, replicated databases, and mirrored disk drives. But we run into Texas-style problems when we believe the claims of vendors that their ingenious solutions have completely eliminated the risk. That leads to complacency where we do not create contingency plans for what to do if the event does happen.

Texas did not reduce the probability, and was not prepared for the impact. Don’t be like Texas.

Listen here or find “Beneficial Intelligence” wherever you get your podcasts.