Sten Vesterli's Blog

Making IT Live up to its Promise

Tag: Risk

Invisible Problems

IT Leadership, ,

“We don’t see a problem.” That is the typical response when a company is investigated for unsafe products or features. Latest car in point: Ford’s BlueCruise self-driving feature.

They might be right in claiming that customers have not reported problems to them. But when the National Highway Traffic Safety Administration (NHTSA) started to take an interest, they could easily compile 2,000 claims of malfunctioning self-driving features.

How are you gathering customer feedback? The fact that you don’t receive any problem reports doesn’t mean there are no problems.

Unknown AI Policies

IT Leadership, , ,

Does everyone in your IT organization know what your rules are regarding AI use? 60% of employees report using AI tools, while fewer than 20% say they know the company’s AI policy.

That is not because the policies don’t exist. More than 80% of IT leaders report that their organizations have formulated polices for AI use.

How are you going to close that gap?

Only Outsiders Can See the Faults

IT Leadership, , ,

Would you design a product with a sleek but potentially deadly feature? Most people wouldn’t, but aerodynamics and design have led to at least 15 people dying in burning Teslas when the electric doors wouldn’t open. The Chinese are no longer having it, and are ordering all cars to have a mechanical door release both inside and outside the vehicle from January 1st next year.

Thousands of trade-offs are made when designing products. But some outcomes are so bad that they ought to disqualify a feature. Product owners want a great product with awesome features, and are not capable of imagining all the bad things that could happen. Even if you don’t have a dedicated Red Team, you need someone outside the product team to probe your products for weaknesses. The people building it can’t see them.

Backup Communication Channels

IT Leadership, , , ,

What is the difference between 30 individual soldiers and a platoon? Leadership and the ability to communicate.

The first step in your resilience planning is to ensure that you can still communicate, even when faced with an onslaught of Russian hackers or American government officials.

That could mean an on-premise open source mail server and a basic web server. Every workstation and company smartphone could have a separate open source mail client and web browser preconfigured for those servers.

There are many other options – the paranoid and those with high threat levels might have spare phones running GrapheneOS and Briar, or even establish their own Meshtastic mesh network.

If you don’t have a backup communication channel, you urgently need to establish one. Especially if you are outside the U.S. and depend on U.S. services.

Rational Decision-making

IT Leadership, ,

As an individual, you are free to make emotional decisions. You can decide to evict some software product from your laptop because you don’t like the vendor’s nationality or stance on today’s hot-button social issue. As an IT professional, you can even set up an open source solution that does almost the same (though invariably with worse UX) in a few days.

But as an IT leader, you are expected to make rational decisions. That’s why you don’t throw out all your Amazon, Microsoft, or Google Cloud on a whim because you are unhappy with U.S. policy. The rational choice is to minimize your risk. That means building new systems outside U.S. clouds so you don’t add to your problems. And migrating away from disfavoured platforms in an orderly, cost-effective manner.

Sovereign Cloud

IT Leadership, , ,

You need to create a delay between a foreign government ordering your cloud provider to cut you off and the actual cutoff. The longer you can make this period, the better. The new AWS European Sovereign Cloud (ESC) is Amazon’s way of offering this. That is a cloud solution running on hardware in Europe, staffed by Europeans, organized under a European daughter company.

That does not protect you against Amazon being compelled by the U.S. government to hand over your data, but all important data should be protected with keys you hold, outside your cloud provider, anyway. But it does make it likely that AWS Europe would contest an order to shut down the service, and that AWS Europe employees would not cut you off at the whim of a foreign dictator.

Because the probability of this happening is still “Rare” (edging towards “Unlikely”), you do not need to act on this risk now. But it is prudent to ensure you have time to react if it should happen.

Risk Evaluation

IT Leadership, ,

Do you have a paper map in your car? No, why would I need that?

If you are a Verizon customer in the U.S., you were just reminded. A large chunk of their network was down for half a day, leaving frustrated customers depending on their atrophied geographical memory. Verizon says the culprit was the usual botched network upgrade, not evil hackers. Some Europeans are better prepared, having routinely been subjected to Russian jamming of GPS and Galileo navigation.

When was the last time you revisited the risk evaluation of your critical systems? The threats are changing and increasing, and your risk evaluation from one year ago no longer applies.

Downside Thinking

IT Leadership, ,

What is the downside? That is the critical question for any kind of decision. Someone has an idea, and they will present the upside. We can save so much money, develop faster, offer better customer service, etc., etc. It is your job as a leader and decision-maker to ferret out the downside.

Lack of downside consideration is behind many questionable business decisions. Twitter seems to offer a new example of Elon’s lack of downside thinking every week. Like “Let’s offer a feature that changes any photo to show the person in a bikini.” Normal people, doing even the most minimal downside consideration, would kill that idea in seconds. But Twitter/X rolled it out – and obviously had to roll it back.

You can do the downside thinking yourself for many decisions. For more complicated scenarios, you might need a dedicated Red Team or outside help to identify the downside. But before any decision, ask yourself: Have we considered the downside?

Digital Sovereignty

IT Leadership, ,

You need to think about Digital Sovereignty. Unless you are in the U.S., of course. For everybody else, this is a very salient topic. Especially for us in Denmark these days.

This doesn’t mean that you have to free yourself from every American cloud provider. But it does mean there is a new item in your risk evaluation: Ending up on the Office of Foreign Assets Control (OFAC) blocklist.

Likelihood is Rare (1) for almost everybody. But if Impact is Catastrophic (5), you end up with a medium risk: Mitigate if cost-effective.

Switching costs almost always make it not cost-effective to transition a running system. But when you are building anything new, you don’t have switching costs. And an effective mitigation is to avoid using U.S. providers.

Rollback plans

IT Leadership, ,

What differentiates a professional software organization from a bunch of amateurs? One thing is the ability to roll back.

It’s not a good day at the office when the Federal Aviation Authority issues an Emergency Airworthiness Directive, grounding 6,000 of the aircraft your customers are flying. A JetBlue flight on autopilot suddenly turned nose-down in mid-flight, and it turns out that the L104 version of the ELAC software was vulnerable to memory corruption due to solar radiation.

But aircraft manufacturers and airlines have procedures in place, and engineers rapidly fanned out to airports around the world, rolling back to the L103 version.

It is impossible to test every situation, and every once in a while, something unforeseen happens. Professional software organizations can rapidly recover. Do you have rollback plans in place every time you roll out new versions of critical software?