Sten Vesterli's Blog

Making IT Live up to its Promise

Patching is Your Responsibility, not Your Customer’s

IT Leadership, Technology That Fits, ,

Face it, you are not even fully patching your own systems. Assuming that your customers or users will try to patch their systems is unrealistic.

If you are delivering any product that contains software, you need to think about how you will patch the thing. Tesla just discovered a problem with the pinch protection in their power windows. Cars with electric windows must have an “automatic window reversal system” that detects if it is about to pinch a finger or worse. Tesla found its system would not always be within the required parameters and pushed out an over-the-air update to more than a million vehicles. Elon Musk took to Twitter to fume about the fact that such a fix is technically a “recall.”

On the other hand, more than a year after the vulnerability was discovered, there are still more than 80,000 vulnerable Hikvision cameras connected to the internet. Besides the fact that everybody can view their footage, the built-in Linux server is probably also mining crypto and sending spam. The owners could not be bothered to pull the thing down from the wall, connect it with a cable, install updated firmware and mount it again.

Be like Tesla, not like Hikvision.

How Bank Customer Data Ended up at Auction

Uncategorized, ,

“But I thought the drives were encrypted!”
“Only if you turn encryption on”

An American bank got off lightly, getting only a $35 million fine. For five years, they simply hired a moving company to get rid of old computers. That company then sold Morgan Stanley’s used hard drives at auction. Too bad that the drives still included information about 15 million customers. The drives did contain an encryption feature, but nobody turned it on…

The whole debacle came to light accidentally when an IT consultant bought a used hard drive for backup and discovered it was full of confidential data.

Organizations keep losing data. It costs money and reputation each time. You must address this problem with employee security awareness, internal procedures, and external security audits. Tell the story of Morgan Stanley at your next IT meeting. It just might remind someone of something they really ought to fix…

A Value-Destroying Technical Innovation

IT Leadership, Technology That Fits, , , ,

The important part is not the technology itself. It is how it interacts with its surroundings.

The big Ethereum upgrade (aka “The Merge”) seems to have been successful from a technical standpoint. But it seems that the Ethereum community focused on the enormous technical challenge of merging the existing Ethereum blockchain with another without stopping either. The problem is that changing from proof-of-work to proof-of-stake turned Ether tokens from a currency into a security. When you “stake” your Ether, you earn interest. And suddenly, the Ethereum ecosystem is subject to U.S. Securities and Exchange Commission (SEC) rules. Consequently, Ether is down 26% this week.

You can implement highly advanced technology with enough skill, time, and money. But unless you have someone skeptical think through how your tech will interact with its environment, all of the tech wizardry might go unused. It might even be destroying value as the Ethereum Merge did.

The Wrong Way to Use Open Source

IT Leadership, Technology That Fits, ,

The glass is less than half full, and that is worrying. One of the focus areas in the 2022 State of Data Science report was enterprise adoption of open source. On every important metric, less than half the respondents gave the answer I was hoping for. For example, “We use a managed repository” got 43%, and “We use a vulnerability scanner” got 36%.

With such a low security maturity, it is obvious that the Log4j debacle and recent hacktivism has dented open source adoption. 40% report scaling back the use of open source in the past year.

Open Source gives you transparency that proprietary software doesn’t. That means you have the ability to verify that it works as promised instead of simply trusting vendor promises. But if you don’t make use of this transparency and instead simply download something because it’s free, you are setting yourself up for problems. Is your organization found in the more than half-empty part of the glass?

How to Avoid Hardwired Credentials

IT Leadership, Technology That Fits, ,

“We’ll add security later,” the developers said.

“We have to ship,” the manager said.

And that’s how 1000s of apps came to be released with hardcoded AWS passwords. That means that hackers can take these credentials and get full access to the AWS back end behind the apps, including reading and changing data for all users. The large number of vulnerable apps does not indicate that thousands of developers have hardcoded credentials. It shows that a smaller number hardcoded AWS credentials into their libraries, and thousands of developers uncritically used these libraries.

We see this kind of IT library supply chain problem all the time. It is much less likely to happen if you have a security QA gate in your workflow. Every system needs to have a security review signed off by an internal employee. It doesn’t help to contract this out. Your security consultant will be long gone when the vulnerability comes to light. The review has to be done by someone whose job is on the line.

Thanks to Kim Berg Hansen for pointing me to this story.

Don’t Replace Production With IT

IT Leadership, ,

The public broadcaster here in Denmark has just announced they’ll be firing 47 journalists to hire more IT people. What the hell are they thinking?!

I’m all for well-staffed IT departments, but IT is supposed to be something that helps the business. When you start firing the people who actually create the product that is the reason your business exists, you are on the wrong path.

If you want more people in IT, present a business case that explains how these people will pay for themselves. That involves calculating a business benefit in dollars and comparing it to the cost of the new people. They should pay for themselves in 12 months. Things are changing much too rapidly to depend on the multi-year repayment schedules used in the past.

Do You Know Where Your Data Is?

IT Leadership, ,

Facebook has no idea where they store your data. In a hearing, two senior Facebook employees admitted that they couldn’t say where user data was stored, much less ensure that it was all turned over to the authorities or deleted if required. The investigator said, “surely someone must have a diagram?” The engineers replied, “no, the code is its own documentation.”

The second law of thermodynamics applies to IT systems just like it applies to the rest of the world. It says that the amount of entropy, or disorder, inexorably increases unless someone spends energy actively trying to diminish it.

That becomes a problem when nobody spends time refactoring or cleaning up but lots of time adding new features, integrations, and dependencies. More than half of all organizations are where Facebook is: They don’t have and cannot establish the full picture of how their systems work. That places them at risk of catastrophic and irrecoverable failure. Can you establish a complete overview of your systems?

The Antidote to Value-Destroying Vanity Projects

IT Leadership, Technology That Fits, , ,

Choosing the solution that is 100 times more expensive sounds absurd. Nevertheless, that is what the U.S. Congress has decided, which is why NASA is struggling to get its SLS rocket off the ground. It has nothing to do with putting a man on the moon and everything to do with keeping the big NASA factory in Alabama running. They don’t have the technology to compete with newer space companies like SpaceX but are cobbling together old Space Shuttle parts. They’ve actually been going around to museums in the U.S., taking out old space shuttle engines for the quixotic project foisted upon them by Alabama Senator Richard Shelby.

Some organizations face a similar challenge: The CEO or someone else in senior leadership has an idea for some technology, and IT is ordered to deliver it. It doesn’t matter if such a vanity project is practical, feasible, or cost-efficient. You cannot fight this kind of project individually because they are highly connected with the ego of one individual.

The solution is to establish a standard evaluation process for technology projects. Every project needs a business owner responsible for calculating the business benefit. Every project also has a technical owner responsible for calculating the cost, including the ongoing running costs after completion. If the benefit comfortably exceeds the cost, the project is qualified to enter the competition with other claims on company investment.

You might not have such a process because a rational decision might also kill some of the IT department’s most beloved resumé-enhancing projects…

Thinking is Better Than Buzzwords

IT Leadership, Technology That Fits, ,

Thinking is hard. That’s why we substitute platitudes whenever possible. I was reminded last week because the election campaign has started here in Denmark. The young politicians on the left don’t know what to do about the increasing grocery prices – nobody does. But they do know that monopolies are bad. IF something bad happens AND monopolies are bad THEN monopolies cause the bad thing. The problem is that Denmark is the country most over-supplied with supermarkets in all of Europe. Calling the six different chains a monopoly means you are substituting critical thinking with a buzzword.

The same thing happens in IT. I keep seeing IT organizations whose strategy is decided by the buzzword of the day instead of actually thinking about your business needs versus the skills and resources available.

Learning from the Mistakes of Others

IT Leadership,

Learning from your mistakes is good. Learning from the mistakes of others is better. One mistake that often occurs in projects that include any kind of software is the lack of sanity checks.

For example, hackers broke into Yandex Taxi yesterday. They ordered every available car in all of Moscow to the same address. The result was a humongous traffic jam on Kutuzovsky Prospekt that took more than two hours to resolve. A sanity check would limit the number of cars that could be sent to the same address. Many of the recent failures of self-driving cars and trucks would not have happened if the software had sanity checks.

Do you have a process for sharing stories of things that went wrong? Whenever you have a meeting, have someone tell a story about something that went wrong and discuss what you can learn from that in your organization.