Sten Vesterli's Blog

Making IT Live up to its Promise

Category: IT Leadership

A Value-Destroying Technical Innovation

IT Leadership, Technology That Fits, , , ,

The important part is not the technology itself. It is how it interacts with its surroundings.

The big Ethereum upgrade (aka “The Merge”) seems to have been successful from a technical standpoint. But it seems that the Ethereum community focused on the enormous technical challenge of merging the existing Ethereum blockchain with another without stopping either. The problem is that changing from proof-of-work to proof-of-stake turned Ether tokens from a currency into a security. When you “stake” your Ether, you earn interest. And suddenly, the Ethereum ecosystem is subject to U.S. Securities and Exchange Commission (SEC) rules. Consequently, Ether is down 26% this week.

You can implement highly advanced technology with enough skill, time, and money. But unless you have someone skeptical think through how your tech will interact with its environment, all of the tech wizardry might go unused. It might even be destroying value as the Ethereum Merge did.

The Wrong Way to Use Open Source

IT Leadership, Technology That Fits, ,

The glass is less than half full, and that is worrying. One of the focus areas in the 2022 State of Data Science report was enterprise adoption of open source. On every important metric, less than half the respondents gave the answer I was hoping for. For example, “We use a managed repository” got 43%, and “We use a vulnerability scanner” got 36%.

With such a low security maturity, it is obvious that the Log4j debacle and recent hacktivism has dented open source adoption. 40% report scaling back the use of open source in the past year.

Open Source gives you transparency that proprietary software doesn’t. That means you have the ability to verify that it works as promised instead of simply trusting vendor promises. But if you don’t make use of this transparency and instead simply download something because it’s free, you are setting yourself up for problems. Is your organization found in the more than half-empty part of the glass?

How to Avoid Hardwired Credentials

IT Leadership, Technology That Fits, ,

“We’ll add security later,” the developers said.

“We have to ship,” the manager said.

And that’s how 1000s of apps came to be released with hardcoded AWS passwords. That means that hackers can take these credentials and get full access to the AWS back end behind the apps, including reading and changing data for all users. The large number of vulnerable apps does not indicate that thousands of developers have hardcoded credentials. It shows that a smaller number hardcoded AWS credentials into their libraries, and thousands of developers uncritically used these libraries.

We see this kind of IT library supply chain problem all the time. It is much less likely to happen if you have a security QA gate in your workflow. Every system needs to have a security review signed off by an internal employee. It doesn’t help to contract this out. Your security consultant will be long gone when the vulnerability comes to light. The review has to be done by someone whose job is on the line.

Thanks to Kim Berg Hansen for pointing me to this story.

Don’t Replace Production With IT

IT Leadership, ,

The public broadcaster here in Denmark has just announced they’ll be firing 47 journalists to hire more IT people. What the hell are they thinking?!

I’m all for well-staffed IT departments, but IT is supposed to be something that helps the business. When you start firing the people who actually create the product that is the reason your business exists, you are on the wrong path.

If you want more people in IT, present a business case that explains how these people will pay for themselves. That involves calculating a business benefit in dollars and comparing it to the cost of the new people. They should pay for themselves in 12 months. Things are changing much too rapidly to depend on the multi-year repayment schedules used in the past.

Do You Know Where Your Data Is?

IT Leadership, ,

Facebook has no idea where they store your data. In a hearing, two senior Facebook employees admitted that they couldn’t say where user data was stored, much less ensure that it was all turned over to the authorities or deleted if required. The investigator said, “surely someone must have a diagram?” The engineers replied, “no, the code is its own documentation.”

The second law of thermodynamics applies to IT systems just like it applies to the rest of the world. It says that the amount of entropy, or disorder, inexorably increases unless someone spends energy actively trying to diminish it.

That becomes a problem when nobody spends time refactoring or cleaning up but lots of time adding new features, integrations, and dependencies. More than half of all organizations are where Facebook is: They don’t have and cannot establish the full picture of how their systems work. That places them at risk of catastrophic and irrecoverable failure. Can you establish a complete overview of your systems?

The Antidote to Value-Destroying Vanity Projects

IT Leadership, Technology That Fits, , ,

Choosing the solution that is 100 times more expensive sounds absurd. Nevertheless, that is what the U.S. Congress has decided, which is why NASA is struggling to get its SLS rocket off the ground. It has nothing to do with putting a man on the moon and everything to do with keeping the big NASA factory in Alabama running. They don’t have the technology to compete with newer space companies like SpaceX but are cobbling together old Space Shuttle parts. They’ve actually been going around to museums in the U.S., taking out old space shuttle engines for the quixotic project foisted upon them by Alabama Senator Richard Shelby.

Some organizations face a similar challenge: The CEO or someone else in senior leadership has an idea for some technology, and IT is ordered to deliver it. It doesn’t matter if such a vanity project is practical, feasible, or cost-efficient. You cannot fight this kind of project individually because they are highly connected with the ego of one individual.

The solution is to establish a standard evaluation process for technology projects. Every project needs a business owner responsible for calculating the business benefit. Every project also has a technical owner responsible for calculating the cost, including the ongoing running costs after completion. If the benefit comfortably exceeds the cost, the project is qualified to enter the competition with other claims on company investment.

You might not have such a process because a rational decision might also kill some of the IT department’s most beloved resumé-enhancing projects…

Thinking is Better Than Buzzwords

IT Leadership, Technology That Fits, ,

Thinking is hard. That’s why we substitute platitudes whenever possible. I was reminded last week because the election campaign has started here in Denmark. The young politicians on the left don’t know what to do about the increasing grocery prices – nobody does. But they do know that monopolies are bad. IF something bad happens AND monopolies are bad THEN monopolies cause the bad thing. The problem is that Denmark is the country most over-supplied with supermarkets in all of Europe. Calling the six different chains a monopoly means you are substituting critical thinking with a buzzword.

The same thing happens in IT. I keep seeing IT organizations whose strategy is decided by the buzzword of the day instead of actually thinking about your business needs versus the skills and resources available.

Learning from the Mistakes of Others

IT Leadership,

Learning from your mistakes is good. Learning from the mistakes of others is better. One mistake that often occurs in projects that include any kind of software is the lack of sanity checks.

For example, hackers broke into Yandex Taxi yesterday. They ordered every available car in all of Moscow to the same address. The result was a humongous traffic jam on Kutuzovsky Prospekt that took more than two hours to resolve. A sanity check would limit the number of cars that could be sent to the same address. Many of the recent failures of self-driving cars and trucks would not have happened if the software had sanity checks.

Do you have a process for sharing stories of things that went wrong? Whenever you have a meeting, have someone tell a story about something that went wrong and discuss what you can learn from that in your organization.

You Need an Agile End Result More Than an Agile Process

IT Leadership, Technology That Fits, ,

An agile development process is not important. An agile end result is. If your organization realizes a benefit from an agile methodology, that only helps you during the relatively short development process. But if you build something that can easily be re-configured and changed, that benefits you for the year or decades that you are running the system.

You would think that a digital billboard would be agile. The whole point is that it can display whatever you want. But German advertisers and shops have just realized that their display screens have very narrow agility. A new law requires these energy-guzzling billboards to be switched off at night to save electricity. It turns out that all the devices were built on the assumption that they would always be on, and they do not take kindly to store employees simply yanking the power cord when they go home.

To achieve agility in your products and systems, you need to avoid hard-wiring your assumptions into them. The only thing you can safely assume is that everything will eventually have to be changed.

What do you incentivize?

IT Leadership, ,

You get what you reward. Twitter decided to primarily reward user growth, and things went downhill from there. Recently, their head of security quit. Then he filed a whistleblower complaint with the authorities, complaining that Twitter’s security is bad and not getting better. Now there is likely to be a very interesting congressional hearing.

White-hat hacker Peiter Zatko (aka “Mudge”) was hired after a 2020 security breach but could not implement the changes he felt were necessary to fight spam and automated bots. The reason is the incentive structure at Twitter.

You see, Twitter management bonuses are based on user growth. There is no bonus for reducing spam or automated bots. You get what you reward, with no exception. You can reward with money, perks, promotions, or other recognition, but you have to incentivize the behavior you want. If all your incentives are based on quantity and not quality, you will get ever-increasing quantity and ever-decreasing quality.