Sten Vesterli's Blog

Making IT Live up to its Promise

Category: IT Leadership

It’s Expensive to Try to Get By With the Cheapest Resources

IT Leadership, , , , , ,

Talent is expensive. Not paying for talent is more expensive. Microsoft gets that. The U.S. Department of Defence doesn’t.

The Microsoft bug hunting program has a maximum payout of $250,000, and they did pay out $200,000 this year. You would think a crucial national defence vulnerability would merit a bigger bounty that finding a flaw in the Microsoft hypervisor, wouldn’t you? The DoD pays out $500 for a high-severity bug, and a whopping $1,000 for a critical issue.

Your developers are rewarded for shipping functionality. They don’t have the mindset to find the vulnerabilities. To build secure systems, you need to offer a bug bounty, or hire outside experts to do security review, or create your own internal white-hat hacker team. It does cost money. But security breaches cost much more.

Why Projects Without Business Cases are Shot Down

IT Leadership, Technology That Fits, , , ,

I just had a customer attempt to start a project without a business case. Such projects are usually driven by the desire to use a specific technology and with a vague idea that this would somehow benefit the end user.

If the IT department is strong, some of these orphan projects get started. They might be successful. However, since the organization has no idea of the business benefit, it is blind luck if the benefits exceed the cost.

If the business prevention department (compliance/legal) is strongest, they are shot down. There is always a reason not to make any changes. A project without a business case can be mortally wounded by any objections about compliance, GDPR, security, etc.

That is why every project needs a business case. It prevents IT from wasting money on something that will not add value, and it prevents compliance & legal from killing projects with a positive business impact.

Do your projects have solid business cases? If not, get in touch, and I’ll help you.

Who is in Charge of Outside-the-Box Thinking?

IT Leadership, Technology That Fits, ,

Everyone can track your license plate – not just the cops. A Belgian security researcher noticed that most parking apps do not validate that you actually own the license plate you add to your app. That means a stalker can add his victim’s license plate to his app and immediately be notified whenever that person parks anywhere…

This is another example of the inside-the-box thinking that developers are prone to. The developers of the Kryptonite bike lock had made it out of extra reinforced steel. Too bad a weakness in the lock allowed a hacker to open it with half of a ballpoint pen.

Finding holes in a system is not just securing the login and checking the encryption. It involves examining the system and its environment and users. That is a skill most developers lack. You need a “red team” who can find the holes before you roll out something embarrassingly insecure.

Do You Still Need an IT Department?

IT Leadership, ,

Is it time to get rid of the IT department? Some people advocate that IT should be fully distributed in individual business units. Dr. Joe Peppard argued this case, and I just read his reply in the Wall Street Journal to the various objections he received.

I find the binary “IT department yes/no” discussion to be a useless polemic. Every organization has some mix of central and decentralized IT. The question is whether each IT capability is correctly placed.

Systems of record make up the foundation of your business and do not provide a competitive advantage. That’s things like email, accounting, and HR. Those fit well in a centralized IT function. You want to focus on reliability, and you don’t want each department or country running its own accounting or email server.

Systems of engagement that provide you with competitive advantage need to be closer to the business and the customers and move faster. That requires a different mindset with more risk-taking. That’s why these systems are often a poor fit in centralized IT. But letting each department roam free means you end up with a dozen incompatible cloud services without synergy.

Conundrums like this cannot be solved at the organizational level where they occur. Because the choice pits the agenda of the CIO/CTO against the CFO, CMO, and CSO, the final arbiter has to be the CEO. He or she is impervious to technical arguments. If you are a CIO or CTO, you owe it to your organization to be able to speak the language of the CEO.

Patching is Your Responsibility, not Your Customer’s

IT Leadership, Technology That Fits, ,

Face it, you are not even fully patching your own systems. Assuming that your customers or users will try to patch their systems is unrealistic.

If you are delivering any product that contains software, you need to think about how you will patch the thing. Tesla just discovered a problem with the pinch protection in their power windows. Cars with electric windows must have an “automatic window reversal system” that detects if it is about to pinch a finger or worse. Tesla found its system would not always be within the required parameters and pushed out an over-the-air update to more than a million vehicles. Elon Musk took to Twitter to fume about the fact that such a fix is technically a “recall.”

On the other hand, more than a year after the vulnerability was discovered, there are still more than 80,000 vulnerable Hikvision cameras connected to the internet. Besides the fact that everybody can view their footage, the built-in Linux server is probably also mining crypto and sending spam. The owners could not be bothered to pull the thing down from the wall, connect it with a cable, install updated firmware and mount it again.

Be like Tesla, not like Hikvision.

A Value-Destroying Technical Innovation

IT Leadership, Technology That Fits, , , ,

The important part is not the technology itself. It is how it interacts with its surroundings.

The big Ethereum upgrade (aka “The Merge”) seems to have been successful from a technical standpoint. But it seems that the Ethereum community focused on the enormous technical challenge of merging the existing Ethereum blockchain with another without stopping either. The problem is that changing from proof-of-work to proof-of-stake turned Ether tokens from a currency into a security. When you “stake” your Ether, you earn interest. And suddenly, the Ethereum ecosystem is subject to U.S. Securities and Exchange Commission (SEC) rules. Consequently, Ether is down 26% this week.

You can implement highly advanced technology with enough skill, time, and money. But unless you have someone skeptical think through how your tech will interact with its environment, all of the tech wizardry might go unused. It might even be destroying value as the Ethereum Merge did.

The Wrong Way to Use Open Source

IT Leadership, Technology That Fits, ,

The glass is less than half full, and that is worrying. One of the focus areas in the 2022 State of Data Science report was enterprise adoption of open source. On every important metric, less than half the respondents gave the answer I was hoping for. For example, “We use a managed repository” got 43%, and “We use a vulnerability scanner” got 36%.

With such a low security maturity, it is obvious that the Log4j debacle and recent hacktivism has dented open source adoption. 40% report scaling back the use of open source in the past year.

Open Source gives you transparency that proprietary software doesn’t. That means you have the ability to verify that it works as promised instead of simply trusting vendor promises. But if you don’t make use of this transparency and instead simply download something because it’s free, you are setting yourself up for problems. Is your organization found in the more than half-empty part of the glass?

How to Avoid Hardwired Credentials

IT Leadership, Technology That Fits, ,

“We’ll add security later,” the developers said.

“We have to ship,” the manager said.

And that’s how 1000s of apps came to be released with hardcoded AWS passwords. That means that hackers can take these credentials and get full access to the AWS back end behind the apps, including reading and changing data for all users. The large number of vulnerable apps does not indicate that thousands of developers have hardcoded credentials. It shows that a smaller number hardcoded AWS credentials into their libraries, and thousands of developers uncritically used these libraries.

We see this kind of IT library supply chain problem all the time. It is much less likely to happen if you have a security QA gate in your workflow. Every system needs to have a security review signed off by an internal employee. It doesn’t help to contract this out. Your security consultant will be long gone when the vulnerability comes to light. The review has to be done by someone whose job is on the line.

Thanks to Kim Berg Hansen for pointing me to this story.

Don’t Replace Production With IT

IT Leadership, ,

The public broadcaster here in Denmark has just announced they’ll be firing 47 journalists to hire more IT people. What the hell are they thinking?!

I’m all for well-staffed IT departments, but IT is supposed to be something that helps the business. When you start firing the people who actually create the product that is the reason your business exists, you are on the wrong path.

If you want more people in IT, present a business case that explains how these people will pay for themselves. That involves calculating a business benefit in dollars and comparing it to the cost of the new people. They should pay for themselves in 12 months. Things are changing much too rapidly to depend on the multi-year repayment schedules used in the past.

Do You Know Where Your Data Is?

IT Leadership, ,

Facebook has no idea where they store your data. In a hearing, two senior Facebook employees admitted that they couldn’t say where user data was stored, much less ensure that it was all turned over to the authorities or deleted if required. The investigator said, “surely someone must have a diagram?” The engineers replied, “no, the code is its own documentation.”

The second law of thermodynamics applies to IT systems just like it applies to the rest of the world. It says that the amount of entropy, or disorder, inexorably increases unless someone spends energy actively trying to diminish it.

That becomes a problem when nobody spends time refactoring or cleaning up but lots of time adding new features, integrations, and dependencies. More than half of all organizations are where Facebook is: They don’t have and cannot establish the full picture of how their systems work. That places them at risk of catastrophic and irrecoverable failure. Can you establish a complete overview of your systems?