Sten Vesterli's Blog

Making IT Live up to its Promise

Tag: Leadership

Do You Still Need an IT Department?

IT Leadership, ,

Is it time to get rid of the IT department? Some people advocate that IT should be fully distributed in individual business units. Dr. Joe Peppard argued this case, and I just read his reply in the Wall Street Journal to the various objections he received.

I find the binary “IT department yes/no” discussion to be a useless polemic. Every organization has some mix of central and decentralized IT. The question is whether each IT capability is correctly placed.

Systems of record make up the foundation of your business and do not provide a competitive advantage. That’s things like email, accounting, and HR. Those fit well in a centralized IT function. You want to focus on reliability, and you don’t want each department or country running its own accounting or email server.

Systems of engagement that provide you with competitive advantage need to be closer to the business and the customers and move faster. That requires a different mindset with more risk-taking. That’s why these systems are often a poor fit in centralized IT. But letting each department roam free means you end up with a dozen incompatible cloud services without synergy.

Conundrums like this cannot be solved at the organizational level where they occur. Because the choice pits the agenda of the CIO/CTO against the CFO, CMO, and CSO, the final arbiter has to be the CEO. He or she is impervious to technical arguments. If you are a CIO or CTO, you owe it to your organization to be able to speak the language of the CEO.

Patching is Your Responsibility, not Your Customer’s

IT Leadership, Technology That Fits, ,

Face it, you are not even fully patching your own systems. Assuming that your customers or users will try to patch their systems is unrealistic.

If you are delivering any product that contains software, you need to think about how you will patch the thing. Tesla just discovered a problem with the pinch protection in their power windows. Cars with electric windows must have an “automatic window reversal system” that detects if it is about to pinch a finger or worse. Tesla found its system would not always be within the required parameters and pushed out an over-the-air update to more than a million vehicles. Elon Musk took to Twitter to fume about the fact that such a fix is technically a “recall.”

On the other hand, more than a year after the vulnerability was discovered, there are still more than 80,000 vulnerable Hikvision cameras connected to the internet. Besides the fact that everybody can view their footage, the built-in Linux server is probably also mining crypto and sending spam. The owners could not be bothered to pull the thing down from the wall, connect it with a cable, install updated firmware and mount it again.

Be like Tesla, not like Hikvision.

How Bank Customer Data Ended up at Auction

Uncategorized, ,

“But I thought the drives were encrypted!”
“Only if you turn encryption on”

An American bank got off lightly, getting only a $35 million fine. For five years, they simply hired a moving company to get rid of old computers. That company then sold Morgan Stanley’s used hard drives at auction. Too bad that the drives still included information about 15 million customers. The drives did contain an encryption feature, but nobody turned it on…

The whole debacle came to light accidentally when an IT consultant bought a used hard drive for backup and discovered it was full of confidential data.

Organizations keep losing data. It costs money and reputation each time. You must address this problem with employee security awareness, internal procedures, and external security audits. Tell the story of Morgan Stanley at your next IT meeting. It just might remind someone of something they really ought to fix…

How to Avoid Hardwired Credentials

IT Leadership, Technology That Fits, ,

“We’ll add security later,” the developers said.

“We have to ship,” the manager said.

And that’s how 1000s of apps came to be released with hardcoded AWS passwords. That means that hackers can take these credentials and get full access to the AWS back end behind the apps, including reading and changing data for all users. The large number of vulnerable apps does not indicate that thousands of developers have hardcoded credentials. It shows that a smaller number hardcoded AWS credentials into their libraries, and thousands of developers uncritically used these libraries.

We see this kind of IT library supply chain problem all the time. It is much less likely to happen if you have a security QA gate in your workflow. Every system needs to have a security review signed off by an internal employee. It doesn’t help to contract this out. Your security consultant will be long gone when the vulnerability comes to light. The review has to be done by someone whose job is on the line.

Thanks to Kim Berg Hansen for pointing me to this story.

Don’t Replace Production With IT

IT Leadership, ,

The public broadcaster here in Denmark has just announced they’ll be firing 47 journalists to hire more IT people. What the hell are they thinking?!

I’m all for well-staffed IT departments, but IT is supposed to be something that helps the business. When you start firing the people who actually create the product that is the reason your business exists, you are on the wrong path.

If you want more people in IT, present a business case that explains how these people will pay for themselves. That involves calculating a business benefit in dollars and comparing it to the cost of the new people. They should pay for themselves in 12 months. Things are changing much too rapidly to depend on the multi-year repayment schedules used in the past.

Do You Know Where Your Data Is?

IT Leadership, ,

Facebook has no idea where they store your data. In a hearing, two senior Facebook employees admitted that they couldn’t say where user data was stored, much less ensure that it was all turned over to the authorities or deleted if required. The investigator said, “surely someone must have a diagram?” The engineers replied, “no, the code is its own documentation.”

The second law of thermodynamics applies to IT systems just like it applies to the rest of the world. It says that the amount of entropy, or disorder, inexorably increases unless someone spends energy actively trying to diminish it.

That becomes a problem when nobody spends time refactoring or cleaning up but lots of time adding new features, integrations, and dependencies. More than half of all organizations are where Facebook is: They don’t have and cannot establish the full picture of how their systems work. That places them at risk of catastrophic and irrecoverable failure. Can you establish a complete overview of your systems?

Thinking is Better Than Buzzwords

IT Leadership, Technology That Fits, ,

Thinking is hard. That’s why we substitute platitudes whenever possible. I was reminded last week because the election campaign has started here in Denmark. The young politicians on the left don’t know what to do about the increasing grocery prices – nobody does. But they do know that monopolies are bad. IF something bad happens AND monopolies are bad THEN monopolies cause the bad thing. The problem is that Denmark is the country most over-supplied with supermarkets in all of Europe. Calling the six different chains a monopoly means you are substituting critical thinking with a buzzword.

The same thing happens in IT. I keep seeing IT organizations whose strategy is decided by the buzzword of the day instead of actually thinking about your business needs versus the skills and resources available.

Learning from the Mistakes of Others

IT Leadership,

Learning from your mistakes is good. Learning from the mistakes of others is better. One mistake that often occurs in projects that include any kind of software is the lack of sanity checks.

For example, hackers broke into Yandex Taxi yesterday. They ordered every available car in all of Moscow to the same address. The result was a humongous traffic jam on Kutuzovsky Prospekt that took more than two hours to resolve. A sanity check would limit the number of cars that could be sent to the same address. Many of the recent failures of self-driving cars and trucks would not have happened if the software had sanity checks.

Do you have a process for sharing stories of things that went wrong? Whenever you have a meeting, have someone tell a story about something that went wrong and discuss what you can learn from that in your organization.

What do you incentivize?

IT Leadership, ,

You get what you reward. Twitter decided to primarily reward user growth, and things went downhill from there. Recently, their head of security quit. Then he filed a whistleblower complaint with the authorities, complaining that Twitter’s security is bad and not getting better. Now there is likely to be a very interesting congressional hearing.

White-hat hacker Peiter Zatko (aka “Mudge”) was hired after a 2020 security breach but could not implement the changes he felt were necessary to fight spam and automated bots. The reason is the incentive structure at Twitter.

You see, Twitter management bonuses are based on user growth. There is no bonus for reducing spam or automated bots. You get what you reward, with no exception. You can reward with money, perks, promotions, or other recognition, but you have to incentivize the behavior you want. If all your incentives are based on quantity and not quality, you will get ever-increasing quantity and ever-decreasing quality.

Do You Want Fancy, Cheap, or Usable?

IT Leadership, Technology That Fits, , ,

Do you prefer fancy, cheap, or useful? Car manufacturers have found that touchscreens look fancy and are cheaper than physical buttons. That’s why modern cars have touchscreens and no buttons. I prefer buttons, and science is on my side. A survey shows that it is almost twice as fast to perform common car tasks with buttons than with a touchscreen.

Unfortunately, a car is not required to be easy to operate. Aircraft cockpits, on the other hand, are full of physical buttons. An aircraft manufacturer accepts the extra cost of physical controls to provide an optimal user experience for the pilot.

Marketing wants fancy. Finance wants cheap. User Experience wants usability. Who has the last word on product design in your organization?