Sten Vesterli's Blog

Making IT Live up to its Promise

Criminally Bad Project Management

IT Leadership, ,

Sometimes, failed IT projects cost real money. Like it just did for British bank TSB, who was fined about $60 million for their shambolic IT migration. The disaster locked people out of their accounts for weeks, and the total cost to the bank is now approaching $500 million with payments to customers, project post-mortems and IT cleanup.

“The firm failed to plan for the IT migration properly, the governance of the project was insufficiently robust and the firm failed to take reasonable care to organise and control its affairs responsibly and effectively, with adequate risk management systems,” the report from the banking authority concluded.

Those words don’t apply to any of your IT project, do they?

Once you Grow up, you Need to Stop Moving Fast and Breaking Things

IT Leadership, Technology That Fits, , ,

Moving fast and breaking things can be fine for a startup. They might need to iterate several times and maybe even pivot once or twice before they achieve product/market fit. It is not OK for an established business. Facebook has long since given up on this strategy, but Twitter, under Elon Musk, has rediscovered it. By thrashing around and changing direction daily, they are alienating both the users and the advertisers who were supposed to pay. If you want to move fast, roll out changes to a small percentage of your users. A mature continuous delivery organization practices blue/green deployment, but even if you are not doing CI/CD, you can still test changes with a small subset of your users. Don’t uncritically inflict the latest great idea on your entire user population. #itleadership #innovation #makeitliveuptoitspromise

Beware of Asymmetric Risk/Reward Profiles

IT Leadership, Technology That Fits, , , , ,

Would you continue to sell a lock based on technology that has been known for 14 years to be trivially easy to hack? Of course not! But Scantron in Denmark has merrily been foisting insecure locks on unsuspecting Danish apartment administrators. Even after a worried renter told them about the problem in several emails and even physical letters (!), they ignored the problem. It took a media shitstorm to make them realize the errors of their ways.

Digital locks have an asymmetric risk/reward profile. The reward is small – you save a little by not having to administer physical keys and re-key locks. The risk is huge – someone might copy a key, turn it into a master key, and rob hundreds of apartments.

When you are evaluating digitalization projects, be very careful about those with such an asymmetric profile. Almost every organization has digitalization projects with a better risk/reward balance than digital locks…

Be Prepared for Every Eventuality

IT Leadership, , ,

My latest favorite German word: Bargeldnotversorgung. It means “emergency supplying of cash.” The careful Germans are preparing for the worst-case scenario of widespread blackouts. Part of their plan is to make sure that everybody has an adequate supply of little colored pieces of paper, also known as “cash.”

Meanwhile, in Denmark, authorities, and shops are merrily careening towards a cashless society that will collapse as soon as the power goes out.

There are many things that can go wrong, and a well-run organization is prepared for all of them. In cases of risk management, be like Germany, not like Denmark

IT Leadership has to Harness the Power of AI

IT Leadership, Technology That Fits, , , , ,

AI has finally gotten really useful inside the IT organization. Most of the examples on the internet are frivolous and amusing, like how to remove a peanut butter sandwich from a VCR, written in the style of the King James Bible. But ChatGPT is helpful for mundane tasks in IT as well.

I’ve been fixing open issues in a small open-source project recently. One of the issues was that part of the code would concatenate strings to build SQL statements. That’s a classic SQL Injection vulnerability. ChatGPT can fix these bugs faster than I can. So I tell the AI, “please rewrite the following to use bind variables,” and give it the code.

Another example is working we legacy shell scripts. My sed/awk skills are rusty, but I can give a convoluted shell statement to ChatGPT, and it will patiently explain all the options and exactly how it works.

Many of your programmers are already playing with ChatGPT, GitHub Copilot, and other AI tools. You might as well embrace it. Set up a knowledge-sharing community for those curious about how AI can help IT. Have them present to you and the rest of the IT department. You’ll be amazed if you haven’t played with ChatGPT and its ilk.

If you Think There are no IT Workers to be had, Look Again

IT Leadership, , , , , ,

There is no shortage of IT workers. But there is a shortage of workers like the ones you have already. That is, young, white, and male. We are making some progress against racism and sexism in hiring, but ageism still seems to be a hidden bias.

While the IT industry is lamenting that they can’t find the people they need, I know many people in my age segment (50+) who have been laid off and can’t find their next job in IT. You need to cast your net wider if you are short of IT professionals. They are out there.

Blockchain is Still a Solution Looking for a Problem

IT Leadership, Technology That Fits, ,

It turns out nobody wanted a blockchain solution. There are still crypto enthusiasts hodling their Bitcoin, but enterprise blockchain was a solution in search of a problem.

I did believe Danish shipping giant Maersk Lines and IBM had found a place where it made sense to build something blockchain-based when they announced their TradeLens platform. The idea was that all the many, many people involved in shipping a container of plastic bric-a-brac from Shenzen to Long Beach would all put their information on a blockchain. That would provide an immutable history of everything about that container.

After IBM closed down its entire blockchain business earlier this year, it was a matter of time before Maersk pulled the plug. Today, they admitted that “TradeLens did not reach commercial viability,” and the project is officially dead.

I believe a land register in a corrupt country somewhere was also planning to use blockchain, but it’s been a while since I last heard about it. In all likelihood, the existing corrupt businessmen and politicians have killed it.

If you know of any successful enterprise blockchain project, I would love to hear about it.

You Don’t Have to Move Just Because You’re Ready

IT Leadership, Technology That Fits, ,

I was worried when I saw Denmark ranked no. 4 in “The Global Cloud Ecosystem Index 2022.” I was afraid that we had somehow stumbled into the cloud trap without my noticing. But it turns out the index is not about actual cloud adoption, only cloud readiness.

Being ready for the cloud means having affordable, fast internet connections, digital public services, data protection regulations, and a well-educated workforce. I’m all for that.

But the fact that we can doesn’t mean we should. Just like the fact that you could move some of your services to the cloud is not an argument for doing it. There are some systems where there is a sound business case for moving to the cloud. But for most existing systems, attempting to move to the cloud destroys value.

Good Intentions are not Enough

IT Leadership, Technology That Fits, , , ,

“We have the ambition to test disaster recovery twice a year.” That’s not something anybody in a professional IT organization would say, is it? Ambition? I have the ambition to create a spam- and hate-speech-free Twitter alternative powered by unicorns and rainbows, but unless I act on my ambition, nothing will happen.

Nevertheless, critical Danish infrastructure was operated on that principle. The common login system that everything from banks to tax authorities to municipalities uses is operated by a company called Nets. They apparently got to write their contract with the state themselves because it contains the ridiculous “ambition” instead of an actual requirement.

They did run a test on May 28, 2020. They did not run a test in November 2020, as was their ambition. Nor in May or November 2021. Not even in May 2022 did they test it. So when they crashed the system in June 2022 due to undocumented changes and other unprofessional shenanigans, the disaster recovery unsurprisingly failed.

Please tell everyone this story. When you are done laughing at the incompetence of central Danish authorities and their vendors, make sure you are testing your own disaster recovery…

Do you Need People to Run Your Systems?

IT Leadership, ,

If everybody in IT left, would your software systems still run? Of course they would. Any professional IT organization strives for hands-off, lights-out operation.

In the short term, a running system should not need any human intervention. It should automatically allocate more disk space and apply routine vendor patches. If you have a variable workload, your system should auto-scale or auto-throttle. User provisioning should be automated, as should routine password resets. System privileges should automatically follow the organizational role of an individual.

In the medium term, however, an unattended system will collapse. There will be emergency security patches that need manual attention. There will be changes in APIs you depend upon.

It remains to be seen if Elon Musk has retained enough talent to stave off the medium-term collapse of Twitter. How about you? Do you have the talent you need to maintain all your systems? Or are some of them left totally unattended, waiting for an implosion?