Sten Vesterli's Blog

Making IT Live up to its Promise

Category: IT Leadership

Another Large IT Project Failure – and How it Could Have Been Avoided

IT Leadership, ,

The City of Birmingham can be added to the long list of organizations that went bankrupt trying to replace their ERP system. They were running a heavily customized SAP system and tried to implement Oracle Fusion. As often happens in this kind of project, the costs exploded from the initial estimate of $25 million to $125 million by the last count. They are not done yet, and since they’ve stopped paying their bills, they might never be.

When you are faced with a legacy system no longer fit for purpose, don’t fall prey to the dangerous illusion that you can run one large project to replace it. A project is a collaborative enterprise intended to reach a well-defined goal. But for a large IT project, the project duration alone (four years and counting in Birmingham) ensures that the goalposts will have moved several times before you are done. Your Program Manager is not likely to be among the few hundred people in the world with the exceptional project and change management skills needed to pull off such a project.

A series of smaller projects to carve out and replace functionality in smaller chunks does not promise to solve all your problems in one fell swoop. But it has a much higher chance of success.

Would You Notice the Quality of Your AI Dropping?

IT Leadership, ,

You know that ChatGPT is getting more politically correct. But did you know that it is also getting dumber? Researchers have repeatedly been asking it to do tasks like generating code to solve math problems. In March, ChatGPT 4 could generate functioning code 50% of the time. By June, that ability had dropped to 10%. If you’re not paying, you are stuck with ChatGPT 3.5. This version managed 20% correct code in March but was down to approximately zero ability in June 2023.

This phenomenon is known to AI researchers as “drift.” It happens when you don’t like the answers the machine gives, and take the shortcut of tweaking the parameters instead of expensively re-training your model on a more appropriate data set. Twisting the arm of an AI to generate more socially acceptable answers has been proven to have unpredictable and sometimes negative consequences.

If you are using any AI-based services, do you know what the engine behind the solution is? If you ask, and your vendor is willing to tell you, you will find that most SaaS AI solutions today are running ChatGPT with a thin veneer of fine-tuning. Unless you continually test your AI solution with a suite of standard tests, you will never notice that the quality of your AI solution has gone down the drain because OpenAI engineers are pursuing the goal of not offending anyone.

Do Your Employees Follow your AI Guidelines?

IT Leadership, , ,

Unless you override it, your organization’s policy for AI-driven tools is “anything goes.” That’s because your developers want to get their job done as quickly as possible. If that involves having Github Copilot write part of the code or copying a code block into ChatGPT for debugging help, so be it.

If you don’t have secrets, maybe that’s fine with you. But even though OpenAI is not training ChatGPT on user prompts, they have not been very diligent about keeping them safe. You should assume that everything your developers paste into ChatGPT will eventually leak.

That includes your data. AI tools are very good at data cleaning and visualization. Your Data Scientists are surely pasting data into ChatGPT and getting back fully functional Python code to run in a Jupyter Notebook. Unless you tell them not to.

If I asked one of your developers or Data Scientists about your policy on AI tools, would they know it? And would they follow the rules or would they take the 10x or 100x productivity boost?

How Do We Make IT Projects More Successful?

IT Leadership, ,

At least nuclear waste storage is worse. In his book “How Big Things Get Done,” professor Bent Flyvbjerg ranks 25 categories of projects by their average cost overrun. IT projects are the fifth worst offender, better than nuclear but worse than buildings, rail, airports, tunnels, and many others. We all know many public IT failures (Denmark has its fair share), and the private sector has suffered many more, even if less publicized.

What can we do about it? One chapter in the book is dedicated to creating better estimates. The problem with our estimating today is that we treat every project as unique. We then estimate each bit, and our usual how-hard-can-it-be optimism leads to the underestimation so common in IT. Flyvbjerg argues that we should start by identifying the class of projects this new project belongs to. The average for this class of projects is then the starting point for our estimate, adjusted up or down.

For example, you estimate an ERP project by looking at other ERP projects. If the cost in your industry is $20 million on average, that is your initial value. Then adjust up or down depending on whether your project is smaller or larger – or more straightforward or more complex – than the members of the reference class.

Bring this book with you to the beach this summer so that you can help our industry move forward when you return from vacation. IT projects exceed their budgets by an average of 73%. We can do better.

AI Will Not Destroy Humanity

IT Leadership, Technology That Fits, , ,

AI doesn’t pose an extinction risk. And it has already created brand new jobs in the catastrophizing industry.

The only reason AI industry leaders like Sam Altman and Demis Hassabis jump on that bandwagon is to encourage more government red tape. If you are a powerful incumbent, asking for as many constraints to your industry as possible makes sense. The EU, ever happy to regulate industries originating elsewhere, is delighted to oblige. With compliance departments of thousands, these massive organizations can handle any amount of regulation thrown at them. But a lean startup will get regulated out of business.

The most fascinating part of AI is local, small-scale AI. We currently have massive, centralized AI running in enormous data centers. But since LLaMA escaped from the Facebook lab, tinkerers and hobbyists have already built Large Language Models on their local computers. But, of course, OpenAI, Microsoft, and Google would like small competitors to be regulated away.

Did You Hear the One About the Gullible Lawyer?

IT Leadership, Technology That Fits, , ,

You need the best arguments to win a discussion, get a project approved, or win a court case. But, if you are short of preparation time, you might take a shortcut like the New York Lawyer who asked ChatGPT for help.

Ever willing to help, ChatGPT offered six cases supporting the lawyer’s argument. Unfortunately, they were entirely made up. That might work if you write a marketing blog post, but it does not hold up in court. The gullible lawyer claims he did not know that ChatGPT might be hallucinating but is, of course, facing sanctions for lying to the court.

IT professionals know that ChatGPT cannot be trusted to answer truthfully. It is not much of a problem for a programmer because the compiler or the unit tests will catch defective answers. But the rest of the world doesn’t know.

Now is the time to remind everyone in the organization of your company policy on using ChatGPT and its ilk (you do have such a policy, right?). Tell the story of the gullible New York lawyer to make the point clear.

Does it Pay to Move to the Cloud? Or Back?

IT Leadership, ,

Most organizations that decide to move workloads to the cloud are missing a crucial piece of information: What it costs to run the system on-premise. In a viral blog post, David Heinemeier Hansson shared his specific calculations for Basecamp and HEY. Moving back from the cloud makes perfect business sense for him. Of course, your calculation will be different, but unless you know what it costs to run on-premise, you are comparing an uncertain cloud cost with a completely unknown on-premise cost.

As a CIO, you are expected to make sound business decisions. You can only do that if you have both numbers.

Offering Alternatives

IT Leadership, ,

Are you building critical software? Then you know to offer a fallback option if something – despite all your testing – does not work. That is often not a concern in organizations that can simply force users to suffer their app. Like the public sector in Denmark, where every parent of a schoolchild in Denmark must use the “Aula” app. Unfortunately, a botched upgrade means that many cannot log in.

Having only smartphone apps makes you vulnerable. The app stores do not older versions, so once you have rolled out a defective version, you (and your users) are up the creek. The mitigation for this risk is to also offer a responsive web application with only the most crucial features.

Take a look at the smartphone apps your organization offers to its customers. Are any of them critical? If so, do you have an alternative ready?

In Praise of (Useful) Managers

IT Leadership, , , , ,

You do need some managers. Elon Musk is trying to prove that Twitter can be run with only himself and the people who write code, and it’s not going well. It turns out that it takes a little more to run an organization than just coding and tweeting.

For example, Elon had announced that only enterprise customers who would pay $$$ would have access to the API. But he had fired everyone who was able to process an application for an enterprise license. So when the last overworked API engineer committed the change that implemented the limit, there were no paying customers because there was nobody to take the money of the few tool vendors willing to pay up.

Your overhead grows inexorably. Unless you pay very close attention, the fraction of total headcount actually writing code goes lower and lower. To avoid ending up having to take a chainsaw to your organization as Elon has done, calculate your coder percentage today and keep track of it.

Cloud Means Aomeone Else is in Control

IT Leadership, Technology That Fits, ,

Cloud services mean you are at the mercy of someone else. It is bad enough that hackers broke into Western Digital’s My Cloud service and encrypted their customer’s data. But many private customers are now learning what it means to use WD’s cloud-based login service. It means that even though your data is stored on your own NAS device in your own basement, you still cannot get at it when WD is down.

If you are using any cloud-based login service in your organization, ask your CISO how people would log in and access ressources if that service is down.