Sten Vesterli's Blog

Making IT Live up to its Promise

Tag: AI

Losing the Picture

IT Leadership,

The fear of every Air Traffic Controller is “losing the picture.” That is the situation where your mental model of where everyone is and where they are going starts breaking down. It is very hard to regain the picture, and the controller will ask for help. Usually, the supervisor will split the sector and assign the relief controller to one half of it.

I have often seen IT organizations that have “lost the picture,” but there is never a relief controller available to step in. They simply continue struggling to keep the system running while the bug reports and enhancement requests pile up.

The number one reason why a team loses the picture used to be that the complexity grows until it reaches the limit of what the team can handle, and then the most experienced person leaves. That is hard enough to handle. Today, some teams are happily letting AI tools build code they don’t understand, even from the beginning. They are rapidly losing the picture, as the emerging horror stories about teams drowning in AI code clearly show.

The Missing AI Business Model

IT Leadership,

OpenAI has started running ads in ChatGPT. For now, they say they are “testing” the feature in the U.S., but there is little doubt it will eventually be rolled out globally.

Just like all the other AI companies, OpenAI is burning cash at an unsustainable rate – using more than 10x what they make on the compute they consume. Some of them will fail.

If you are using external AI tools in any of your systems, make sure your developers are plugging in the AI in a way that makes it easily replaceable. As the shakedown starts, you’ll be seeing price hikes on the paid plans as well, and you need to be able to quickly and easily change to another provider. Or run an Open Source model in-house.

Keeping Up With AI

Spiritual Programmer, ,

How are you keeping up with developments in AI? There are several major AI players releasing new versions with new capabilities every few months. They have different strengths and weaknesses, and we are all inundated with news about how AI will take away our jobs. So how can we keep up when we have a day job?

If your organization doesn’t have an AI knowledge-sharing program, establish one with colleagues or friends. Meet over beer and pizza, share your current knowledge, and assign responsibilities. Someone might have the task of keeping up with Claude Code. Someone else might be responsible for investigating Gemini CLI. Meet up regularly and informally share what you’ve found.

The AI field is too big and fast-moving for you to keep up with it on your own.

Unknown AI Policies

IT Leadership, , ,

Does everyone in your IT organization know what your rules are regarding AI use? 60% of employees report using AI tools, while fewer than 20% say they know the company’s AI policy.

That is not because the policies don’t exist. More than 80% of IT leaders report that their organizations have formulated polices for AI use.

How are you going to close that gap?

Measuring Productivity

IT Leadership,

How do you measure productivity? The research suggests that, with AI tools, perceived programmer productivity increases, while objective productivity decreases. In addition, maintainability decreases as more of your code base has never been seen by the human sent in to clean up the AI-generated messes.

That should indicate that we need to prove a massive increase in productivity to justify the use of AI tools. But how to measure it?

Lines of Code (LoC) was already a bad measurement. In the days of AI, it is a totally random measure, and a higher LoC might just as well indicate a decrease in functionality as an increase.

If you’re in the big enterprise/government world, you might use Function Points or Use Case Points. If you are running Agile, you can use team velocity (within each team). Ideally, you would measure business value. Unfortunately, few organizations can articulate and calculate the business value of their IT.

If you want to argue that AI tools increase productivity, you need to put a number on the productivity you claim to increase.

AI is not Coming for Your Job

IT Leadership,

In a real-world test, the best AI completed 1 in 40 full tasks satisfactorily. Researchers put together the Remote Labor Index, a set of 100 typical remote working tasks – data visualization, architecture drawing, game development, etc. These are the kind of tasks you would normally give to remote gig workers through an online task platform.

The results are sobering. The best AI was Manus, delivering 2.5% acceptable results. Gemini 2.5 Pro managed only 0.8% task completion.

Don’t be blinded by the fact that AI can solve a few very specific tasks well. In the real world, AI is very, very far from taking anybody’s job.

Another Place Not to Use AI Chatbots

IT Leadership, , ,

Alaska wanted an AI chatbot to give legal advice. Anybody care to guess how that went?

Yes, not well. They are now 15 months into a 3-month project, but expect it to go live this month.

I’ll make a prediction: This project will end up in the bucket of IT projects that sink without a trace, leaving only cost and no business benefit whatsoever. Sadly, one in four IT projects still end in that bucket. My hunch is that this number is increasing, as AI is incorrectly applied to more and more use cases.

If you are going to implement AI with an LLM, don’t do it in a critical application like giving legal advice to citizens. There is no way to stop an LLM from hallucinating. The only way to automate advice and be sure it is correct is with old-school expert systems.

Good AI Advice

IT Leadership, , ,

Who advises you on AI? Don’t say ChatGPT. Also, don’t take advice from random blowhards on LinkedIn. You need advice from someone who has a realistic view of your situation and your business.

As a consultant, I’m all for having external advisers to bring you an outside perspective. But it is equally important to have a well-founded inside perspective.

I recommend establishing an internal AI advisory board within IT. Appoint some people interested in AI and give them a modest time budget to keep up to date with what is happening in the field. Make it as diverse as possible – juniors, seniors, developers, sysadmins. If you are fortunate enough to have diversity in gender and ethnicity in your IT organization, also utilize that. Have your AI board meet with the CIO/IT leader regularly, and also have them present at department meetings.

Inside people are much more invested in finding AI tools that can truly help in your specific situation. They are also the ones who will suffer if you implement bad AI. That gives them a very good ability to see through exaggerated vendor claims.

The Right and the Wrong Way to Use LLMs

IT Leadership

There are two ways to use Large Language Models. One works well, the other much less well. Over the holidays, I’ve been talking to a lot of family and friends about AI, and it turns out that many people conflate these two approaches.

The way that works well is to use LLMs deductively. That means starting with a lot of text and distilling some essence or knowledge from it. Because you are only asking the AI to create something from a text you have given it, it has much less room to run off on a tangent, making stuff up. At the same time, it can show off its superhuman ability to process large amounts of text. In an IT context, this is when you give it dozens of interlinked files and ask it to explain or find inconsistencies or bugs.

The way that doesn’t work well is using LLMs inductively. That is when you ask it to produce text based on a short prompt and its large statistical model. This allows it to confabulate freely, and the results are hit-or-miss. In an IT context, this is where you ask it to write code. Sometimes it works, often it doesn’t.

Whenever you discuss LLMs with someone, set the stage by defining the inductive/deductive difference. If people already know, no harm done. If they don’t have this frame of reference, establishing it makes for much better conversations.

Learning From People, Not From Documents

IT Leadership, , ,

Implementing AI has a critical and often-overlooked problem that Raman Shah just reminded me of in another discussion: It can only learn what is documented.

When you teach an AI to perform a process in your organization, you train it on all the internal documents you have. But these describe the RAW – Rules As Written. They do not describe the RAU – Rules As Used.

It takes a lot of care to do process discovery properly. You need to have a human being interview the people doing the work to find out how business is actually done.

Work-to-rule is a classic form of industrial action where workers do exactly what they’re told in order to stop production without going on strike. If you train an AI on just your documents, you are asking it for a work-to-rule stoppage.