Sten Vesterli's Blog

Making IT Live up to its Promise

Tag: Processes

Did You Hear the One About the Gullible Lawyer?

IT Leadership, Technology That Fits, , ,

You need the best arguments to win a discussion, get a project approved, or win a court case. But, if you are short of preparation time, you might take a shortcut like the New York Lawyer who asked ChatGPT for help.

Ever willing to help, ChatGPT offered six cases supporting the lawyer’s argument. Unfortunately, they were entirely made up. That might work if you write a marketing blog post, but it does not hold up in court. The gullible lawyer claims he did not know that ChatGPT might be hallucinating but is, of course, facing sanctions for lying to the court.

IT professionals know that ChatGPT cannot be trusted to answer truthfully. It is not much of a problem for a programmer because the compiler or the unit tests will catch defective answers. But the rest of the world doesn’t know.

Now is the time to remind everyone in the organization of your company policy on using ChatGPT and its ilk (you do have such a policy, right?). Tell the story of the gullible New York lawyer to make the point clear.

In Praise of (Useful) Managers

IT Leadership, , , , ,

You do need some managers. Elon Musk is trying to prove that Twitter can be run with only himself and the people who write code, and it’s not going well. It turns out that it takes a little more to run an organization than just coding and tweeting.

For example, Elon had announced that only enterprise customers who would pay $$$ would have access to the API. But he had fired everyone who was able to process an application for an enterprise license. So when the last overworked API engineer committed the change that implemented the limit, there were no paying customers because there was nobody to take the money of the few tool vendors willing to pay up.

Your overhead grows inexorably. Unless you pay very close attention, the fraction of total headcount actually writing code goes lower and lower. To avoid ending up having to take a chainsaw to your organization as Elon has done, calculate your coder percentage today and keep track of it.

How Bank Customer Data Ended up at Auction

Uncategorized, ,

“But I thought the drives were encrypted!”
“Only if you turn encryption on”

An American bank got off lightly, getting only a $35 million fine. For five years, they simply hired a moving company to get rid of old computers. That company then sold Morgan Stanley’s used hard drives at auction. Too bad that the drives still included information about 15 million customers. The drives did contain an encryption feature, but nobody turned it on…

The whole debacle came to light accidentally when an IT consultant bought a used hard drive for backup and discovered it was full of confidential data.

Organizations keep losing data. It costs money and reputation each time. You must address this problem with employee security awareness, internal procedures, and external security audits. Tell the story of Morgan Stanley at your next IT meeting. It just might remind someone of something they really ought to fix…

The Wrong Way to Use Open Source

IT Leadership, Technology That Fits, ,

The glass is less than half full, and that is worrying. One of the focus areas in the 2022 State of Data Science report was enterprise adoption of open source. On every important metric, less than half the respondents gave the answer I was hoping for. For example, “We use a managed repository” got 43%, and “We use a vulnerability scanner” got 36%.

With such a low security maturity, it is obvious that the Log4j debacle and recent hacktivism has dented open source adoption. 40% report scaling back the use of open source in the past year.

Open Source gives you transparency that proprietary software doesn’t. That means you have the ability to verify that it works as promised instead of simply trusting vendor promises. But if you don’t make use of this transparency and instead simply download something because it’s free, you are setting yourself up for problems. Is your organization found in the more than half-empty part of the glass?

How to Avoid Hardwired Credentials

IT Leadership, Technology That Fits, ,

“We’ll add security later,” the developers said.

“We have to ship,” the manager said.

And that’s how 1000s of apps came to be released with hardcoded AWS passwords. That means that hackers can take these credentials and get full access to the AWS back end behind the apps, including reading and changing data for all users. The large number of vulnerable apps does not indicate that thousands of developers have hardcoded credentials. It shows that a smaller number hardcoded AWS credentials into their libraries, and thousands of developers uncritically used these libraries.

We see this kind of IT library supply chain problem all the time. It is much less likely to happen if you have a security QA gate in your workflow. Every system needs to have a security review signed off by an internal employee. It doesn’t help to contract this out. Your security consultant will be long gone when the vulnerability comes to light. The review has to be done by someone whose job is on the line.

Thanks to Kim Berg Hansen for pointing me to this story.

The Antidote to Value-Destroying Vanity Projects

IT Leadership, Technology That Fits, , ,

Choosing the solution that is 100 times more expensive sounds absurd. Nevertheless, that is what the U.S. Congress has decided, which is why NASA is struggling to get its SLS rocket off the ground. It has nothing to do with putting a man on the moon and everything to do with keeping the big NASA factory in Alabama running. They don’t have the technology to compete with newer space companies like SpaceX but are cobbling together old Space Shuttle parts. They’ve actually been going around to museums in the U.S., taking out old space shuttle engines for the quixotic project foisted upon them by Alabama Senator Richard Shelby.

Some organizations face a similar challenge: The CEO or someone else in senior leadership has an idea for some technology, and IT is ordered to deliver it. It doesn’t matter if such a vanity project is practical, feasible, or cost-efficient. You cannot fight this kind of project individually because they are highly connected with the ego of one individual.

The solution is to establish a standard evaluation process for technology projects. Every project needs a business owner responsible for calculating the business benefit. Every project also has a technical owner responsible for calculating the cost, including the ongoing running costs after completion. If the benefit comfortably exceeds the cost, the project is qualified to enter the competition with other claims on company investment.

You might not have such a process because a rational decision might also kill some of the IT department’s most beloved resumé-enhancing projects…

Somebody Else’s Problem

IT Leadership, Technology That Fits, ,

Things that are Somebody Else’s Problem (SEP) are invisible. Douglas Adams famously joked about this in “The Hitchhiker’s Guide to the Galaxy,” but the effect is serious and real.

For example, local British politicians were falling over each other trying to attract data centers. They were focusing on the cachet of having Google or Facebook in their town, and the half-dozen jobs for the electricians and plumbers maintaining them. Supplying these energy-hungry behemoths with power was Somebody Else’s Problem.

Now they have so many data centers in West London that their electrical grid is overloaded, and they won’t be able to build more housing until they have upgraded their main cables. That’ll be sometime in the 2030s.

As an IT leader, it is your job to ensure that each team knows the problems they might cause for other parts of the organization.

Documentation is Unnecessary Until You Need It

IT Leadership,

If you have a fire in your server room, your insurance pays out. Insurance is expensive, but a necessary part of your risk management strategy. For many risks, there is a way to get almost free insurance. Yet few people take it. I am talking about documentation.

A chocolate factory in Belgium didn’t follow its own processes and did not document its production. When kids started falling sick with salmonella all over Europe, suspicion fell on the Kinder egg factory in Arlon. The authorities asked for the production documentation. Because the factory couldn’t provide it, the whole plant was shut down. If they had had documentation, they would have been insured against this risk. They could have shut down just one production line instead of the whole plant.

So the reason you might not be able to get chocolate eggs this Easter is bad documentation.

Are You Monitoring Your Automated Systems?

IT Leadership, Technology That Fits, , ,

It is hard to anticipate the real world. I’m sure the wet concrete on the road in Japan looked just like solid ground to the delivery robot. Consequently, it happily trundled into the urban swamp and got stuck. The story does not report whether the delivery company managed to get their robot out before the concrete hardened…

This is why you need careful monitoring of all the fully automated systems you are deploying. The first line of defense is automated metrics and their normal interval. For a delivery robot, the distance covered over a minute should be greater than zero and less than 270 (if you have limited the robot to e.g. 10 mph). The second line of defense consists of humans who will evaluate the alarms and take appropriate action. The third line of defense are developers who will fix the software and the alarms.

Too many automated systems are simply unleashed and depend on customers to detect that something is wrong and complain. You want to figure out you have a problem before the image of your robot encased in concrete starts trending on Twitter.

Shooting the messenger

IT Leadership,

Even though the clueless Governor of Missouri tried to shoot the messenger, he missed. Last year, a reporter published his findings that private data on more than 100,000 teachers was available to anyone who knew how to click “View Source” on a web page. The Governor held a widely-ridiculed press conference where he vowed to prosecute the “hackers” who had told the world about the incompetence of the state IT department.

A thorough report by law enforcement now roundly exonerates the journalist. It also exposes that personal information on more than half a million people had been available for a decade to anyone who care to look.

Even professional IT organizations occasionally fail like the state of Missouri did here. You have a little simple system, you are under schedule pressure, and you forgot to book time with the security team. So you roll it out without a security review. The antidote to this is to maintain a complete systems inventory with a field for the name and email of the person who did the security review. That will show you if this step got skipped, and allow you to quickly ask questions about any alleged security issues before you start shooting at the messenger.