Sten Vesterli's Blog

Making IT Live up to its Promise

Tag: Processes

How Bank Customer Data Ended up at Auction

Uncategorized, ,

“But I thought the drives were encrypted!”
“Only if you turn encryption on”

An American bank got off lightly, getting only a $35 million fine. For five years, they simply hired a moving company to get rid of old computers. That company then sold Morgan Stanley’s used hard drives at auction. Too bad that the drives still included information about 15 million customers. The drives did contain an encryption feature, but nobody turned it on…

The whole debacle came to light accidentally when an IT consultant bought a used hard drive for backup and discovered it was full of confidential data.

Organizations keep losing data. It costs money and reputation each time. You must address this problem with employee security awareness, internal procedures, and external security audits. Tell the story of Morgan Stanley at your next IT meeting. It just might remind someone of something they really ought to fix…

The Wrong Way to Use Open Source

IT Leadership, Technology That Fits, ,

The glass is less than half full, and that is worrying. One of the focus areas in the 2022 State of Data Science report was enterprise adoption of open source. On every important metric, less than half the respondents gave the answer I was hoping for. For example, “We use a managed repository” got 43%, and “We use a vulnerability scanner” got 36%.

With such a low security maturity, it is obvious that the Log4j debacle and recent hacktivism has dented open source adoption. 40% report scaling back the use of open source in the past year.

Open Source gives you transparency that proprietary software doesn’t. That means you have the ability to verify that it works as promised instead of simply trusting vendor promises. But if you don’t make use of this transparency and instead simply download something because it’s free, you are setting yourself up for problems. Is your organization found in the more than half-empty part of the glass?

How to Avoid Hardwired Credentials

IT Leadership, Technology That Fits, ,

“We’ll add security later,” the developers said.

“We have to ship,” the manager said.

And that’s how 1000s of apps came to be released with hardcoded AWS passwords. That means that hackers can take these credentials and get full access to the AWS back end behind the apps, including reading and changing data for all users. The large number of vulnerable apps does not indicate that thousands of developers have hardcoded credentials. It shows that a smaller number hardcoded AWS credentials into their libraries, and thousands of developers uncritically used these libraries.

We see this kind of IT library supply chain problem all the time. It is much less likely to happen if you have a security QA gate in your workflow. Every system needs to have a security review signed off by an internal employee. It doesn’t help to contract this out. Your security consultant will be long gone when the vulnerability comes to light. The review has to be done by someone whose job is on the line.

Thanks to Kim Berg Hansen for pointing me to this story.

The Antidote to Value-Destroying Vanity Projects

IT Leadership, Technology That Fits, , ,

Choosing the solution that is 100 times more expensive sounds absurd. Nevertheless, that is what the U.S. Congress has decided, which is why NASA is struggling to get its SLS rocket off the ground. It has nothing to do with putting a man on the moon and everything to do with keeping the big NASA factory in Alabama running. They don’t have the technology to compete with newer space companies like SpaceX but are cobbling together old Space Shuttle parts. They’ve actually been going around to museums in the U.S., taking out old space shuttle engines for the quixotic project foisted upon them by Alabama Senator Richard Shelby.

Some organizations face a similar challenge: The CEO or someone else in senior leadership has an idea for some technology, and IT is ordered to deliver it. It doesn’t matter if such a vanity project is practical, feasible, or cost-efficient. You cannot fight this kind of project individually because they are highly connected with the ego of one individual.

The solution is to establish a standard evaluation process for technology projects. Every project needs a business owner responsible for calculating the business benefit. Every project also has a technical owner responsible for calculating the cost, including the ongoing running costs after completion. If the benefit comfortably exceeds the cost, the project is qualified to enter the competition with other claims on company investment.

You might not have such a process because a rational decision might also kill some of the IT department’s most beloved resumé-enhancing projects…

Somebody Else’s Problem

IT Leadership, Technology That Fits, ,

Things that are Somebody Else’s Problem (SEP) are invisible. Douglas Adams famously joked about this in “The Hitchhiker’s Guide to the Galaxy,” but the effect is serious and real.

For example, local British politicians were falling over each other trying to attract data centers. They were focusing on the cachet of having Google or Facebook in their town, and the half-dozen jobs for the electricians and plumbers maintaining them. Supplying these energy-hungry behemoths with power was Somebody Else’s Problem.

Now they have so many data centers in West London that their electrical grid is overloaded, and they won’t be able to build more housing until they have upgraded their main cables. That’ll be sometime in the 2030s.

As an IT leader, it is your job to ensure that each team knows the problems they might cause for other parts of the organization.

Documentation is Unnecessary Until You Need It

IT Leadership,

If you have a fire in your server room, your insurance pays out. Insurance is expensive, but a necessary part of your risk management strategy. For many risks, there is a way to get almost free insurance. Yet few people take it. I am talking about documentation.

A chocolate factory in Belgium didn’t follow its own processes and did not document its production. When kids started falling sick with salmonella all over Europe, suspicion fell on the Kinder egg factory in Arlon. The authorities asked for the production documentation. Because the factory couldn’t provide it, the whole plant was shut down. If they had had documentation, they would have been insured against this risk. They could have shut down just one production line instead of the whole plant.

So the reason you might not be able to get chocolate eggs this Easter is bad documentation.

Are You Monitoring Your Automated Systems?

IT Leadership, Technology That Fits, , ,

It is hard to anticipate the real world. I’m sure the wet concrete on the road in Japan looked just like solid ground to the delivery robot. Consequently, it happily trundled into the urban swamp and got stuck. The story does not report whether the delivery company managed to get their robot out before the concrete hardened…

This is why you need careful monitoring of all the fully automated systems you are deploying. The first line of defense is automated metrics and their normal interval. For a delivery robot, the distance covered over a minute should be greater than zero and less than 270 (if you have limited the robot to e.g. 10 mph). The second line of defense consists of humans who will evaluate the alarms and take appropriate action. The third line of defense are developers who will fix the software and the alarms.

Too many automated systems are simply unleashed and depend on customers to detect that something is wrong and complain. You want to figure out you have a problem before the image of your robot encased in concrete starts trending on Twitter.

Shooting the messenger

IT Leadership,

Even though the clueless Governor of Missouri tried to shoot the messenger, he missed. Last year, a reporter published his findings that private data on more than 100,000 teachers was available to anyone who knew how to click “View Source” on a web page. The Governor held a widely-ridiculed press conference where he vowed to prosecute the “hackers” who had told the world about the incompetence of the state IT department.

A thorough report by law enforcement now roundly exonerates the journalist. It also exposes that personal information on more than half a million people had been available for a decade to anyone who care to look.

Even professional IT organizations occasionally fail like the state of Missouri did here. You have a little simple system, you are under schedule pressure, and you forgot to book time with the security team. So you roll it out without a security review. The antidote to this is to maintain a complete systems inventory with a field for the name and email of the person who did the security review. That will show you if this step got skipped, and allow you to quickly ask questions about any alleged security issues before you start shooting at the messenger.

We are Still Building our IT on Shaky Ground

IT Leadership,

Once again, researchers have demonstrated the shaky foundation under our IT infrastructure. A lot of modern code is being built with Node.js, making use of Node Package Manager (npm) to pull in libraries that your code depends on.

There is no evidence that evil Russian hackers built npm. But if it didn’t exist, it would be a priority for the cyber-warfare command of our adversaries to build something like it and tempt us to use it.

The problem is that it is easy to use npm wrong, and hard to use it right. We’ve already seen many cases where organizations simply pull the latest packages from npm when they build. That means that as soon as the central package in the npm repository is corrupted or taken down, that failure will ripple through many layers of code that depend on it.

The latest discovery is that 8,000 packages have maintainers with an expired email domain. That allows any hacker to purchase that domain, re-create the maintainer email and take over the package.

Ask your CISO or other security function how your IT organization makes sure that every project only pulls npm packages from your official repository of security-vetted packages.

Do you have control over the libraries that go into you projects?

IT Leadership, , ,

Yet again, a rogue developer took down thousands of applications that depended on his library. Unhappy with the fact that open source developers work for free and companies use open source to make lots of money, he deliberately broke the faker.js and colors.js NPM libraries.

Interestingly, the more than 20,000 projects that depend on these two libraries download them almost 30 million times per week. That means a lot of projects are downloading the code from the NPM repository for every build.

In a professional IT organization, all your projects don’t just pull the latest version, they pull a specific version. And you don’t pull straight from the internet, but from the “blessed repository” with the officially approved version of everything. Are you sure you don’t have projects that just pull the latest libraries down from wherever?