Sten Vesterli's Blog

Making IT Live up to its Promise

Category: IT Leadership

We are Still Building our IT on Shaky Ground

IT Leadership,

Once again, researchers have demonstrated the shaky foundation under our IT infrastructure. A lot of modern code is being built with Node.js, making use of Node Package Manager (npm) to pull in libraries that your code depends on.

There is no evidence that evil Russian hackers built npm. But if it didn’t exist, it would be a priority for the cyber-warfare command of our adversaries to build something like it and tempt us to use it.

The problem is that it is easy to use npm wrong, and hard to use it right. We’ve already seen many cases where organizations simply pull the latest packages from npm when they build. That means that as soon as the central package in the npm repository is corrupted or taken down, that failure will ripple through many layers of code that depend on it.

The latest discovery is that 8,000 packages have maintainers with an expired email domain. That allows any hacker to purchase that domain, re-create the maintainer email and take over the package.

Ask your CISO or other security function how your IT organization makes sure that every project only pulls npm packages from your official repository of security-vetted packages.

Reliability Engineering

IT Leadership, Technology That Fits

Coinbase just spent 16 million dollars on a 30-second Superbowl ad. It seems like the ad worked, because their website was promptly overwhelmed with traffic and crashed. Maybe they should have spent a bit on resilient network infrastructure as well.

The problem with many of the IT infrastructures I see is that they are brittle. Each component can be resilient with load balancing and database failover without making the overall system robust. Reliability engineering is a cross-domain discipline, and it is not enough that each team builds robustness into their little piece of the total landscape. Who is responsible for the overall reliability of your systems?

The Horror of not Testing

IT Leadership, Technology That Fits

In the classic 1983 John Carpenter horror movie “Christine,” the radio on the possessed 1958 Plymouth Fury can only play old rock and roll stations. Owners of 2016 Mazdas in Washington State now have the same experience. They don’t even get rock’n’roll but are instead forced to endure NPR.

Their cars are not possessed by evil spirits but suffer from a software bug. It turns out that the local NPR station sent out “now playing” album images without a .jpg extension. That was enough to send the radio and navigation unit into an endless loop, making it impossible to use navigation or Bluetooth – or change the station. Embarrassed, Mazda is offering a free replacement of the $1,500 connectivity master unit.

This incident illustrates the dangers of casual testing. A professional tester would have sent the unit all kinds of corrupted or misnamed files, files with zero length, and very large files. That would have uncovered the bug. Do you have testing professionals on your teams? If you let developers test their own software, you’ll end up where Mazda is – or worse.

Don’t Embarrass Yourself

IT Leadership, Technology That Fits,

Face recognition is a serious intrusion of privacy. The American tax authorities thought they could force it on the restive public, but was proven wrong.

The plan was to use controversial identification service ID.me. This requires users to submit scans of driver’s licenses, copies of utility or insurance bills, and to provide a live video feed of their face. Some states are using it to fight benefit fraud, but forcing it on everyone was a bridge too far. After a public outcry (and the news that Washington State just lost data on millions of citizens), this plan has been shelved.

This was a stupid idea from the outset. Beguiled by fast-talking sales people, officials lulled themselves into thinking this made sense. If they had someone on the team whose job it is to provide contrarian advice, this would never have happened. How do you ensure you get contrarian feedback before you embarrass yourselv before the entire nation?

Google Just Challenged You

IT Leadership, Technology That Fits, , ,

Google just challenged your IT organization. They created a free version of their Workspace plan where users get collaboration spaces, chat, video conferencing, and the usual Google programs Sheets, Slides, and Docs.

This dramatically increases the risk that people in your organization will create a free Google Workspace Essentials account and run their projects from there. That means all your data is under the control of Google instead of you. If the person who set up the Workspace forgets to appoint another administrator and leaves the company, your data is stuck on Google servers with no option to apply the corporate data governance.

To face this challenge, you need a stick and a carrot. The stick is an official policy prohibiting unauthorized collaboration spaces on third-party servers. The carrot is officially approved collaboration software with great usability. It’s easy to create the stick, but it doesn’t work without the carrot. Do you have the carrot?

Pay attention to the rules

IT Leadership, Technology That Fits, , , , ,

It’s probably time to start paying attention to the rules. Inspired by the Silicon Valley ethos of moving fast and breaking things, many organizations have been rolling out technology without much concern for existing rules and regulations.

Uber, Airbnb, and the myriad e-scooter startups are on the back foot all over Europe as the state reasserts its authority. Even in the U.S., regulators have started to put their foot down. Tesla is having to reprogram 50,000 vehicles that were intentionally programmed to disrespect stop signs. If the car was driving slowly and couldn’t see anybody else around an intersection, it would ignore the stop sign and continue into the intersection. That’s illegal, but humans do it all the time. It turns out authorities were less than thrilled to see bad human behavior programmed into Tesla’s cars.

We have rules for a reason. Some of them are ridiculous (like the ubiquitous cooking consent), but good citizenship includes adhering to the rules until you can persuade the rule-maker to change them. Don’t be like Tesla.

Engineering a Crisis

IT Leadership, Technology That Fits, ,

After imposing a loss of several hundred million dollars on airlines and annoying millions of passengers, the FAA has now stopped its publicity stunt. 90% of U.S. aircraft are now cleared to perform instrument landings even at airports near 5G towers.

They could have done this any time in the two years since the 5G licenses were awarded. However, quietly doing their job was not on the FAA’s agenda. After their failures led to hundreds of deaths in the Boeing 737-MAX8 disasters, they wanted to prove that they now take their job insuring safety seriously. They, therefore, engineered a crisis that put them on the front pages of newspapers nationwide before eventually doing what they should have done more than a year ago.

Don’t let corporate image considerations lead you to fail your customers. In short, don’t be like the FAA.

Do you have control over the libraries that go into you projects?

IT Leadership, , ,

Yet again, a rogue developer took down thousands of applications that depended on his library. Unhappy with the fact that open source developers work for free and companies use open source to make lots of money, he deliberately broke the faker.js and colors.js NPM libraries.

Interestingly, the more than 20,000 projects that depend on these two libraries download them almost 30 million times per week. That means a lot of projects are downloading the code from the NPM repository for every build.

In a professional IT organization, all your projects don’t just pull the latest version, they pull a specific version. And you don’t pull straight from the internet, but from the “blessed repository” with the officially approved version of everything. Are you sure you don’t have projects that just pull the latest libraries down from wherever?

Don’t Trust Phones

IT Leadership

Don’t bring your personal devices to China. The Olympic committees of several European countries are issuing burner devices to their athletes and strongly discouraging them from bringing their personal devices to next month’s Winter Olympics in Beijing. That has long been an established practice for some Western companies sending employees to China.

If you attend the annual Black Hat hacker conference in Las Vegas in person, you probably also shouldn’t take your personal device.

Since personal devices are often the second part of the two-factor authentication required to access your network, you need to establish rules about where those devices can or cannot go. Or better still, get hardware tokens and don’t trust smartphones at all.

Don’t Ignore Bad News

IT Leadership, ,

You should not create products that kill people. And when you find that you’ve accidentally done so, you should not continue to sell them.

After initially refusing the strong recommendation from the US Consumer Product Safety Commission, Peloton has now stopped sales and started offering refunds. Their treadmill product has injured 72 children and pets and has tragically caused one death. This product is so much more dangerous than other treadmills because it has to look cool. Peloton didn’t like the look of a regular treadmill that has skirts to prevent children and pets from getting pulled under the machine. Instead, they created a dangerous object and wrote in the manual to keep children and pets away.

When you have invested a lot of time and money and built something you are proud of, you don’t want to hear that it doesn’t work. That’s why Peloton didn’t immediately recall their deadly treadmills. That’s Kryptonite was in denial for months even after it was proven that their expensive bicycle locks could be opened with a Bic pen.

As a CIO or CTO, you need to have someone who can talk to you honestly about the problems with your IT systems and products. It is your leadership decision what to do about it. But you can’t make a good decision if you don’t know there is a problem. That’s why my customers get me to help them evaluate systems that don’t provide the expected business benefit.