Sten Vesterli's Blog

Don’t Ignore Bad News

You should not create products that kill people. And when you find that you’ve accidentally done so, you should not continue to sell them.

After initially refusing the strong recommendation from the US Consumer Product Safety Commission, Peloton has now stopped sales and started offering refunds. Their treadmill product has injured 72 children and pets and has tragically caused one death. This product is so much more dangerous than other treadmills because it has to look cool. Peloton didn’t like the look of a regular treadmill that has skirts to prevent children and pets from getting pulled under the machine. Instead, they created a dangerous object and wrote in the manual to keep children and pets away.

When you have invested a lot of time and money and built something you are proud of, you don’t want to hear that it doesn’t work. That’s why Peloton didn’t immediately recall their deadly treadmills. That’s Kryptonite was in denial for months even after it was proven that their expensive bicycle locks could be opened with a Bic pen.

As a CIO or CTO, you need to have someone who can talk to you honestly about the problems with your IT systems and products. It is your leadership decision what to do about it. But you can’t make a good decision if you don’t know there is a problem. That’s why my customers get me to help them evaluate systems that don’t provide the expected business benefit.

Think About the End at the Beginning

Your risk of getting hit by space debris just went up. The Chinese have launched the first module of their space station. Like last time, they have left their launch booster in uncontrolled orbit. Other nations plan a controlled deorbit so they can splash their used rockets in the sea. Private companies reuse them. The Chinese just lets it hit whereever.

All object have a lifecycle. In modern production, manufacturers are starting to think about how to ensure that as much as possible of products can be reused, recycled, or disposed of safely. In IT, we’re not good at thinking about end-of-life. That’s why we have decades-old mainframe systems that we can’t figure out how to get rid of.

As a CIO or CTO, next time you greenlight a new system, ask the architects and designers how they plan to decommission it. How will useful data be extracted from the system? Will historic data need to be saved? How will the business logic be extracted and reused many years into the future? The system works to spec now, but in less than a month, the system and the documentation will have diverged. Think about the end at the beginning. Don’t be like China and leave it to chance.

Are you a Manager or a Leader?

Basecamp lost a third of their employees after management put the foot down hard on political and diversity discussions. Coinbase got of lighter, losing only five percent when they implemented a “no politics” rule.

You might agree or disagree with the rules that management have imposed at these companies. But they do show something rare in the IT industry: Leadership.

Managers make sure that jobs are filled, projects are staffed, software is released, bugs are fixed, and time sheets are filled in. Leaders set direction for the company. Because top IT specialists are in short supply and can have a very large impact on a project or a company, they know they are valuable. That encourages them to speak their mind freely, on IT matters and other important issues on their mind. That can turn into heated political arguments, or even suppression of other opinions.

It is a leadership task to create a productive environment where each employee can make a meaningful contribution. The leader must make sure everybody gets heard, and people with unpopular opinions are not bullied. Getting that balance right is hard, and will look very different in different organizations, countries and cultures. But leadership is a necessary precondition for creating a high-performing IT organization.

As a CIO or CTO, are leading your organization or just managing it?

Consider the Failure Scenarios

The cable snapped, and a 25-tonne undersea mining vehicle is now stuck at the bottom of the Pacific Ocean.

Having one cable is the proverbial “single point of failure.” Just like in IT, it might not make business sense to pay the extra cost for full redundance. But in a professional IT organization, somebody has examined the failure scenarios. If the database server crashes, we might lose this much data, and we will restore operations in this way.

Sending a robot to the bottom of the ocean without implementing a feature that allows it to autonomously return to the surface seems like an over-optimistic strategy. Do you allow similar unwarranted optimism in your IT organization?

Article from BBC: https://www.bbc.com/news/science-environment-56921773

Public Incompetence

California shows why you don’t want to entrust your government with more data than it absolutely needs. California is building a database with all donations to political and non-profit organizations. Free-speech advocates from a Charles Koch foundation to the NAACP are against it, and the Supreme Court will soon rule whether this is okay. As a European, I don’t much care either way.

The IT security part is interesting, though. It is fairly sensitive information whether someone dontated to Donald Trump or Black Lives Matter. California of course promises to protect it, but they have built a website where you can simply change the URL to get access to every donation record in the entire database.

That is mind-boggling incompetence. I am not blaming the individual developer, though an experienced lead programmer could have spotted this glaring vulnerability. But I am blaming the IT leaders in the organization that have failed to put any kind of security review in place, even for highly sensitive data. That is a firing offense in my book. Maybe a competent CIO should run against Governor Newsom in the California recall election.

Are you Trusted?

Trust is a fragile thing. The University of Minnesota (UMN) broke it and are now struggling to contain the fallout. How do you ensure your IT organization remains trustworthy?

UMN researchers submitted a Linux patch containing a known bug in order to see whether the Linux community would spot the problem. This kind of uninvited security testing is ethically a gray area and is frowned upon by most IT professionals. The researchers wrote a paper with their findings and then submitted more buggy patches.

The Linux community was not amused by being experimented on, and decided to block all further contributions from the University of Minnesota. Furthermore, they decided to rip out several hundred past contributions to the Linux kernel by the UMN because they no longer trusted them. University leaders are belatedly scrambling to apologize, but the community has rejected their apology.

As a CIO or CTO, do you know if your users trust the systems you roll out? One unhelpful supporter unable to explain a data discrepancy can make the entire Finance department lose trust in your expensive Business Intelligence dashboard. IT only creates business value if it is used and trusted. Make sure you measure trust.

Are your AI Projects Legal?

Because the IT industry has failed to agree on any meaningful guidelines for AI usage, regulators are now stepping in. In order to get the attention of the global giants, the proposed EU regulation is threatening with GDPR-style fines of up to 6% of global sales. The rules outlaw some usage, like real-time facial recognition, and place strict limits on other uses. For “high-risk” use by police and courts, companies must provide risk assessment and documentation of how the system comes to its recommendations.

In the US, the Federal Trade Commission has also just weighed in. In a blog post, they clarified that selling or using biased AI might constitute “unfair or deceptive practice” and be subject to fines.

As a CIO or CTO, check who is responsible for ensuring your AI projects adhere to all relevant regulations. Each individual project cannot be responsible for keeping up with rapidly developing global regulations. If you have not appointed someone to keep watch over your AI project, the blame will end on your desk when your organization is found to violate AI regulations you weren’t even aware of.

Beware of Un-updatable Devices

A hundred million IoT devices are open to hacking. It turns out there is a whole slew of flaws in four different basic TCP/IP implementations. Since many IoT devices don’t have auto-update capabililty, and many don’t have updatable firmware at all, all of these devices are simply waiting to be subverted by hackers.

In order news, a startup has produced an autonomous robot that drives around the farmer’s field all by itself, zapping what it considers weeds with lasers. What could possibly go wrong?

If you are deploying any IoT technology, consider carefully how the devices will be updated with new software. Parts of the IoT industry have a sell-and-forget mindset, and that will embed ticking timebombs in your infrastructure.

The Intern Did It!

The intern did it! Solarwinds’ new CEO just added another top contender to the pantheon of bad excuses. This one is right up there with “the dog ate my homework” and is destined to become an instant classic.

Testifying before a U.S. Congressional Committee, Solarwinds came out looking like bungling amateurs. First, they had a system that allowed a password like solarwinds123. Second, they had an externally accessible system where that password worked. Third, they didn’t do anything about it when security researchers pointed it out. Fourth, they try to pin the blame on an intern that created that password.

As a CIO, you can either isolate your public-facing systems completely from the internal ones, and allow username/password access. Or you can use two-factor authentication or other additional security. The time when you could secure a non-trivial, externally-facing system with just a username and password are long gone.

Beware of Data Collectors

The days of untrammelled data collection “for the common good” might be ending. At least the watchdogs are yapping a little louder, as a new court case in London shows. The British National Health Service apparently has a contract with secretive American data analyst company Palantir, and they are being sued by a private privacy watchdog.

I would probably not choose to name my company after the device with which the dark lord Sauron subverted the wizard Saruman. However, the Lord of the Rings reference is obviously lost on investors, who value Palanatir at around $50 billion even though they have yet to make money.

As a CIO, you need to know where your data goes. If you have third parties analyzing them, you need to have someone make an independent assessment of whether you can expect your data to be safe with them. You need to balance the reward against the reputational risk if you are found cooperating with shady operators.