Are You Still Building Things That Don’t Scale Automatically?

There is no excuse for a modern system to be slow. I’m at a 5,000-people conference this week, and their official networking app is totally overloaded and almost unresponsive.

You might still have legacy systems with scalability issues, but everything you build today should be cloud-native. As a first-class citizen of the cloud, a modern app has access to automatic scaling, monitoring, robustness, and many other features.

Ask the architects building new systems in your organization about how the application will scale. If the answer is that it will scale automatically, good. If the answer is that somebody has to notice response time increasing and manually do anything, you are still building to the old paradigm.

Do You Understand What You are Running?

Don’t run systems you don’t understand. Some people had placed billions of dollars into a cryptocurrency called TerraUSD. They were told this was a “stablecoin” that would keep a value of $1. Underlying this claim was a clever algorithm that interacted with investors and another cryptocurrency in complex ways. Until its magic no longer worked and the supposedly stable TerraUSD dropped 80%. Trading in it is now halted.

In the global financial crisis of 2008, people had invested in complex financial instruments that they didn’t understand. Many billions were lost and large institutions went bankrupt. The banks who came out of the crisis unscathed were those who had stuck to simple banking products that everyone could understand.

Take a look at your IT landscape. Can you find somebody who understands your operating infrastructure? Or have generations of DevOps engineers just googled problems and tweaked your Kafka and Kubernetes configuration until it somehow seemed to work?

Why Employee Surveillance Doesn’t Work

Do you know what a “mouse jiggler” is? Your most innovative employees do. It is not a device to shake a rodent in a cage. It is a small USB device that sends random mouse movements to a computer.

Who would want such a thing? Employees subjected to tracking software, that’s who. With the mouse moving, the software will record “productivity.” The pandemic led to a boom in surveillance tech, euphemistically called “employee productivity software.” As workers return to the office, that tech is not removed from corporate laptops. But workers are pushing back, in accordance with Newton’s Third Law of IT systems: Whenever the organization implements a policy, the employees will implement an equal and opposite workaround.

Techno-optimists keep trying to replace humans with technology. There are some places where that works. Replacing human leadership with surveillance technology is one of the places where this strategy doesn’t work.

Do You Trust Amazon?

The default is no trust. You shouldn’t trust a random USB stick you pick up in the parking lot, and your customers and users don’t trust you. If you want trust, you have to be transparent in a way your users understand and appreciate.

Somewhere in the Amazon terms & conditions it probably says in illegible legalese that everything you say to your Alexa smart speaker can and will be used against you. Researchers have shown that your interactions with Alexa are reported to dozens of advertisers, and Amazon says the research is flawed. Who do you believe?

Amazon have hundreds of lawyers and are probably within the law. The problem is that they are not complying with users’ expectations. If you want any kind of goodwill from your users and customers, you have to meet their actual expectations. Hiding behind reams of legalese doesn’t cut it.

Unnecessary Complexity

Why use a proper screwdriver when you have a multi-tool? It is true that it is a lousy screwdriver, but it can do a dozen other things. That’s the thinking behind using Microsoft Windows for Point-of-Sale terminals. It turns out to be a bad idea. It can take up to 40 minutes for a Windows 11 machine to install the latest update, and in the meantime you are unable to do business.

The problem is not throwing an overpowered machine at the task. A Raspberry Pi works fine for a home weather station even though it is only using 0.01% of its capacity. The problem is adding unnecessary complexity. A Windows 11 workstation is running literally hundreds of services, 98% of which are not necessary for Point-of-Sales functionality. The more components you have, the more potential problems you will have, and the harder it will be to find them when they occur.

You would never allow your IT architects to use over-complicated components with dozens of unnecessary interactions, would you?

Vulnerability Chains

Are you sure you own your devices? Or do you just have a temporary ability to use them that could vanish any second?

Smart home enthusiasts taken by Insteon marketing found out the hard way that their devices function at the suffering of the Insteon servers. When the company abruptly shut down, users found none of their devices worked because everything depended on a connection to servers that were no longer there.

This is an example of a vulnerability chain where the Insteaon servers proved the weakest link. Every networked device has a vulnerability chain from the client endpoint through multiple network devices until it reaches the server. Are you aware of the vulnerability chain from the card readers that control access to your building? Don’t be blindsided by a risk you hadn’t even considered.

Optimization to Powerlessness

Here in Denmark, we were surprised to find that the Russians have rendered our military combat ineffective. When NATO asks what we can provide, we can offer a hundred special forces soldiers, some past-due-date antitank weapons, and an armored brigade without armor. The reason is not lack of money. We spend many millions. We just don’t spend it on things that matter.

The Russians did not have to attack us kinetically or subject us to a devastating cyber-attack to achieve this. They simply needed to infiltrate the Ministry of Defence with spreadsheet-wielding MBAs supported by a fifth column from McKinsey. We have now optimized our way to warfighting impotence.

Many organizations have similarly found that they have optimized themselves to powerlessness. A ship stuck in the Suez or a war in Ukraine will bring their entire production to a halt.

The only way to resilience, as any capable army knows, is to have extra. You have more supplies on hand than the absolute minimum, and more different suppliers than you need. You have spare warehouses and production capacity. If you let the MBAs with their spreadsheets run the business, you might suddenly find you have no business.

Are You Monitoring Your Automated Systems?

It is hard to anticipate the real world. I’m sure the wet concrete on the road in Japan looked just like solid ground to the delivery robot. Consequently, it happily trundled into the urban swamp and got stuck. The story does not report whether the delivery company managed to get their robot out before the concrete hardened…

This is why you need careful monitoring of all the fully automated systems you are deploying. The first line of defense is automated metrics and their normal interval. For a delivery robot, the distance covered over a minute should be greater than zero and less than 270 (if you have limited the robot to e.g. 10 mph). The second line of defense consists of humans who will evaluate the alarms and take appropriate action. The third line of defense are developers who will fix the software and the alarms.

Too many automated systems are simply unleashed and depend on customers to detect that something is wrong and complain. You want to figure out you have a problem before the image of your robot encased in concrete starts trending on Twitter.

Do You Let Convenience Trump Security?

Personal data on anyone is available from all the large U.S. social media platforms and ISPs to anyone who cares to ask. The mechanism is an Emergency Data Request (EDR). When law enforcement doesn’t have time to wait for a court order because someone’s life is in imminent danger, they send an EDR. This is simply an email from a law enforcement mail address. To send a fake EDR, you simply purchase a legitimate government email address from a hacker who has breached one of the more than 15,000 police forces in the U.S.

You would never divulge information on your customers based on just a plausible-looking email. But how do you ensure that expediency has not trumped security somewhere in your organization?

What Happens Then?

There is an easy way to avoid making stupid decisions: Asking “what happens then?” A decision is exposed as stupid when it turns out that the decision-maker did not carefully think through the consequences. Bad decisions occur when someone only looks at the immediate result.

New York City dodged a bullet when they started implementing bike lanes in the narrow streets of Manhattan. They could easily have made the stupid decision of simply marking a part of the street as a bike lane. Fortunately, someone clever at City Hall asked herself: What happens then? If you had simply painted bike lanes on streets, thoughtless New Yorkers would have wiped out bicyclists by the thousands with their car doors. So New York decided to paint a separation area between the car parking area and the bike lane. Clever.

Next time you are faced with a decision, try asking “what happens then?” several times. You might find this saves you from doing something stupid.