Sten Vesterli's Blog

Making IT Live up to its Promise

Tag: Testing

Good Intentions are not Enough

IT Leadership, Technology That Fits, , , ,

“We have the ambition to test disaster recovery twice a year.” That’s not something anybody in a professional IT organization would say, is it? Ambition? I have the ambition to create a spam- and hate-speech-free Twitter alternative powered by unicorns and rainbows, but unless I act on my ambition, nothing will happen.

Nevertheless, critical Danish infrastructure was operated on that principle. The common login system that everything from banks to tax authorities to municipalities uses is operated by a company called Nets. They apparently got to write their contract with the state themselves because it contains the ridiculous “ambition” instead of an actual requirement.

They did run a test on May 28, 2020. They did not run a test in November 2020, as was their ambition. Nor in May or November 2021. Not even in May 2022 did they test it. So when they crashed the system in June 2022 due to undocumented changes and other unprofessional shenanigans, the disaster recovery unsurprisingly failed.

Please tell everyone this story. When you are done laughing at the incompetence of central Danish authorities and their vendors, make sure you are testing your own disaster recovery…

How to Avoid Techno-Blindness

IT Leadership, Technology That Fits, , ,

Techno-blindness is a dangerous affliction. It is a disease of over-optimism mainly affecting people in the technology industry. The symptom is overconfidence that a system works as intended and a lack of awareness of what might go wrong.

Somebody in Moscow thought it was a cool idea to have a computer play chess with children, using a robotic arm to move the pieces. Until a child made an unexpected movement. The robot grabbed his hand and broke his finger. TuSimple is building autonomous trucks, and one of them accidentally executed an old instruction, causing it to turn left in the middle of the highway. Fortunately, nobody was injured as the truck veered across the I-10 and slammed into a barrier.

Important systems need independent safeguards. That means a completely separate piece of code that can intervene if the output of an algorithm lies outside some boundary. A truck shouldn’t be able to turn left at high speed. A robotic arm shouldn’t move on the chessboard until the player’s hands are off the pieces.

As a CTO, it is your job to ensure there are safeguards around important systems. You cannot depend on techno-blind developers to do this by themselves.