Are You Sure Your Backup System Works?

Why did all the trains in Denmark stop on Saturday? Russian hackers may or may not have been involved, but Danish incompetence was.

The Danish State Railways (DSB) has digitized all the paper that a train driver used to carry. That’s temporary speed restrictions, track works, and deviations from the standard schedule. They have also outsourced their digital solution to an amateurish vendor, and neither the vendor nor DSB had a backup solution. So when the vendor shut down the system due to an unspecified “security issue,” the trains stood still.

I’ve boarded a Delta Airlines flight with a hand-written boarding card on a scrap of paper. A professional organization continues to run, though slower, without its computers. An unprofessional organization like DSB is paralyzed. Are you like Delta Airlines or like the Danish State Railways?

Why Should the Business Trust You With Their Money?

“Give us a bag of money and go away.” That seems to be the thinking of most in the #NoEstimates movement. They have, of course, misunderstood the original concept, just like people who claim to do Agile when all they’ve done is to do away with the documentation. I agree that estimation is hard and software is complex, but asking the business to commit money for unknown benefits in the uncertain future represents monumental hubris. The real world works by comparing costs and benefits, even though both cannot be evaluated exactly.

I’ll be meeting some of the best and brightest IT architects in Denmark at the annual Software Architecture Open Space next week. This is an open-format conference, and I noticed some of the other participants have already brought up estimation and #NoEstimates as a topic. I’m looking forward to an interesting discussion. If you are in the vicinity of Copenhagen on Nov 3rd, I encourage you to participate in SAOS as well. You’ll surely learn something.

The Problem is the Humans, Not the Technology

The weakest link is the human. Microsoft does keep the software in their Azure cloud up to date with the latest patches but still managed to lose 2.4 terabytes of data belonging to 65,000 customers in 111 countries. The reason is that someone at Microsoft misconfigured a storage container.

This story became public because a security company wanting to sell its scanning solution posted it. They also informed Microsoft, who quickly secured the container. But for every white-hat hacker scanning the internet for unsecured storage, there are ten black-hat hackers siphoning off your secrets and selling them.

By buying a high-level cloud service from a reputable vendor, you can be sure that it runs on well-patched servers without known vulnerabilities. But you’ll have no idea when your cloud vendor fails to secure some lower-level service until you read about it in the news.

There are Many Reasons Not to Move to the Cloud

You don’t save anything by moving to the cloud. Ask around – how many of the organizations you know who moved to the cloud have reduced operations headcount? Some things are simpler in the cloud, but many others are more complicated.

You enforce some good security practices because there is no way to NOT install the latest security patches. And you can quickly spin up an extra testing environment.

But unless you really have a highly variable load, or you are starting something new where you don’t have a clue how much power you’ll need, the cheapest option is to buy some hardware and put it in your server room.

The next time one of the vendors tells you how much you save by moving to the cloud, take a really good look at the calculation. I’ll be happy to help you. You will likely find out that there isn’t a business case for moving.

Can You Trust Your Vendor?

Did you invite the hackers in yourself? Hundreds of German companies are waking up to the revelation that “German” cyber-security company Protelion is a front for a Russian company with links to Russian intelligence.

The hapless boss of the German IT Security agency even invited Potelion to sit on the German Cybersecurity Council. He is facing an unceremonious sacking…

Being able to roam freely inside the firewall and install agents with admin privileges is the dream of any hacker. There are at least one million devices running Protelion’s “security” software and the companies who invited Protelion in face a wholesale scrubbing of their entire IT infrastructure.

How do you plan to ensure that your security audits do not worsen your security?

Are you Dependent on Freelancers?

Using freelancers is dangerous. It starts innocently enough with just a single developer experienced in your chosen tool. But soon, you’ll be hiring a few more freelancers to fill positions you couldn’t hire anyone to do. Suddenly you wake up to the fact that the only people who know how to use half of the cloud services in your product are freelancers, who will be gone next time there is a funding squeeze.

I’m in favor of temporarily using freelancers to augment your team – I’ve been an external consultant most of my working life. But use them responsibly. Freelancer.com showed a 40-50% increase on a year-over-year basis last quarter for various categories, while postings for permanent employees on other sites grew only 12%. That sounds like many organizations are becoming dependent on freelancers. So ask yourself if you can maintain and run your systems without freelancers.

Well-led Employees Don’t Leave

When a company changes hands, the tree is shaken. Some of the employees that weren’t really attached to the company leaves. That’s what is happening at Twitter right now, and that will happen whenever there is uncertainty in your organization.

But have you noticed that some teams experience a lot of turnover in turbulent times, and some teams are solid as a rock? The difference is leadership. Members of a well-led team will continue to do their job through whatever tsunami of social media opprobrium, whereas members of badly led team will jump ship at the first sign of trouble.

Are you tracking the turnover in each part of your organization? It will tell you something important about the leaders.

It’s Expensive to Try to Get By With the Cheapest Resources

Talent is expensive. Not paying for talent is more expensive. Microsoft gets that. The U.S. Department of Defence doesn’t.

The Microsoft bug hunting program has a maximum payout of $250,000, and they did pay out $200,000 this year. You would think a crucial national defence vulnerability would merit a bigger bounty that finding a flaw in the Microsoft hypervisor, wouldn’t you? The DoD pays out $500 for a high-severity bug, and a whopping $1,000 for a critical issue.

Your developers are rewarded for shipping functionality. They don’t have the mindset to find the vulnerabilities. To build secure systems, you need to offer a bug bounty, or hire outside experts to do security review, or create your own internal white-hat hacker team. It does cost money. But security breaches cost much more.

Why Projects Without Business Cases are Shot Down

I just had a customer attempt to start a project without a business case. Such projects are usually driven by the desire to use a specific technology and with a vague idea that this would somehow benefit the end user.

If the IT department is strong, some of these orphan projects get started. They might be successful. However, since the organization has no idea of the business benefit, it is blind luck if the benefits exceed the cost.

If the business prevention department (compliance/legal) is strongest, they are shot down. There is always a reason not to make any changes. A project without a business case can be mortally wounded by any objections about compliance, GDPR, security, etc.

That is why every project needs a business case. It prevents IT from wasting money on something that will not add value, and it prevents compliance & legal from killing projects with a positive business impact.

Do your projects have solid business cases? If not, get in touch, and I’ll help you.

Who is in Charge of Outside-the-Box Thinking?

Everyone can track your license plate – not just the cops. A Belgian security researcher noticed that most parking apps do not validate that you actually own the license plate you add to your app. That means a stalker can add his victim’s license plate to his app and immediately be notified whenever that person parks anywhere…

This is another example of the inside-the-box thinking that developers are prone to. The developers of the Kryptonite bike lock had made it out of extra reinforced steel. Too bad a weakness in the lock allowed a hacker to open it with half of a ballpoint pen.

Finding holes in a system is not just securing the login and checking the encryption. It involves examining the system and its environment and users. That is a skill most developers lack. You need a “red team” who can find the holes before you roll out something embarrassingly insecure.