Several dozen Danish banks were down for five hours yesterday. Due to incompetence, not Russian hackers.
They were running on robust mainframe systems, because these have proven over decades that they never go down. But it turns out that running critical systems takes both hardware and skill. And the skill was lacking.
The reason mainframes have historically had very high availability is that they are really well-engineered and they’ve been run by really competent people. But those people have reached retirement age, and their jobs are gradually taken over by people with a different mindset. That’s how the mainframe hosting provider managed to run a poorly tested capacity management system that accidentally deallocated resources from all their customers.
There is mainframe hardware, and there is the mainframe mindset. The “this can never, ever, be allowed to fail” mindset. Which is retiring.
Are you sure you are transferring not only skills but also attitude when training new people to take over your critical systems?