When in the 1990s I was researching the impact of technological change on the Hong Kong Telephone Company (HKTC) over the period to the late 1980s, I came across a lesson from history that seems to require continual relearning. HKTC had seen uninterrupted growth in the demand for basic telephones throughout the post-war period. Demand was insatiable. HKTC was a locally-owned monopoly since the 1920s and, despite at least one General Manager absconding with funds, had served the community quite well. Well, that is, until the 1970s. Western economies were hit by a severe economic recession in 1973-75, and Hong Kong did not escape the fallout. For the first time in living memory the demand for new telephone lines dried up. People just could not afford the luxury when unemployment rose and earnings fell. This would cause no undue problem to HKTC because they had installed a perfectly good monitoring mechanism for new applications, and their investment cycle was linked directly to the data coming out of the system. Only one problem. Top management refused to believe the incredible. It just was not in their experience. So they over-rode the system and authorized investment to continue. Within six months HKTC was facing such a dramatic liquidly and profits crisis the company had to turn to government for a bail-out. They got it, but also got a lot of heavy regulation on top of it which was not lifted until the market was opened to competition in the 1990s.
In early April 2015 in London’s well-to-do Hatton Garden district, thieves spent a weekend abseiling down a lift shaft ‘Ocean’s 11’-style into a safety deposit box vault, helping themselves to a considerable chunk of the loot, an estimated 72 safety deposit boxes containing around USD292 million worth of jewels. They had planned it well, cutting off surveillance cameras and alarm systems. What they did not know was that one alarm system still caught them at work and duly set-off an emergency call to the local police station. Lucky for them that evening on duty was a stand-in for Police Inspector Jacques Clouseau of the Pink Panther movies. He (or was it she?) duly ignored the call and marked it for no action. The police subsequently announced an award of USD20,000 for information leading to an arrest. Peter Sellers would have earned more.
The list of failsafe systems that are ignored or over-ridden could go on. And we all tend to do it when things seem counter-intuitive. It’s human. After all, that alarm had never gone off before, so probably a false positive. The demand for telephones never falls, so a blip in the numbers is just a quaint error term. The simple fact is that machines, or the programmes that run the machines, are getting smarter all the time, but we aren’t. Individuals can get smarter by learning through experience and thereby not repeating the same mistakes, but that does not seem to scale up very easily to society. Technologies can scale up geometrically, but we are lucky to scale up arithmetically.
Before reaching the dangerous conclusion that we should start handing everything over to machines, let’s quality that. Programmed machines, or ICTs in general, are more efficient than humans, and with AI will become our equals or better in making value judgments according to specified criteria. Forget the hoo-ha about ‘them’ taking over, the fundamental problem is that as humans we are divided between ourselves as to what we want the technologies to do. With drones, for example, some of us want them to drop medical supplies over targeted areas, others want them to drop bombs. There really isn’t much we can ever do to avoid this human pathology, but what we can do is to design systems, both human and technical, that automatically react in a responsible manner. If that means the shut-down of operations and operating systems which display irregularities beyond pre-determined limits, then so be it. The cost of reacting to false-positives is inconvenience and maybe some financial penalties, although back-up systems could reduce both. The cost of not reacting is likely to involve ever increasing risks to life and property. There seems to be no other way.
