The Rise of Overconfident Machines

At the time of writing, a number of eminent scientists have raised concerns about the potential adverse consequences of the belated rise of…

At the time of writing, a number of eminent scientists have raised concerns about the potential adverse consequences of the belated rise of Artificial Intelligence. Belated because it has been anticipated since the 1960s, but, until a few years ago, it had not materialised except in toy (albeit challenging) worlds, such as, for example, chess. The last few years have, however, been game changing. Amazon’s robotic warehouses, Google’s visual search, Skype’s automatic voice translation all hint towards what is known in academic circles as ‘strong AI’, i.e., the ability of a machine to respond to real-world stimuli in ways indistinguishable, or superior, to human behaviour. Put simply, machines can increasingly beat us. And this is significant.

It is important for two reasons. First, because whenever a piece of technology beats humans in something vital, this usually causes a revolution. When fire beat human saliva in making food safe and digestible, the modern human evolved. When stone tools beat human hands in warfare and hunting, the Stone Age began. Then iron beat stone. Steam. Electricity. The Digital Age. And now Artificial Intelligence.

It is hard for us to envisage the ‘next day’ where thinking machines routinely outperform humans. Massive unemployment, the potential for asymmetric warfare and a deepening rich-poor divide have all featured as potential elements of a dystopian AI future. Equally, eradication of disease, poverty and social injustice have also been put forward as a utopic alternative.

Hard to tell. Even harder to tell is whether machines will eventually dominate the human race, which some thinkers see as theoretically inevitable…geeky end-of-the-world enthusiasm aside, this is not as extreme as it sounds.

But there is a more practical, immediate concern, with real implications about the choice of methodology that AI companies like us are building right now. Our worry is that our immediate future will be plagued by overconfident machines. Put differently, if true wisdom is to know what you don’t know, machines are still pretty stupid.

Machine translation will always translate the input text, even if it has to clutch at straws and output junk. A bit like an overeager translation intern, it lacks the wisdom to say “I am not quite sure what this means, sorry”.

This carries risks that are all-too-real. Our foremost experience of AI will not be in the form of a charming golden robot with a British accent and a lovable sidekick, but rather in the form of hidden automation. And just like a human, an unsupervised machine that thinks too highly of itself can prove catastrophic.

So is it within our ability to code up “modesty” and “self-criticism”? Yes, and no.

Yes, because, to the relief of statistically-minded machine learning researchers most ML is now probabilistic, which simply means that answers are given in the form of a probability, rather than a straight yes/no. A picture might be a picture of a cat with 90% probability, or with 50% probability. It is then a matter of deciding how best to use this estimated uncertainty on a case-by-case basis, and, when it matters, one can always choose not to report the 50/50s.

However, uncertainty is subtler than that. Most Machine Learning is model-based, which means that its output (e.g., the probabilities above) is computed itself on the basis of certain assumptions about the data-generating process. These assumptions will, almost surely, be violated at some point. A thinking machine should then recognise that fact and report higher uncertainty.

Unfortunately, honest reporting of uncertainty remains challenging. Lets briefly examine why that is the case in the two flavours of AI that are ‘hottest’ right now: Bayesian AI, and Neural Networks (of which Deep Learning is the latest branch).

Bayesians are proud in relying on a coherent, all encompassing mathematical framework for expressing uncertainty, and they follow best practices in always reporting probability distributions, rather than point estimates. A Bayesian who is worried about a certain assumption will invariably express that assumption in a “fuzzier” manner, by introducing uncertainty about it in the form of yet another probability distribution — assumptions about assumptions. Unfortunately, even the most carefully stacked Bayesian model might still be betrayed by the real world, especially since the computational burden of the Bayesian calculus often force over-simplistic assumptions for the sake of faster computations. Now, assumptions are violated al the time — this is not a problem in itself, and to over-engineer the assumption set is not always the right decision. What is problematic, though, is that, if the model assumptions are wrong, it is really hard to know whether the answers are still somewhat valid. Ironically, for more complex models, the problem gets worse, because complex models fail in complicated ways. This results in the following well-kept secret: 40-year old tools such as linear and logistic regression are still the workhorses of business intelligence and advanced analytics. Users will call them “reliable”. The correct technical term is “robust”, which, in statistics parlour, means, roughly speaking, a model that tends to remain pretty accurate even when its assumptions somewhat break down. Not all is lost of course — Bayesians are aware of this problem, and are working on it. But to this day off-the-shelf AI will not generally come with strong robustness guarantees, these have to be engineered into the system separately.

What about deep learning? Such methods are often referred to as ‘black-boxes’ , and typically involve a highly parameterised input-output relationship fitted to the data via optimisation using a training dataset. These algorithms are fascinatingly complex and correspondingly powerful (indeed at Mentat we are building our own deep learning library for IoT cybersecurity right now). However, their complexity, and the fact that parameters of deep learning models are not really meaningful in themselves (technically speaking, they are not generally viewed as random variables or population parameters), makes it incredibly difficult for them to report their own uncertainty. Deep learning is in many ways similar to deep thinkers, or our lizard brains: it knows the answer instinctively, but cannot explain why.

That this should be a challenge is not surprising. Self-criticism, or, more broadly, self-reflection is one of the most elusive, and important, aspects of intelligence. Indeed, a certain school of thought in the philosophy of mind lists the ability to self-reflect as the defining characteristic of consciousness. Needless to say, humans do not always exhibit such intelligence themselves: cognitive biases, racism, fanaticism are all in many ways consequences of the inability to question one’s assumptions. However, the human race as a whole has historically demonstrated exquisite abilities to self-reflect, and reason about assumptions about assumptions, a little like the nested Bayesian model above.

So how can we build “self-critical” thinking machines? Our answer here at Mentat is that robustness and diagnostics are the unsung heroes in real-world deployments of AI.

The former consists in introducing failsafes into models that allow them to decrease their confidence when they feel overstretched (shrinkage in classical machine learning is a wonderful example of a simple solution to a difficult problem, and analogous techniques are applicable in almost all AI/ML algorithms, albeit with much greater care and technical difficulty at times). Failsafes against the possibility that the data-generating process changes over time is in fact a core innovation in our Machine Learning In Motion toolbox, and an unrecognised performance bottleneck in many competing offerings. Model diagnostics complement robustness and stem from an approach that was all-the-craze in the golden era of classical statistics, consisting in tracking performance indicators of the model, and taking suitable action when they take unreasonable values. This may sound mundane (who doesn’t keep track of their model’s performance?) but the art of diagnostics is in constructing the right statistic, and understanding the range of values it should be allowed to take without raising an alarm — a challenging mathematical problem, when done correctly. Thankfully, recent methodological developments can allow us to do both tasks with generality.

There is, of course, a lot more to “self-reflection” than what was explored here, but self-diagnostic, robust model-based ML is an excellent starting point.

Some time ago, a member of our university study group, after a long period of intense, silent thinking aimed at solving an infamous maths past paper question, announced: “Am I talking cr@p? That is the question.” Simply put we should not fully trust Thinking Machines until we can witness them spontaneously make a similar claim.

Originally published at www.ment.at on 17-Mar-2015