The Paradox of Predictability

Predictability is one of the pillars of AI safety, but for complex learning systems, unpredictability is unavoidable. How do we assure the safety of systems that are inherently unpredictable?

Image credit: https://unsplash.com/@drmakete

“The creative unpredictability of intelligence is not like the noisy unpredictability of a random number generator.”
Eliezer Yudkowsky, 2008

Public narratives around AI technology are becoming more prominent. These conversations have surfaced a collection of terms associated with the safety of these technologies. One term that is commonly linked with safety is predictability. Predictability can be described as the ability to precisely and consistently predict the actions or outputs of an AI system. The concept of predictability is often paired with the concept of understandability as, theoretically, the more you understand how a system works the more predictable the outputs will be. The key difference between the two terms is that predictability is a quantitative quality and understandability is a subjective quality that is dependent on human capacity for understanding.

In static environments, unpredictability is less of a concern. In these cases, through restrictions of inputs and the elimination of outcomes, reasonable predictions can be assured. However, more dynamic and complex environments will ultimately entail unforeseen inputs that a system may not have been trained against. How a system responds to an unforeseen input will be influenced by the system's capacity for adaptation and learning. For learning systems operating in complex environments with changing conditions, it will be exceptionally difficult to predict what the outputs of the system may be in situations that are deemed unfamiliar or atypical. 

Predictability is one of the pillars of safety for AI systems. We want to be able to anticipate the actions or outputs of a system we deploy or interact with. However, as we progress towards more “intelligent” systems, unpredictability will become an inherent property of the system. Intelligence is intangible, subjective and multifaceted. It consists of elements like understanding, perception, reasoning, learning and problem solving. AI attempts to capture and translate human intelligence and cognition in machines and technical platforms, a task that is deeply complex and still unresolved. 

If we are to reach a point of technical advancements where systems increasingly resemble elements of human cognition, we need to reconsider how we measure safety in these systems. As systems become increasingly more intelligent, they will also become increasingly more unpredictable. Predictability will quickly become a paradoxical measure of safety for complex intelligent systems that are inherently unpredictable. Mandating predictability in these systems will limit their development and potential. 

Moving forward, we need to reconsider the measures of safety for AI systems. While arguments can be made for why predictability is an important measure of safety, the reality is that as technological capabilities progress, we will be faced with more “black box” systems, such as deep neural networks, in which the inputs are known; however the computational process to achieving outputs is difficult to ascertain. 

Claiming that a system meets a particular standard of predictability does not make that system safe, as there are many examples that demonstrate that complex software can be made to behave in unexpected ways under specific conditions. A recent example of this is the incident in which a Tesla veered away from a direct path and did not stop for pedestrians entering a crosswalk [1]. 

Unpredictability in complex systems is something that is unavoidable. Instead of designing against it, we should be designing around it. That is, we need to be considering how we respond to unpredictability in a timely and safe way. When the Boeing 737 Max malfunctioned, the pilot reached for the airplane’s Quick Reference Handbook (QRH) which contains a series of simple checklists that are designed to help pilots rapidly assess and manage “non-normal” situations. Nothing in this handbook seemed to apply to their situation [2]. The combination of software malfunction and gaps in human knowledge and fail safes led to a near catastrophic situation that could have been avoided had more appropriate measures been put in place. 

Rather than attempting to mandate predictability in intelligent systems that fundamentally cannot meet such a requirement, perhaps we need to re-evaluate different measures of safety to identify attributes that are both achievable and justifiable indications of safety. We also need to explore how we can implement more robust fail safes for complex AI systems. Fail safes designed around human operators can support the interweaving of human capacity for reasoning and judgement into the operation of complex systems. 

The progression of intelligent systems is uncharted territory in many ways. How we address the safe implementation and adoption of these systems will influence how these systems are designed and developed. If we continue to pursue predictability and fail to embrace unpredictability, we will likely see more consequences when intelligent systems behave unpredictably and these consequences will be heightened in environments that lack appropriate measures.

References

[1] https://hothardware.com/news/tesla-fsd-autopilot-crosswalk-dmca-video-takedown 

[2] https://www.theverge.com/2019/5/2/18518176/boeing-737-max-crash-problems-human-error-mcas-faa