AI: Black Magic inside a Black Box
If a product works, do you care what’s inside? And if it does not work, should it matter what’s inside whether you have a right to a valid explanation?
If you enjoy sausage, do you really want to know what went into it? Vegans do, but carnivores probably prefer not to be aware of all the gory details. If you are interested in buying a car, do you care about its powertrain? Many people are more concerned with fuel efficiency or the range of the battery than the exact specifications of the engine, transmission, and so on. If you see a recommendation on Netflix, Spotify, or Amazon, would it make a difference if you knew an AI came up with it or not?
With physical products, you can often inspect the interior: slice the sausage or pop the bonnet (a.k.a. hood) of the car. Whether you are any the wiser is another matter entirely, but you can, in principle, look inside. For digital products, the inner workings are often hidden. Even if you could look inside, it might not be obvious how a product or feature worked, especially when powered by complex machine learning models. Essentially, you have black magic inside a black box.
The real question is of course: does it matter how the digital sausage gets made? If the product is fit for purpose, that is, there is a market for it, but the ‘AI’ were a series of coin flips or a witch reading tarot cards, who cares? More importantly, how would anyone know? Such a component would still be wrapped inside the black box of the product’s internals and the data that powers it. Even if you knew that transaction, customer preference, or social data went into it, it means little, particularly when data brokers are involved. But that is true even for products that do not rely on AI at all.
The right to an explanation
Scant transparency is common in many products and services. That opaqueness is not unique to artificial intelligence: most digital products suffer from a lack of transparency and accountability. That includes the right and means to settle disputes or request meaningful explanations.
And if you got an explanation for a decision in a product or service, how would you even know if that explanation were true? Even today there is no guarantee, not even in human interactions. Job rejections are, for instance, often accompanied by standard phrases such as “not a good fit” or “many qualified applicants who matched our requirements better”, when in reality it may be based on your raucous social media posts, political affiliation, skin colour, the lack of a degree from a top university, gender, experience at a competitor or a certain industry, or just because the company did not like your accent, face, accessories, or shirt. Subconscious judgements are common, fast, and frequently rationalized. The explanation people give may not reflect reality.
Sure, automated solutions are vastly more scalable, and so is the potential harm when things go wrong, but your fundamental right to an explanation should not depend on implementation details of a product or service. The right to an explanation is, nevertheless, primarily talked about in the context of the proposed AI Act and the non-binding AI Bill of Rights. Biden’s executive order, which the courts can still overturn or the next US president revoke, merely encourages regulatory agencies to request explanations on “the usage of AI models”. It does not specify the right to an explanation, let alone a truthful one.
Unregulated AI itself does not increase the possible damage to society and individuals, automation without accountability does. If your bank account is closed down, do you expect to have a different recourse based on if the decision was made by a human, an algorithm, or an AI? If your job application is automatically rejected, does it matter if a complex neural network or simple time filter on the gap between jobs was responsible? Or if your insurance claims are rejected without proper care and attention, does your right to view claim files, which health insurers routinely refuse to hand over on time, depend on how an insurer implemented their service?
None of these cases involved any AI, yet we accept these as the way things are.
A human or buggy algorithm decides our fate and that’s that.
It is peculiar that current regulatory proposals single out AI, especially when the definition of AI is pretty broad: any machine-based system that influences its environment, physical and/or virtual, by producing outputs (e.g. predictions, recommendations, content, or decisions) based on data.
Deep neural networks as well as basic machine learning methods, such as linear regression models or decision trees, are obviously AI.
But any automated decision-making system that uses data, even if it redirects that data to /dev/null, or any prediction resulting from a series of nested conditional statements based on specialized human knowledge (i.e. data) are technically under the AI umbrella.
Trust
Words such as trustworthiness and reliability are thrown around a lot in AI regulations, though trust in a system depends on how its decisions affect you. If a product works, few people question its actions or rationale. It is when you are negatively affected by a product that you care about its inner workings. That is not unique to AI.
Suppose your bank made a mistake to the tune of $10,000 in your favour. They have no idea it happened. The specifics, human error or technical glitch, do not matter. If you keep quiet, it is yours to spend. How do you feel about your bank after this sudden windfall? Pretty good, probably. Trust in your bank has either remained the same or gone up ever so slightly. Free money!
What if instead $10,000 was missing from your account, and the bank claimed it never happened? You’d be livid and rightly so. You will never trust that bank again.
Trust in a system is therefore not only a function of your prior beliefs but also on how you are affected by its decisions.
‘Cause I say so
And that makes explanations tricky, whether for AI or not. A psychology experiment tried to find out what sort of explanations people accept. Picture a series of people in line to the copier. Asking whether you can jump to the head of the queue leads to success in 3 out of 5 cases. Just by asking! Kind of makes you feel like a schmuck for standing in line with the other schmoes, doesn’t it?
When you add a non-explanation, such as “because I want to copy stuff,” your success rate goes up to 93%. That explanation is worthless, because everyone in line to the copier wants to make copies. If you add you are in a rush, one percentage point more.
The moral of the story: people accept terrible explanations. At least from other people. Does that mean any automated system can claim “because the algorithm said so”? I doubt that holds up in a court of law. But it might be enough to get the majority of customers off your back. Moreover, what is understandable to one person may be complete gibberish to another.
Who cares?!
If the generative AI incumbents have seen further, it is not only because they have stood on the shoulders of giants but also because they have trampled on the little people, profiting off the legal grey zone around fair use and copyright. A single case of plagiarism is intellectual theft; millions of cases are innovation.
It therefore seems that only executives and investors care about the AI inside digital technology: it is the technology of the hour. That is not to say, artificial intelligence as it currently exists is not useful. There are legitimately good uses of the technology, but without human oversight there are also severe societal risks, such as the concentration of power and money. The companies that demanded regulation were the ones that had raced to release their tech into the wild, then backtracked, and subsequently asked for funds to exacerbate the problems they warned against.
Regulators care to the point they have been lobbied. The EU’s proposed AI Act claims to protect people from harmful applications, including, but not limited to, indiscriminate mass surveillance. The EU, however, prefers to have the monopoly on mass surveillance with a proposal to break end-to-end encryption. They also refused to divulge their list of experts for similar proposals under the guise of “Won’t somebody please think of the children?!” Most experts turned out to be from outside of the union and stood to benefit directly from the proposed legislation. The GDPR has, likewise, barely dented data privacy: exorbitant penalties do not deter certain companies from repeated violations. This makes it hard to swallow the AI Act as another piece of well-intentioned legislation and not politicians overreacting to what might be in the distant future instead of what already is.
This is a common problem in product management: to solve for imagined future problems rather than actual current ones. The Bletchley Declaration on AI safety is a prime example of this. The risks it describes are all couched in speculative language. It should have focussed on the harm and risks of current artificial intelligence systems instead. For instance, a clear ruling on whether training models on pirated content and privately profiting from it constitutes plagiarism or is fair use at hyperscale is more pressing than protecting people from imagined, future problems that are more science fiction than steeped in reality.
Conformity assessments cannot fix the black-magic-inside-a-black-box problem. All such regulatory requirements will achieve is to protect the incumbents from challengers who may not have the resources to go through lengthy procedures with government agencies, as they see their cash dry out and talent leave for pastures green with hills of dollar bills. Innovation requires many different approaches to be tried, many of which will not lead to success. Regulatory sandboxes with priority lanes for SMEs and oversight from so-called competent authorities, as the EU proposes, may not be sufficient. Such centralized resources can easily become a bottleneck to innovation, though the intent of ensuring innovation in high-risk applications can safely continue with limited legal liability is noble.
Without accountability transparency is pointless, and legislation without enforcement toothless. And overly generic legislation that singles out AI is downright suspicious.