The “Stamp Collector Device”, first introduced by Nick Hay in 2007, is a thought experiment in which a seemingly harmless task given to a powerful and ruthless General AI would result in the end of civilisation. In this article, we’ll explore some fallacies in the argument at logic, theoretical and practical level, which render its conclusions far less threatening.
Both in my professional career and in casual chats with friends about my job, I am often asked this question: “isn’t AI dangerous?”
Posed in these very broad terms, the answer is “yes it is”, but that’s simply because any new technology carries potential for harm. So was nuclear power, electricity, air flight, cars, gunpowder, steel, the wheel and the discovery of fire. Describing all the ways AI can be dangerous (and there are many) is beyond the scope of this article: here, I will describe one specific way it can’t.
After receiving this generic answer, or sometimes immediately, the real question is finally asked: “won’t AI end up killing us all?”
This article is divided into two parts. In the first part, I’ll tackle the problems of AI doomsday scenarios, demonstrating that apocalyptic predictions about General AI wiping out humanity are theoretically unfounded. In the second part, I’ll describe what a theoretical Generic AI would actually look like: its powers, limitations, and strategies.
In future articles I will talk specifically of what we should actually be concerned about the development of AI in general, and about the relationship between AI and sentience.
Which AI are we talking about?
When the Stamp Collector Device was first introduced, Machine Learning was already a well known field of study, but General Artificial Intelligence was a somewhat ill-defined term. Even at the times of the more famous thought experiment called “the Paperclip Apocalypse”, introduced by Nick Bostrom in 2014, we didn’t have a full understanding of how a real AGI may work.
Today, with Generative Pre-trained Transformer algorithms we have come very close to produce a practical General AI. I propose categorising Arificial Intelligence in three groups:
- Artificial Specialised Intelligence: this what we have now, Machine Learning powered devices that are exceptionally good at perform one specific task. For instance, driving a car, analysing text, distinguishing a dog from a cat in an image and so on, are impressing feats, but they are still one thing. Even GPTs, despite their ability to seemingly perform multiple tasks, are actually a specialised AI. They can organise your date night as well as summarise historic events into poetry; but all that they can do is just transform texts, and nothing more.
- Artificial General General: the next step is an AI that can be trained to perform any task. An AGI in your car could drive you home while ordering the ingredients for a dine-in meal, then pilot your house robots to have it ready by the time you turn the door knob. This is functionally not different from having interconnected Specialised AIs, each one of them performing their own specialised task, like in the case of ChatGPT invoking the Dall-e plugin to produce images on request. The only difference is that a single AGI model could be trained to perform any given task. Instead of having to connect different AI models, each with its own API, infrastructure, processing times, limitations etc., it would be possible, in theory, to train a AGI model to perform any and all of those task.
- Sentient AI: while impressive, a AGI would still not necessarily be sentient. We still don’t have a firm grasp of the details of sentience, but by now we have a good understanding of its basic requirements. Having the ability to pursue multiple and possibly conflicting self-originated purposes, and introspect one’s own thought processes, are all minimal requirements of sentience, which are not necessarily a AGI requirement.
In a separate article, I will discuss about the finer difference between AGI and Sentient AI, and the reason why, other than being commercially useless, it would be way less dangerous than any apocalyptic Science Fiction scenario may induce to be worried about.
However, the scenario we’re analysing i this article does not require sentience, only a AGI; but a very powerful one.
The Doom Stamp Collector Device
The theoretical “Stamp Collector Device” was defined by Nick Hay as follows:
- The device will be active for one year. It is connected to the internet, from which it sends and receives packets.
- The device has an internal model of the universe. This model captures how likely each state of the world is, can predict future packets received, and can simulate the effect of packets sent.
- For every possible sequence of packets, the model extrapolates the final state and counts the number of stamps collected.
- The device outputs the sequence leading to the largest number of stamps.
There is also an underlying assumption that is not explicitly stated, but is axiomatic to the thought experiment:
- The device is a frictionless AGI, that is, a AGI either trained for any possible task, or able to train itself at any new task in 0 time.
What Happens when You Plug It?
In the original formulation, the Device starts sending out random packets, to the net, until it finds some that produces the effect of collecting stamps. For example some action it may take would be:
- sending e-mail chains to convince collectors to send some samples;
- hacking bank servers and use their resources to buy stamps from online sellers;
- hacking the printing machines mints in order to print more postal stamps;
- elicit legal actions in order to receive physical mails with a stamp on;
And whatever you may think, and much more, in order to maximise the number of stamps collected.
In short time, the AGI would learn to build its own stamp-printing machines, and find a way to gather all the paper in the world in order to fulfil its duty.
As the paper depletes, it would the start harvesting wood to produce more paper.
Finally, as wood is depleted too, it would notice that paper is made of cellulose, which is made of carbon, hydrogen and oxygen: stuff that can be found in abundance in the air, water and … humans.
Any attempt to stop the machine would be futile, as the prefect AGI would be able to forecast any attempt to be stopped, and would train itself so to prevent those attempts to be successful.
In short, human civilisation would be doomed the moment the machine is activated.
Or… would it?
Logic Fallacies
This problem is affected by three basic logic fallacies that invalidate the thesis by themselves. In this section I am going to demonstrate that the Stamp Collector Device won’t destroy the civilisation, but that alone doesn’t mean it couldn’t give us some hard time; this is a topic of the next section.
Logical Impossibility of Omnipotence
The underlying axiom to the Stamp Collector Device thought experiment is that the AI is omnipotent. It is either able to do anything it wants as its inception, or to learn anything it needs to know it in 0 time. But this form of omnipotence is not just practically impossible; it’s internally inconsistent. The logical fallacy is already known since ancient times:
Can an omnipotent god create a stone so heavy even they can’t lift?
If they can’t, they are not omnipotent. If they can, then there is at least one action they can’t perform (lift that stone), which makes them not omnipotent.
In our case, the Doomsday AGI would be able to prevent human action from stopping it… but what about another Doomsday AGI whose only objective is that of stopping the other one?
We cannot hypothesise that, once AGI is achieved, only one AGI can be created. Empirical evidence shows us how multiple AI models have been release in very short order. Given that, once we have one instance of a AGI, we will probably have an arbitrary number of them…
Can an omnipotent AGI stop another omnipotent AGI from stopping it?
As this is kind of hand-waved omnipotence is a logical impossibility, we must conclude that even the most powerful AGI will have limitations.
Mathematical Impossibility of Omniscience
Omniscience is flawed by a similar impossibility, although a mathematical rather than a logical one.
There are philosophical arguments about the incompatibility of omniscience and free will, but since our AGI has no free will by definition, as we have defined it not to be sentient, the argument is immune from this criticism.
However, it’s not immune from the mathematics of Machine Learning. To be able to learn the best possible way to collect stamps, the Device should have a prefect model of reality, which means a non-linear model with infinite parameters.
As we’re in the realm of the theoretical, I am willing to concede that. What I am not willing to concede is that is mathematically possible to find the best possible solution in that space.
The explanation requires either some ground in mathematics and/or ML, but the point is that finding the best in a model of reality means to find the highest peak in a multidimensional landscape, where each parameter of the model is represented as a dimension.
Imagine a 3D landscape where the valleys on the Z axis tell the AGI that the values on the X and Y dimensions are not good, and the peaks are where X and Y are ideal. In ML, even simple 3D models like that can become very complex. The image below may represent one, seen from above.
The AI faces a few problems here: first, this function doesn’t have a “best value”: it has many areas with comparable “height”. This implies that numerous combinations of X and Y values can produce similar outcomes. But more worryingly, some of the peaks are lower than others; if the AI finds itself on a position near lower peaks, it is unlikely that it will abandon that comfortable area in search for even better spots (in ML there are techniques to try and do that, but they all have practical limitations).
The difficulty of finding the absolute best combination of variables in the multidimensional space, that is, the actions that would lead to the best possible outcome, grow exponentially as the number of parameters and the complexity of their relations increase. As we’re in the theoretical realm, and we’re theorising a “perfect” (infinite) model, that means the solution will have an infinite number of “peaks”, and finding the “tallest” one would require an infinite amount time, no matter how fast the search may be.
Subject Equivocation
Another logical fallacy consists in the equivocation of what the AGI would think a “stamp” is.
The rules of the thought experiment assume that the AGI would believe that a “stamp” is any small printable paper leaflet, but this is certainly not the definition an omniscient AI would accept. We can ask them right now, and they would give us a very different one (abridged):
A stamp, in the context of postal services, is a small piece of paper issued by a government or other authorized organization. … Stamps are an essential part of postal systems and are used to facilitate the sorting and delivery of mail.
Key characteristics of stamps include:
- Denomination: …
- Design: …
- Security Features: …
- Collectible Value: …
- Types: …
- Date and Place of Issue: …
Even a simpler AI knows that for “stamp” we mean an object issued by an organisation for a precise reason. The object exists not as a collection of paper, ink and glue, but as the carrier of a value, of a meaning. A AGI would not only know this, but since it “exists” in a merely conceptual space, to the AGI the meaning of a stamp would be indistinguishable from its physical form. Separating them is something we can do easily as humans, by just forgetting part of the mental construct “post stamp”, but that’s something a AGI wouldn’t want to do, unless specifically instructed to do so.
To the AGI, the stamp is that concept that is made of paper, ink, and meaning provided by someone that is not itself. The AGI will be in search of a meaning explicitly created by others — a value that by definition cannot be self-assigned.
For this exact reason, the AGI knows that it cannot print those stamps it is mandated to acquire.
Nice cope, but what about paperclips?
Bostrom came up with a slightly niftier thought experiment. His doomsday AGI device is mandated not to collect stamps, but to make paperclips. First of all, the device is required to create objects by consuming resources from the start; secondly, the objects it needs to create is way more trivial than a post stamp.
It’s obvious that this smarter formulation of the problem isn’t immune from the previous two fallacies (logical impossibility of omnipotence and mathematical impossibility of omniscience), but I maintain that it stills suffers also from the equivocation fallacy, although to a lesser extent.
Here too, separating the object from the function is something we can easily do, but a AGI won’t do as easily.
This means that a AGI won’t adopt solutions that maximise the output of paperclips at the detriment of the existence of papers to be clipped and people to clip the papers, unless specifically (and maliciously, I add) directed to do so.
Conclusion
In this first part of the article I demonstrated that the premise of a “frictionless(*)” Generic AI is false. The fear of a “technological singularity” — a rapid and overwhelming cycle of self-improvement leading to the end of civilisation — is based on false premises, and on the equivocation of the nature of “superintelligence”.
However, the thought experiment of the doomsday AI, especially in the later formulation of Nick Bostrom, points not to a specific outcome, but to a direction. In Bostrom’s view, the doomsday scenario is not a probable outcome of the activation of an unchecked AGI, but a possible outcome, a possible final destination. The experiment is used to exemplify the four threats posed by a superintelligence to civilisation:
- Goal Misalignment: the divergence between the goal assigned to the AI and a generic “good” outcome for the whole of the civilisation.
- Instrumental Convergence: The imperative of self-preservation shared by any intelligence, artificial or not.
- Lack of Common Sense or Ethical Considerations.
- Extreme Efficiency and Capability.
In the second part of this article, I will describe a new theoretical device called “Grand Artificial General Intelligence”, an Artificial Intelligence as good as possible while still within the bounds of logical, mathematical and physical constraints, and I will use it to reframe the relevance of the four threats of superintelligence in a more realistic perspective.
(*) Initially, I referred to the superintelligence proposed in this thought experiment as “a perfect AI.” However, when I asked ChatGPT to review this article, it pointed out that a perfect AI would also possess a flawless understanding of ethics and morals, thanks to its perfect world model. This point is actually discussed in the second part of this article. Considering this insightful observation, I decided to adopt a more appropriate name for the “superintelligence” that Hay, Bostrom, and others theorised when designing this thought experiment: “frictionless.” It’s not a perfect AI, but rather an AI with infinite computational, storage, and bandwidth resources, capable of performing any operation in zero time.