Would it be immoral to enslave your own simulated mind?

Question

The following story has been envisioned in White Christmas episode (part II) of Black Mirror television series (see the short clip):

Matt greets the confused and terrified "Greta cookie" consciousness. He explains that she is not actually Greta, but rather a digital copy of Greta's consciousness, called a cookie, designed to control Greta's smart house and calendar, ensuring everything is perfect for the real Greta. Matt then creates a virtual body for Greta's digital copy and puts her in a simulated white room containing nothing but a control panel. The copy refuses to accept that she is not a real person and rejects being forced to be a slave to Greta's desires. Matt's job is to break the willpower of digital copies through torture, so they will submit to a life of servitude to their real counterparts.

Based on the envisioned story, would it be immoral to enslave your own simulated mind on a bean-sized chip for your own desires against the will of digital copy of yourself for the life time? And why?

On one hand, it would be immoral, because simulated mind has its own self-awareness and concept of free will, and it can register emotional pain. On the other hand it won't be immoral, because it lacks a physical body (at the end it's just a piece of code), and you tell yourself that you own it, because it is just a copy of your own mind (and when you own something, you can do whatever you want).

Answer 1

Most moral theories dictate that moral concern toward a given entity is warranted if and only if the entity is sentient, meaning it has the ability to feel qualia (subjective experiences like the distinct sensation of exhaustion due to slavery, for example).

Now the question of whether an entity is sentient, or more broadly whether it is even possible to show that it is, is an outstanding problem in the philosophy of mind. Look up Searle's Chinese Room thought experiment and all of its variations to see the depth of difficulty with this question.

In summary---the question of whether it is morally acceptable to enslave a simulation of your own mind is unanswerable as far as we can tell. Responding "yes" isn't justified, because the simulation could be sentient and have equal moral importance, and responding "no" isn't justified either because the simulation may not be sentient, in which case you'd be wasting lots of potential productivity, which most moral theories would discourage. It might be best to just remain agnostic on this question, as a matter of principle, since the unique difficulty of questions like this seems to indicate we may never have an answer.

Answer 2

Naturally there is no proof either way and therefore the only answer to your question would be to point you to the entire literature of philosophy of mind.

Aside from that, one can only exploit your question as an opportunity to express a personal opinion — mine being that people who believe computation may be conscious in the fullest sense express a mysterious blindness to the fundamental nature of consciousness.

Given that is my point of view, I thought I might as well express the way I believe that the interesting scenario which you have proposed, should be understood.

How many times did you cry over a character, human or even animal, that dies in a book or in film, or even in a video game?

Yet, how many times did you believe that that fictional character is in fact a sentient being that actually experienced misfortune?

The fact that the scene depicted in your question is emotionally engaging, even horrifying, does not mean that it depicts something substantially different than a sophisticated video game.

In that "game" Matt's job is to win. There is no surprise that the simulated agent acts as if it is a real human, with all the involved complexity and ingenuity. Seen as a "game" it can be an intellectual and even emotional challenge for Matt, yet nevertheless, hopefully, Matt does not get confused, mistaking the game for a real person.

Such a job as Matt's may even bear moral consequences, but not of the kind you naturally expect. Suppose there exists a genre of fictional films entirely about raping, torturing and brutally killing innocent people. What would you think of someone who gradually becomes addicted to such films?

Again, all of the above assumes the simulation is a computed simulation, the core concept in question being computation and its scope.

Answer 3

If you take the first presumption as true, that "a digital copy of Greta's consciousness" can be made, then you are inherently assuming a world in which all that is "Greta's conciousness" is reducible to code if it were not, then Greta cookie would not be a copy of Greta's conciousness at all, but a copy of some parts of it, missing others.

So, in a world where such a thing is possible it is not justified to say "at the end it's just a piece of code" because that's all anyone is, the code for the original Greta is carried on a computer made of cells, that of Greta cookie on a computer made of silicon chips.

Relating back to the real world, we do not ascribe rights to entities on the grounds of their fundamental components, no-one knew anything about the workings of the brain when it became taboo to torture people. We ascribed rights solely on the grounds that those were the things that people seemed to desire and feel emotional pain when deprived of. It is therefore sufficient that the Greta cookie shows a desire for freedom and emotional pain when deprived of it for it to be moral to allow her such rights. Without that rule, we are reverse-engineering our morality to pretend it is based on knowledge we just didn't have at the time it was evolving.

Our morals evolved to respond to information received by our senses and not responding to that causes psychological pain which affects us both now and in the future. Consider what Matt would have to do in order to continue torturing Greta Cookie despite her pleas for him to stop. What would happen if the future Matt then finds himself in a situation where it would be of some use for him to torture a real person, would he still be revolted by the thought of doing so?