0:37
Intro. [Recording date: March 12, 2025.]
Russ Roberts: Today is March 12th, 2025 and my guest is author and philosopher Jeff Sebo of New York University [NYU]. Our topic for today’s conversation is his new book, The Moral Circle: Who Matters, What Matters, and Why.
Jeff, welcome to EconTalk.
Jeff Sebo: Yeah, thanks so much for having me.
00:59
Russ Roberts: What is the moral circle?
Jeff Sebo: The moral circle is a metaphor for our conception of the moral community. So, when we make decisions, when we select actions or policies, to whom are we accountable? to whom do we have responsibilities? who do we extend consideration? Normally, we might extend consideration only to some humans, though many of us now recognize we really owe consideration at least to all humans and many animals, like mammals and birds affected by our actions and policies. So, this book is about: Should we go farther than that? And if so, how far should we go?
Russ Roberts: You start with a provocative, entertaining scenario that you come back to now and then, and maybe we will as well in our conversation. You’ve got some roommates. One is Carmen, the other is Dara. Or maybe you pronounce it differently; I don’t know, it’s in print. Tell us about Carmen and Dara, and before we know more about them, your initial introduction to the group. Then we’ll talk about how that gets more complicated once we learn about their nature.
Jeff Sebo: Yes, absolutely. I should say, by the way, that this thought experiment is inspired by a similar one from the philosopher Dale Jamieson, but I take it a little bit farther.
So, imagine that you live with a couple of roommates, Carmen and Dara. You get along really well. Obviously, you have agreements and disagreements, and you have to sort through some tensions because you live together and have different preferences. But, on the whole, you have good relationships.
One day, the three of you for fun decide to take ancestry tests to learn a little bit more about where you come from. To your surprise–to your collective surprise–your roommate Carmen turns out not to be a member of your species at all: she turns out to be a Neanderthal. You thought Neanderthals were extinct, but it turns out a small population has still survived and exists to this day, and Carmen is one of their members. And, your roommate Dara, it turns out, is not even a being of your kind. Dara is a Westworld-style robot. You thought that, at best, these kinds of beings would exist only in the farther future, but it turns out that a small population already exists in data mode, and Dara is a member of their population.
The question that I ask in this thought experiment is: how does this revelation affect your attitudes towards, but more importantly, your moral relationship with you roommates? Do you still feel that you have a responsibility to consider their interests and strive to find a fair and equitable way to live together in your household, in spite of the fact that Carmen is a member of a different species and Dara is a being of a different substrate? Or, do you now feel that you have a right, as long as you can get away with it, to treat them however you like, and impose your own beliefs, and values, and decisions on them, even if it seems to be against their will?
Russ Roberts: I like the modest, undemanding example of playing music late at night or early in the morning, if we have different wake up and work times. We could also imagine them having different kinds of ancestry than the ones you chose. One of them could have a parent who was a guard at Auschwitz, one of them could be a founder of Ku Klux Klan’s offspring. We could ask questions whether that should change things. We could discover things about Carmen and Dara in their own past, not just their parents’ past, that disgust us or we think is morally reprehensible.
I think it’s a very interesting idea to think about how we treat people in general. Often, I think in our conversation, we might go back and forth between how we think we ought to treat them versus what does morality demand of us. And, they may not be the same, for a variety of reasons.
5:23
Russ Roberts: But, let’s start with the Carmen and Dara that you talked about. Summarize what you think are some of the range of responses people could have in that situation, how you think we ought to respond.
Jeff Sebo: Yeah. There are a lot of options, even within modern ethical theory. And then, of course, in society people are going to have an even wider range of responses. Most people, at least these days in philosophy, would accept that you do still have moral responsibilities to your roommate Carmen. Carmen is the Neanderthal. Yes, Carmen is a member of a different species, but apparently this species has co-evolved with humanity in such a way that we now have broadly the same capacities, and interests, and needs, and vulnerabilities. And so, Carmen, you can presume, is still conscious. It feels like something to be her. Is sentient; she can feel pleasure and pain, and happiness and suffering, is agentic. She can set and pursue her own goals based on her own beliefs and desires. And she still has all the same projects and relationships that she had yesterday, before you had this revelation.
The mere fact, in and of itself, that Carmen is a member of a separate reproductively isolated–but very close–species, is not enough to strip away any intrinsic moral significance she has and her interests have. And I think pretty much everybody would agree about that. There might be subtle differences now, in terms of what she wants and needs and how you relate to her, but fundamentally you do still have responsibilities to her.
Now, Dara is a whole separate question. Dara appears to be conscious, and sentient, and agentic, and have projects and relationships. But, Dara is a product of science, not of evolution, and Dara is made out of silicon-based chips, not carbon-based cells. So in this case, you might have real uncertainty. Philosophers and other experts have real uncertainty about whether a sufficiently advanced, sophisticated, silicon-based being like a Westworld-style robot, like your roommate Dara, whether it really can feel like anything to be that being. Whether they really can experience pleasure and pain, and set and pursue their own goals in a morally significant way.
And so, while we might have broad consensus that you still have responsibilities to Carmen, with Dara, we might have a lot of disagreement and uncertainty. And then, you are going to have to make decisions about how to treat her, despite that disagreement and uncertainty.
8:00
Russ Roberts: So, before we go further on this, talk about the Welfare Principle that you write about and how that might inform how we deal with this new information.
Jeff Sebo: The Welfare Principle is a plausible and widely-accepted idea in philosophy that holds: if you have a capacity for welfare, then you also have moral standing. So, this is what that means. The capacity for welfare is understood as the capacity to be benefited and harmed, to be made better off or worse off for your own sake. My car could be damaged, but my car is not really capable of being harmed–made worse off for its own sake–so my car lacks the capacity for welfare.
And, moral standing means that you have a certain kind of intrinsic moral significance: that you matter for your own sake, and that I have moral responsibilities to you. I owe them to you.
The Welfare Principle basically holds that welfare is sufficient for moral standing. If you have the capacity for welfare, that is enough for you to matter for your own sake and for me to have responsibilities to you when making decisions that affect you.
Russ Roberts: Just two comments, the first whimsical. We will link to this clip–one of my favorite moments in Fawlty Towers is when Basil Fawlty, in a hurry to get somewhere, his car breaks down and doesn’t restart at a red light or somewhere. And he gets enraged, and he gets out of the car. He goes and picks up a large branch by the side of the road and he starts hitting the car with it, saying, ‘How many times have I told you?’ It’s funny. It’s very funny. But, it illustrates unintentionally this principle. He could damage the car. He could dent it, he could hurt its paint, he could incapacitate it permanently with a set of actions, but he can’t harm the car in its own sense of self.
Just to be clear, because we’re going to turn to consciousness inevitably in this conversation: Going back to Dara, if I notice that Dara’s batteries are running low and I boost her up, or vice versa–I unplug her or block her access to electricity, similarly to keeping Carmen from eating stuff out of the fridge, taking away her keys so she can’t go buy groceries–we would be comfortable saying that it’s cruel, it’s harmful to Carmen. Dara would be, I think, more complicated.
So, you want to add anything, in terms of the Welfare Principle for Dara, in terms of suffering, or wellbeing, or happiness? Because in one of the formulations, I thought it might include this, but I’m not sure.
Jeff Sebo: Yeah. What I can add–and by the way, I love that example. The philosopher Derek Parfit has a similar example. He used to talk about how he would always feel the strong urge to hit and punish his computer when the computer stops working. Then he would have to try to psychologically overcome that.
In any case, part of what is interesting and complicated about the Dara case is that it reveals disagreement and uncertainty, both about ethics and about science. Both about the values and about the facts.
On the ethics side, we could have disagreement and uncertainty about: what is the basis for welfare and moral standing in the first place? Do you need to be sentient, capable of consciously experiencing pleasure and pain? Or is it enough to be conscious without being sentient? To be able to have subjective experiences, even if they lack a positive or negative valence. Or, is it enough to be agentic without being conscious–to be able to set and pursue goals, even if it feels like nothing to be you? Philosophers disagree about that. And based on your answer to that question, that sets a different standard that Dara would need to meet.
And then on the science side, we might also have disagreement and uncertainty about what it takes to meet those standards. Is a sufficiently sophisticated silicon-based being capable of having feelings of their own?
Both of those are contested sets of issues. That is part of what would probably make you feel really confused if you learned that your roommate Dara is a silicon-based robot, after all.
12:26
Russ Roberts: You use the phrase–I forget exactly how you used it–the experience of what it’s like to be you, something like that. That’s a reference, I assume, to Thomas Nagel. You want to take a minute and step back, and give listeners and viewers a little bit of that background as an example of one way of thinking of consciousness and sentience?
Jeff Sebo: Yeah, thank you. So, Thomas Nagel wrote a famous paper called “What Is It Like to Be a Bat?” This was now decades ago. Basically, this paper was helping people to understand what we now call phenomenal consciousness. And, this is helpful because the word ‘consciousness’ can be used in many ways. Sometimes we can use it to mean being awake instead of being asleep. Or, being self-conscious, self-aware, instead of not having that kind of meta-cognition.
But, in this paper, Tom Nagel was focusing on a particular phenomenon, which he used ‘what is it like to be you’ to identify. The basic idea here is that our brains do a lot of processing. Some of it corresponds to subjective experiences and some of it might not. Right? So, when our brains have perceptual experiences or affective experiences–when I see the color red, when I hear the sound of a trumpet, when I feel pleasure and pain–those are all subjective experiences that feel like something to me. But then, when my brain helps my body regulate heartbeat or digestion, that might not feel like anything at all.
The question here is, first of all: What is it like to be a radically different kind of being? What are their subjective experiences, those kinds of conscious experiences like? And then second of all: What kinds of beings can have those experiences in the first place? How far does it extend in the tree of life, and then beyond the tree of life?
So, yeah: when we ask about consciousness in this context, we are focusing on phenomenal consciousness: What is it like to be a different kind of being?
Russ Roberts: This is sometimes called qualia: your perception of things. A lot of interesting papers, at least interesting to me, on this. Many listeners of yours may not find it of interest. The Nagel paper, which we’ll link to, is mostly accessible to a non-philosopher. If I remember correctly, there’s some hard parts.
But I want to reference another work of philosophy, which I’m going to forget the name of, but you might remember it. It’s by Harry Frankfurt. I looked it up a second ago. I think it might be “Necessity and Desire,” but it might not be. In that paper, if I’m getting it right and we’ll link to the right one, he talks about that we have desires about our desires. So, an animal might have a desire for shelter, reproduction, food, warmth, all kinds of things on a cold, rainy night. And, we have those things, too; so in that sense, we share a certain level of consciousness with animals.
But we also have desires about our desires. I might desire ice cream, but I might desire that I didn’t like it as much as I do. And, this opens, I think–it sounds kind of trivial, but it’s actually I think quite important–it opens a way that I think about this question of AI [artificial intelligence], of robots and Westworld characters. Do you imagine the possibility that Dara will have regrets? That Dara will wish her hair were a different color, or wish she had chosen, been assigned to someone other than me or you as her roommate. Or wishes she hadn’t been cruel to Carmen unintentionally earlier that morning in an interaction over the volume level of the stereo.
For me, since it’s going to be–well, you write a lot about the fact that it’s hard to know what level of consciousness anything feels–what level of suffering and happiness anything feels–whether it’s an ant up to a dog, say, for example. And, we already have the experience of Claude and other LLMs [large language models] that act in language the way humans do. And we presume humans are like us, and we feel suffering and happiness. So, we might assume that Claude does. But, if Claude does not have regret–if Claude doesn’t have longing–like, I didn’t use Claude, say, yesterday, does he sit there? He doesn’t sit there. But, when I come back to him, he might say, ‘Gee, I was so sorry you didn’t talk to me yesterday.’ But, does that have any meaning if he’s a machine?
For me, at the current level, it certainly has no meaning–to me. You might disagree, and we might disagree on the probability that Claude will become something different. What are your thoughts on these issues of regret, desire, longing, sadness, and so on, other than their verbal manifestations, and whether that tells us anything about LLMs and other types of silicon-based things?
Jeff Sebo: Yeah. A lot there. One is about this concept of second-order desire–desires about other desires. Another is about these complex emotional states, like regret. Then a third is about the present and future state of large language models and other AI systems, and how these ideas all fit together.
So, briefly on each of these points, and then you can tell me which one you want to pursue, if any of them.
With respect to second-order desire, and then these more complex states like regret, there is no reason in principle why those should be unavailable not only for non-human animals in certain forms, but also and especially for AI systems. So my dog, for example, might not have desires about desires in the same kind of linguistic way that I do, and he also might not experience regret in the same kind of way that I do. But, he can have his own kind of meta-cognition and that can still carry some ethical weight.
So, for example, he can attend to his own perceptual experiences, as well as the perceptual experiences of others; and that kind of attentiveness can allow him to tune in to some sorts of mental states, and have different kinds of experiences, and make different kinds of decisions. And then that can affect his interests, and his goals, and what I owe him in order to make sure that I treat him well and promote his welfare.
So, that version of meta-cognition and its ethical significance can be available even to my dog. The same can be said about more complex emotional states. Perhaps not regret, because that really is tied into our language and reason. But, emotional states that are adjacent to regret.
Why does this matter for ethics? Well, there are two ways it might matter for ethics. One concerns our moral agency and the other concerns our moral patient-hood.
So, moral agency is when you have duties and responsibilities to others, and moral patient-hood is when others have duties and responsibilities to you. So, I do think that having sophisticated forms of higher order states, like belief and desire, and emotions like regret, are necessary for moral agency–for having duties and responsibilities to others. This is part of why my dog does not really have duties and responsibilities to me in the same kind of way that I do to him.
But, those complex types of higher order states and emotions are not, in my view, requirements for moral patient-hood. You can still have a life that matters to you, you can still be capable of being benefited and harmed, even if you lack the cognitive sophistication that ordinary adult humans have.
So, those are a few general remarks about I think the ethical significance of those states.
Russ Roberts: I totally agree with you on animals. We might disagree on where the–there might be a line I’d draw–I don’t think you would draw; we’ll talk about it perhaps–for animals, for non-human carbon life forms. I get in my Twitter feed videos about people tricking their dogs. Putting their hand over something and the dog makes a choice, and the dog is misled by the person. And the dog is troubled. You don’t know literally the dog is troubled because the dog can’t literally communicate, but the facial expressions, the behavior, the posture of the dog suggests disappointment, sometimes resentment. Of course, it could just be a passing state that looks like that, therefore the video gets popular on Twitter, but I’m open to that reality.
21:37
Russ Roberts: I think it’s much harder with Dara, so I want to push you there. Then we’ll talk about probabilities. But, start with the strong case for why I could imagine having to care about Dara’s welfare.
Jeff Sebo: Great. Yeah. I think that is the tough question.
As a starting point: there is no reason in principle why AI systems in the near future will be incapable of many of the types of cognitive states that humans and other animals can have. So, we already are creating AI systems, not only with physical bodies in some cases, but also with capacities for perception, attention, learning, memory, self-awareness, social awareness, language, reason, flexible decision-making, a kind of global workspace that coordinates activity across these modules. So, in terms of their functional behavioral capacities, as well as the underlying cognitive mechanisms that lead to those functional and behavioral capacities, we can expect that we will, within the next two, four, six, eight years, have AI systems with advanced and integrated versions of all of those capacities.
And that can extend to cognitive capacities that play the functional role of desires about desires, of emotions like regret.
So, I think the only question is: Will these types of cognitive capacities in AI systems come along with subjective experiences? Will it feel like something for AI systems to have desires about their own desires, or to have the functional equivalent of regret? And: Does it need to feel like something in order for AI systems with those cognitive capacities to have intrinsic moral significance and deserve respect and compassion?
So, what I think about that right now is: We can expect that there will in fact be AI systems with advanced and integrated versions of these cognitive capacities, functionally and behaviorally speaking. And, we are not right now in a position to rule out a realistic possibility that it will feel like something to them. Right now, there is enough that is unknown about the nature of consciousness–about phenomenal consciousness–that it would be premature to have a very high degree of confidence that it will feel like something to be those AI systems, or that it will not feel like anything to be those AI systems. I think right now, we can presume that such systems will exist, and we should be fairly uncertain whether and at what point it will feel like anything to be them.
That is our predicament, when we need to make decisions right now, about whether and how to scale up this technology.
24:30
Russ Roberts: So the only thing I disagree with–the first part of your remarks about that–is the self-awareness. I don’t know if we have any–I’m totally agnostic. Well, that’s not true. I’m skeptical. I wouldn’t say it’s a zero chance, which is fun because we’ll talk about the role probability plays in this. But, I’m skeptical that they’ll develop self-awareness. I might be surprised and turn out to be wrong.
It’s interesting, I think, to think about how I might come to revise that view. Right? So, if my only interface–you know, you put Claude into a physical body, a thing that looks like a human, and Claude could easily express regret. I talk to Claude in two places. I talk to him on my phone: He’s not inside my phone. He’s an app. And similarly, on my laptop on a webpage on a browser. But, if he was embodied in some dimension, in a physical thing called a robot, I’d be more likely to be fooled by Claude’s claims of self-awareness. But, I don’t know how I would ever assess whether those professions of self-awareness were real. So, I want to challenge you with that and see what you think.
But, again–and I also want to bring this back to this question of suffering and pleasure. So, it might be sentient. It might be conscious. I think the crucial question for our moral responsibilities is the one you identify, which is the Welfare Principle. Are you–is it enough that Claude has the kind of responses you talked about? Is that enough to invoke the Welfare Principle for you?
Jeff Sebo: Yeah. Those are great questions.
And by the way, I agree with you about Claude. I think that if we placed Claude in a physical body capable of navigating an environment, we might start to experience Claude as having not only self-awareness, but also morally significant interests of various kinds. And, that might be a false positive. We might be anthropomorphizing–
Russ Roberts: I have that already–
Russ Roberts: I have that already. It’s embarrassing.
Russ Roberts: I can taste it. I can’t quite–
Jeff Sebo: We all have it. We all have it. Yeah. People had it two years ago. People had it four years ago with even much, much more basic large language models. So I agree with you that that could be a false positive. That could be over-attribution of these capacities.
It is worth noting however, that even near-future AI systems might not work in the same kinds of ways that current large language models do. Current large language models do generate realistic text, realistic language outputs based on text prediction and pattern matching. And so, when they say, ‘I am self-conscious,’ or, ‘I am conscious,’ or, ‘I am morally significant,’ then we should not treat that as strong evidence that they are and that they in fact do have self-knowledge.
But, it might be that, in the near future, AI systems not only produce realistic behaviors, but produce them via the same types of cognitive mechanisms that humans and other animals use to produce similar behaviors. So, representations that function like beliefs do, like desires do, like memories do, like anticipations do, fitting together in the same kind of way. And then when those AI systems profess having a certain kind of self-awareness, then we might need to take that a little bit more seriously.
Now it is worth also noting that self-awareness, as with animals, can come in different shapes and sizes, different kinds and degrees. It might not be helpful to ask do they have self-awareness, yes or no? It might be helpful to ask: What kinds of meta-cognition do they have and lack, and what are the moral significance of those forms of meta-cognition?
But, one area where AI systems are going to outstrip animals is that they will, at least functionally, behaviorally, have human-like versions of all of these cognitive capacities, and then some. So then that goes back to your question: Is that enough for moral significance? My own personal answer is no. I really think phenomenal consciousness is a key ingredient for moral standing, intrinsic moral value. And so for me, a lot really does rest on that further question: Fine, they have language, they have reason, they have self-awareness. We can stipulate that for the sake of discussion. Does it correspond to subjective experience? Does it feel like anything to be them? Can they feel happiness and suffering? For me, intuitively, that is what everything rests on.
Russ Roberts: Yeah–
Jeff Sebo: Do I feel–sorry, go ahead.
29:20
Russ Roberts: No, no. My reaction to that is very common sense. I’m an untrained philosopher, which sometimes is an advantage. Most of the time, it will be disadvantage, I concede. But, my first thought in this setting is: It’s a machine.
Now, the fascinating part about that common sense reaction is of course is that maybe I’m a machine. I happen to be made out of flesh and blood, but I am at the mercy of algorithms, I’m at the mercy of my genes, I’m at the mercy of physical manifestations of my nervous system and endocrine system that maybe are analogous to what is going on inside of a Westworld-type robot. I don’t think so, but maybe I’m wrong. Because when you said, ‘Oh, it will have the same cognitive,’–I forget how you worded it–I’m thinking, ‘No, it won’t.’
It’ll be vaguely analogous in that there’s electric stuff in my brain as neurons fire, and there’s electric stuff in Claude’s responses in zero/one settings. And, I’m also kind of, maybe finishing sentences as I go along; I just don’t realize it. I’m looking for the next word just like Claude does, etc., etc.
But they’re not the same. I would argue that’s an illusion. Do you want to agree or push back on that? Before we get–I want to come back to the Welfare Principle.
Jeff Sebo: Great. Yeah. I guess I would both agree and push back on that.
So, in terms of pushing back, I do think that there will be at least broadly analogous cognitive capacities in AI systems in the near future at the level of cognitive representations that play the same functional role as beliefs, and desires, and memories, and anticipations, and so on and so forth.
Now, as you say, that might not mean that there is an exact one-to-one correspondence between how it works in our brains and how it works in these silicon-based systems.
For example, Peter Godfrey-Smith and other really smart philosophers and scientists point out our brains play all these roles by producing these very specific kinds of chemical and electrical signals and oscillations that at present are possible in carbon-based brains, but not in silicon-based chips. Right?
So that then leads to this further question: How fine-grained do these similarities and capacities need to be in order to realize the relevant kinds of welfare states and the relevant kinds of moral significance? Does it need to work exactly like it does in human, or mammalian, or avian brains in order to generate the relevant kinds of interests and significance? Or, is it enough for different kinds of brains to play broadly the same functional roles in different kinds of ways?
I think this is a real open question that is very difficult to answer. But I will caution us about racing to one extreme or the other extreme. On the one hand, it would be a mistake to be too coarse-grained. If we specify these in too broad a way, then any animal, any plant, any fungus, microscopic organisms can trivially satisfy these requirements. And that might be too broad. But, if we specify it in too fine-grained a way, then we might be ruling out even the possibility of consciousness or moral significance in reptiles, amphibians, fishes, octopuses; and that would be a mistake. We should be open to the possibility that different kinds of cognitive systems can realize broadly similar forms of value in different kinds of ways and not rule that out by fiat.
33:06
Russ Roberts: So, let’s turn to the basis for the Welfare Principle–which you don’t provide. Nobody does. It’s not a personal criticism.
It seems self-evident that it’s wrong to harm things and it’s good to help things. But, I want to ask why. In particular–this is not a gotcha show and it’s not much of a gotcha, and I’m sure you’ve thought about these things. I would suggest the possibility that our belief in the Welfare Principle–the ethical demands to be kind and not be cruel–comes from a religious perspective. A religious perspective that many philosophers, of course, disagree with or are uncomfortable with, either intellectually or personally.
I just want to raise the possibility–I’m curious how you’d react to it–that it’s a leftover. It’s a leftover from a long tradition of a few thousand years–3000 years in Western thought. There’s parallels of course in Eastern thought, maybe we’ll talk about those as well. It crossed my mind while I was reading your book that there are many elements of Eastern religion. There’s elements of both in your ethical principles–meaning not yours, Jeff Sebo’s, but the discipline’s–philosophy’s–ethical principles. And your book is a very nice survey of the different ways philosophers look at these questions.
But: Why should I care? If I don’t believe in God, and I think that the so-called Judeo-Christian–or Buddhist–pick your choice–or Islamic principles, are about how–about, say,–animals, or our obligations. If you don’t accept those, why should I care about how I treat other people?
Carmen–forget Carmen and Dara. [?How about?] you? I’m your roommate, but you get on my nerves, Jeff. You play the stereo late at night when I want to sleep. And I don’t like the smell of the food you cook. Whatever it is.
Now, I may try to impose my will on you and fail, but I’m more interested in the questions that your book is about. Which is: Why do I have an ethical obligation other than to my own pain and pleasure? I think I do, just to be clear. I’m asking a thought question. But, why?
Jeff Sebo: Yeah, great question. And I think we might make good roommates, because I tend to go to sleep pretty early, so I think we would get along as far as that goes.
Now this is a question in meta-ethics. So, meta-ethics is: What is the status of ethics? So, when we have ethical disagreement, ethical uncertainty, what are we doing in those moments? Are we disagreeing about an objective truth, or are we shouting our preferences at each other, and one of us will win and the other will lose through sheer force of will? [More to come, 36:15]
