1. Introduction

One of the claims made by Picard to justify the interest in affective computing was the importance of emotion in reasoning, deliberating and decision making (Picard 1997, 2015). In the last 30 years or so the role of emotion in reasoning has been under considerable debate (Damasio 1994; De Sousa 1987; Frank 1988; Goldie 2004; Brady 2013; Tappolet 2016). In this paper I argue that, if one accepts that: 1) emotion has an important role to play in reasoning and deliberation and 2) we design Artificial Intelligence Systems (AISs) with access to agents’ emotions and behaviour (in terms of recognising and predicting chains of emotions and associated behaviour), this ultimately allows cases of agent manipulation. In turn, this manipulation has direct implications for autonomous agency.

AISs are applications or technologies using Artificial Intelligence tools and techniques to perform specific tasks e.g. to find patterns in data, make predictions or decisions, recognise images etc. (Peters 2022). To give some specific examples, recommender systems determine communications to users on social media feeds or provide scores for interviewes during interviews, algorithms operate fitness wearable devices and conversational AIs are used in customer services (Peters 2022; Klenk 2021; Bogen 2019). This paper focuses on possible cases where AISs could use emotion recognition techniques to make predictions on agents’ emotions and associated behaviour with the aim to encourage or discourage certain behaviours. This influence on an agent’s emotions and behaviour admits degrees from cases of minimum influence to cases of manipulation. It would be good to note here that the manipulation claim does not imply that there is no possible positive outcome from AISs having the ability to influence emotion. Instead, I accept that such a capability can have positive applications and implications: for example, in health environments where the individual for whatever reason cannot control the intensity of an emotion, it may be beneficial to have an application which can warn the human agent or nudge them away from certain emotion/behaviour chains to help the human agent control or regulate this emotion. In these situations, the individual would prefer to be nudged away or in any case be more able to control the intensity of those emotions when it has been associated with inappropriate or life endangering behaviour.

In this paper, I focus on cases of manipulation rather than cases of pure influence. In everyday life situations, we influence each other by acting one way or another or recommending actions to other agents without this influence being considered outright morally bad. For example, even though you do not normally wake up early to go for a run, you have told me in the past that you think it would be beneficial for you to run regularly and that you do not like to run on your own. One morning I send you a text to invite you to join me for a morning run and you accept. My action is influencing your behaviour and I am letting you know that I am doing this to help you achieve a goal. Or alternatively you ask me to let you know when I do go for a morning run each time I go so that you are motivated to go for a run. It is plausible to think in this and similar cases that the agent is influenced but not manipulated. On the other hand, in cases of manipulation, an agent is manipulated when the link between her mental states and associated behaviour is interfered with in such a way that she suffers an injustice (Klenk 2021). Borrowing from Klenk’s account, I take manipulation to occur in cases where an injustice causes an agent to have mental states which would not have existed but for the injustice occuring. To give an example used typically in the literature on manipulation, in Shakespeare’s Othello, the main character is manipulated by Iago’s lies to feel jealous and in an enraged state, kills his wife Deisdaimona. Othello is manipulated by Iago in the sense that his mental states of anger, jealousy or rage are caused by and are based on falsehoods; injustice occurs because he is obstructed from knowing the real reasons behind Iago’s words and actions.

Although an agents’ behaviour is frequently influenced by their environment and other agents, what separates influence from manipulation on this account is that in manipulation cases, agents do not have access to the reasons behind the influencer’s behaviour. This may be because the influencer refuses to give them or conceals them, as in Iago’ s case. However, we can also think of cases where these reasons are not visible because of negligence or a lack of care associated with whoever is causing the injustice (Klenk 2021). For example, one may be spreading misinformation on social media which influences others because they have a complete lack of care about whether this information is true or not. This is different to cases of disinformation where an agent knowingly spreads lies on social media (Nguyen 2020) to cause specific behaviours or beliefs to others. However, in both cases an injustice is caused because the manipulated agents have certain mental states e.g. desires, beliefs, emotions which they would not have had, had manipulation not occured. Although manipulation can occur through influencing both cognitive and affective mental states, in this paper I focus on influencing affective mental states and more specifically manipulating affective mental states to encourage or discourage behaviour.

If one wishes to manipulate someone’s emotions, it would be helpful if one had the ability to recognise these emotions both before and after any attempt to manipulate to confirm the attempt was successful. In everyday life interactions, we typically recognise other’s emotions through information provided from facial expressions, an agent’s behaviour or their testimony. In lab environments or when using technological devices, one can also have access to information relating to physiological responses e.g. skin conductivity or brain activation information through FMRi, EEG etc. (Fragopanagos and Taylor 2005). These types of information can be used for recognising human emotions as in identifying what emotional state a human agent is experiencing. There is considerable debate on the accuracy of emotion recognition by both humans and AISs and associated challenges such as cultural variability and context. See for example Waelen (2024); Cabitza et al. (2022); Vaccaro et al. (2020); Barrett et al. (2019); Kragel et al. (2018); Kreibig & Gross (2017); Jack (2016); Hess & Hareli (2016); Scarantino et al. (2015); Gendron et al. (2014); Hassin et al. (2013); Lindquist et al. (2012); Aviezer et al. (2008). If accurate emotion recognition is an important condition for the ability to manipulate affective states efficiently, in cases where this is missing, so in cases of ‘knowing too little’, it is not guaranteed that manipulation will occur in an efficient way e.g. in the sense that the manipulator recognises that manipulation has or has not occurred.

Largely, cases of ‘knowing too little’ can result from the complex nature of emotion, a degree of unpredictability associated with human behaviour, and the lack of current technological capability to correctly recognise or classify human emotion in the wild and so outside laboratory and experimental spaces. See Cabitza et al. (2022) for a review of the current accuracy of emotion recognition techniques used by AISs. However, cases of ‘knowing too little’ may dissolve with the advance of technological progress. For current studies showing progress in this area, see Kuipers et al. (2023); Nicolai et al. (2022); Saffaryazdi et al. (2022); Küntzler et al. (2021); Loaiza (2021); Storbeck et al. (2015). For older studies showing that some emotions are recognised with enough frequency in certain contexts, see Ekman (1972); Ekman & Friesen (1971); Ekman & Cordaro (2011). For example, emotion recognition accuracy can improve by using personalisation techniques (Nwadike et al. 2024; Pearson 2019). These techniques aim to improve emotion recognition by training AISs to recognise the facial expression of a single individual; the hope is that by learning the individual’s typical facial expressions rather than her exaggerated facial expressions, this may help increase accuracy scores. As such, it is possible that by using improved techniques we will be able, at some point in the future, to accurately recognise human emotion and predict the behaviour arising from or associated with them.

In the main part of the chapter, I accept this possibility and deal with issues arising from future AISs being able to accurately recognise human emotion and robustly predict actions associated with those emotions. I call these cases of ‘knowing too much’ to highlight issues relating to manipulation. I show that in these cases there is a restriction of agency such that it becomes difficult to claim that the agent is autonomous. The degree of manipulation can be exacerbated because 1) human agents are not aware of what types of AISs are used and to what purposes and 2) the belief that manipulation can only occur when the influencer intends to manipulate. As argued by Klenk (2021), the latter is not necessary because manipulation can also occur because of lack of care or negligence. I conclude that solving the first problem of ‘knowing too little’ may inadvertently lead to the creation of the second problem of ‘knowing too much’ and suggest some important questions for further research. Even if one is pessimistic about technological progress in the future, if our current technological tools can recognise human emotion in some cases and can create associations between emotions and an agent’s behaviour, this still allows for cases of manipulation to be possible. This means that the main claim of this paper also applies to cases where AISs have these capabilities on the basis of information available to them currently.

I include below my claim as a set of premises which, if likely to be true, entail the conclusion:
P1. AISs (properly designed) can manipulate the emotions and behaviour of human agents including inducing or regulating emotion.
P2. Emotion influences practical deliberation in human agents viewed as the ability for self-reflective evaluation.
P3. From 1 and 2 it follows that ‘AISs can manipulate (restrict or interfere with) practical deliberation of human agents’ (by manipulating their emotions and associated behaviour).
P4. Practical deliberation provides justification for attributions of autonomous agency.
____________
Conclusion: Artificial Intelligence Systems (properly designed) can restrict autonomous agency by reducing or removing the capacity to provide justification for actions.

In the following sections I provide evidence to support premises 1 and 2 included above by discussing the role of emotion in self-reflective evalution and autonomous agency and by looking at possible cases where knowing too much about an agent’s emotions and being able to manipulate these states, restricts an agent’s autonomous agency.

2. Autonomous Agency and Self-Reflective Evaluation

Personal autonomy is identified as the capacity to evaluate one’s motives, values and goals by considering one’s beliefs and desires (Christman 1991; Mele 1993). If this evaluation shows that the motives do not align with one’s beliefs and desires, the motives need to be adjusted accordingly. An agent is a being with the capacity to act intentionally such that their actions are causally related to their intentions (Davidson 1980; Bratman 1987; Dretske 1988; Mele 2003). For alternative accounts of autonomous agency, see Ginet (1990) or Lowe (2008) who do not define agency in terms of causal relations between agent-involving states and events. Combining these definitions, an autonomous agent is a being with the capacity to act intentionally and a capacity to evaluate the motives of intentional action to consider whether they align with held beliefs and desires. This account of autonomous agency allows cases where the agent can still act autonomously even when they hold false beliefs about the world.

The capacity to evaluate one’s motives in light of held beliefs and desires is also described as a capacity for self-reflective evaluation. Through self-reflection agents can check whether an action aligns with their motives and if not, correct any misalignment cases. If someone or something interferes with the agent’s ability for self-reflective evaluation, for example because of conditioning or indoctrination, this removes or cancels out the agent’s autonomy (Buss and Westlund 2018). The capacity for evaluation presupposes a capacity for self-reflection; an agent must reflect on their own values, goals, desires and beliefs when deliberating how to act. Self-reflective evaluation takes place when one deliberates about how to act; it provides evidence for autonomous agency and the exercise of freedom of the will (Frankfurt 1971; Bilgrami 2006). If one finds a way to cancel or interfere with this capacity, one can restrict autonomous agency.

Importantly, and as discussed above, a big part of one’s ability for evaluative self-reflection during deliberation is based on their ability to have good knowledge of their self in the sense of knowing their emotional traits and dispositions as well as knowledge of their values, goals etc. This knowledge is important during deliberation because it helps agents determine which action aligns with their values and goals and which does not. This knowledge also includes knowing what actions they are usually motivated towards when undergoing an emotion and what actions/behaviours they are motivated to avoid. To give an example, I may avoid spending time with an individual who expresses sexist or racist ideas because I do not wish to experience the emotions associated with this interaction e.g. anger, frustration etc. or the behaviours I portray in these encounters e.g. having to explain that racism is bad etc.

The importance of emotion for self-reflective evaluation is also evident in the relation between emotion and personal identity or character (Goldie 2000). An important part of what makes us who we are, is the ability to control or regulate our emotions but also the ability to choose what emotions we allow ourselves to engage with, in what manner and to what degree. Losing control over our emotions and the behaviours associated with them means that we lose control over our personal identity. Having control over our emotions according to these views gives us control of the self-reflective evaluation process and of our personal identity. The relevance of emotion here is that according to views focusing on personality or moral motivation (e.g. Goldie (2000); Tappolet (2016)), emotions are a fundamental feature of what makes a person the person that she is; her emotional repertoire along with her attitudes towards it, are a big factor in how she behaves and what kind of personality she displays. If an individual’s identity ‘derives from the role she plays as the central character in the story of her life’ (Kind (2015, page 125); Schechtman (1996)), then not playing that central character would mean that the agent loses control of their personal identity. To show how emotion can influence an agent’s actions in the example mentioned above, if I know that a certain individual will make me feel angry because of racist comments, I can avoid meeting that individual because I do not want to experience the emotions usually elicited by this encounter and the actions I choose to perform e.g. disagree, get into a fight etc. Generally speaking, the specific environments we find ourselves in are very important for the emotional repertoire we are encouraged to develop and maintain in the sense that they either encourage or discourage specific emotional behaviours (Colombetti 2017). Where we have control over the environments we find ourselves in, this facilitates the control over our emotions. However, where we have minimal or no control over the environments we find ourselves in, this removes our ability to control our emotions.

One can claim that regardless of the environment, if the agent is not forced to do an action suggested by someone else, this allows for some degree of autonomous agency. According to this objection, if there is a degree of autonomy still available, this ensures that they are autonomous agents and that they can be held responsible for their actions. This objection goes against a wealth of research in philosophy and psychology showing that the environment one is living in shapes one’s character and behaviour. For example, situationist studies show that in cases where we do actually act freely, the degree of freedom is significantly reduced by the situation and environmental context we find ourselves in (Darley and Latane 1968; Milgram 1963; Haney et al. 1973). In such cases, primes in the environment may induce certain behaviour or decision without conscious acknowledgement by the agent that this is taking place (Wegner and Wheatley 1999; Wegner 2002). In the later cases, as shown in the experiments by Wegner & Wheatley (1999), the agent may report that they consciously willed a certain behaviour/action when we know that they did not. This makes it possible that manipulation can occur even when the agents are not explicitly forced to choose a specific behaviour.

2.1 Emotion and Self-Evaluative Reflection

There are different ways one can influence an agent’s ability for self-reflective evaluation. One way is to stop the process of evaluation outright for example by persuading the agent that such a process is not necessary so that they act without exercising this capacity. An alternative way would be to change the process of evaluation itself such that it still goes on but not as a guiding principle towards the decision; instead, its function is to provide justification for the decision after the decision has been reached. To explain how this can take place, I borrow from accounts of moral deliberation which take emotions to be essential in this process. According to these accounts, when we deliberate about what the moral thing to do is, what we are actually doing is trying to find justification for a moral judgement existing already. This moral judgment is identical to how we feel, and so to our emotions about what we are deliberating. Moral deliberation in this sense is a process we undertake when we are trying to rationalise judgments we have reached already on the basis of our emotions. According to this view, emotions are not complimentary to the process of moral deliberation; instead they are actually constitutive of or identical with moral judgements (Haidt 2001; Richardson 2018). When we are deliberating about what to do, emotions and reason are not co-creating this decision; instead, how we feel about things just is how we select to act. Reason provides a justification for the selected action as a post hoc exercise engaged to justify a decision we have already made rather than an exercise to help us reach this decision. If these strong views about the role of emotion in moral deliberation are correct and emotions are indeed constitutive of and identified with judgements, the ability to induce or encourage specific emotions to agents can lead to controlling the process of moral deliberation itself. If one controls the process by which agents decide how to act, they restrict autonomous agency. Although the examples used in this paper focus on moral reasoning, because of the motivational force of emotion any implications may be applicable to reasoning in general.

Alternative accounts of moral deliberation accept that emotion plays an important part in this process without also accepting the stronger view that emotions are identical with moral judgments. According to these accounts, emotions help us during the evaluation process by making our values more salient; they are perceptions of our values (Tappolet 2016). In this respect, they help motivate a specific action rather than another and so ultimately help us align actions with our values and beliefs. Because of the direct link between our emotions and our values or desires, we are in a better position to know which actions are compatible with them (Tappolet 2016). Another way to describe a person’s values, motives and goals is to talk about someone’s traits and dispositions, what is sometimes called one’s character (Goldie 2000). In order for an agent to be in a position to self-reflectively evaluate their actions, they need to have knowledge of their own character which includes values, goals, motives, traits and dispositions. In this way knowledge of ones’ own character along with the capacity for self-reflection are essential features of human agency (Taylor 1985).

If one accepts this latter account, one way to interfere with the process of self-reflective evaluation is to control or influence the desires, values or goals the agent accepts as their own. In this case, the agent would still be allowed to reflect on whether the motives align with her desires, but she does not freely choose these desires as her own. Here the discussion moves from the agent being free to evaluate whether her motives align with her desires during deliberation to whether the agent freely chooses which desires, goals and values she will consider her own. Typically, the latter is considered as a fundamental feature of someone’s personal identity in the sense that, in theory, we can select which goals, desires or values we wish to associate with our personality or character. However, and as argued by Meyers and others, societal influences can interfere in such a way that some agents cannot choose freely which goals and desires they can adopt in forming their own personal identity (Meyers 2002; Neisser and Jopling 1997).

Regardless of which view on the role of emotion one accepts from the views discussed above, the role of emotion during self-reflective evaluation is important: a) if one accepts the strong account, manipulating one’s emotions provides a direct route for manipulating their moral judgements. b) if one accepts the alternative account, manipulating one’s emotions has a direct effect on the adoption of goals, values, motives and ultimately the agent’s character. To conclude, if one accepts that emotion plays an important role for the capacity of agents to evaluate actions using self-reflection, manipulating emotion allows a direct route to cancelling or controlling the process of self-reflective evaluation during delibration.

3. Manipulation of Emotions by AISs

Technologies oriented towards emotion recognition can use different types of information from textual analysis, clicking, velocity of scrolling, eye tracking to information on facial expressions of emotion by using Facial Expression Recognition Systems (Khare et al. 2024; Samadiani et al. 2019). For example, FERS are used in healthcare to help diagnose anxiety (McClure et al. 2003), or in virtual and augmented reality technologies to enhance the quality of the interaction (Hickson et al. 2017; Chen et al. 2015). For a more recent example, FERS are used in research projects producing Adaptive Interactive Movies (AIM) such as the film ‘Before we Disappear’ (2024) where machine learning technologies coupled with computer vision applications interpret each spectator’s moods and emotions to adapt the film in real time and choose one of possible three endings. This AIM was co-produced by Dr Ramchurn and Blueskeye AI in 2024, see Icke (2023).

In this section, I look at cases where through successful emotion recognition and personalisation AISs could manipulate emotions and associated behaviours in agents. Agreeing with Baumann & Döring (2011), I argue that emotion oriented technologies could restrict autonomous agency. This restriction becomes problematic in cases where 1) the user would not consent to it if consent was required, 2) the user is completely unaware that it takes place or 3) the user may be aware that automated emotion recognition is taking place but is unaware of associated operations e.g. predictions of behaviour and the reasons or purposes of those operations. In the latter cases, the owner of the application or the owner of the platform the application is being used on, may state that they are using this technology to provide more tailored experiences to users by recognising their preferences through emotion recognition or to monitor performance or wellbeing in the workplace (Suni Lopez et al. 2019; Gal et al. 2020; Spataro 2020). To show why these cases may restrict autonomous agency and the capacity for self-reflection, I discuss a hypothetical case below. Although this is a hypothetical scenario, I include references showing cases where AISs are already used to perform some of the activities mentioned. This moves the scenario from the realm of fictional cases to the realm of plausible contemporary cases.

3.1 Hypothetical case

We can think of a possible scenario where an AIS, for example a recommender feeder on a digital platform, is tasked with eliciting as many behaviours/actions of type x in the human agent (HA) without having acquired relevant consent by the HA or without revealing the purposes behind using this technology. For a comprehensive review of recommender systems and the ethical risks associated with them see Milano et al. (2020). The AIS is not designed with any ethical or moral code which would stop it from using any available means to generate specific types of behaviour. In this scenario, and as is typically the case in digital platforms, the end goal of the recommender system is to elicite specific behaviours by controlling both the content of the feed and the time allocated to each feed with the ultimate goal to increase commercial profit. For a discussion of the different types of recommender systems and the tasks they usually perform see Burr et al. (2018). There are several ethical risks associated with using recommender systems such as risks to privacy and opacity. In this paper, I focus on the risks associated with autonomous agency and personal identity.Milano et al. (2020) suggest a taxonomy for risks associated with recommender systems. The risk to autonomous agency and personal identity discussed in this paper falls under the ‘rights violations’ category in their taxonomy. It is reasonable to accept that the AIS has the ability to scaffold the human agent’s environment e.g. by recommending specific events, facts, opinion articles, comments, videos, images etc. and attract the individual’s attention to specific content. This scaffolding creates specific affordances for the user in the sense that the content chosen by the recommender system encourages the elicitation of specific emotions to the user. This is similar to the impact that comments or information from users of social media platforms can have on the social platform users. As Steinert & Dennis (2022, page 36) describe, social media technologies ‘facilitate online emotions because of emotional affordances’. In general, affordances are interpreted as possible actions enabled by the environment (Gibson 1979). More specifically affordances in social media technologies (SMTs) are ‘the relational properties of SMTs that are likely to induce an emotional state or emotion-related behavior’ (Steinert and Dennis 2022). If the AIS through using automated recognition can identify that specific contents elicit emotions in agents and others do not, they may use the former to capture the agent’s attention. Automated emotion recognition in this case would be based on processing information on sentiment analysis referring to text, clicking, velocity of scrolling or FERSs. It currently is not likely that many users of online platforms would consent to giving access to facial expressions during use of platforms but automated emotion recognition can be successful by processing other types of available information as mentioned earlier. The ability to attract the agent’s attention is an important step in this process. If the AIS can also be trained via personalisation to make successful associations between identified emotional states and correlating behaviour, they can then predict an agent’s behaviour which then gives them a direct route to bringing this behaviour about.

If the above scenario is possible, the AIS is in a position to attract the HA’s attention and elicite a specific emotion to the HA. Which emotion the AIS will select, depends on the behaviour associated with that emotion. For example, if the HA spends more time on a specific digital platform when feeling fear or anger, the AIS could select to elicite fear or anger in the HA via some communication so that the HA spends more time on the digital platform. Alternatively, the AIS may identify that the HA is currently undergoing a specific emotion, e.g. anger and then send them photos or articles or comments from specific platforms to enhance this anger and promote engagement with these platforms such as reading comments by others, making comments etc.

In the scenario mentioned above, AISs could induce specific emotions in agents where they have been trained to know that these emotions are associated with specific behaviours in these agents and they are tasked with bringing about a specific behaviour. In other words, if they know that this behaviour robustly follows a specific emotional state, and they also know what cues usually bring this emotional state about, they can use these cues to bring the emotional state about. To give an example, if the AIS is tasked with increasing agent’s A screen time on a particular digital platform and they are in a position to know which emotional states increase screen time on the platform and also which cues bring about these emotional states in agent A, they can increase A’s screen time by recommending content which elicits these emotional states. The capability for manipulation can increase through personalisation techniques when AISs become very efficient in recognising what emotions a specific agent is experiencing and can associate these emotions with behaviours e.g. with experiencing other emotions, spending time on specific websites etc.

3.2 Manipulation of autonomous agency

The case above would qualify as a case of manipulation because the agent is not aware of the reasons why the recommender system is recommending this content to them. Injustice in this case can arise because of lack of consent, lack of knowledge of the mechanisms underlying this interaction, or lack of declaring the commercial profit motivating the manipulation.

The risk to autonomous agency in these cases is associated with the ability of the AIS not only to predict certain behaviours but also to cause them. For example, the AIS can induce feelings of anger or frustration by recommending an article on a specific issue which has caused you feelings of anger on a previous occasion. In that occasion, you posted an angry comment on social media which caused controversy and then resulted to your comment getting many responses. With this knowledge, the AIS can decide that it is advantageous for the task it is currently performing e.g. to increase your screen time on the platform, to induce the same type of behaviour. To give other examples, the AIS can recommend an article asking you to sign a political petition out of a newly induced moral judgment about the content of it, or to vote towards a certain goal, or to sign up to a particular group advocating the newly induced moral judgement etc. Knowing that you are prone to do these things whilst feeling angry, may give it the power to manipulate your behaviour one way or another. As such, it becomes pertinent to ask whether agents lose their autonomous agency, in part or wholly, when they are manipulated into specific behaviours by AISs.

Cases where AISs accurately recognise and predict an agent’s behaviour show that they know ‘too much’ about an agent; in these cases, AISs can be used to restrict autonomous agency. If emotion plays an important part in self-reflective evaluation during deliberation either by interfering or by controlling it, the combination of 1) the ability to induce emotions to an agent and 2) knowledge of associations of emotion-action in an agent, can lead to manipulation and restriction of autonomous agency. If we design AISs with a capability to recognise and predict human emotion, perhaps by combining applications already operating in this area, this will enable their human owners to be in a position of power compared to the human agents these AISs are developed to interact with. In those cases, manipulation takes place when the AIS either induces or enhances an emotion and in this way is able to increase the probability of specific motivations and actions associated with this specific emotion. I accept that the manipulation claim does not need to apply to all cases and that there will be some human agents who will not be manipulated. In addition, some agents may be manipulated in some instances but not in others. However, the claim is not that this manipulation takes place in all cases. Instead, the claim is that this manipulation is possible in some cases for some agents. This ability goes in both directions e.g. it is an ability to bring about certain actions, behaviours or choices, but also it is an ability to stop or nudge away from certain emotions and actions, behaviours of choices arising from these emotions.

If the scenario above is plausible, by improving currently existing emotion recognition techniques and combining them with other techniques such as personalisation, we allow manipulation cases in environments where AISs interact with human agents. Agreeing with Milano et al. (2020), such manipulation cases can be considered as rights violations because they restrict or abuse autonomous agency. Further research in this area would help determine rights and responsibilities for all stakeholders in these environments along with necessary steps to avoid violation of rights and harmful interactions.

4. Conclusion

Affective states are considered as either influencing or constituting moral judgments and their importance for moral reasoning and self-reflecctive evaluation is widely accepted in the literature (Haidt 2001; Tappolet 2016; Richardson 2018). If one recognises an agent’s emotional dispositions and knows which action typically follows an emotion and vice versa, one can restrict autonomous agency by interfering with the process of moral deliberation and self-reflective evaluation. If one designs and uses AISs to recognise and induce emotions and associated behaviour to human agents, this has direct implications for the agent’s capacity for self-reflecive evaluation during moral deliberation and ultimately for autonomous agency. This can be used as another restrictive tool which exacerbates existing oppression and power inequalities. Further research is needed in this area to determine the checks and balances that need to be put in place to avoid cases of manipulation and other harmful types of interaction.

Note

I would like to thank the anonymous reviewers of this issue for the constructive comments and the attendees of ISRE2024 Conference at Queen’s University Belfast for the helpful suggestions on an earlier version of this paper.

References