One of the claims made by Picard to justify the interest in affective computing was the importance of emotion in reasoning, deliberating and decision making (Picard, 1997, 2015). In the last 30 years or so the role of emotion in reasoning has been under considerable debate (Brady, 2013; Damasio, 1994; De Sousa, 1987; Frank, 1988; Goldie, 2004; Tappolet, 2016). In this paper I argue that, if one accepts that: 1) emotion has an important role to play in reasoning and deliberation and 2) we design Artificial Intelligence Systems (AISs) with access to agents’ emotions and behaviour (in terms of recognising and predicting chains of emotions and associated behaviour), this ultimately allows cases of agent manipulation. In turn, this manipulation has direct implications for autonomous agency.

AISs are applications or technologies using Artificial Intelligence tools and techniques to perform specific tasks e.g. to find patterns in data, make predictions or decisions, recognise images etc. (Peters, 2022). To give some specific examples, recommender systems determine communications to users on social media feeds or provide scores for interviewes during interviews, algorithms operate fitness wearable devices and conversational AIs are used in customer services (Bogen, 2019; Klenk, 2021; Peters, 2022). This paper focuses on possible cases where AISs could use emotion recognition techniques to make predictions on agents’ emotions and associated behaviour with the aim to encourage or discourage certain behaviours. This influence on an agent’s emotions and behaviour admits degrees from cases of minimum influence to cases of manipulation. It would be good to note here that the manipulation claim does not imply that there is no possible positive outcome from AISs having the ability to influence emotion. Instead, I accept that such a capability can have positive applications and implications: for example, in health environments where the individual for whatever reason cannot control the intensity of an emotion, it may be beneficial to have an application which can warn the human agent or nudge them away from certain emotion/behaviour chains to help the human agent control or regulate this emotion. In these situations, the individual would prefer to be nudged away or in any case be more able to control the intensity of those emotions when it has been associated with inappropriate or life endangering behaviour.

In this paper, I focus on cases of manipulation rather than cases of pure influence. In everyday life situations, we influence each other by acting one way or another or recommending actions to other agents without this influence being considered outright morally bad. For example, even though you do not normally wake up early to go for a run, you have told me in the past that you think it would be beneficial for you to run regularly and that you do not like to run on your own. One morning I send you a text to invite you to join me for a morning run and you accept. My action is influencing your behaviour and I am letting you know that I am doing this to help you achieve a goal. Or alternatively you ask me to let you know when I do go for a morning run each time I go so that you are motivated to go for a run. It is plausible to think in this and similar cases that the agent is influenced but not manipulated. On the other hand, in cases of manipulation, an agent is manipulated when the link between her mental states and associated behaviour is interfered with in such a way that she suffers an injustice (Klenk, 2021). Borrowing from Klenk’s account, I take manipulation to occur in cases where an injustice causes an agent to have mental states which would not have existed but for the injustice occuring. To give an example used typically in the literature on manipulation, in Shakespeare’s Othello, the main character is manipulated by Iago’s lies to feel jealous and in an enraged state, kills his wife Deisdaimona. Othello is manipulated by Iago in the sense that his mental states of anger, jealousy or rage are caused by and are based on falsehoods; injustice occurs because he is obstructed from knowing the real reasons behind Iago’s words and actions.

Although an agents’ behaviour is frequently influenced by their environment and other agents, what separates influence from manipulation on this account is that in manipulation cases, agents do not have access to the reasons behind the influencer’s behaviour. This may be because the influencer refuses to give them or conceals them, as in Iago’ s case. However, we can also think of cases where these reasons are not visible because of negligence or a lack of care associated with whoever is causing the injustice (Klenk, 2021). For example, one may be spreading misinformation on social media which influences others because they have a complete lack of care about whether this information is true or not. This is different to cases of disinformation where an agent knowingly spreads lies on social media (Nguyen, 2020) to cause specific behaviours or beliefs to others. However, in both cases an injustice is caused because the manipulated agents have certain mental states e.g. desires, beliefs, emotions which they would not have had, had manipulation not occured. Although manipulation can occur through influencing both cognitive and affective mental states, in this paper I focus on influencing affective mental states and more specifically manipulating affective mental states to encourage or discourage behaviour.

If one wishes to manipulate someone’s emotions, it would be helpful if one had the ability to recognise these emotions both before and after any attempt to manipulate to confirm the attempt was successful. In everyday life interactions, we typically recognise other’s emotions through information provided from facial expressions, an agent’s behaviour or their testimony. In lab environments or when using technological devices, one can also have access to information relating to physiological responses e.g. skin conductivity or brain activation information through FMRi, EEG etc. (Fragopanagos & Taylor, 2005). These types of information can be used for recognising human emotions as in identifying what emotional state a human agent is experiencing. There is considerable debate on the accuracy of emotion recognition by both humans and AISs and associated challenges such as cultural variability and context.1 If accurate emotion recognition is an important condition for the ability to manipulate affective states efficiently, in cases where this is missing, so in cases of ‘knowing too little’, it is not guaranteed that manipulation will occur in an efficient way e.g. in the sense that the manipulator recognises that manipulation has or has not occurred.

Largely, cases of ‘knowing too little’ can result from the complex nature of emotion, a degree of unpredictability associated with human behaviour, and the lack of current technological capability to correctly recognise or classify human emotion in the wild and so outside laboratory and experimental spaces.2 However, cases of ‘knowing too little’ may dissolve with the advance of technological progress.3 For example, emotion recognition accuracy can improve by using personalisation techniques (Nwadike et al., 2024; Pearson, 2019). These techniques aim to improve emotion recognition by training AISs to recognise the facial expression of a single individual; the hope is that by learning the individual’s typical facial expressions rather than her exaggerated facial expressions, this may help increase accuracy scores. As such, it is possible that by using improved techniques we will be able, at some point in the future, to accurately recognise human emotion and predict the behaviour arising from or associated with them.

In the main part of the chapter, I accept this possibility and deal with issues arising from future AISs being able to accurately recognise human emotion and robustly predict actions associated with those emotions. I call these cases of ‘knowing too much’ to highlight issues relating to manipulation. I show that in these cases there is a restriction of agency such that it becomes difficult to claim that the agent is autonomous. The degree of manipulation can be exacerbated because 1) human agents are not aware of what types of AISs are used and to what purposes and 2) the belief that manipulation can only occur when the influencer intends to manipulate. As argued by Klenk (2021), the latter is not necessary because manipulation can also occur because of lack of care or negligence. I conclude that solving the first problem of ‘knowing too little’ may inadvertently lead to the creation of the second problem of ‘knowing too much’ and suggest some important questions for further research.4

I include below my claim as a set of premises which, if likely to be true, entail the conclusion:P1. AISs (properly designed) can manipulate the emotions and behaviour of human agents including inducing or regulating emotion.P2. Emotion influences practical deliberation in human agents viewed as the ability for self-reflective evaluation.P3. From 1 and 2 it follows that ‘AISs can manipulate (restrict or interfere with) practical deliberation of human agents’ (by manipulating their emotions and associated behaviour).P4. Practical deliberation provides justification for attributions of autonomous agency.____________Conclusion: Artificial Intelligence Systems (properly designed) can restrict autonomous agency by reducing or removing the capacity to provide justification for actions.

In the following sections I provide evidence to support premises 1 and 2 included above by discussing the role of emotion in self-reflective evalution and autonomous agency and by looking at possible cases where knowing too much about an agent’s emotions and being able to manipulate these states, restricts an agent’s autonomous agency.

Personal autonomy is identified as the capacity to evaluate one’s motives, values and goals by considering one’s beliefs and desires (Christman, 1991; Mele, 1993). If this evaluation shows that the motives do not align with one’s beliefs and desires, the motives need to be adjusted accordingly. An agent is a being with the capacity to act intentionally such that their actions are causally related to their intentions (Bratman, 1987; Davidson, 1980; Dretske, 1988; Mele, 2003).5 Combining these definitions, an autonomous agent is a being with the capacity to act intentionally and a capacity to evaluate the motives of intentional action to consider whether they align with held beliefs and desires.6

The capacity to evaluate one’s motives in light of held beliefs and desires is also described as a capacity for self-reflective evaluation. Through self-reflection agents can check whether an action aligns with their motives and if not, correct any misalignment cases. If someone or something interferes with the agent’s ability for self-reflective evaluation, for example because of conditioning or indoctrination, this removes or cancels out the agent’s autonomy (Buss & Westlund, 2018). The capacity for evaluation presupposes a capacity for self-reflection; an agent must reflect on their own values, goals, desires and beliefs when deliberating how to act. Self-reflective evaluation takes place when one deliberates about how to act; it provides evidence for autonomous agency and the exercise of freedom of the will (Bilgrami, 2006; Frankfurt, 1971). If one finds a way to cancel or interfere with this capacity, one can restrict autonomous agency.

Importantly, and as discussed above, a big part of one’s ability for evaluative self-reflection during deliberation is based on their ability to have good knowledge of their self in the sense of knowing their emotional traits and dispositions as well as knowledge of their values, goals etc. This knowledge is important during deliberation because it helps agents determine which action aligns with their values and goals and which does not. This knowledge also includes knowing what actions they are usually motivated towards when undergoing an emotion and what actions/behaviours they are motivated to avoid. To give an example, I may avoid spending time with an individual who expresses sexist or racist ideas because I do not wish to experience the emotions associated with this interaction e.g. anger, frustration etc. or the behaviours I portray in these encounters e.g. having to explain that racism is bad etc.

The importance of emotion for self-reflective evaluation is also evident in the relation between emotion and personal identity or character (Goldie, 2000). An important part of what makes us who we are, is the ability to control or regulate our emotions but also the ability to choose what emotions we allow ourselves to engage with, in what manner and to what degree. Losing control over our emotions and the behaviours associated with them means that we lose control over our personal identity. Having control over our emotions according to these views gives us control of the self-reflective evaluation process and of our personal identity. The relevance of emotion here is that according to views focusing on personality or moral motivation (e.g. Goldie (2000); Tappolet (2016)), emotions are a fundamental feature of what makes a person the person that she is; her emotional repertoire along with her attitudes towards it, are a big factor in how she behaves and what kind of personality she displays.7 To show how emotion can influence an agent’s actions in the example mentioned above, if I know that a certain individual will make me feel angry because of racist comments, I can avoid meeting that individual because I do not want to experience the emotions usually elicited by this encounter and the actions I choose to perform e.g. disagree, get into a fight etc. Generally speaking, the specific environments we find ourselves in are very important for the emotional repertoire we are encouraged to develop and maintain in the sense that they either encourage or discourage specific emotional behaviours (Colombetti, 2017). Where we have control over the environments we find ourselves in, this facilitates the control over our emotions. However, where we have minimal or no control over the environments we find ourselves in, this removes our ability to control our emotions.

One can claim that regardless of the environment, if the agent is not forced to do an action suggested by someone else, this allows for some degree of autonomous agency. According to this objection, if there is a degree of autonomy still available, this ensures that they are autonomous agents and that they can be held responsible for their actions. This objection goes against a wealth of research in philosophy and psychology showing that the environment one is living in shapes one’s character and behaviour. For example, situationist studies show that in cases where we do actually act freely, the degree of freedom is significantly reduced by the situation and environmental context we find ourselves in (Darley & Latane, 1968; Haney et al., 1973; Milgram, 1963). In such cases, primes in the environment may induce certain behaviour or decision without conscious acknowledgement by the agent that this is taking place (Wegner, 2002; Wegner & Wheatley, 1999). In the later cases, as shown in the experiments by Wegner & Wheatley (1999), the agent may report that they consciously willed a certain behaviour/action when we know that they did not. This makes it possible that manipulation can occur even when the agents are not explicitly forced to choose a specific behaviour.

Technologies oriented towards emotion recognition can use different types of information from textual analysis, clicking, velocity of scrolling, eye tracking to information on facial expressions of emotion by using Facial Expression Recognition Systems (Khare et al., 2024; Samadiani et al., 2019). For example, FERS are used in healthcare to help diagnose anxiety (McClure et al., 2003), or in virtual and augmented reality technologies to enhance the quality of the interaction (Chen et al., 2015; Hickson et al., 2017). For a more recent example, FERS are used in research projects producing Adaptive Interactive Movies (AIM) such as the film ‘Before we Disappear’ (2024) where machine learning technologies coupled with computer vision applications interpret each spectator’s moods and emotions to adapt the film in real time and choose one of possible three endings.9

In this section, I look at cases where through successful emotion recognition and personalisation AISs could manipulate emotions and associated behaviours in agents. Agreeing with Baumann & Döring (2011), I argue that emotion oriented technologies could restrict autonomous agency. This restriction becomes problematic in cases where 1) the user would not consent to it if consent was required, 2) the user is completely unaware that it takes place or 3) the user may be aware that automated emotion recognition is taking place but is unaware of associated operations e.g. predictions of behaviour and the reasons or purposes of those operations. In the latter cases, the owner of the application or the owner of the platform the application is being used on, may state that they are using this technology to provide more tailored experiences to users by recognising their preferences through emotion recognition or to monitor performance or wellbeing in the workplace (Gal et al., 2020; Spataro, 2020; Suni Lopez et al., 2019). To show why these cases may restrict autonomous agency and the capacity for self-reflection, I discuss a hypothetical case below. Although this is a hypothetical scenario, I include references showing cases where AISs are already used to perform some of the activities mentioned. This moves the scenario from the realm of fictional cases to the realm of plausible contemporary cases.

Affective states are considered as either influencing or constituting moral judgments and their importance for moral reasoning and self-reflecctive evaluation is widely accepted in the literature (Haidt, 2001; Richardson, 2018; Tappolet, 2016). If one recognises an agent’s emotional dispositions and knows which action typically follows an emotion and vice versa, one can restrict autonomous agency by interfering with the process of moral deliberation and self-reflective evaluation. If one designs and uses AISs to recognise and induce emotions and associated behaviour to human agents, this has direct implications for the agent’s capacity for self-reflecive evaluation during moral deliberation and ultimately for autonomous agency.16 Further research is needed in this area to determine the checks and balances that need to be put in place to avoid cases of manipulation and other harmful types of interaction.

I would like to thank the anonymous reviewers of this issue for the constructive comments and the attendees of ISRE2024 Conference at Queen’s University Belfast for the helpful suggestions on an earlier version of this paper.