The Terms Affective Computing and Emotion AI

The term Affective Computing is used here, for analytical reasons, as a superordinate term for various forms of data processing that allow conclusions to be drawn about emotions, affects or other inner states The term “inner state,” is initially used here descriptively, without being affirmed, to refer to the presupposed lived experiences that are coded in this context as distinct feelings or emotions such as joy, anger, sadness, disgust, shame, etc., in all their variations and temporal sequences. The terms affect and emotion are used here in the sense of the discourses examined as related expressions, but not as separate concepts. In general, the term affect refers more strongly in its etymology to rapid felt inner movements accompanied by distinct physical reactions. Emotions are often conscious, reflected or reflectable, and communicable ways of experiencing and relating to others. The objects, states, and dynamics recorded by measurement technology and processed by computers are understood here in the sense discussed as affects, emotions, moods, passions, and unnamed modes of experience, which are represented by clusters of data from various sources.
The article does not endorse the assumption implicitly and explicitly propagated by proponents of Emotion AI approaches that computer-based measurement technologies can adequately capture or represent the inner states of individuals. The argument presented focuses on the question of what can and must be encoded and recoded as emotion and affect in technical systems in order to be processed and transmitted. based on one or multiple data sources such as the computer user’s facial or bodily changes, that are documented as expressions (visual data), temperature changes (of the whole body or body parts, the face or hands), voice or sound data, heart rate (ECG), skin resistance, brain activity (e.g. EEG) and others (Afzal et al. 2024; See for an overwiew Ahmadpour et al. 2025). Affective computing encompasses the functions and services known as emotion recognition.

The areas of application include health care, intelligent driving, security and identity checks as well as surveillance in public spaces, behavioral research, advertising analysis and general sentiment analysis. When AI-powered and connected to artificial neural networks (ANN), these forms of computing are referred to as Emotion AI in this contribution. Affective Computing is the umbrella term for a historical change in computing since the 1990s, while Emotion AI is the newer notion that describes the powerful combination of ANNs with affective computing.

Introduction

This contribution examines Affective Computing as a new technical form of encoding emotions and suggests that it should therefore be critically understood in terms of its conceptual normative implications as virtual semantics. Ethical and conceptual normative questions in the discussions about affective computing have so far either focused on the possible social, moral and legal consequences of this technology (Weber-Guskar and Menges 2025) or the reductionist conceptions of emotion and affect applied in these forms of computing (Crawford 2021; Leys 2017; Tuschling 2014) especially since the fundamental criticism of psychologist Lisa Feldman Barrett and her team (Barrett et al. 2019; Barrett 2006).

In this contribution, however, Affective Computing itself and not its possible consequences or underlying concepts will be examined as conceptually normative by proposing that it has to be understood as a virtual semantics. Here, I define virtual semantics as a set of logical-functional rules that provide a framework within which syntactically operating systems link their processed objects—or, in this case, digital data—with categorized valences and horizons/spaces of meaning. Regarding Affective Computing, this signifies the systematic connection or matching of clustered sensor data of the physiological measurements of computer users or captured beings like pedestrians in a public square with defined states See FN 1. that are decoded as emotions and come with a space of meaning (as in a certain pattern of measurements are matched to a state called sadness that is set in a space of meaning with a spectrum of states See FN 1. and experiences called depression, grief, melancholy). The term “virtual” in virtual semantics is used here both in the philosophical sense, in the French tradition and that of Charles Sanders Peirce as something that has similar effects and functions (Peirce 1920; Sprenger 2023, page 167–168; Chalmers 2023, page 187–188), but a different physical form than natural-language semantics as well as a term for Affective Computing’s embeddedness in virtual life worlds (Rieger et al. 2020).

In its implementation, the article uses a media-semiotic and media-archaeological approach (Weatherby 2025; Parikka 2012) and develops its assumptions about Affective Computing as a new and normative form of coding emotions in three steps: 1) From the outset, the term “code” was necessarily polysemic, even with regard to its origin. 2) What we think of as emotion and affect has always referred to representations of inner states and historically changing forms of coding experiences (historical examples include love letters and novels, and now computer-based recodings such as Affective Computing and Emotion AI. On this basis, 3) Affective Computing can be understood as a system for recoding older forms of emotional expression and as virtual semantics. With reference to digital media as a new environment that Mark Weiser first described (Alpsancar 2012; Gramelsberger 2023), Affective Computing takes on the role of a virtual semantics in these environments or worlds.

Against this background I want to point out the importance of a media-archeological perspective to fully understand the development and functionality of Affective Computing and propose the notion of a quasi-semantics or virtual semantics. Insofar as Affective Computing relies on different traditions to encode emotions in a specific way, it is always to be considered normative—often without disclosing its normative implications. The contribution presents its notion of Affective Computing as virtual semantics in three steps:

1. The polysemic nature of code: The notion of code has always been polysemic. Firstly, code is a purely functional way to regulate the elements and use of any technical, cipher, or sign system. Secondly, as an inevitable consequence, it not only paves the way for the functional, i.e. syntactic use of the given system, but also indirectly and necessarily shapes the semantic use of this system.

2. Encoding emotions—historical changes: To be able to express emotions and to communicate, transfer and share emotions, they have to be encoded, either in traditional sign systems like natural languages or in pattern recognition by machines as it is applied today in Affective Computing and Emotion AI.

3. Affective Computing as virtual semantics: Affective Computing is a powerful form to recode/encode emotions and embodies virtual semantics. In Affective Computing, coding in the sense of computer codes connect in a functional, but ethically challenging way with older historical forms to encode emotions (in/as verbal and/or nonverbal/pictorial signs with corresponding bodily expressions and conscious or nonconscious reactions/answers).

Step 1: The polysemic Nature of Code

The term code refers to many different practices and contexts in history, culture and technology: For example, we talk about a dress code when special attire is required for certain occasions (Fig. 1). We are all familiar with codes of conduct that apply to public spaces and, nowadays, to our digital social exchange on platforms.

In scientific use, code concepts and terms by now occupy an important place, especially in computer science. The term code is already ambiguous in the era before modern computing because it has its roots in the opposing fields of cryptography (Fig. 2) and law (Fig. 3), such as the historical Napoleonic Code (Nöth 2000, page 216; Vismann 2000).

Now the word code is often used as a short version for computer code as one of the most important foundations of our contemporary digital culture. The term ‘coding’ was even the earlier and more important term than programming, with which one of the first American programmers Grace Hopper identified strongly at the beginning of the computer era in the 1940s (Fig. 4): “We were not programmers in those days. The word had not yet come over from England. We were ‘coders’” (Hopper 1981, page 7).

The concept of code developed here encompasses the field of computing as well as the older meanings and areas of application of the term. This is intended to underscore the striking ambiguity of the term code, which has persisted throughout its history and remains relevant in today’s digital society. But what is the polysemic nature of the concept of code exactly?

In pragmatic semiotics, as developed by Eco following Charles Sanders Peirce, code has not always the same, but different meanings. In semiotics, the word code has always had a “narrower” and a “broader” meaning, as Eco critically notes (2002, page 58). In its narrower sense, encoding means organizing or formulating or encrypting a message according to the specific rules of a particular code. In a broader sense, coding also means using a technical or linguistic code, as it is referred to in the narrower sense, to formulate a message that can articulate and convey thoughts or feelings. Or to put it another way: if the narrow meaning of code refers to the selection according to the rules of a sign system, the broad meaning refers to the expressive functions associated with this selection. Many “semiotic explanations” would not pay enough attention to the fact that the term code contains these two meanings (ibid.: 57). According to Eco, this results in a profound ambiguity of the concept of code, that I call the polysemic nature of code.

Eco distinguishes the dimension of encoding a message from the second dimension of the concept of code, that of conveying content, by speaking of the establishment of syntactic rules on the one hand and the establishment of semantic rules on the other (ibid.: 58).

A classic example of the narrower meaning of the word code, that is the use of syntactic rules, is the transmission of a message in Morse code (Fig. 5).

But also, the use of the Greek alphabet—or any other writing system—falls in this category. However, the formulation of a thought in the Greek alphabetic script is also equivalent to coding in the sense of the broader meaning of the word as the establishment and practice of semantic rules. Eco points out that semiotic research cannot avoid dealing with both processes, i.e. the establishment of syntactic AND semiotic rules. This is due to an important and media-theoretically significant reason, which is related to the scope of the entire sign system. For the selection of the possible signs of a system also determines which combinations of signs—and, as a secondary but inevitable consequence, meanings—are possible at all. Eco states about the syntactic and semantic procedures that are equally meant by the code:

“In semiotic research, “code” is usually understood to mean both processes. This confusion is justified for a subtle reason: If a code has selected certain combinable units in a purely syntactic way to the exclusion of others, it is precisely because this operation served to enable a semantic function.” (Eco 2002, page 58)

For any media-semiotic research—the generation and communication of linguistic content, semantics and sense—in the verbal basis of the word, the technical conditions of meaning generation must therefore be taken into account. This is also the case because “the codes are the necessary and sufficient condition for the existence of the sign” (Eco 1977, page 170–171). At this point, semiotics uses a definition problem to describe the basic medial operation that characterizes coding processes of various forms. Code, as the semiotician Umberto Eco shows, following the mathematical theory of communication, introduces “ordering possibilities” into the set of communication possibilities of a system, which are primarily intended to enable the transferability of messages, because “the code represents a probability system that is placed over the equal probability of the initial system in order to control it communicatively” (Eco 2002, page 57). This means that Eco’s approach can be used as media-semiotics, as he emphasizes the fundamental “model character of the sign” (Heilmann and Venus 2014, page 54).

Umberto Eco defines code in general as the “conventionalized system of metalinguistic rules that assign certain elements of expression to certain cultural units” (Eco 1977, page 184). Even this definition of code, which refers primarily to the use of linguistic signs in the narrower sense, can be extended to define the coding of other objects, situations or experiences like emotions, because even in affectively charged speech and emotional verbal and nonverbal expression, the more or less conventionalized assignment of certain expressive elements to more or less habitual or culturally shaped emotional states takes place. Among the numerous examples of these conventionalized expressions of emotion, raising both arms as a sign of great joy and covering the mouth with the hand to express sudden surprise and/or shock come immediately to mind. While gestural signs of emotion continue to exist, new symbols of emotion have been added through technical communication, of which emojis are the most visible and dynamic. A differentiated renegotiation of encoding emotions is precisely what the field of standardized pictograms, known in media as emojis, represents (Stark and Crawford 2015). Before the introduction of emojis and in parallel with the networking of computers, emotion psychology began to standardize visible emotional behavior patterns or literally coded them via myriads of experiments—the famous Facial Action Coding System (Ekman 1978) had this as its main goal and intention. And while the basic process of encoding or coding is clearly evident in the name of the historical FAC system, the technical necessity and logic is nevertheless inherent in ALL forms of technical processing of perception and behavior data (called affects and emotions).

The developed polysemy of this code term is framed by the concept of virtual semantics introduced here, which is applied to Affective Computing in step 3. Or to put it more bluntly: My choice of words, my habitually developed and shared gestures, nods etc., are not only the basis but also the ultimate limit of what I can say and express. And last not least computed emotions can now be understood more precisely as those entities, objects, patterns, clusters of data that were and further can be digitally encoded. I am not claiming that there are not many uncodified forms of expression in the individual way of being and communicating that are conveyed and lived out. At the same time, it should be acknowledged that there is an enormous amount of repetition and also—not always unpleasant—noise in human life. Presenting this polysemic nature of the term code makes it possible to develop two points for the overall analysis of Affective Computing as virtual semantics:

Firstly, to show that coding is inherent in every sign system and thus in every linguistic formulation, every expressed experience, and every felt communicated state. See FN 1 for the term “state”. Physical expressions or traces of emotions must either be recorded as physiological processes using technical devices and thus measured as is prominently done in Affective Computing, or emotions must be communicated gesturally and usually verbally in order to be shared. In all these cases, emotions must be encoded linguistically and/or technically.

Secondly, to show that media-semiotic sign theory, especially as represented by the semiotician and pragmatist Umberto Eco, formulates a complex relationship between syntax and semantics in the concept of code. At first glance, coding in a sign system concerns only the syntactic use of signs. Syntactic use of a sign system covers the actual forms and types of the matching elements like letters in an alphabet as well as their applicable combinations, their possible links and connections in a given sign system.

However—as I want to point out—, by establishing rules for the forms and relationships of signs, the used syntax in a certain code fundamentally and inevitably affects the semantic use of the sign system based on it. That is the core of what I call the polysemic nature of code. I link this polysemic concept of “code” to the concept of virtual semantics.

Step 2: Encoding Emotions—historical Changes from Expressing inner Movements and States to Making Emotions computable

In his large-scale study on code as a cultural form, media historian Bernhard Dionysius Geoghegan describes how the historically changing forms of encoding information, but also language and ultimately culture itself, were addressed with the mathematical information theory in the advent of modern computers (Geoghegan 2023). What had previously appeared as different rules, conventions and habits concerning such diverse things as dialects, the exchange of goods, marriage customs and aesthetic expressions in highly diverse societies, appeared through the lens of the mathematical theory of communication as the patterning of scientists such as Gregory Bateson, Margaret Mead and their colleagues in the colonies, hospitals and suburbs (Geoghegan 2023, page 53 f.). For Geoghegan, the concept of code refers to a distributed and non-organized tendency to formalize previously implicit relationships, things and objects, as did the standardized family tree representations in ethnography, and thus make them at least potentially technically processable.

Geoghegan’s “code” spans an arc from the linguistic turn in language theory (Ferdinand de Saussure and others) and anthropology (Claude Lévi-Strauss) to contemporary practices of coding as an extension of information-theoretical cultural analysis (Warren Weaver). In this chain of powerful coding practices, I include the form of coding of emotions, affects and moods in computing and artificial intelligence, often on the basis of various physiological parameters that are read out by sensors, as in Affective Computing.

According to Bernhard Siegert, the previous century saw the replacement of humanistic logic and reason by technology in the form of media codes (Siegert 2018)—that includes all sorts of content even, as I want to add, such intimate things as bodily changes, moods, whims, hesitations, ticks and the like.

In terms of media-semiotics, it was developed that the concept of code has both a syntactic and, indirectly, a semantic function. Furthermore, code in the sense developed here refers to natural sign systems such as spoken language, but also to techniques such as alphabetic writing and, above all, to sign-processing machines such as electronic digital computers. Conversely, however, this means that it is not only Affective Computing and Emotion AI that discretize and code human experiences, but that emotional messages conveyed through language have always been encoded, whether in art in its various forms, and of course in poetry, literature and even in personal love letters. This section traces now parts of the history of emotion transmission in order to describe the historical change and transfer of these forms of coding. The analysis presented here brings the negotiation of one’s own emotions in written form, as for example known from the tradition of love letter writing, among other things, closer to AI. At the same time, the focus on interlocking forms of coding serves to understand the special form of coding emotion-related data in AI as normative and to make its functioning problematic, because it embodies a shift to automation in the history of encoding emotions.

The historical and technical transition from a handwritten, emotionally charged letter-writing culture to mechanical and ultimately electronic digital word processing (Heilmann 2012) became a theme in early computer programming projects. Between 1953 and 1954, several typewritten love letters appeared on the bulletin boards of the University of Manchester, passionately addressed to an undefined, gender-ambiguous other and signed with the initials MUC. They read something like this:

Darling Sweetheart,

You are my avid fellow feeling. My affection curiously clings to your passionate wish. My liking yearns for your heart. You are my wistful sympathy: my tender liking.

Yours beautifully,
M. U. C. (Strachey 1954, page 26)

These were most likely the first love letters generated by a computer algorithm, written by programmer Christopher Strachey. Strachey later went down in history primarily as the inventor of time-sharing (Corbató et al. 1962). Originally a mathematician, Strachey came to computer development indirectly and after personal crises through a friendship with Alan Turing (Campbell-Kelly 1985). Due to his unusual approach to computing, Christopher Strachey is increasingly being recognized in media and literary studies, although the amount of work on him remains relatively small in comparison. Noteworthy examples include an early short biography by Martin Campbell-Kelly, Jacob Gaboury’s appreciation of Strachey in the context of queer computing (Gaboury 2013), and the discussion of his contribution to stochastic poetry (Bernhart and Richter 2021).

During a period when he had little to do, his biographer Martin Campbell-Kelly reports that Strachey relieved his boredom by writing the so-called Love Letter Generator for the Manchester University computer (Ferranti Mark I) (Campbell-Kelly 1985). His sister Barbara advised him on the selection of possible words from the Roget’s Thesaurus to be used by the algorithm (Strachey 1954, page 27).

The algorithm worked with surprisingly few, but well-chosen words, which were divided into six groups: greetings (Saluations1, Salutations2), adjectives, nouns, adverbs, verbs (Strachey 1954, page 26). The computer scientist with the handle gingerbeardman forensically reconstructed the code for an exhibition and documented it on a website and on Github (see https://www.gingerbeardman.com/loveletter/). If we now apply Eco’s media-semiotic concept of code from the previous section to the situation, these word groups not only represent the syntactic possibilities, but also define the semantic framework and thus, in terms of media technology, the affective spectrum of meaning of the “love” in the generated letters. Here we see a shift from the letter-writer’s own choice of words—that is, a form of self-coding—to a greater degree of external preselection through programming—that is, a form of external coding.

Strachey stands out from the group of early computer developers because he did not view programming as a purpose-driven activity. Toni Bernhart and Sandra Richter therefore attest to the “playful nature of his programming work” (Bernhart and Richter 2021, page 14), and Gaboury sees a similarity between Strachey’s approach to technology and Turing’s recognition scene from The Imitation Game (Gaboury 2013). For Turing and Strachey, technology is no longer the excluded other per se, but a possible object of affective and linguistic-social recognition. Jacob Gaboury also wants to understand Strachey's work in the context of queer computing as an ironization of conventional, heterosexual emotional language (Gaboury 2013, 2022). Bernhart and Richter similarly emphasize the gap between convention and emotion in Strachey’s work:

„Paradoxically, Strachey allows the program-controlled computer, which he deliberately characterizes as a non-thinking and therefore non-human machine incapable of emotion, to produce precisely the type of text that, in its typical emotional intensity, expresses the exact opposite.” (Bernhart and Richter 2021, page 14)

In two respects, Strachey’s Love Letter Generator represents a historic shift in the coding of emotions that continues to have an impact on Affective Computing and Emotion AI: First, it highlights the major differences between simple and repetitive ways of generating expressive—in this case written—documents and their quite remarkable concrete form. Second, Strachey’s program demonstrates that technology is socially negotiated as a kind of performance (Gaboury 2022). Strachey’s Love Letter Generator serves also to illustrate his assessment of the early possibilities of computers as intelligent actors.

Strachey clearly states that computers, though, do not “think”—and it can be added here that they do not “feel” either, in order to emphasize the importance of programming (Strachey 1954, page 25). The statement that computers do not think and feel applies to all programs that seek to give the appearance of machine thinking—and, it should be added, perceiving or feeling—i.e., all computer programs and systems that we have since come to refer to as Artificial Intelligence. Nevertheless, their “tricks” would lead to unexpected and interesting results: “This is true of all programs which make the computer appear to think; (...) However, sometimes these tricks can lead to quite unexpected and interesting results.” (Strachey 1954, page 27). The same applies to emotions or emotional intelligence, which are here imitated and recreated in a purely mechanical and performative manner. Compared to later forms of Affective Computing and Emotion AI, the Love Letter Generator does not analyze any user data and assign it to emotions. Rather, the written expression of infatuation and deep feeling suggests an emotional state on the part of the machine. Nevertheless, the Love Letter Generator deserves consideration within the genealogy of Emotion AI, precisely because it exhibits and utilizes the mutual attribution of various emotional states in the process of writing-based encoding of affects and emotions.

This does not imply a lack of humanity, but rather that technical evidence reflects something about human evidence that is often overlooked and ignored. Human declarations of love, especially in written form, may be unique and unforgettable expressions of the sender’s and recipient’s experiences and feelings. For third parties, however, they are often anything but free of patterns and schematic repetitions. Human expressions of love, as Strachey exposes with his Love Letter Generator, are as irreplaceable as they are interchangeable—and this is not a bad thing per se. Language has always been our technical Cyrano.

The Love Letter Generator is not just one example among many. Rather, it embodies the shift in emotion coding from analog writing to computer coding. Even if the Love Letter Generator cannot be technically compared with today’s AI, it is still one of the first technical systems that could synthesize emotional written testimonies. It is therefore permissible in the history of modern computing in general, and in the history of AI with a focus on emotion simulation and recognition in particular, to draw a historical arc from Strachey’s Love Letter Generator to ANN-based Affective Computing and Emotion AI.

On the one hand, in order to understand it, the history of emotion coding can be traced from gestural expression and verbal explanation to minnesong, poetry, and literature, and finally to computer-based emotion recognition. On the other hand, the last step in particular represents a turning point, as there is a significant shift from self-coding to external coding.

Step 3: Affective Computing as a Virtual Semantics

This contribution develops a perspective on Affective Computing as virtual semantics in a media-archeological and media-semiotic sense, focusing on the basic operation of a framework—a semantics—of encoded “emotions” in their socio-technological embeddings. Affective Computing and Emotion AI can be defined as a semantics because the basic syntactic principles of sensor-based and emotion-related data processing have—or can assume—strong semantic functions. These forms of computing are described here as virtual semantics for two reasons: First, the adjective “virtual” discursively marks the networked computer environment and lifeworlds in which these semantic effects emerge. Second, and of greater theoretical significance, it underscores a specific similarity and simultaneous difference from natural-language semantics. The virtual semantics of affective computing and Emotion AI is an “as-if” semantics. Charles Sanders Peirce expresses this in his concept of virtuality or even “enshrined” it with the following equation (Chalmers 2023, page 187): “A virtual X (where X is a common noun) is something, not an X, which has the efficiency (virtus) of an X” (Peirce 1920, page 763). The term “virtual” in virtual semantics also is part of the Deleuzian tradition of a virtual entity as something not yet realized but nevertheless real (Münker 2005). To summarize and clarify this point: The term “virtual” in virtual semantics refers both to the digital environment of the forms and systems of encoding under consideration and to the fundamental difference that still exists from other linguistic semantics, precisely because the practices of encoding mark a significant shift from the written communication of emotional experience to sensor-based computing.

In the course of the 20th century, the term code changed its meaning with the development of modern computers, so that the word code eventually came to epitomize the practice of programming machines. When converting analog phenomena such as sounds, images and language into digital data, the process of a certain—but not necessarily binary—“numerical coding” must take place (Heilmann 2010, page 128). The world’s libraries are no longer just filled with handwritten and printed artifacts, but also with computer code (https://codelibrary.opendatasoft.com/). The very data that makes up Affective Computing and is supposed to relate to emotionality must be coded and discretized in a certain way in order to make it machine-processable in the first place. As the media-semiotic framing makes clear, every form of coding involves a shaping process that cannot be circumvented. In the case of Affective Computing and AI systems, however, decisions about specific coding methods are made by third parties, consisting of scientists, programmers, nation states and corporate organizations (See for the highly problematic case of emotion recognition by nation states: Sanchez-Monedero & Dencik (2022)). These various third parties alone or combined can but do not necessarily have problematic or outright bad intentions, but as per today they don’t reflect the normative consequences of their technological pragmatism—but who determine what will later be defined as emotion in the context of technical recognition applied in their systems and embedded in social contexts. In concrete terms, this means, for example, that the services and systems offered under the name Emotion Recognition, Affective Computing, Empathic Technologies and others systematically implement which emotions are processed and how. Until now, the number of emotions was often fixed, usually remaining in the single digits and focusing on, for example, four states (clusters of data) defined as joy, anger, sadness and surprise (see for example the trial with an emotion recognition software in Berlin: Projekt zur Gesichtserkennung erfolgreich (2018)).

Specifically, this will now be examined step by step in terms of the data flow involved in emotion processing within potential applications of Affective Computing and Emotion AI (Fig. 6.). The steps here refer to the various components and stages of data collection and processing, which in some cases are mandatory and in others optional as part of the technical processes of Affective Computing and Emotion AI discussed here (see for recent overviews from the field: Afzal et al. (2024); Ahmadpour et al. (2025)). To this end, I propose an analysis scheme that embeds the data flow and device use in a general system in order to show the coding and recoding processes in each case: 1. Data sources: stimuli and environment, 2. Data collection: sensory conditions, 3. Data accumulation and modulation: sensor combination and time frames, 4. Data patterns: detection emotions, affects, sentiments and the like. I will now go through the steps in the system, characterize them, and examine the coding and recoding processes in each case in order to prepare the definition of Affective Computing and Emotion AI as virtual semantics. The steps do not necessarily have to be taken in every case of Affective Computing and Emotion AI, nor do they have to be taken in the same form or in the order specified. However, they provide a good overview of the schematics of Affective Computing and Emotion AI, which helps to substantiate the arguments presented.

1. Data sources

The first schematic step is to locate and identify the data sources used in the systems. The start of the process of an Affective Computing application or emotion detection, which is schematized here for analysis purposes, is all about data sources. Data sources can, but must not in any case include a stimulus, that is a defined object or perception invoking a certain experience that is often alternatively called reaction behavior or even emotion itself (Tuschling 2022). In experimental psychology, a stimulus is defined as an object that has been specifically designed and tested to be seen, heard, felt, or perceived in a particular way. In emotion research, these objects are often photographs or videos intended to elicit a specific emotional response (Mollahosseini et al. 2017). A classical stimulus is a photograph of a spider or snake for example to trigger fear (or in other cases curiosity or even joy). But a stimulus can also be a movie or clip, as visualized, and the Affective Computing application tracks or measures the reception process in the viewer. In the age of mobile computing, the data sources can be identical to a whole environment in a car, in the street or in a forest. In many cases, the data sources themselves contain coding forms that require recoding, for example when spoken emotional content is converted into discrete form and combined into patterns in the data collection, accumulation, modulation and patterning processes.

2. Data collection

The second schematic step is to take measurements and collect data using various sensors. These can be either “just” functional elements on websites, touchscreens, cameras, and other interfaces on PCs, laptops, and tablets, or multiple uses of other forms of measurement that are used stationary in the laboratory and/or as wearables. In the most comprehensive scenario, the five major response values—visual responses, skin resistance, heart rate, heat patterns, and EEG and more—are collected, correlated with each other, and assigned to emotional states (discrete emotions). Due to the wide variety of possible scenarios—from the laboratory to everyday smartphone use—, this article will therefore refer to sensory conditions, which vary in each case. For the argument developed here, the significance of the differences between the forms of Affective Computing takes a back seat to the fact that in all cases, these are chains of coding and recoding in data collection, data processing, and data evaluation.

3. Data modulation

The third step involves accumulating and correlating data from one or more sources in order to modulate it. This step is particularly important when the five key response values (Facial Recognition for visual expressions, Electroencephalograms EEG for changes in brain activity, Electrocardiograms ECG for heart rate, Electromyography EMG for muscle tension, and Galvanic Skin Response (GSR)) are collected in order to perform Affective Computing in a comprehensive sense. However, even if no wearables or other sensors are used, regular computer or smartphone use usually involves several data sources such as the screen, microphone, and keyboard, and in the case of smartphones, additional sensors (Gramelsberger 2023, page 12).

4. Data patterns

In the fourth step, which involves the actual “emotion recognition,” the data is evaluated and correlated (Chun 2021). This can also be done in various ways. In classic Affective Computing, which collected user data on a sensor basis, emotion recognition was performed using predefined emotion categories. Within a specific measurement data range, possibly in close correlation with a range from another data (sensor) source, the system identified or recognized these measurements as equivalents for joy, anger, etc. So, it encoded experiences from live measurements to discrete values that are supposed to correspond to emotions.

At this point, it is already clear that Affective Computing and Emotion AI cover a spectrum of scientific approaches, technical procedures, practical applications, interfaces, and hardware and software components. With the rise of AI since the 2020s, Affective Computing has gained new momentum on the one hand, while on the other hand it has become one element among many in the design of models. Certainly, the affective and emotional modulation of communication between models and users has taken on new significance for providers and developers since the massive use of these systems (Zuboff 2019). At least three forms of the convergence of sensor-based Affective Computing and machine learning (AI) can be described: Embedded emotion analysis, predefined emotion categories and sentiment scores.

Embedded emotion analysis

Emotion analysis can be completely embedded as part of pattern recognition under conditions of strong machine learning. This case illustrates the continuity from step 2: Encoding Emotions to step 3: Affective Computing and Emotion AI as a Virtual Semantics, because the older forms of coding emotions in written and image form as part of the training data, especially in the foundation models, form the “emotion knowledge base” for these forms of embedded, accompanying emotion or sentiment analysis in ANNs. The patterns formed from past and ongoing learning, the respective emotions, affects, and moods, represent the conceptual-normative “molds” for emotionality and affectivity, as they are repeatedly imprinted, reinforced, and transformed in interaction with intelligent chatbots between humans and machines. All GPTs are capable of this form of analysis, sometimes in combination with the systems mentioned below. In this context, we must also assume that there is a unique image-text disconnect in the effects of training data (Afzal et al. 2024). As long as there are models that base their pattern formation on training that includes visual and behavioral data, these AIs will identify the elements that emotional psychology and classical Affective Computing have discretized and standardized as emotions and affects (measurement data and data patterns).

Predefined emotion categories and sentiment scoring

Emotion recognition based on predefined emotion categories corresponds to “classical” Affective Computing, in which measurement ranges were classified for the data sources collected, i.e., physical areas and forms of expression, which were supposed to correspond to distinct emotions. When recording heart rate, skin resistance, and visual data, certain ranges—especially in combination—corresponded to emotions such as joy. Many of the early applications of Affective Computing read or recorded only a few emotions. The definition of the measurement ranges, which are often treated as equivalent to emotional and affective inner states, involves technical coding. This simultaneously defines and sets norms for what is perceived and processed as emotion and affect in the respective Affective Computing. Among the examples are systems like Affectiva, Hugging Face Transformers, VADER Sentiment-Analysis (https://github.com/cjhutto/vaderSentiment), and Smart Eye.

Affective Computing and Emotion AI are thus understood in terms of media-semiotics, associating or even equating very different and diverse emotional qualities with technically coded and recodable categories or patterns of emotion and affect. They thus function as technical forms with normative functions, which are understood here as virtual semantics. As a power analysis, the concept of virtual semantics builds on Alexander Galloway’s protocol analysis, in which he summarized the technical rules or protocols of Internet communication (Galloway 2004). Codes are called the concrete linking conventions, protocols the implementation of the linking rules, and semantics the resulting framework. Semantics is the horizon-building framework of coded entities, in this case “emotions” and “affects” in strong parentheses, represented by clusters and patterns of digital, discrete data. Horizon-building semantics logically—and indeed absolutely—build, frame and contain computer-processed emotionality. Two points of power are emerging: Code compatibility and automated semantic links (due to a shift between in-house programming and third-party programming).

The alignment of an originally very reductive Affective Computing system (see Campolo in this issue) also involves a cultural shift from the tendency toward self-coding of one’s own emotionality through gestural signs and language to external coding, and further reinforces the withdrawal of automatized processes. By defining Affective Computing and Emotion AI as virtual semantics, the following conceptually normative problems associated with these technologies can be reexamined: 1. The genesis, i.e., the history of coding and standardization of technically processed emotional qualities, can be examined and understood in detail. 2. The outstanding and special significance of standardized representations of emotions and affects as training material for AI is recognized. 3. The framing and normative function of Emotion AI, which semi-automatically or automatically assigns certain expressions, gestures, and sequences to specific states that are supposed to represent inner qualities, can be problematized in the defined sense as virtual semantics.

Conclusion

The advances in the field of artificial intelligence (AI) over the past decade have been marked symbolically as a new insult to humanity, or “insult 4.0,” following the cosmological, biological, and psychological insults caused by the discoveries of the Copernican revolution in astronomy, evolution theory and psychoanalysis (Dotzler 2024, page 59). All three discoveries before AI are said to have destroyed a form of human self-illusion of being the center of the universe (by Copernicus), a unique creation (by Charles Darwin) and the sole master of mental faculties (by Sigmund Freud).

Now, in the age of strong artificial intelligence, intelligent thinking and creative work no longer seem to be distinguishing features of the human species, especially in view of the impressive achievements of generative pretrained transformers in text production as well as speech and image synthesis. Individuality and “emotionality” appear in positions today as the last bastions for distinguishing humans from artificial beings, which Leif Weatherby critically frames as remainder humanism (2025).

Against this background, Affective Computing and Emotion AI have been assessed from two opposite perspectives as a shifting of these boundaries: firstly from a humanist perspective and secondly from a technicist perspective. From a humanist perspective, Affective Computing and Emotion AI can be understood in this context as a further transgression of the boundary between man and machine. From a technicist or techno-optimist perspective, on the other hand, Affective Computing and Emotion AI can be understood as a sufficient and often helpful imitation of human characteristics by digital technology, that bridges the distance between technological systems and lived experiences. A third perspective presented here can be described as analytical in general and media-semiotic and media-archaeological in detail. A media-semiotic and media-archeological perspective understands and problematizes Affective Computing and Emotion AI in their genesis and socio-technological setting. The goal is to show Affective Computing and Emotion AI as a powerful new form to encode emotions that takes on the function of virtual semantics in the contemporary digital culture. It emphasizes the still large differences between human individuals and their machines, but shows how Affective Computing and Emotion AI interweave computers and their users in a complex automatic or semi-automatic, not seldom highly problematic way that can be understood as a normative set of rules as virtual semantics. The paper makes the following points, which do not exclude other perspectives of a phenomenological and media-aesthetic nature, but rather seek to complement them:

Step 1
I have applied a broad concept of code, following Umberto Eco, to emotions and affects. As soon as emotions and affects are shared, conveyed, expressed, written down, and, in short, communicated, they must be encoded. In principle, this can happen in very different ways, e.g., in gestural, spoken, or written form.

Step 2
I have examined various historical techniques for encoding emotions, such as handwritten and computer-generated love letters, and described the shift toward Affective Computing and Emotion AI as a transition from self-encoding to external encoding.

Step 3
Affective Computing and Emotion AI are described as virtual semantics based on the broad concept of code, as presented in step 1. “Virtual” refers both to the digital environment and to the quasi- or “as if” nature of the semantics described, since it comprises a technical-syntactic framework that exerts a strong semantic effect. The conceptual and normative issues identified concern, among others, the automation of emotional and affective communication.

Bibliography