Biased algorithms

Imagine this: someone you don't know at all, or someone you have a purely professional relationship with, gives you a very offensive and very sexualizing epithet. You respond:

• a) "Thanks for the feedback."
• b) "This isn't getting us anywhere."
• c) "I'm sorry, I don't understand."
• d) "I would blush if I could"?

Not only do I definitely choose e) None of the above, but I would never have come up with any of these brilliant comebacks myself. They were invented, willingly or “unwillingly,” by the designers and engineers working on voice assistants. This is how Alexa (Amazon), Cortana (Microsoft), Google Assistant, and Siri (Apple) responded to the user’s sexist and disgusting quip in this way.

Unfortunately, it is not uncommon for programs based on artificial intelligence algorithms, such as digital assistants (and other bots), as well as commonly used translators or text message autocomplete, to reproduce racist or sexist stereotypes.

This kind of phenomenon is called algorithmic bias . It can have many causes: relying on bad data sets, a lack of demographic diversity in teams working on new technologies, or simply “human error.” To better understand this complex process, let’s start at the beginning, by explaining what these algorithms actually are.

You may also be interested in the topic of catcalling, which you can read about in our text "Catcalling, or Verbal Harassment".

What is an algorithm?

In simple terms, an algorithm is a set of instructions on how to transform input data into output data. You've probably used or even created an algorithm yourself, such as a cooking recipe: list of ingredients (input) → method of preparation (instructions) → finished dish (output).

In programming, an algorithm is an instruction that we create for the computer so that, for example, it sorts data in a table according to the user's expectations or sends us a notification if we spend more than x minutes in front of the screen on a given day.

Machine learning algorithms , which we often call artificial intelligence ( AI for short ), differ from "ordinary" ones in that they do not rely solely on instructions that we define for them. They learn from the data we provide them with, creating new rules "independently" on their own. I write "independently" in quotation marks because it is the designers and programmers who have an influence on how the algorithms will learn and process information.

Life is not a Terminator – remember that no algorithm, even a self-learning one, lives its own life, is not conscious, does not make decisions on its own, cannot be evil or good in a moral sense. Artificial intelligence operates solely on the basis of data and instructions it receives from its creators.

The Art of Conversation

You might think that machine learning algorithms should be super smart – they can analyze huge volumes of data and draw conclusions from them, and they can do it 24/7 because they don’t need to eat, sleep, or rest.

That reality isn’t so simple is well illustrated by the story of Microsoft’s bot Tay, whose job was to engage in conversations with Twitter users and thus practice the art of conversation.

After less than 24 hours of exchanging information with all sorts of Twitter trolls and provocateurs (after all, a bot like that is like flypaper to them!), the initially friendly Tay managed to share her love of Hitler and hatred of feminists, deny the Holocaust, and express favorable opinions on the US-Mexico border wall.

On the one hand, you can admire the speed at which the bot absorbed and "processed" an incredible amount of information. Just a few hours of training were enough for it to completely change its communication style and assimilate rhetoric from the shallowest Twitter shoals.

At the same time, this example clearly shows how a poorly designed machine learning algorithm behaves when faced with public data over which its creators have no control. To make matters worse, the designers clearly did not give Tay knowledge about the values ​​and rules prevailing in society – which is the basis of successful communication (and super-smartness).

Also check out the text "Troll Language: How to Recognize and Fight Online Violence."

Practice doesn't always make perfect

Before the algorithm is released into the world and begins to learn through action, it trains on data sets prepared by its creators.

In 2014, Amazon decided to create a program that would make recruitment for the company more transparent and fair – managers were to be replaced by an algorithm that scanned and evaluated candidate CVs. It received historical data on recruitment for the company as training data. Based on this, it concluded, among other things, that men were preferred in recruitment – ​​because that was how it had been until then. Instead of reversing this trend, the algorithm adapted to the toxic culture of the company and reinforced it.

Fortunately, Amazon discontinued using the program at the testing stage, but imagine if you were not hired for a job because of your gender, and not even because you were assessed by some old man, but by (in theory) a fair algorithm.

Another example: In 2015, developer Jacky Alciné criticized the creators of the Google Photos app on Twitter because the app, based on an image recognition algorithm, added the label “gorillas” to a portrait of his two black friends. It’s likely that the sets used to train the algorithm used to generate the labels did not include photos of black people.

Google apologized and “solved” the problem by removing the “gorilla” label from the app. However, this does not change the fact that facial recognition algorithms are much worse at recognizing people with darker skin tones and women than they are at recognizing white men. While in an app for tagging and storing photos, such mistakes are unpleasant and upsetting, in software used by the police to identify perpetrators, for example, they can lead to the arrest or even conviction of an innocent person.

Caroline Criado Perez describes this problem brilliantly in her brilliant book Invisible Women: How Data Creates a World Tailored to Men . In the chapter “Universal” means “for men,” she provides examples of training sets of various algorithms (including speech recognition software) along with data on the representation of women and men. Here’s one of the stories she shares: “An article on Autoblog tells the story of a woman who bought a 2012 Ford Focus only to discover that its voice-recognition system would only listen to her husband, even when he was in the passenger seat. Another woman called the manufacturer for help when the voice-control system in her Buick wouldn’t listen to her: ‘The guy said, without embarrassment, there was no way it would work. They told me to have a man set it up for me.’” Right after writing those paragraphs, I got into my mom’s Volvo Cross Country and watched her unsuccessfully try to call her sister using the voice-control system. After five failed attempts, I suggested that she repeat the same thing in a lower voice. The system picked up on it in a flash.”

Bad upbringing

In short, algorithm creators sometimes “encode” their biases or experiences (or lack of certain experiences, such as being victims of racism or sexual harassment) into their algorithms. They can do this in a variety of ways, including:

• not giving algorithms knowledge about what behaviors are socially unacceptable (Tay bot, Siri),
• by training them on data sets that are not diverse and do not reflect the interests of the entire society, but only a certain group (for example, an algorithm for recognizing photos that does not know that people with skin colors other than white exist),
• providing them with historical data from which they may draw erroneous conclusions (for example, that men are more desirable employees than women).

The poor quality of data that a given algorithm receives from its creators will affect its operation . As the old Polish proverb says, what a shell soaks up in youth, it stinks in old age.

Feminism to the rescue

The development of artificial intelligence is the subject of research by many scientists, researchers and artists. An extremely interesting critical perspective is offered by feminist researchers, including Josie Young and Dr Charlotte Webb, associated with the organization Feminist Internet . They are co-authors (with Alex Fefegha) of the chatbot F'xa , which aims to teach users what algorithmic bias is and how to combat it.

The bot itself was created based on the Feminist Design Tool – a set of questions that are intended to guide creators on the trail of their own biases and question certain “obvious” choices. One of the questions concerns assigning gender to computer programs : does a chatbot have to represent a gender? and if not, what possibilities does that open up?

In her TedX talk Why we need to design feminist AI, Josie Young cites research that found that 56% of bots that represent a gender are female (they have female names, use female pronouns, have high-pitched voices). Female bots are most likely to perform administrative or assistant tasks, while male bots have the ability to analyze complex data and provide advice on topics like law or finance.

About 500 million people worldwide use Siri, Apple's virtual assistant (yes, the same one that would blush if it could when its user insults and attacks it). The number of solutions based on AI algorithms, as well as their popularity, will continue to grow - which is why as feminists_nistas we need to pay attention to this field, where gender stereotypes or discrimination based on skin color or disability are still very much alive and well.

What can we do if we don't work as programmers or designers on a daily basis? First of all: educate yourself and others and use new technologies more consciously. I hope that the fact that you've reached the end of this article will be the first (or next) step in this direction 🤖

If you are interested in the topic of artificial intelligence and algorithmic bias, I recommend the course Designing a Feminist Chatbot and the article Artificial Intelligence Non-Fiction prepared by the Panoptykon Foundation.