How Does Bias Enter the System?

You know that bias can enter an AI system through a product’s creators, through training data (or lack of information about all the sources that contribute to a dataset), or from the social context in which an AI is deployed.

Assumptions

Before someone starts building a given system, they often make assumptions about what they should build, who they should build for, and how it should work, including what kind of data to collect from whom. This doesn’t mean that the creators of a system have bad intentions, but as humans, we can’t always understand everyone else’s experiences or predict how a given system will impact others. We can try to limit our own assumptions from entering into a product by including diverse stakeholders and participants in our research and design processes from the very beginning. We should also strive to have diverse teams working on AI systems. Check out the Business Value of Equality Trail to learn more about the advantages of hiring diverse teams.

Training Data

AI models need training data, and it’s easy to introduce bias with the dataset. If a company historically hires from the same universities, same programs, or along the same gender lines, a hiring AI system will learn that those are the best candidates. The system will not recommend candidates that don’t match those criteria.

Model

When you create a machine learning model, the factors that you use in the model, such as race, gender, or age, can result in recommendations or predictions that are biased against certain groups defined by those characteristics. You also need to be on the lookout for factors that function as proxies for these characteristics. Someone’s first name, for example, can be a proxy for gender, race, or country of origin. For this reason, Salesforce Einstein does not use names as factors in its Lead and Opportunity Scoring model.

Human Intervention (or Lack Thereof)

Editing training data directly impacts how the model behaves, and can either add or remove bias. We might remove poor-quality data or overrepresented data points, add labels or edit categories, or exclude specific factors, such as age and race. We can also leave the model as-is, which, depending on the circumstances, can leave room for bias.

The stakeholders in an AI system should have the option to give feedback on its recommendations. This can be implicit (say, the system recommends a book the customer might like and the customer does not purchase it) or explicit (say, the customer gives a thumbs up to a recommendation). This feedback trains the model to do more or less of what it just did. According to GDPR, EU citizens must also be able to correct incorrect information a company has about them and ask for that company to delete their data. Even if not required by law, this is best practice as it ensures your AI is making recommendations based on accurate data and is ensuring customer trust.