Speed, Scale, and Access to Large Datasets

AI systems are trained to optimize for particular outcomes. AI picks up bias in the training data and uses it to model for future predictions. Because it’s difficult if not impossible to know why a model makes the prediction that it does, it’s hard to pinpoint how the model is biased. When models make predictions based on biased data, there can be major, damaging consequences.

Let’s take one example highlighted by Oscar Schwartz in his series for the Institute of Electrical and Electronics Engineers on the Untold History of AI. In 1979, St. George’s Medical School in London began using an algorithm to complete the first-round screening of applicants to their program. This algorithm, developed by a dean of the school, was meant to not only optimize this time-consuming process by mimicking human assessors, but to also improve upon it by applying the same evaluation process to all applicants. The system made the same choices as the human assessors 90–95 percent of the time. In fact, it codified and entrenched their biases by grouping applicants as “Caucasian” and “non-Caucasian” based on their names and places of birth, and assigning significantly lower scores to people with non-European names. By the time the system was comprehensively audited, hundreds of applicants had been denied interviews.

Machine learning techniques have improved since 1979. But it’s even more important now, as techniques become more opaque, that these tools are created inclusively and transparently. Otherwise, entrenched biases can unintentionally restrict access to educational and economic opportunities for certain people. AI is not magic; it learns based on the data you give it. If your dataset is biased, your models will amplify that bias.

Developers, designers, researchers, product managers, writers—everyone involved in the creation of AI systems—should make sure not to perpetuate harmful societal biases. (As we see in the next module, not every bias is necessarily harmful.) Teams need to work together from the beginning to build ethics into their AI products, and conduct research to understand the social context of their product. This can involve interviewing not only potential users of the system, but people whose lives are impacted by the decisions the system makes. We discuss what that looks like later in this module.

Resources

• Trailhead: Artificial Intelligence Basics
• Trailhead: Get Smart with Salesforce Einstein
• Salesforce Acceptable Use Policy