Bias and ethical issues in machine-learning models

The success stories that have gathered around data analytics drive broader adoption of the newest artificial-intelligence-based techniques—but risks come along with these techniques. The large numbers of freshly anointed data scientists piling into industry and the sensitivity of the areas given over to machine-learning models—hiring, loans, even sentencing for crime—means there is a danger of misapplied models, which is earning the attention of the public. Two sessions at the recent MinneBOS 2019 conference focused on maintaining ethics and addressing bias in machine-learning applications.

To define a few terms: modern analytics increasingly uses machine learning, currently the most popular form of the field broadly known as artificial intelligence (AI). In machine learning, an algorithm is run repeatedly to create and refine a model, which is then tested against new data.

MinneBOS was sponsored by the Twin Cities organization Minne Analytics; the two sessions were: "The Ethics of Analytics" by Bill Franks and "Minding the Gap: Understanding and Mitigating Bias in AI" by Jackie Anderson. (Full disclosure: Franks works on books for O'Reilly Media, which also employs the author of this article.) Both presenters pointed out that bias can sneak into machine learning at many places, and both laid out some ways to address the risks. There were interesting overlaps between the recommendations of Franks, who organized his talk around stages, and of Anderson who organized her talk around sources of bias.

When we talk about "bias" we normally think of it in the everyday of sense of discrimination on the basis of race, gender, income, or some other social category. This focus on social discrimination is reinforced by articles in the popular press. But in math and science, bias is a technical term referring to improper data handling or choice of inputs. And indeed, the risks in AI go further than protected categories such as race and gender. Bias leads to wrong results, plain and simple. Whether bias leads to social discrimination or just to lost business opportunities and wasted money, organizations must be alert and adopt ways to avoid it.

Franks based his talk on the claim that ethics are intuitive and inherently slippery. As a simple example, consider the commandment "Thou shalt not kill." Although almost everybody around the world would acknowledge the concept, few would agree on exactly when you could violate the rule. Another example would be the famous "right to be forgotten", now mandated by the GDPR in the European Union. Sometimes, the need to keep data for legal conformance or law enforcement overrides the right to be forgotten. At other times, Franks said, it's just infeasible—how can you maintain a relationship with a health care provider or insurance company if you want them to forget something about you?

Franks offered five stages in machine learning at which to look for and correct bias. I'll describe them here, noting where Anderson's three sources of bias match up.

Modeling targets and business problems

This stage is where the two companies mentioned earlier parted ways on the question of offering customer rewards. As another example, Franks cited the privacy issues tied up in the notorious case where Target sent pregnancy-related offers to a 17-year-old who was trying to hide her pregnancy from her father. This public relations disaster highlights the differences between laws, ethics, and plain good sense about business goals. Legally, Target was perfectly entitled to send pregnancy-related offers. Although there's a difference between personal medical information and routine retail sales like potato chips, Target was probably within reasonable ethical bounds in sending the pregnancy information. Where it fell short was in considering what customers or the general public would find acceptable.

This stage in Franks's taxonomy aligns with Anderson's first source of bias, "Defining the problem". Anderson mentioned, as an example, college analytics about what potential students to target for promotional materials. (Recruitment costs per student start around $2,300 and can go over $6,000.) If one assumes that the students getting promotions are more likely to apply, matriculate, graduate, and ultimately benefit the institution (certainly that's the reasoning behind sending out the materials), avoiding bias is important for both business and social reasons. Anderson said that, before running analytics, the college has to ask what it's trying to find out: Who is most likely to accept, who is most likely to graduate, who will give the largest donations later in life, who will meet diversity goals, etc.

Modeling input data

Here the core problem is that nearly every algorithm predicts future performance based on existing data, so it reproduces whatever bias was used in the past. Anderson calls this a form of circular reasoning that excludes important new candidates, whether for a college, a retail business, or a criminal justice probe. Franks pointed out that any model can be invalidated by a change in the environment or the goal.

Franks turned here, as Anderson had, to an example of predicting college admissions. If your model was trained at a time when 80% of your students were liberal arts majors, but you have changed strategies so that 80% of your students are now business majors, your old model will fail. Another example Franks offered of a changed environment pertains to criminal sentencing. U.S. society has recently changed its idea of how to punish criminals, so "fair sentencing" software based on the old ways of sentencing has become invalid.

Two of Anderson's sources of bias align with this stage: "Selecting data" and "Cleaning/preparing the data". Anderson cited a problem in preparing data, involving a retail firm that was trying to determine which customers were most profitable. When cleaning the data, the firm excluded small purchases because it assumed it should aim for customers who purchased expensive items. Later it found out that it had totally missed a large and loyal clientele that spends quite a lot—but in small amounts.

Often, she said, you have look at results to uncover hidden sources of bias. In one case she cited, Amazon's HR department designed an algorithm to help find the best programmers among applicants. The department started without any checking for bias, which naturally led to a model that discriminated in favor of men because they are currently most common in the field. After looking at results and realizing that the model was biased against women, the department created another model after changing obvious gender markers such as pronouns and names. But the model was still biased. It turns out that word choices in job applications are gender-specific too.

Modeling transparency and monitoring

Franks endorsed the idea of transparency in models, commonly called "explainability": model users have to know what is triggering a decision. In one example, an image-recognition program was trained to distinguish huskies from wolves. It seemed to perform extremely well until researchers delved into a failed case. They discovered that the program didn't look at the animal at all. If there was snow, it assumed the animal was a wolf. This was due to the kinds of images it was shown; the wolves were outdoors and the huskies were mostly indoors.

The strategy used by these researchers to determine the exact features that led to a decision fall under the term Local Interpretable Model-Agnostic Explanations (LIME). In the case of image processing, a typical LIME approach is to focus on several different parts of the image and run them through the model to see which parts of the image truly predict the result. This process is model-agnostic.

Some fields have legal requirements for transparency. For instance, when a bank denies credit, it has to list the precise criteria that led to the decision. There are no such regulations in health care, but transparency is necessary here too. Most clinicians would be very nervous making a diagnosis based on a black-box model.

Franks suggested that LIME be a factor in government approval for the use of analytics models. If a robust LIME procedure shows that the algorithm is looking at relevant features, the model should have a stronger chance of being approved. Franks also brought up the importance of emergency brakes on algorithms. In one example, a programmed trading algorithm wiped out one company's stock price for a short time. The programmers should have built in a check for such sudden or bizarre behavior, and made the program shut itself down, just as a factory has a button any employee can push to stop the assembly line.

Modeling usage

At this stage, Franks said, you must understand whether the context in which you're using a model makes it fair. Often the researcher spends 10% of the time creating a reasonably good algorithm and 90% error-proofing it.

He complained that the public is overly spooked by exceptional events in the use of new analytics. Every time an autonomous vehicle is involved in an accident, there are bans and public calls to abandon the technology. The real question for Franks is: over a certain interval (say, 100,000 miles driven), how many accidents were caused by human drivers versus self-driving cars?

Defining policies

Franks said organizations should define clear policies regarding the use of machine learning and publish them. The famous case where Apple Computer refused to unlock an accused criminal's cell phone led to the benefit of Apple publicly stating its views about privacy. Apple may have gained some customers and lost others, but now everyone can judge the company on the basis of this policy.

Recommendations

Franks's concluding recommendations were to create an ethics review board consisting of people from different relevant disciplines, like research institutions' Institutional Review Boards (IRBs), to write out policies, and deal firmly with violations. He called on the analytics community as a whole to take ownership of bias.

Anderson's main recommendation for fixing bias was to ensure team diversity. You have to experience life a certain way in order to understand how people with that life behave—and how they're potentially excluded. She also advised companies to collaborate, overcoming the barriers they set up out of worries over trade secrets. Many organizations don't even share the information they have across internal company borders. They should join forums and collaborative initiatives to recognize bias and improve diversity. Organizations focused on this include the Algorithmic Justice League (AJL). There are also toolkits such as IBM's AI Fairness 360.

The talks by Franks and Anderson showed that, about a decade into the new epoch of machine learning, researchers and practitioners are aware of bias and are designing practices that try to correct it. One remaining question is how much we as a society can depend on the competency and goodwill of the organization designing or using the model. Where can regulation fit in? And how much responsibility lies on the researchers who designed the model, versus the user who applies it in real life, or even the regulators who approve the model's use? Hopefully, as we learn the efficacy of practices that correct bias we can also answer the question of how to make sure they are used.