Algorithmic Discrimination

1. How Discrimination Enters AI Systems

Biased Training Data

AI trained to hire new employees tended to hire white males.

Proxy Variables

  • features that are formally race-neutral or gender-neutral but strongly correlated with protected characteristics.
  • Zip code is a common example
  • race becomes a ghost variable

    • Flawed Target Variables

  • hiring systems predict who resembles past successful employees
  • healthcare systems predict past healthcare spending

    • Feedback Loops

  • Self-fulfilling prophecies
  • A system trained to predict where crime will occur sends police to those areas. Police presence in an area increases the probability of arrests in that area.

    • 2. The Legal Framework

  • Disparate treatment- intentional discrimination
  • Disparate impact- a rule or process appears neutral but disproportionately harms members of a group
  • discrimination does not require bad intentions

    • 3. A Philosophical Analysis of Discriminatory AI

  • Some things can be legally permissible and morally wrong.
  • what makes algorithmic discrimination wrong, when it is wrong, and why?
  • Choosing a fairness metric is a moral and political decision about which kind of error matters most and who should bear that risk

    • The Formal Equality View

  • treat like cases alike (don't discrimiate)
  • the problem with algorithmic discrimination is that even when race is not an explicit input, its proxies reproduce the same irrelevant criterion
  • this framework struggles with the legitimacy defense.
  • If a feature genuinely predicts a relevant outcome (zip code predicts loan repayment rates), it is not formally irrelevant, even though using it reproduces racial inequality.

    • The Substantive Equality View

  • what makes discrimination wrong is not formal irrelevance but the perpetuation of unjust hierarchy.
  • view shifts the question from "is race being used?" to "what effect does this practice have on existing patterns of inequality?"
  • we ask not just whether an algorithm is formally neutral, but whether it reproduces or reinforces unjust structural conditions.

    • The Capabilities Approach

  • evaluates practices by their effect on people's ability to live flourishing human lives and to exercise the central capabilities that human dignity requires
  • algorithmic discrimination is wrong when it undermines people's core capabilities
  • discriminatory systems that affect access to housing are more seriously wrong than those affecting access to luxury goods,

    • 4. Who Is Responsible?

  • Developers
  • Data collectors and institutions
  • Deployers
  • Users
  • Regulators