AI tool can detect discrimination on basis of race and gender

AI systems - those involved in selecting candidates for a job or for admission to a university - are trained on large amounts of data.

Published: 11th July 2019 02:36 PM  |   Last Updated: 11th July 2019 02:37 PM   |  A+A-

Artificial Intelligence

For representational purposes


NEW YORK: A team of US researchers has developed an Artificial Intelligence (AI) tool for detecting unfair discrimination - such as on the basis of race or gender.

AI systems - those involved in selecting candidates for a job or for admission to a university - are trained on large amounts of data. "But if these data are biased, they can affect the recommendations of AI systems," said Vasant Honavar, Professor at Pennsylvania State University.

For example, if a company historically has never hired a woman for a particular type of job, then an AI system trained on this historical data will not recommend a woman for a new job.

The team at Pennsylvania State and Columbia University created the AI tool for detecting discrimination with respect to a protected attribute, such as race or gender. "For example, the question, 'Is there gender-based discrimination in salaries?' can be reframed as, 'Does gender have a causal effect on salary?,' or in other words, 'Would a woman be paid more if she was a man?'" said Aria Khademi from Penn State.

The researchers tested their method using various types of available data, such as income data from the US Census Bureau to determine whether there is gender-based discrimination in salaries. They also tested their method using the New York City Police Department's stop-and-frisk programme data to determine whether there is discrimination against people of colour in arrests made after stops.

"We found evidence of gender-based discrimination in salary. Specifically, we found that the odds of a woman having a salary greater than USD 50,000 per year is only one-third that for a man. This would suggest that employers should look for and correct, when appropriate, gender bias in salaries," explained Honavar.

"Our tool," he said, "can help ensure that such systems do not become instruments of discrimination, barriers to equality, threats to social justice and sources of unfairness." The results appeared in Proceedings of The Web Conference 2019.

Stay up to date on all the latest Tech news with The New Indian Express App. Download now
(Get the news that matters from New Indian Express on WhatsApp. Click this link and hit 'Click to Subscribe'. Follow the instructions after that.)


Disclaimer : We respect your thoughts and views! But we need to be judicious while moderating your comments. All the comments will be moderated by the editorial. Abstain from posting comments that are obscene, defamatory or inflammatory, and do not indulge in personal attacks. Try to avoid outside hyperlinks inside the comment. Help us delete comments that do not follow these guidelines.

The views expressed in comments published on are those of the comment writers alone. They do not represent the views or opinions of or its staff, nor do they represent the views or opinions of The New Indian Express Group, or any entity of, or affiliated with, The New Indian Express Group. reserves the right to take any or all comments down at any time.

flipboard facebook twitter whatsapp