Critics question federal prison agency's AI plans for inmate assessments

Experts are urging Correctional Service Canada to slow down after it claimed using artificial intelligence to help create criminal profile reports for prisoners would save time and maintain accuracy, despite using just six text cases in a trial.

OTTAWA—Experts are urging Correctional Service Canada to slow down after it claimed using artificial intelligence to help create criminal profile reports for prisoners would save time and maintain accuracy, despite using just six text cases in a trial. The warnings come as CSC considers rolling out generative AI tools that would summarize sensitive documents used to write influential assessment profiles of offenders going to prisons, as the Star has reported, raising alarm about the potential for the summaries to be tainted by AI-generated hallucinations or biases. Criminal profile reports are “foundational documents” prepared by CSC staff during a prisoner’s intake process that identify risks and play a role in major decisions like access to programs and likelihood of parole.

Drawing from scores of other documents, they include details about: an offender’s criminal history; the circumstances of their crimes; patterns of violence; behavioural, mental health and addiction issues; family and social background; trauma history; education and employment records; and victim impact statements. Sorelle Friedler, a professor of computer science at Haverford College who helped develop AI policy for former U.S. president Joe Biden, said “there’s no way to prove anything” with a trial of just six cases. Because of the unique circumstances of all offenders and their crimes, Friedler argued a proper test for biases and errors would have to include thousands of “hand-validated” cases covering every single crime in Canada more than once.

“It’s just absurd,” she told the Star. “You would want to have enough individuals to genuinely cover the wide range of circumstances that individuals coming in have experienced.” And even that might not be enough because of the higher risk of using AI in a criminal justice context, she said, including “embedded demographic biases.”

On top of AI potentially adding false information to document summaries, using AI to generate them would be even more difficult and time-consuming to spot crucial data being omitted, she said. After a small-scale trial using fictional and anonymized data, CSC told the Star the tools produced “measurable time savings” and were “also found to maintain the accuracy of information, as parole officers still manually input the compiled information and conduct a final review to ensure its accuracy.” “CSC is currently reviewing the outcomes of this evaluation and considering potential next steps,” spokesperson Esther Mailhot said.

She said there is no “specific timeline” for a decision, including on whether a more extensive pilot project would be conducted. Mailhot had previously suggested further testing would be needed to assess potential biases in AI-generated summaries, because doing so was “constrained” in the initial limited test. Jenny Kwan, the NDP’s public safety critic, said the government’s talk of safeguards is not sufficient.

She is calling for an independent assessment, broad stakeholder engagement, and checks and balances on any potential rollout. “I think that the issues and concerns around biases that are embedded into the AI system is still there. They have not addressed that issue,” Kwan told the Star.

“The government needs to be open and transparent with this.” Public Safety Minister Gary Anandasangaree declined to comment when reached by the Star. Justice Minister Sean Fraser said he acknowledged concerns but also generally sees an opportunity with AI, which the Carney government is increasingly adopting.

AI is “rapidly changing the world” and “the right path forward will involve careful steps where we use new tools to ensure that we can deal with backlogs and systems to ensure that we’re reducing the reliance on taxpayers’ money to deliver services to Canadians,” Fraser said. “But at the same time, we need to have guardrails in place that give different communities across Canada confidence that their privacy interests won’t be violated, that they won’t be discriminated against on the basis of the community from which they come.” Jennifer Evans, principal at the consulting and research firm PatternPulse AI, said the information provided by the government following the trial is “really unclear” and lacking in detail.

She said there are other tools, such as optical character recognition, that can summarize documents without the same risk of error as AI tools based on pattern recognition. Where AI excels, she said, is in its speed and in determining trends in the data. “There’s no human capability or sort of traditional computing capability that can do pattern recognition at the scale or rate that AI can, but the correctness of the information is the biggest question,” Evans said.