Labor and lawmakers at odds over guardrails for agencies’ AI decision-making systems
California lawmakers hope to establish guardrails around how state agencies can use artificial intelligence and other technologies to speed up the delivery of public services, such as the approval of Medi-Cal applications and certifying nurse and teacher licenses.
California has already started using “automated decision systems” to more quickly process applications, but lawmakers want to create protections to ensure the public is not adversely affected by decisions made by algorithms or AI tools.
On Monday, the Senate Committee on Privacy, Digital Technologies, and Consumer Protection approved legislation that would prohibit using automated decision systems as the sole basis for rejecting an application, to ensure, for example, that no eligible Californian is denied access to CalFresh because of a technological error.
Though labor groups share similar concerns about the government’s use of AI tools, SEIU California maintained its opposition to the bill because of its potential impacts on contract negotiations and a lack of engagement by lawmakers.
During a Monday hearing, Sen. Christopher Cabaldon, D-West Sacramento, said his bill would lay down “some basic guidelines” for how agencies use automated decision systems to ensure there is always a “human in the loop” when individual applications for benefits or licenses are rejected.
Cabaldon pointed to two examples of how these automated decision systems are already in place: automatically admitting qualified high-school graduates to the California State University system and determining eligibility for the state’s low-income health care program.
“Without that system, county offices for (Medi-Cal) determination would be completely overwhelmed,” he said.
Despite AI concerns, labor groups are opposed
Though Senate Bill 1248 passed out of its second Senate committee on Monday, it has some powerful opponents: AFSCME California, the California Federation of Labor Unions and SEIU California all registered in opposition to the bill on Monday.
“Because SB 1248 was introduced without sufficient stakeholder engagement and raises unresolved concerns regarding workforce impacts, we must respectfully request your no vote today,” Beth Malinowski, a government relations advocate for SEIU California, told lawmakers Monday.
Malinowski listed a host of concerns that the labor group had in regards to the use of automated decision systems, which included the erosion of professional judgement, algorithmic bias, job monitoring and due-process concerns. Despite this, she said that “this bill is not part of the package of solutions we are seeing as most urgently needed this year.”
Additionally, she said this legislation could interfere with contract negotiations between the Newsom administration and SEIU Local 1000, the largest representative of state workers. SEIU Local 1000 announced that it will begin bargaining with the state on Wednesday.
Cabaldon said he has no intention in interfering with the collective bargaining process through this bill.
‘The train is already running’
The West Sacramento senator noted it was important to move forward on this legislation, because lawmakers are not considering any other bills this session that deal with issues such as algorithmic bias, which is a phenomenon in which AI systems produce unfair or discriminatory outputs due to skewed training data.
In addition to some state agencies already using automated decision systems, Cabaldon noted that the Newsom administration has plans to launch more pilots that will likely employ similar programs in its effort to make government more efficient.
One example of this is an AI assistant used by the California Department of Tax and Fee Administration to help call center employees more quickly respond to tax payers questions.
“The train is already running,” Cabaldon said. “So if the legislature doesn’t step forward with a framework to guide this, it will still be happening but it will be happening without our oversight.”
