A critical flaw raises questions about the validity of a study that credits a Sacramento County nonprofit with reducing child abuse and neglect.

Commissioners for First 5 Sacramento, which uses tobacco tax revenue to fund early childhood intervention programs, enthusiastically received the report on Birth & Beyond last month. The agency goes to the homes of troubled parents to teach them better child-rearing skills.

That same day, commissioners approved $4.7 million for Birth & Beyond for the coming fiscal year.

The study by LPC Consulting Associates looked at 900 families receiving Birth & Beyond services in the last fiscal year. Of those families, 48 percent had a referral to Sacramento County Child Protective Services for suspected abuse or neglect prior to entering the Birth & Beyond program, the report says.

While in the Birth & Beyond program, the referral rate "drops dramatically" to 19 percent, the report concludes.

The problem is the report compares vastly different lengths of time – making it hard to support its conclusion, said Jessica Utts, chairwoman of the statistics department at the University of California, Irvine, and author of the textbook "Seeing Through Statistics."

The consultants looked at referrals to CPS in the lifetime of family members before they entered Birth & Beyond – including abuse and neglect complaints when the parents were children.

That time period, which could run decades, was compared against the time the family was in Birth & Beyond, a period ranging from three to 12 months.

"It's hard to tell how effective the program is, based on this report," said Utts, who has served as chairwoman of the American Statistical Association's education committee. The report would need to consider similar time periods, such as one year before entering the program and one year after, she added.

The study's author, Lynne Cannady, has heard similar questions about past reports on Birth & Beyond. This is the 10th year her company has done the study, employing a similar methodology each time, she said.

"That's a perfectly acceptable critique of the work," she said.

Yet, Cannady said, she is still comfortable with her conclusion because previous studies have found that a majority of abuse and neglect referrals were made in the year before entering the Birth & Beyond program, with positive outcomes for families up to two years after they leave the program.

Also, Cannady noted, a study conducted by two professors found positive results by Birth & Beyond. That study, like Cannady's work, was funded by the Child Abuse Prevention Center in North Highlands, which paid Cannady about $15,000 for this year's report, said Sheila Boxley, the center's chief executive officer.

In 2009, Ed Byrnes of Eastern Washington University and Michael Lawson of the University of California, Davis, reported that 36 percent of families receiving Birth & Beyond services had a new CPS referral in one year, compared to 49 percent for a similar group of families not receiving those services.

The authors concluded that Birth & Beyond influenced the rate of referrals "in the desired direction, but not to a great extent."

None of the shortcomings of Cannady's research were discussed when she presented her latest report to First 5 commissioners. It was the first time her report was presented to First 5, though her previous studies have been shown to Sacramento County and other agencies.

Cannady told the commission that the findings in this year's report repeated what previous studies have found about the program's effectiveness.

First 5 officials lavished the findings with praise at their May meeting.

"Nothing short of remarkable," said Commissioner Scott Moak, the public address announcer for the Sacramento Kings. "It's amazing to see."

"I was so impressed with these findings," said Toni Moore, First 5 Sacramento's executive director, shortly before presenting a budget that would make Birth & Beyond a top First 5 recipient. "It just reinforced for me why we are identifying this as a high priority program for the commission."

© Copyright The Sacramento Bee. All rights reserved.

• Read more articles by Brad Branan

Order Reprint