In this podcast recorded at the Courts,

Community Engagement, and Innovative Practices in a Changing Landscape symposium held

in Anaheim in December 2015, Susan Turner, professor in the department of Criminology,

Law & Society at the University of California-Irvine, explains how risk assessment tools are developed and

discusses the strengths and limitations of risk assessment. 

RAPHAEL

POPE-SUSSMAN: This is Raphael Pope Sussman, with the Center for Court Innovation. This podcast is part of a series

of dispatches from the Court’s Community Engagement and Innovative Practices in a Changing Landscape Symposium

held in Anaheim in December 2015. The conference focused on justice reforms, including recent developments in California,

public safety realignment and Proposition 47. Public safety realignment refers to changes brought about by 2011 legislation

that shifted responsibility for certain populations of offenders from the State to the county level. Proposition

47, a ballot initiative passed by referendum in 2014 reclassified certain low level felonies as misdemeanors. I hope

you enjoy listening.

Hi, this is Raphael Pope Sussman, with the Center for Court Innovation, and

I’m here today at the Court’s Community Engagement and Innovative Practices in a Changing Landscape Conference

in Anaheim. Right now I’m sitting with Susan Turner, professor in the Department of Criminology, Law and Society

at University of California, Irvine. Susan, thank you for speaking with me today.

SUSAN TURNER:

Glad to be here.

POPE-SUSSMAN: What’s the process for developing a new risk assessment tool?

TURNER: Well, risk assessment tools can be developed by an organization. There are also ones that are available

that have been developed by others, so some of the first decisions that jurisdictions make is whether to do it home

grown or whether or not to buy something that’s been developed and is available for purchase, or is perhaps

free. If you’re interested in developing a risk tool for your jurisdiction, one of first things you need to

decide is what are you interested in predicting? Most jurisdictions in risk assessment are interested in predicting

some kind of recidivism behavior, such as arrest or incarceration or conviction. Then we generally use a set of variables

that are related to an offender’s prior record, age or gender, and then some other factors that may be related

to their drug use, mental health status, their association with criminal peers. You basically need to make sure that

you have the data for your left hand and your right hand side of the equation.

You gather the

information on an individual level, so for every person you’re interested in, you gather information on whatever

outcome and then their record, their variables such as their gender, their age, and their prior record. You want

to have a sample that’s large enough. You put them generally in a regression equation, where you try to figure

out whether or not these factors are predictive. You’re able to gather weights for the variables based on these

mocks. For example, if you’re predicting whether somebody would be convicted, you might have age and you’d

have a weight of 5, gender you’d have a weight of 3, different prior record variables would have various weights.

You can just imagine sort of scoring everybody up, and the higher the scores, generally the more likely someone will

recidivate. Once you do that on one half of your sample, you build your model. Then you do something called validation

on a separate set of data that you haven’t looked at before and that tells you whether or not what you’ve

developed on your first sample validates or is predictive on your second sample. That’s basically a kind of

simplistic view, but the overall gist is that you have data available, you develop your instrument on one half of

the sample and test it on the second.

POPE-SUSSMAN: What are the limitations of these tools?

TURNER: Well, there’s a lot of good things about the tools and there are, of course, always limitations

with any kind of tool. Risk assessment for recidivism is, like many other risk tools, we have risk tools in our regular

lives. You think of your car insurance premiums that you pay are based on risk assessment tools. Risk assessments

tools are only as good as the data that are behind them, number one. Another limitation to risk assessments tools

that we all need to be aware of is that they’re never 100% accurate, and probably will never be 100% accurate.

They generally are what’s referred to as a moderately good predictor, but because of the nature of human behavior

and variables that we don’t know why people do the things we do, we can never gain 100% accuracy, which sometimes

gets us into trouble with say politicians who have very high standards for whether or not you want to use an instrument.

Sometimes so high that they’re really not practical.

POPE-SUSSMAN: What are the great strengths?

TURNER: Great strengths are that we normally discuss risk assessments tools contrasted with what we refer

to clinical judgment, or some people call it gut. The research has shown that actuarial tools or risk prediction

instruments are more accurate than gut or clinical experience. Many people don’t like to believe that because

they’re often trained in a clinical background, or you can imagine a parole agent with 30 years of experience

is sometimes looking askew at risk assessments tool, but they’re more accurate. They can also result in decision

making that’s more consistent, because you can imagine the great variability if everyone follows their own gut.

You have lots of different ways the same person might be scored, so there’s consistency. There’s also,

with some risk assessments tools the advantage of time. Some of them are automated so for large numbers of people

they can be scored pretty quickly, allowing an agency to be able to get actuarial tool on a number of offenders without

the resources that may be required for, say, 45 or 50 minute instruments that are collected individually.

POPE-SUSSMAN: What roles do you see for risk assessments tools in the larger project of decarceration?

TURNER: One of the things we’ve talked about here today was the use of risk assessment and one of the

things I mentioned was that risk assessment’s been around for a long time. It’s got a resurgence nowadays,

I think partly because we’re trying to reduce the current populations that we have incarcerated and we don’t

have enough money to be able to keep everyone locked up. Money is driving a lot of decisions and one of them I think

actually may be risk assessment. We need to decide who are the highest risk people, the ones that we need to devote

our resources to, our public safety resources, and to not spend the resources on the ones who are the lower risk.

POPE-SUSSMAN: How do we ensure that risk tools don’t show bias against individuals and communities

that may have disproportionate rates of established risk factors, like housing or employment or disability?

TURNER: Very good question. I think there are a number of people who are concerned with risk assessments

tools based on dimensions that may impact certain groups unfairly. Many risk assessments tools use an offender’s

prior record as a major variable in their recidivism outcomes. Then there’s obviously the question, if prior

records are somehow biased in their very creation within communities of color, what does that mean in terms of risk

assessments tools. It’s a very valid question. One of the ways that we help address this is when we develop

tools, we take a look at and see how well the tools works with the different populations of interest.

For

example, when we developed a tool for the California Department of Corrections and Rehabilitation, we were particularly

concerned about whether the tool predicted for females as well as males, because there’s a lot of work suggesting

that some of the risk factors for women are different and for men. Also, we were concerned about whether the risk

tool worked equally well for blacks and Hispanics and whites and other groups. What we do with those is that we separate

the samples into the groups of interest and check to see whether the tool is as predictive in each of the groups.

If you find a tool that doesn’t work very well in one of the segments that you’re interested in, then you

need to go back to the drawing board.

POPE-SUSSMAN: I’ve heard that misdemeanor populations

are particularly difficult to assess. Why is that?

TURNER: Well, the discussion of misdemeanor

risk assessments has really come into the light here in California with proposition 47. Traditionally, most of us

deal with risk assessments in the felony world. All the risk assessments tools I’ve done are pretty much predicting

felony recidivism. There is a question, are these tools good for misdemeanors. There are tools out there that have

been developed for misdemeanors. There’s a tool for misdemeanor DUIs and some tools that are developed by the

University of Cincinnati team have established one that’s for misdemeanors. They basically work the same as

the felony tools. They’re predicting a different kind of an outcome, which is maybe a lower level of behavior.

What we find with our risk tools is that these tools can be very robust to different populations and different kind

of outcomes.

POPE-SUSSMAN: Do you have any big vision going forward, the next step for risk assessment?

TURNER: Well, one of the risk assessment has always been used pretrial for a long time. It’s been used

the last 20 or 30 years at the probation stage. I think more recently it’s being considered at the sentencing

stage, which I think in many ways has a little more ethical issues, whether or not in sentencing actually considering

risk as opposed to the sentence type.

POPE-SUSSMAN: What do you mean by ethical issues?

TURNER: I mean, what’s the purpose of sentencing and some people say the purpose of sentencing is that

it should be proportionate to the crime that was committed and that our statutes are based, sentences are based on

perceived severity of violent offences being more serious that property offenses and you reserve the highest sentence

lengths for the more violent. If you move to a risk based system and you say, well as we found in our research, people

convicted of violent crimes are not necessarily the most likely to re-offend, so you basically a conflict going with

what is the justice based, in terms of the offense, versus a risk based, which is about future behavior. That’s

sort of conflict that you’ve got to figure out how you’re going to deal with. In our own development of

the risk tool for the State, in our training and discussion of the tool, we’ve come across people who are very

perplexed by our findings that show that people who have convictions for violent offenses, there’s actually

what we refer to as negative weights on that, so that people with violent offenses are actually less likely to have

recidivism than those that don’t, which is very counter-intuitive.

POPE-SUSSMAN: I guess

that would also get to people’s ideas about what the justice system is supposed to do. Is it supposed to be

punitive? Is it supposed to be rehabilitative? Is it supposed to be utilitarian?

TURNER: Yeah,

I think, as I teach my students, a couple things you have to remember. If you remember a couple of things, you go

out after school it’ll be very helpful. One of the them is to remember always the different purposes of punishment.

One is deterrence, and that’s what we we think risk based decision making is based on. But there’s retribution,

incapacitation. Yeah, there’s many philosophies of what the goal of punishment is. I think risk prediction sets

you straight in the cross hairs of some of the other theories of punishment.

POPE-SUSSMAN: Well,

thank you. I really appreciate you taking the time to speak with me.

TURNER: Thank you very much

for the opportunity.

POPE-SUSSMAN: This is Raphael Pope Sussman, of the Center for Court Innovation,

and I’ve been speaking with Susan Turner, Professor in the Department of Criminology, Law and Society at the

University of California, Irvine. For more information on the Center for Court Innovation, visit www.courtinnovation.org