Rate of Wrongful Convictions – You Can’t Improve What You Don’t Measure – Six Sigma and the US Justice System

(Editor’s note:  some readers may find this post a little “dry”, but I believe it’s none the less relevant.)

Just what is the rate of wrongful convictions in the US?  Nobody knows for sure.

Estimates of the rate of wrongful convictions in the US span a range from 0.5% to 5.0%, based upon several recent studies done by law schools, and cited on this blog.  SCOTUS Justice Antonin Scalia happens to believe that it’s 0.027%, but he is clearly not connected with reality.  Anything in the 0.5% to 5.0% range is a staggering number, and would cry out for remedy.

What if we were confident that the justice system produced the correct result 99.99966% of the time, and we had the data to prove it?  Another way to say this is that the justice system would get the verdict ‘wrong’ only 3.4 times per million cases, or a 0.00034% wrongful conviction rate.  That would be a wonderful thing.

This is exactly the kind of performance we have come to expect from the companies that manufacture all the products we buy and use very day.  You might be old enough to remember (unfortunately, I am) the days when it was not that uncommon to buy a new product, get it home, take it out of the box, and have it not work, or be missing a part, or have cosmetic blemishes.  This unacceptable level of product defects was costing industry a huge amount of money, because they would have to make good on warranties, and they were also losing customers.  However, it took them quite a while to finally realize this.

A perfect example of how an intolerable situation with process output defects was turned around is Japan in the early 50’s.  At that time, anything with ‘Made in Japan’ on it was globally, universally synonymous with “junk”.  W. Edwards Deming started working with Japanese industry in the early 50’s, teaching them the concepts of statistical quality management.  The Japanese embraced this, because they saw and understood its value, and by the 80’s, the Japanese were universally recognized as producing flawless products.  Their reputation for quality became known around the world.

In the early 80’s, Motorola recognized that they were losing billions of dollars (that’s “billion” with a “B”) on product quality issues.  In response to that situation, they invented a thing called Six Sigma.  Six Sigma started with Deming’s principles of statistical quality control and wove them into a structured methodology for improving the output defect rate of any process.  And by the way, the “justice system” is a process – it has inputs and outputs, the ultimate output being a verdict in a criminal case.

The nominal goal of a “six sigma” effort is to improve a process to the point where the number of process output defects statistically falls within the sixth standard deviation from the mean – there’s lots of statistics here, but that basically means 3.4 defects per million; and, it doesn’t have to stop there.  The statistical symbol for “standard deviation” is the lower case greek letter sigma, σ.  Hence, the name “six sigma”.  This methodology has been widely adopted by industry; the result being that when you bring a new product home today, you absolutely expect it to work right out of the box.

Why not apply six sigma to the justice sytem?!  Imagine a justice system with a wrongful conviction rate of 0.00034%.  By applying six sigma principles, this kind of result, and even better, is realistically achievable.  Well …. there are some practical reasons why this can’t be done in general – not the least of which is that the key components of the justice system are comprised of politically elected positions – legislatures, judges, prosecutors.  And every state has its own unique version of a “justice system”.  But there are six sigma concepts that can be applied in “pieces”.  To start, the most basic underlying principle of six sigma is MEASUREMENT.  If you can’t (or don’t) measure it, you can’t improve it.  You have to have the DATA to be able to determine what is broken, how badly it’s broken, and where; before you undertake to fix it.

Referring back to the opening paragraph – we can only estimate what the wrongful conviction rate in the US is, and only then within a broad range.  That’s because the data doesn’t exist to do anything better.  We really don’t even understand the real size of the problem today.

Implementing six sigma also requires some means of taking the resulting output data, determining the root cause of problems, and feeding it back into the process design to effect the desired improvement. In the case of the justice system, the “process design” function is accomplished by state legislatures (or Congress) – probably the least effective and least efficient mechanism to get anything done I can think of.  As an example – I would suggest that one of the root causes of wrongful convictions is having politically elected prosecutors.  If it were determined by actual data that this is true, then the feedback into the process design would result in a law to make ‘prosecutor’ a non-partisan appointed or hired position.

In fact, undertaking any kind of six sigma related changes to the justice system would require passage of a law to even get the process started.  It would have to be funded, and somebody would have to be “in charge”; all with the sanction of the legislature, and we know what a Herculean task that is.  However, Ohio’s SB77 is an outstanding example of what is possible.  This law has become a nationwide model for justice system reform, and it’s passage was justified with the limited data that we do have today (and an incredible amount of hard work by a group of highly dedicated people).  Imagine the case that could be presented to a legislature if we had comprehensive, hard data.

The new National Registry of Exonerations is a huge step in the right direction.  It’s the best source of hard data on wrongful convictions I have seen to date.  The Innocence Project also maintains a data base on exonerations based upon DNA.  But this is just a start.  We need much more comprehensive data collection at the state level, because this is where the laws get made.  We have no idea how many wrongful convictions and innocent incarcerated people disappear into the shadows, and nobody ever knows about them.  If we just had the data to know what to fix or change, and to prove the case, we could start bringing some light to the shadows.  Maybe  ….  maybe  …  we could even start approaching six sigma quality in our justice system.

You can’t fix what you don’t measure.