Bringing data analysis into the courtroom
In recent years, data analysis has been applied in almost every facet of our societies, from science, technology and business, to health and government. But the legal profession remains an exception.
A common belief shared among judges and legal scholars is that the law is an art, not a science. Specifically, the common law is ‘self-contained’ in the sense that careful reading of cases can provide the answers for all legal questions, hence there is little need to invoke data analysis or other scientific methods to solve legal problems.
The view that data analysis and scientific methods have no place in law is difficult to justify.
On the one hand it is true that law is different from science. Science aims to find out what the world is and will be, whereas law dictates what the world should become.
On the other hand, however, law shares important commonalities with the natural and social sciences. Some of the most fundamental values in law, such as logic, consistency and predictability are also the very things scientific methods are perfectly apt to achieve.
Therefore, data analysis can play a crucial role in many areas of law. Take fact-finding as an example. Bayesian statistics can help us ascertain past facts, and regression analysis can establish causal relations based on association.
Furthermore, some legal questions cannot be answered without adopting data analysis. For instance, the doctrine of disparate impact under the United States discrimination laws provides a remedy when a practice that may be neutral on its face has an unjustified adverse effect on members of a protected class. In these cases, statistical significance tests or other similar methods must be applied to assess the existence and level of disparate impact.
The neglect of data analysis in law has received a fair amount of criticism, in particular from social science disciplines. Contemporary legal analysis has been described as “data-free social science”. Some social scientists even refuse to view law review articles as “proper scholarship”.
Some complaints about the unscientific nature of law come from within the legal profession. The legendary American Circuit Judge Billings Learned Hand once asked: “How long we shall continue to blunder along without the aid of unpartisan and authoritative scientific assistance in the administration of justice?”
Lack of data-analytical skills among lawyers
Integrating data analysis in the judicial process will not be easy.
In contrast to many other professions, lawyers are notorious for lacking numerical skills. Michelle Obama once said that she became a lawyer because she was bad at science and maths. “We (lawyers) can’t add or subtract, so we argue.”
In the past few years, I have been using the well-known Linda Problem to gauge the level of maths skills of my LLM students. The problem is as follows:
Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.
Based on the above facts, which of the following has a higher probability?
(a) Linda is a bank teller.
(b) Linda is a bank teller and is active in the feminist movement.
The correct answer is (a) because the probability of one event (being a bank teller) is always higher than (or equal to) the probability of two events occurring together (being a bank teller and being active in the feminist movement). Every year, however, the majority of students (70-90%) chose (b).
This result, while disappointing, should not come as a surprise to anyone who is familiar with the common law legal education. A basic level of competency in statistics and quantitative methods is not required for admission to, or graduation from, law schools.
Lack of numerical skills is not a problem per se, as long as it is recognised and remedied. Most of us are not medical experts, but we all know we should see a doctor if we are sick.
The real hurdle is the reluctance of judges to accept mathematics and data analysis in court. In Gill v Whitford 138 S Ct 1916 (2018), the plaintiffs adopted a mathematical test called ‘efficiency gap’ to prove that Wisconsin’s Republican legislative leadership had drawn the state’s legislative district map to make Republican control inevitable. During oral arguments, Chief Justice John Roberts of the United States Supreme Court said that the plaintiffs were trying to “throw [the reapportionment issues] into the court pursuant to, and it may be simply my educational background, but I can only describe as sociological gobbledygook.”
If the court, he added, had used a mathematical test to gauge the extent of partisan gerrymandering, “the intelligent man on the street is going to say that’s a bunch of baloney”.
Data analysis does not fare any better in English courts. Milton Keynes Borough Council v Nulty  EWCA Civ 15 is a case involving an insurance claim for a fire.
One party claimed that the cause of the fire was a discarded cigarette. In the judgment, Lord Justice Toulson stated that “the process [of judging whether the balance of probabilities test has been met] is not scientific … and to express the probability of some event having happened in percentage terms is illusory.” Furthermore, “you cannot properly say that there is a 25% chance that something has happened … either it has or it has not.”
This remark is astonishing to say the least, because the court has effectively rejected the entire field of Bayesian statistics, something developed and perfected by mathematicians over the past 200 years.
In criminal cases, ignorance of mathematics can lead to serious consequences. One of the most notorious examples is the murder trial of Sally Clark in the United Kingdom.
She was accused of murdering her two infant sons who had died one year apart. At trial, the prosecution’s expert witness testified that the chance of one child dying as a result of sudden infant death syndrome (SIDS) was 1 in 8,543, hence the chance of two SIDS occurring was 1 in 73 million (1/8543 x 1/8543). This calculation wrongly assumed that the two SIDS deaths in the same family were independent.
During the appeal, this error was exposed by probability experts, and Ms Clark’s conviction was overturned. Unfortunately, by that time Ms Clark had spent more than three years in prison.
There are several ways to address the deficit of data analytical skills in the judiciary. The most obvious one is to provide necessary training and guidance to sitting judges.
In the United States, the Reference Manual on Scientific Evidence issued by the Federal Judicial Centre contains chapters on statistics and multiple regression; the Judicial College in the UK offers workshops on statistical reasoning from time to time.
However, there is some doubt as to the effectiveness of providing training to judges. Judges are under enormous pressure to manage an ever-increasing caseload; hence it could be very challenging for them to acquire new quantitative skills to which they have had little prior exposure. As an alternative, it has been proposed that an independent statistical expert system should be established so that judges can readily seek professional advice when needed.
Finally, to address the root of the problem, a paradigm shift in legal education is required. A new empirical orientation, along with statistical methods, would have to be integrated throughout the law school curriculum.
In this regard, there are some promising early signs. Several dozen top US law schools, including Harvard, Yale and Stanford, now offer elective courses on statistics, although this trend is yet to arrive in New Zealand.
Back in 1894, Oliver Wendell Holmes, one of the giants in US legal history, wrote: “For the rational study of the law the blackletter man may be the man of the present, but the man of the future is the man of statistics and the master of economics.”
Today, with the world turning increasingly digital, traditional legal methods alone are no longer adequate, and the need for wider adoption of data analysis in the legal field is bound to grow. One may hope that Mr Holmes’s long-ago prediction will finally become reality, and the practice of law will be more like science than like art.
Dr. Benjamin Liu email@example.com is a senior lecturer at Auckland University. He is in the process of creating a new course teaching law students basic principles of probability, statistics and elementary programming in the legal context.