The World Justice Project (WJP) produces an annual Rule of Law Index. The Index brings together information from a wide range of sources. The 2016 Index covered 113 countries and jurisdictions and provides an authoritative analysis of how the rule of law is faring in each. While New Zealand is always well placed, there has been some discussion of what it means when we are overtaken in the rankings. The New Zealand Law Society’s Rule of Law Committee looks at how the Index works and what can be concluded in relation to New Zealand.

The Rule of Law Index provides scores and rankings based on eight factors: constraints on government powers; absence of corruption; open government; fundamental rights; order and security; regulatory enforcement; civil justice; and criminal justice. Each of these is assessed by reference to 44 sub-factors, scores for which are also published.

The methodology giving rise to these indexed results turns upon responses to questionnaires specific to each factor. These are completed by general members of the public and “experts”, namely in-country practitioners and academics with expertise in civil and commercial law, criminal justice, labour law and public health. So they reflect lay and informed perceptions and experiences that, WJP says, are equally weighted “in most cases”.

Limitations to methodology

As WJP acknowledges, there are inherent limitations to the methodology underpinning the Index that should condition the interpretation of its results.

First, the Index does not purport to offer policy insights of a kind offered through other international evaluation mechanisms, such as “mutual evaluation reports” undertaken by the Financial Action Taskforce (in the money laundering / counter terorrist financing context) or by the OECD Working Group on Bribery (in the anti-corruption context). Rather, the Index provides a snapshot of a country’s preceived performance. The questionnaires do not explore underlying dynamics, so results do not purport to provide insights into even perceived cause and effect. As WJP states, the Index “does not identify priorities for reform and is not intended to establish causation or to ascertain the complex relationship among different rule of law dimensions in various countries”.

Secondly, being dependent upon subjective responses to questionnaires, sample size and statistical methods (such as the “z-score” method) are important in terms of mitigating the impact of skewing, outlier responses. WJP claims to use local polling companies to obtain a representative sample of 1,000 respondents and to identify, on average, more than 300 local experts per country. Spread across the eight factors, this translates to only 37 or 38 experts per factor.

Thirdly, as expected of a worldwide survey the questionnaires appear designed to assess rule of law maturity in both undeveloped and developed contexts. To understand why one country is scoring highly and another poorly, it is helpful to understand the sort of universal metrics that the questionnaires target.

Respondents are generally given a hypothetical scenario and either asked to rate the likelihood of a certain consequence of that scenario on a scale of “very likely” to “very unlikely” or to identify the most likely of a number of given consequences. For example, to cater for the undeveloped end of the criminal justice spectrum expert respondents are asked about the likelihood of a detained suspect being tried and convicted in a secret trial or of an arrested drug cartel member being severely physically harmed during police interrogation or killed by police before trial. Not all questions so bluntly target the potential for such outcomes but all questions do permit responses reflective of a very low level of rule of law maturity.

Comparing scores

Given that questionnaires target different factors, it is probably too much of a stretch to use comparisons between scores across factors as a means of indicating of where a country’s limited resources should be deployed. Is a bruised apple worse than a slightly dented orange?

For these sorts of reasons, WJP emphasises that “the Index is generally intended to be used in combination with other instruments, both quantitative and qualitative. Just as in the areas of health or economics, no single index conveys a full picture of a country’s situation”.

All that said, a particular country’s improvement or decline can be gauged by comparing its results from year to year. Although questions do change (to some extent) from year to year, WJP states it has “normalised” the 2016 results to a 2015 base year. For the past few years, WJP has also used a “bootstrapping” procedure to illustrate whether changes in factor scores are statistically significant at the 95% level of confidence.

That additional analysis is telling for practitioners, academics and policy-makers – particularly here in New Zealand.

Comparing New Zealand’s 2016 and 2015 scores

Compared to 2015, our scores are now down for the factors of “fundamental rights”, “order and security” and “criminal justice” but up for “constraints on government powers” and “open government” and unchanged for “absence of corruption”, “regulatory enforcement” and “civil justice”. Further comparison going back to 2013 indicates that scores across all factors bob up and down from year to year but within very small margins. The only statistically significant changes in that period have been as follows:

• the factor “criminal justice” dropped from 0.79 in 2013 to 0.72 in 2014; and
• the factor “constraints on Government power” dropped from 0.88 in 2014 to 0.85 in 2015.

These changes were not indicative of any trending decline. “Criminal justice” climbed to 0.77 in 2015 and currently sits at 0.75, still well up from the 2014 drop. “Constraints on Government power” is also now up, sitting at 0.86.

New Zealand’s “overall score” is currently 0.83. If you presumed that this is a good score, you’d be right. The highest possible score is 1.00. A score of 0.83 places us first in the WJP’s “East Asia and Pacific Region” and eighth of 113 countries worldwide. This (arguably) represents a drop from 2015, when we were sixth out of 102 countries, but it has to be asked whether the challenge is to maintain or improve our own scores or to be the best in the world at all cost. It can be said, though, that our scores are not demonstrative of steady improvement.

Doubtless, New Zealand’s lack of corruption is a significant driver of our high performance and on that front, according to Transparency International’s Corruption Perceptions Index, we are the best in the world (along with Denmark) with a score of 90 out of 100. We have jostled for that number one spot with Denmark and Finland since that index was first published in 1995. At least since 2013 our “absence of corruption” scores on the Rule of Law Index have all been in the range of 0.9 – 0.92.

In terms of rule of law, this means more than a good score on one factor. As Transparency International New Zealand notes, top performers on the Corruption Perceptions Index share key characteristics: “high levels of press freedom; access to budget information so the public knows where money comes from and how it is spent; high levels of integrity among people in power; and judiciaries that don’t differentiate between rich and poor, and that are truly independent from other parts of government.” These characteristics clearly avail any country’s performance in terms of adherence to the rule of law.

The NZLS Rule of Law Committee consists of Austin Forbes QC (convenor), Gregor Allan, Isaac Hikaka, Professor Philip Joseph, Sir Geoffrey Palmer QC and James Wilding.