This week we ask whether judges' personal preferences affect how cases are decided — and what that means for how courts should be structured.
Courts are supposed to treat like cases alike. But research consistently finds that judges' backgrounds and beliefs can influence their decisions. Most such research focuses on courts where individual votes are public — but what about systems where only a collective judgment is published?
A new study uses innovative statistical analysis of Swiss court data to shed light on exactly this problem, with implications for how judicial institutions everywhere should be designed.
Joining host Alan Renwick are two of the study's authors, both from the UCL Department of Political Science: Ben Lauderdale, Professor of Political Science, and Judith Spirig, Associate Professor of Political Science.
Mentioned in this episode:
[00:00:04] Alan Renwick: Hello, this is UCL Uncovering Politics, and this week we are looking at the impact of judges' personal preferences. Do those preferences affect how judges decide the cases before them? And what are the implications for how judicial processes should be structured?
Hello, my name is Alan Renwick, and welcome to UCL Uncovering Politics, the podcast of the School of Public Policy and Department of Political Science at University College London.
Courts are supposed to act consistently. The principle that like cases should be treated alike goes back at least to Aristotle. But cases are decided by judges and a wealth of research in many countries has found differences among judges in the decisions they reach, differences that appear to be related to judges' backgrounds, identities, and where we know them, their ideological positions. So it's important that we look at the impact of judges' preferences on the decisions they hand down.
And insofar as some differences between judges are unavoidable. It's important also to consider how decision making processes can be structured to minimise the impact of these differences and maintain consistency.
Yet research investigating the effect of judges individual preferences on decisions has long been constrained. Most of it has been done in countries where judges, individual votes and opinions are made public. We might think famously of the Supreme Court decisions in either the UK or the US, but in many places things aren't like that. Cases are heard by a panel of judges, but only a collective judgement is published and no public record of individual votes or dissents is given.
So how can we learn about the role of judges' preferences in places such as these? Well, a new article uses some clever statistical analysis to resolve that problem, and it thereby sheds fresh light on these fundamental questions of justice. Two of its three authors are my colleagues here in the UCL Department of Political Science.
Ben Lauderdale is Professor of Political Science, and Judith Spirig is Associate Professor of Political Science, and I'm delighted that both join me now. Ben and Judith, welcome back both of you to UCL Uncovering Politics and I think it's fair to say that the study that we're talking about was, it's definitely fascinating, is definitely important. It's also quite complex. We've got quite a lot that we're going to be covering here.
We've got three elements to the story. We've got the impact of judges' preferences. We've got the impact of decision making structures, whether for example, panels of judges are deciding by majority or by unanimity or by some other kind of rule.
And we've also got the practical question of how you study these matters in countries that have limited public information on what individual judges think. So we're going to get to all of that and we'll try to, um, go through that in a way that will keep all of our listeners with us.
But let's start with a little bit of background. Ben, do you want to kick us off? What do we already know about judges' preferences and how those preferences affect judicial decisions?
[00:03:31] Ben Lauderdale: Right. We know that the law is not unambiguous, and we know that judges have different understandings of how the law should be mapped into decisions in given cases.
And we know that from a range of studies that have been done in different judicial institutions across different countries. As you noted, a lot of work has been done in the US because the institutions generate data that is easy to access and work with. And a lot of the agreement and disagreement is public, that isn't always the case everywhere.
But what we know in general is that while the details vary in different places, judges' decisions do seem in many contexts to reflect their training, their orientations, their politics. And, you know, how that politics manifests depends on what sort of court it is, whether it's an appeals court, whether it's a sort of court for making initial decisions.
But we nonetheless have developed a sort of language in the political science literature of talking about judicial preferences and preferences here really just means a sort of summary of how judges are making decisions across different kinds of cases. And so when we talk about judges having different preferences, what we are saying, not quite tautologically, is that there are sets of cases on which those judges will make different decisions. And preferences are the language that we use to summarise those differences in decision making patterns.
[00:05:11] Alan Renwick: And could we maybe say a bit more, Judith, about the different countries where studies have been done and the different kinds of judicial systems that are involved here? I guess I'm listening to Ben who's thinking perhaps particularly about the US and we're very familiar with the idea that judicial appointments, at least some judicial appointments in the US are quite politicised.
Whereas, you know, I'm coming from a British politics perspective where the processes of appointing judges are very non-political. And also I think how we talk about judges, you know, we don't have a discourse of thinking about judges as people with political perspectives at all.
Can you take us a bit further around the world just so that we understand the range of different systems that we're talking about and places we have evidence from?
[00:05:54] Judith Spirig: Yeah. No, I think you're absolutely right. There are differences between the systems in different countries.
And so even within common and civil law countries, broadly speaking, their differences in terms of how politicised, let's say the appointment processes of judges or also in how maybe politicised certain topics that judicial decisions deal with are. And that certainly plays a role.
I think another relevant difference is the understanding of the role of the judge. And as you were saying, there is, you know, with dissenting opinions being published or allowed and with individual votes being communicated, there is an understanding that judges can defer in their judgements or their assessment of a case within, and this being a feature of how cases are decided, whereas within that tends to be more civil law countries there's really an idea of the secrecy of the deliberation of a judicial panel and judges' role is being much more interpreted in terms of them really applying the law to the case. And that's all there is to it.
[00:07:08] Alan Renwick: Mm-hmm.
[00:07:08] Judith Spirig: That is not to say that there, you know, that these countries do not have politicised appointment systems, right? And I do think the UK is to some extent an outlier, at least as compared to some other European countries whose systems are different in many ways, but also in the degree of the politicisation of the appointment process.
[00:07:27] Alan Renwick: And we have evidence from a range of countries, it's not just the US that we have evidence from that's suggesting that judges have preferences and that those preferences can influence their decisions.
[00:07:38] Judith Spirig: No, there's some more. But as you were saying, and Ben was saying too, it is predominantly from the US. Also over the past more than 10 years that, you know, we have been working on this, there have been more studies published from other countries too, but it still is predominantly from the US some from Canada, and countries beyond as well.
[00:07:58] Alan Renwick: So in this research, you're wanting to extend findings relating to judicial preferences beyond those countries that have evidence, public evidence about judges' individual decisions and individual votes and individual opinions to countries where we don't have that kind of evidence, and all we have is evidence from the collective judgments.
I mean, I guess the first question to ask before we go into the detail of that is why is it worthwhile? I mean, can't we just kind of assume that the patterns that we find in countries that we do have good evidence for are likely to be replicated in countries that we don't have good evidence for?
[00:08:37] Ben Lauderdale: I think we can't assume that because one of the things as sort of Judith mentioned, you know, the different appointments, processes in different countries mean that you're getting potentially different kinds of people with different orientations becoming judges. The patterns of decisions on a court are really deeply shaped by the kinds of cases that that court considers and different systems bring different bodies of cases to a given court.
And, you know, it depends whether it's an appeals court or a court where they're making an initial decisions. But it also matters whether it's a domain specific court or a court which sort of considers, you know, sort of a wide range of kinds of cases. And so, I think it would be a bit dangerous to assume that sort of, certainly the levels of disagreement or sort of the way that disagreement is processed in a court where we can observe the details of that, is going to apply in a system where you potentially have a different pool of cases, a different selection mechanism for the judges.
And also there are the potential effects of the observability itself. The fact that the disagreement is not going to be public potentially influences how people make decisions sort of within the black box of the process you can't observe so easily.
[00:10:03] Judith Spirig: Maybe just one thing I can add, I think is the structure of the decision making process as such that also varies widely across courts and countries.
So the specific process that we were looking at involves, in practice, let's say no deliberations at all. And that is not always the case. For example, when you look at panels of judges, oftentimes there's an element of deliberation involved.
[00:10:30] Alan Renwick: By deliberation you mean the judges discussing among themselves?
[00:10:33] Judith Spirig: Yes. Yeah, sorry. Yeah.
[00:10:35] Alan Renwick: So in essence what we have here is a complex phenomenon and we need more evidence. Fundamentally, that's what's going on. Great. So in the study that we're talking about and that you have done, you focus on Switzerland. What prompted you to focus on Switzerland?
[00:10:52] Judith Spirig: Well, I think we knew that there were data available on the website with large numbers of cases or decisions that we could look into. We knew that asylum issues are a politicised topic, which means that answering a question about whether... let's say the politicisation of the judicial selection process could play a role in decisions would be particularly, maybe, interesting and important to get some evidence on.
Then I think there's also some, both like substative reasons, in the context of what we just talked about. So we do know that the process is different. There are no individual votes communicated, and we didn't really know whether the, let's say the insights that had been generated from studies in the US would apply to the case that we're looking at. But then also some methodological reasons that would make it feasible for us to actually find out something about judges' individual preferences from the joint decisions, which is like the institutional feature about how cases are allocated to panels of judges or judges are allocated to cases.
[00:12:09] Alan Renwick: And we're talking in Switzerland about judges who are essentially political appointees. Is that fair?
[00:12:14] Judith Spirig: I think the answer to this question is, if you ask, essentially, then yes, but not formally.
[00:12:21] Alan Renwick: Uhuh
[00:12:21] Judith Spirig: So, the formal requirement is just that you are, I think above 18 and have the right to vote in Switzerland.
But then in practice, you are required to be a member or a sympathiser of a political party because the idea is that the body of judges in a particular court reflects the strength of the political parties in the Swiss National Parliament.
[00:12:55] Alan Renwick: Yeah, so I guess that's potentially an aspect of the context that it is worth bearing in mind when we're doing this analysis.
[00:13:01] Judith Spirig: Yes.
[00:13:01] Alan Renwick: Yeah
[00:13:01] Judith Spirig: Very much so. Because it is why I think many, or maybe even most of the, I don't wanna say most, but there are lots of ways in which judicial appointment processes are politicised and many countries do have politicised judicial appointment processes, but I think the way in which the Swiss one is, is rather unique.
[00:13:25] Alan Renwick: Okay. That's great. So really good to have all of that context. Thank you. Now Ben, I'm going to ask you a very big question, which is basically what do you seek to find out in your analysis of the Swiss case, and in terms that I'm gonna be able to follow as a non statistician, how do you go about doing so?
[00:13:44] Ben Lauderdale: Right, so what we observe for each case is a decision, a number of facts about the case that are sort of embedded in the text of the decision about the applicant for asylum and details of their circumstances, country of origin, et cetera.
[00:14:02] Alan Renwick: So we should actually just explain, we haven't quite done so that you're looking specifically at appeals on asylum cases. So these are cases where someone's asylum application has been rejected and they're appealing against that decision. And that's the set of cases that you're focusing on?
[00:14:16] Ben Lauderdale: That's right. So we know the decisions at the previous stage are sort of all the same, but then these appeals, we observe the decision that this court makes on the appeal.
We observe some facts about the appeal itself, and then we observe which judges were selected to hear this case. Three judges, typically, with some details we can maybe go into on cases where we don't observe three judges. And one of those judges has been specifically identified as the chair for that decision and all the judges serve as chairs sometimes.
And we can go into a little bit more about what we know about the process by which the decision is made and the case is heard in a moment, but what we're trying to do in our analysis is to simultaneously figure out how those different judges' views go into the decision, and also some summary of what those judges' views are, their preferences.
And that sounds like a very challenging thing to do, to both figure out what each of them would like and also how each of their views are being combined to make a decision.
[00:15:35] Alan Renwick: Mm-hmm.
[00:15:36] Ben Lauderdale: Um, and I think we'll need to say a bit more about how we're doing it, but that's the task that we're trying to do.
[00:15:42] Alan Renwick: Yeah, so this really intrigued me when I was going through the article because you know who the three judges are who are on the panel, you know who's the chair, you know the outcome, you know the decision that they collectively made, but you don't know the individual preferences of those judges and you don't know how they came to an overall judgement.
So did they decide that they needed to come to a unanimous judgement? Did they just go with the majority? Did they just accept what the chair thought? You don't know that either. And I was thinking, so I can see how you should be able to kind of work out what decision making process they're using if you know the individual preferences of the judges. And I could see that if you knew the individual preferences of the judges, you would be able to work out 'well okay, so how were they coming to collective decision?' But you're doing both together here and that seems like an extraordinary thing to have done. Do you want to just take us through the logic of that?
[00:16:37] Ben Lauderdale: Yeah. So the core approach is to use our theoretical expectations about which kinds of judicial preference aggregation mechanisms are possible.
And that's sort of a lot of words to say how it is that their views get combined into a decision. We have a few different live theories that we want to test about that. And what we can do statistically is sort of take each of those in turn, assume that that theory of the decision making process is correct, estimate based on that what all the judges' preferences are given the data that we have.
And then we can do that for each of the different models of judicial preference aggregation, the decision making process, and then see which of those actually fit the data best. And that does turn out to be possible. With enough cases, you can start to see patterns that indicate one theory is better or another theory is worse.
So to give a sort of concrete example.
[00:17:39] Alan Renwick: Mm-hmm.
[00:17:40] Ben Lauderdale: One of the theories we consider, and I think we'll talk in a moment about features of the decision making process that make this an interesting theory, is that actually it's only the chair's view that matters and the other two judges who are listed on the case are irrelevant. It just gets decided by the chair, the other people sign off.
So if that were the right theory of what's going on in the black box as it were, that has implications for the patterns in the data that you see that, you know, the only thing that determines how a given case is decided is what the preference of the judge who's been assigned to be chair is. And that has implications for figuring out what the preferences of all the judges are based on the cases where they were chair.
So you can fit a model based on that assumption, but you could also fit a model based on a different assumption, which is that all three judges carefully consider the case. They have a vote internally by majority rule. And the winner of that vote is the decision that gets published. And we don't find out whether there was disagreement or not. That is a very familiar kind of problem to political scientists. That's one where the median judge's preferences determine the outcome.
So if there's one judge that is very strict with respect to asylum appeals and one judge who is very lenient and there's someone in the middle, it will always be the one in the middle whose views determine the outcome. And we can fit a model that assumes that that internal process is occurring and that has implications based on this set of cases that we consider for where the preferences of all the judges are and then we can see how well that fits the data.
And so what we did was fit a series of those models, those two included, as well as some others, including some that we knew didn't make much sense, like assuming that the strictest judge, their preferences always prevail, or the most lenient judge, their preferences always prevail. We didn't expect those to fit very well and they don't fit very well.
And sort of maybe to give away the punchline, what we found is that chair model and that median model of the sort of relatively simple theories, those two fit the best. But actually what fits better is a sort of intermediate model that mixes those, which to sort of sum up in words is one where the chair has disproportionate power, but the other two judges on the panel do seem to influence the decision, and so who those judges turn out to be does matter, it's not just the chair, but the chair does seem to matter more.
[00:20:19] Alan Renwick: Judith, do feel free to add anything to that if you want to. I guess one particular point that Ben was hinting at there is it's important that we understand what we do know about the processes that the judges follow in reaching these decisions.
[00:20:32] Judith Spirig: Yeah, exactly. And I think that speaks a lot to your intuition about having to have one of the two things like either the way they make a joint decision based on their individual preferences or their individual preferences in order to figure out the way they aggregate their preferences into the joint decision is spot on, right?
It's like we basically thought, based on the research that exists, what are plausible ways about how judges come to a joint decision? And then we said, okay, let's joint estimate their individual preferences, assuming, you know, every one of them is the right way, the right way to describe how they come up with a joint decision. And then after we'd done all of that, we would compare how well each of these models would describe the actual outcomes that we observe.
[00:21:30] Alan Renwick: Mm-hmm.
[00:21:31] Judith Spirig: And then basically we can say, well, you know, if we have this one rule, like the mixed kind of model where the chair has a disproportionate influence on the decision as we observe it, fits our data best, then that's going to be the best description of what's actually happening and so the preferences that we estimate with this model are going to be the most plausible ones.
[00:21:55] Alan Renwick: Yeah. Okay. I think I follow.
[00:21:58] Ben Lauderdale: I think it might be useful here to sort of walk through how cases pass through the process of these judges and maybe Judith can say something about how that sequence works.
[00:22:09] Judith Spirig: Okay. So, and I was already hitting at that when saying why one of the reasons, and maybe I should have led with that, the Swiss Federal administrative courts that handles the assignment appeals is maybe one a case that is, you know, particularly useful to study this, is that there's a specific way in which they determine which three judges should hear a case.
And that is some software that allocates which judges to a case, that has some constraints, which have to do with Switzerland, like, which kind of languages, you know, and what the language of the first instance decision is and things like this.
But then it tries to just allocate judges in a way that they all have the same workload basically. And so there is this, you know, we call it exogenous, but there's no direct relationship between how strong, and there shouldn't be and we do try and test this to some extent, but there's no relationship between how strong a case is, or in other words, how likely we would think the appeal should be granted given what we know about the case, like the applicant's country of origin and so on and so forth, and the judges themselves, like their party affiliation and things like this.
[00:23:25] Alan Renwick: So we can be confident that if there are differences between judges in the proportion of appeals that they grant or don't grant, then that's a difference in the judges rather than difference in the cases.
[00:23:37] Judith Spirig: Exactly.
[00:23:37] Alan Renwick: That they are deciding.
[00:23:38] Judith Spirig: Yeah.
[00:23:38] Alan Renwick: And I guess Ben is also hinting at the fact that we should be aware that with each case, first it goes to the chair and the chair looks at it and makes a recommendation, and then it goes to the second judge in the panel, and that judge takes a view and then it goes to the third judge in the panel, and that judge takes a view. So it could be that the second and third judge is just kind of, I mean, maybe they're in for a quiet life, maybe they're just very, very overworked, but that they might decide they're just going to accept the chair's view. Or it could be that no, actually they do look very seriously at each case and they kind of fight their corner, so there are various possibilities as to what might happen there, but at least we can rationalise why it might be that the chair would have disproportionate influence over the outcome.
[00:24:21] Judith Spirig: And it's also something that we see in some US courts where there's a designated opinion writer and they appear to have more influence over the case decision than some of the other judges.
[00:24:34] Alan Renwick: Yeah. So we've already talked there about one of your findings. Moving on to what you actually find through this research. So we've talked about the fact that the evidence that suggests that in terms of how they reach their decisions, chairs are disproportionately influential, but other members of these panels are not unimportant and insofar as they matter too, some kind of majority based decision rule is being applied. What about the question of judges' individual preferences and how much of a difference individual judges make to these outcomes?
[00:25:11] Ben Lauderdale: What we discover is that under the models that fit the best, there is, I think it's fair to say, a substantial amount of disagreement among the pool of judges that are hearing these cases about where they should draw the line.
And this is easy to understand in the asylum context because a lot of the law depends on sort of how you evaluate the circumstances in the country that someone is coming from and how that interacts with features of the asylum applicant in making a judgement about whether it's safe for them to return or not. And people can reasonably disagree on where the legal threshold is for it being safe for them to return.
And there are other features of the law that similarly are not straightforward to evaluate, and so, there's reason to expect disagreement, just about that sort of difficult factual evaluation. But we do find evidence that that disagreement is associated with the sort of the parties that the individual judges were affiliated with.
And so, you know, that finding does mirror other countries where we see that sort of the political orientation of judges is predictive of how they will vote and we sort of see what we would expect in terms of which party's judges are more favourable versus less favourable to the asylum applicants.
[00:26:40] Alan Renwick: So the really big effects here, if I understood correctly, in your best fitting model, you estimate that the most lenient individual judge, their preference would be to grant appeals in 34.7% of cases while the least lenient judge would prefer to grant appeals in 4.6% of cases. So there's a massive difference between these judges.
[00:27:07] Judith Spirig: Yeah. I mean, maybe one thing we can add is that they don't decide most of the time themselves, but as Ben mentioned before, it does seem like the mixture model, which says that the median judge also does have some influence, it's not just the chair judge, and that means that there's some moderating influence of the panel over having just the individual judge decide.
So, basically what that would imply is that if we look at the actual case outcomes, they're going to be smaller differences than what we would think given judges' individual preferences to some extent. But yeah, the differences do exist. There's some uncertainty about where they actually are as the preferences are or as you can see in the graph, but there are meaningful differences.
And another thing maybe we could add is that there's not just differences across parties, but also within parties. So it's still the case that judges affiliated, for example, with the Social Democrats do exhibit some variation in their preferences.
[00:28:15] Alan Renwick: Yeah. So the kind of overall passion across the parties is, I guess what we would expect. The Social Democrats on the left are much more likely to want to grant an appeal than judges from the Swiss People's Party on the far right, or at least that's what you're estimating here.
So there are big, big differences between the parties. So Social Democrats, about 20% Swiss People's Party, about 5%, but there is then variation within those parties as well. And I guess some of the parties are showing more variation than others. So if I remember correctly, the Christian Democrats were quite diverse in their
[00:28:52] Judith Spirig: Yeah.
[00:28:53] Alan Renwick: Positions, whereas the Swiss People'sParty judges were all much stricter than most other judges.
[00:29:00] Ben Lauderdale: Yeah, I mean, I think it is really important to note that while we are estimating this substantial variation at the individual judge level, this system is sort of designed to smooth some of that out.
That's the whole idea of having panels of judges rather than individual judges decide cases. And sort of the more that it's going to be difficult to get consistent decisions out of different judges individually, the stronger the normative argument for having a panel based system.
And so, you know, the system is designed in some way to smooth out some of these variations. It can't do it entirely. There's still the implication of our analysis, is that there is a bit of a luck of the draw if you're an applicant that, you know, you might happen to get three judges who are all relatively strict or you might happen to get three judges who are relatively generous in terms of granting successful appeals. And that's both what enables us to identify that there is variation, but it also has some normative implications for the process from the perspective of the applicants.
[00:30:11] Alan Renwick: Yeah. So you had a number in the paper that said that 8.4% of cases would be decided differently with a different panel. And I guess I was trying to get my head around exactly what that meant because presumably there were really quite a lot of cases where with a particular different panel they would've had a different outcome. But you're kind of saying that if we did the randomisation again of which judges are on the panels, then something like 8.4% of cases would end up with different decisions.
[00:30:40] Ben Lauderdale: That's right. This is sort of what's thought of as a test retest reliability in the sort of more statistical jargon.
It's asking the question, okay, if we took these cases and we ran them through the system, again, how many of them would come out differently? As you note, there are more cases than that that could conceivably with some panel come out differently if you found, you know, the most strict or the most lenient panel to get them to come out the other way.
But that's not what that number refers to. That number is sort of just imagine we reran these cases through once each, on average across all of them. What percentage would we expect to come out differently? And that's what that 8% number corresponds to.
[00:31:23] Alan Renwick: Yeah, no, great. Thank you. Judith?
[00:31:26] Judith Spirig: The only thing I could maybe add is if you think the number is small, 8.4% for example, and you thought it would be bigger given our figure about the heterogeneity.
I think one thing that is important to consider perhaps is that this heterogeneity is only going to impact cases that fall in the space where judges would disagree.
[00:31:51] Alan Renwick: Mm-hmm.
[00:31:51] Judith Spirig: Right? So extremely strong or weak cases are going to be decided in the same way by all panels.
[00:32:00] Alan Renwick: Mm-hmm.
[00:32:00] Judith Spirig: That could exist.
[00:32:01] Alan Renwick: Mm-hmm.
[00:32:01] Judith Spirig: I guess it's just another way of saying like the heterogeneity is only going to impact a certain kind of case.
[00:32:08] Ben Lauderdale: On the other hand, you might have reacted to that 8% and thought, boy, that seems really high. You know, one in 12 cases would come out differently if we ran them again. These are people's lives, the course of which was being determined by this process that seems, you know, surely we should want that number to be incredibly low and I think normatively, it's true, we should want it to be very low, but also there's a real challenge in institutional design in creating a system that can actually generate such a low number, given the fact that people who are qualified to be judges on this court disagree.
And people with different political orientations disagree about how to make these evaluations. And there are too many cases for one person or the same set of people to decide all the cases given the detailed facts that need to be worked through. And so there's a real challenge in institutional design in any court system to be able to make decisions at scale and at the same time make those decisions in a consistent and statistically reliable way.
[00:33:11] Alan Renwick: And in a sense, you were saying that actually the Swiss system is set up quite well, you know, given the starting point of an expectation that judges will be reflective of the diversity of opinion across the population as a whole, then the system is set up well in order to kind of moderate the effects of that diversity upon judgments so that you're maximising the consistency to a reasonable degree.
[00:33:39] Judith Spirig: Yeah, I think to some extent, yes. Not having just the chair or individual judges decide these cases, we do think the analysis would suggest that that has some moderating influence.
[00:33:50] Alan Renwick: I guess you might ideally think reduce the influence of the chair more.
[00:33:53] Judith Spirig: I mean that would be
[00:33:54] Alan Renwick: and have pure majority decision making.
[00:33:55] Judith Spirig: One, one. Exactly. So one thing that you could do with these results would be to say, well we do value consistency so much that we would try to reduce the influence of the chair over the majority vote system basically. And I think, you know, while we don't know exactly why the chair has disproportionate influence, the setup of the procedure would suggest that that has to do with the sequencing in which the decision is being reached.
So you could think about, you know, not having the second and the third judge not having information about the draft decision of the chair, but then that would have implications for workload and things like that.
[00:34:42] Alan Renwick: Yeah. So you could basically do three blind reviews and average them and that might be the optimal approach, but it's costly.
[00:34:48] Ben Lauderdale: Yeah. I mean, as academics who mark student work and operate in a system with second marking for some assessments, we've all observed that there are variations in how you implement marking oversight for student work. You can have people mark blind in parallel. So you have two people read the student work, separately come to their views, and then attempt to reconcile that potentially through some process of deliberation. That's time intensive.
Or you can have someone who's designated as the first marker read the student work, put down a mark and an explanation, and then hand it to someone else to review and, you know, one of the consequences of that is that there is an option available to the second reviewer to say, you know, okay, this looks about right and spend a lot less time on it than they would if they had to start from scratch. That might be the kind of process that leads to that first markers views having disproportionate weight.
But there's a straightforward workload trade off. And if you only have one case to decide, or one student paper to mark it's relatively easy to say that everyone should form their own independent view and then deliberate and come to a decision. But if you have hundreds of student papers or thousands of appeals cases, you may not have the institutional capacity to follow that kind of process. And so a process that has a bit of oversight, but doesn't have independent multi-person decision making and deliberation, may be a reasonable compromise. But there are different compromises that could be made.
[00:36:29] Alan Renwick: So I guess from your evidence, you can more or less estimate the amount of time that each judge might be putting in and therefore how many extra judges would be required in order to have a full kind of three independent reviews taking place of each case. And you can also estimate, presumably, how much difference that would make to outcomes?
[00:36:54] Ben Lauderdale: So I think we can sort of simulate how much would it further reduce the variability in outcomes.
[00:37:00] Alan Renwick: Mm-hmm.
[00:37:00] Ben Lauderdale: If we went to the full independent version, assuming the preferences that we estimated based on what we think is going on right now.
So what's a little harder is, and you were perhaps too optimistic about our abilities here, is to figure out
[00:37:17] Alan Renwick: I'm bamboozled by your abilities all the time. They look kind of potentiary.
[00:37:21] Ben Lauderdale: It's not magic. figuring out how much time the second and third judges are spending is I think not straightforward from the data that we have.
[00:37:29] Alan Renwick: Yeah
[00:37:29] Ben Lauderdale: We would need different sorts of data to know, are they getting this level of sort of oversight and tempering of the chair's views through really just a tiny bit of effort? Or is it actually taking a substantial fraction of the amount of effort that they would have to put in to just do an independent evaluation and judgement themselves.
[00:37:47] Alan Renwick: Yeah. No, that seems very fair. We're going to have to wrap up, but final question to you, Judith. I'm just intrigued that these findings are so interesting. I'm wondering if anyone in Switzerland is looking at them and thinking, ooh, should we change our processes? Is the court interested in these findings.
[00:38:02] Judith Spirig: Yeah, the court has been interested in our work and I think their commitment to sharing the data with us is sort of evidence of that too. And we have in the past, it's been a few years, but been invited to present preliminary results.
Like we also say in the article, they're definitely concerned about allegations of, you know, inconsistency in decisions and they do want to, I think, do a good job, obviously.
[00:38:29] Alan Renwick: Mm-hmm.
[00:38:29] Judith Spirig: But they also understand that judges are people. They had to talk in the media about, I mean this is a topic that comes up not just based on our research, but just asylum is a contested issue. They have lots of decisions to make. There is a big backlog. Their decision making comes up in the media at different points in time. They do acknowledge that the way the system is structured, judges have different preferences.
[00:39:01] Alan Renwick: So in many ways it's an example of a really nice piece of collaboration between researchers and an organisation that cares about doing its work well, and is keen to get the evidence that will help it to do that.
Well, thank you so much, Judith and Ben. It has been fantastic to hear about this research. Really, really interesting work.
We have been discussing the article Inferring Individual Preferences from Group Decisions: Judicial Preference Variation and Aggregation on Collegial Courts by Dominik Hangartner, Benjamin E. Lauderdale, and Judith Spirig.
As ever, we will put the details including a link in the show notes for this episode. We will be back next week when Emily McTernan will be joined by our colleague Jeff Howard, to discuss the regulation of AI content online.
To make sure you don't miss out on that or other future episodes of UCL Uncovering Politics all you need to do is subscribe. You can do so on Apple, Google Podcasts or whatever podcast provider you use, and while you're there, we'd love it if you could take a moment of time to rate or review us too.
I'm Alan Renwick. This episode was produced by Matthieu Dinh. Our theme music is written and performed by John Mann.
This has been UCL Uncovering Politics. Thank you for listening.