Trigger warning aside, I wasn’t expecting such a positive response to Part 1. But this one is a harder read. Seriously. Don’t read this if “predicted grades” are your thing.

I’ve tried to be nuanced ; If you feel yourself about to a jump to conclusions, ask questions first.

The key assumption here is that the pressure comes from concerns to find a way to facilitate third level entry in the new academic year. Few calling for the cancellation seem to have acknowledged that cancellation would deprive those who are not going to university of an important qualification. This cannot be a fair situation either. I have assumed that no one is as selfish as to adopt and attitude of  “well I’m alright Jack”. Make of that what you will. While it is reasonable for people to look out for their own interests, affirming whatever rights they have should not come at the expense of someone else’s rights. Balance in all things Virginia…..

….So….grab cup of tea, this one goes on a bit…..

Schools, Standards and Such-like

The denominational nature of Irish education has meant that Irish education has a number of qualities that set it apart from education in some other jurisdictions. Chief among these is school and teacher autonomy. Schools are run in a manner consistent with the schools’ stated ethos. And while there has been a rise in the role of the schools’ Patrons’ since the Education Act 1998, individual schools still retain a very high degree of autonomy. Teacher autonomy continues in the face of the homogenising influence of the Junior Cycle reforms which, despite claims to do away with a ‘one size fits’ all, still manages to be remarkably prescriptive.

This autonomy is vital to the ‘health’ of our education system. It allows schools and teachers to respond with a huge degree of flexibility to the needs of their students. It affirms teacher professionalism in the design and delivery of lessons (although again, the new Junior Cycle is eroding this) and in the care teachers exercise not just towards students’ academic endeavours but in their wellbeing. Irish teachers are highly trained by international standards and are highly sought-after outside Ireland.

But autonomy comes with one particular price which is highlighted in the current crisis. No two teachers deliver the same lesson or lesson-topic in the same way. It is possible, and probably very common that two such teachers would also have different class tests to examine material taught. This is because each teacher might be looking to examine different aspects of a topic depending on the extent of student assimilation of the material in a lesson. Again, it is important to note that this is driven by teacher flexibility in responding to their particular set of students.

In short, there is no standardisation of lessons, class tests, and school end-of-term tests. If we multiply this across the education system we have a vibrant and engaging education system without the shackles of prescribed approaches to teaching and learning (though this is changing, sadly) and in-house testing. And this is a key strength of Irish education. To attempt to change this would produce a stultifying murder-machine that would harm education in Ireland irreparably. The only downside – though I reject it is a downside – is that we do not have a system in which we can use in-house tests collectively to compare children against a set of national norms. The only standardised tests that all students sit equally are the LC examinations. This is plenty.

Growing pains

Teachers take a developmental approach to getting students to the point where they can sit their LC exams. This approach means that in-house tests serve the purpose of pointing students in the direction they need to go. They are summative tests which have a formative purpose. In-house tests are generally not intended to be a fullstop along the way as if to say “well, you know that topic 50%” and that is it. It is more a case of “for now, you know that topic 50% but you need to study it more and make these changes and you will get closer to 100%”.

This is important. A student’s performance on a test is partly determined by how important they think that test is as well as factors such as their cognitive development at the time of the test (lower at the start of the LC course compared to the end), the number of exposures to curriculum content (lower at the start of the LC course compared to the end), their level of motivation (lower at the start of the LC course compared to the end) and effort (lower at the start of the LC course compared to the end) among others.

LC students would generally be acknowledged as attempting to pull out their best possible performance after they have matured through the whole curriculum, not just part of it. To assess them on results retrospectively before they were at their possible best would be unfair to them. And to attempt to adjust for this would introduce a large degree of bias – how exactly would we adjust for each particular student. This is an impossible task across a 60,000 strong cohort.

The only standardised tests that all students sit are the LC examinations. They are the same for everyone and are a statistically valid and reliable, and are a fair and consistent means of testing all students. There are of course some caveats and exceptions to this but as a rule, the LC exams are fair for everyone. And when they are unfair, they are unfair equally for everyone. They are administered independently of schools and teachers to each student and they are marked anonymously. An examiner (the person marking) does not know the school nor necessarily the location of the students they are examining.

This is the important point in the current debate and the heavy campaign to cancel the LC exams and replace it with what some are calling ‘predicted grades’. Inevitably such grades, it is suggested, would be based on the very class tests mentioned above. However, these tests are not standardised against a national cohort, and they are generally not even the same test and so it would not be a fair and consistent basis on which to ultimately allocate scarce third level places.

But, but, but….

If the leaving Certificate was cancelled and “predicted grades” used, a student could say that they would have done better on the LC exam but for the cancellation. This means even if predicted grades were to be used, the LC exam would still have to be offered. Maybe not in August but at some point. Bear in mind the exams would still be needed by those who were not going to college but who needed the qualification. (And on the chance that someone did not get past Year 1 in third level, their LC exam results would become relatively more important again to them).

Again, students could also argue that the use of predicted grades was retrospective and not what they were expecting their university place would be decided upon. As I mentioned below, saying students could sit the exam if they were unhappy with the predicted grade is logically inconsistent with arguments that at the same time are calling for the cancellation of the LC exam!

Bear in mind, that at the time the decision was made, postponement of the exams represented (and arguably may still) the least disruption to the education system for everyone in education. There is already a danger of the system ‘backing up’ like a clogged drain. If, as it must have seemed to the Department of Education, there was the possibility of holding the exams – safely – in August, it would have been less disruptive overall even if 2020 LC students were inconvenienced.

The only valid arguments I can see (comments below if I missed any) for cancelling the August date (bearing in mind the LC exams still needs to happen for some students) is twofold:

1. The mental health of a large number of students is SO adversely affected by the having to sit the exams at a later date that it is a lesser evil to find alternative means to facilitate their access to third level. This argument is separate to issues related to the corona virus issue. It is purely about possible stress of having to wait an extra two months to sit the exam. 
   I’m not sure how such adverse affect nor a relevant number could be determined. I think it would be a waste of time to set about determining this. I’m not convinced that the postponement is, in and of itself, an significant additional mental strain but that having to work throughout the summer could be. I’ve been in university libraries in summer time; students do study during the summer. This is not to minimise that postponement is a very real for some students, in particular those with an already fragile mental health. Where students are acting as carers, the extra time before the exam could be a boon. But nonetheless, the mental health argument is strong and deserves a listen and importantly, consideration in planning.
2. The risk of infection from the virus is not sufficiently reduced (possibly R=0) to enable people to sit in a room – even socially distanced – for any period of time. This for me is the most important reason.
   There can be no question of putting the physical health of any individual at any risk.

Barry’s Brainstorm

Into all this writes Barry O’Callaghan in the Irish Times. He is chair of a committee of the government-funded National Association of Principals and Deputy Principals although he was writing in a personal capacity. While the sub-editor usually chooses the headline, he argues for a cancellation and the use of “predicted grades”. A closer read of the article finds several suggestions so the article reads to me a little more like a stream-of-consciousness than an in-depth proposal but I took it at face value.

Barry makes a number of claims that for me are questionable – including, for example, that high achievers will probably cope with waiting until August for exams. In fact, it is probably the high achievers with the most to lose by not having the exams and whose stress levels are at least as high but probably higher than many.

It is important to note that NO ONE is suggesting that students will do exams with the risk of infection hanging over them. But no one can validly claim that this is what is being asked of them without knowing whether the virus will be under control at the end of July (or knowing that someone has be stupid enough to actually state such a claim!).

So Barry suggests that schools supply expected grades to the State Examinations Commission (SEC). He says “Teachers predict grades based on evidence in filed records, schools’ IT systems, mocks, Junior Cert, CAT4 testing, project/practical work etc.”

This suggestion is, to me, poor on many levels underlined by the use of etc. But let’s take them in turn.

Filed records – which ones? All of them? Some of them? Cherry-picked one perhaps? It doesn’t matter – because filed records – short hand for mid-term and class tests – are not standardised for every student in the country – that is, they are not based on all students doing exactly the same test which has been norm referenced. So to use them to determine college entry would be unfair on this basis alone. In addition to this students would not have necessarily approached these on the basis that they would be used to determine their college entry. Any student could claim in court that they were deprived of a place because their predicted grade did not represent how they would have done had they sat the LC exam and that cancelling the LC exams deprived them of the opportunity to do this.

(As an aside, it is a question of law as to whether a student has a legal right to sit the Leaving Certificate exam. There may well be an implied right, through decades long practice and reasonable expectation, or even a contractual right established by the acceptance by the SEC of the examination application and fee. I have assumed that such a right exists and that a student could have a legal standing in claiming a reasonable expectation of the exam going ahead – albeit with some sort of postponement under the circumstances. More anon).

Schools’ IT systems – same as above.

Mocks – Not all schools hold mocks. Not all school that hold mocks held them before the schools closed. Not all schools use the same mock papers. Mock papers are not standardised for the national cohort (I used to write some). Mock papers are often leaked on social media. So these cannot be a valid source of material to construct ‘predicted grades’.

Junior Cert – It’s hard to know where to begin. Firstly, the Junior Cert was at least two, and for many students, 3 years ago. So the grades students got then are cannot be an accurate indicator of how they would perform in the LC exams. And again, students were not thinking of their Junior Cert in the context of determining their college place. In addition to this, boys tends to achieve less than girls in the Junior Cert but they catch up during the LC course. So using the JC results would be unfair to boys.

CAT4 – I have advocated for *consideration* of CAT4 as a possible alternative. Cognitive Ability Tests test verbal, numerical, spatial and problem solving abilities. They are not IQ tests. The CAT4 has some weaknesses. It does not test determination, persistence and effort as the LC exams do. On the other hand, as a test of cognitive ability it is a good indicator of capacity to learn. It would also be more egalitarian. Being based on ability it would factor out some (but only some) of the advantages of middle class children. But it would still require students to sit in a room – something that would not be safe unless the possibility of infection were reduced to zero.

Project/practical work – Orals and practicals were cancelled and students were awarded 100% so these can’t be included in the consideration of predicted grades. In addition, project work is worth different percentages in different subjects. So there is no way of treating students equally in considering these. Project work has not been cancelled so students with projects are disadvantaged relative to those with practical and oral exam components. And I can’t see this changing on its own because of the different percentages each project is worth.

All told, none of the way’s Barry suggested are, to me, in any way a statistically valid and reliable means of generating a grade that would be a clear, honest and fair way of determining the allocation of scarce university places. Arguments that we have to do something and that therefore predicted grades are the thing is a fallacy. We have to ensure that whatever solution (other than holding the LC exams) is fair to everyone and not likely to create more problems than it solves. Sometimes doing anything for the sake of doing something results in a disaster.

It is easy to say that this is naysaying or that these are all the ways they don’t work but what about ways telling us the ways we can make them work. No amount of rhetoric will produce an outcome in which retrospective predicted grades are fair to all students. As stated above, I have assumed that other than certain exceptions, everyone is in favour of treating everyone equally, and generally as reasonable as possible, being fair to students as a collective.

None of this does not mean that we can’t do “predicted grades” under the circumstances. Anything is possible. Including makey-uppy grades. My argument is not specifically against the concept of predicted grades (although these are problematic and there are calls to abandon the practice of using predicted grades in the UK). This is not my consideration here. But rather, bearing in mind everything in Part 1, that retrospective predicted grades are unfair and not a sound basis on which to allocate university places. My argument is that retrospective predicted grades is less fair to all students than postponing the exams.

Again, no one is suggesting students sit exams with the risk of infection hanging over them.

The letter is in the ex post

Barry goes on to say a lot that really just adds up to constructing a system ex post. This is a serious issue. Unless every students signed away their rights (assuming such was even legally permissible), they could return and say that even after everyones’ consideration, the resultant grade still does not reflect what they would have achieved had they been allowed to sit the LC exam. Put simply, I cannot see any way in which the LC exams can be cancelled at all, in the absence of cast-iron guarantees, without producing negative side-effects; I think they can be merely postponed (lot’s of interesting avenues to go down there but that’s for another time).

A key strength of our current system, is that it is fair on everyone, it is anonymous, there is an appeals process, college places are allocated anonymously and there is no favouritism shown to anyone. Students know they have a shot, they can make it their best shot and even if they didn’t get what they want, they know it is was fair as fair can be (in our current socio-economic context) because everyone was treated the same.

Some have suggested that students could get a predicted grade *and* if they are not happy then sit the exam. But this, logically, invalidates arguments about mental health and infection above. If mental health and/or risk of inflection are reasons why the exams cannot be held, then it is inconsistent to argue that holding the exams could be an option – unless there are other, unspoken, reasons why some people want predicted grades. In addition to this, why ask schools to go through the enormous effort of generating such grades when the exam is still an option. If the exam is still an option, then it is an option. At this point, I should point out that I do not think exams in August is an option. Hold yer horses!

Barry attempts to address potential conflicts of interest saying that “staff will recuse themselves and will be appropriately replaced”. He doesn’t explain what ‘appropriately replaced’ means. But he misses the point teachers have been saying for years: Teachers cannot be advocates and at the same time judges of our students. All of us would have to recuse ourselves. To expect us to be judges of our students would be is a breach of natural justice: You cannot be a judge in your own cause. However, I would have no problem with principals, on their own, and without recourse to teachers, deciding on such grades – other than that they are not fair to all students. That ball would be in the principals’ court but I am sure they would not like such responsibility to rest solely on their shoulders either. I wouldn’t wish it on anyone.

Barry says that data will moderated by the SEC and anomalies investigated but he doesn’t say of how this will be done (I accept there were space limitations) and how it would get over the ‘but for’ test, namely, but for the fact the LC was cancelled a student would have gotten a better grade and a higher CAO offer. This could never be determined without holding the exams. So the exam would have to be held to remove possible court actions based on the ‘but for‘ test on foot of predicted grades. So there is a conflict of reason between the choices of predicted grades and then the exam or predicted grades but cancelling the exams. For the arguments about health to stand up, predicted grades are, in this dichotomy, the only option. But these are unfair and really cannot be made fair.

Another solution would need to be found to satisfy those who want to go to on to third level but who cannot sit the exam. In this case, the choice for students is whether they are prepared to take the risk of success (here, here and here) at college based on something (predicted grades) other than sitting the Leaving Certificate exams while knowing there is a possibility in the future they may come to rely on possessing a set of LC examination results. An as example, how would a 2020 predicted-grade based application to anything stand up against an 2021 set of actual Leaving Certificate results.

Barry says It will be “forbidden” for schools to discuss expected grades with student and parents. He doesn’t say how this will be done. Will there be penalties? Will there be SEC be out looking out for school principals’ nods and winks to every pushy parent they have ever had to deal with? This is not to poke fun at the idea. It is merely to highlight that this suggestion is impractical and far removed from reality of Johnny’s parents angry that he didn’t get med school because the local principal gave him a H2 in something.

Appealing awful

Barry presents two avenues of appeal. One is that students could request a “desk review of documentary evidence and moderation mechanisms” which I presume means they could look at the results the school puts forward and they would be given the opportunity to challenge them. As for the moderation mechanism, this could not be part of a review because it would have to be the same for every student in every school – there would be nothing to review.

Again, this highlights the fundamental problem: If a student is unhappy with their predicted grade, they can always resort to the mantra – their school tests were never intended to be taken as the basis for determining their college entry. Any use of so-called predicted grades would be a retrospective act. All the head-nodding between schools and the SEC could never resolve the but for test.

In this scenario, the only ‘permissible’ predicted grades which would keep everyone happy and not create more problems than it solves is for every student to get predicted grades that give them 625 points. As a thought experiment, ask a student what points they were hoping to get. Then ask them what number of points would they like!

Barry’s second appeal suggestion that students would sit a written paper. However, this is not an appeal of a predicted grade but a punishment for not accepting a grade that a student might have been correct in rejecting. In addition, it brings us back full circle: it invalidates the health arguments. It begs the question, why not just sit the LC in August. It would be hard in this context not to conclude that some seek cancellation merely for the convenience of cancellation. I don’t necessarily have a problem with that suggestion either but I think people should be open about it.

I appreciate there are difficulties in terms of access to technology while schools are closed – and these issues are across all socio-economic groups; I appreciate that generally students are under stress and yes, there are wellbeing issues. But I don’t think that the mere postponement adds significantly to this; rather, it is each individual’s particular circumstances, primarily familial, which adds to the stress. I think this would be there anyway. For sure there are exceptions and the government should do something here. But there are also some who tweeted and asked if the government would refund the holiday they had planned in July! Still other students are just getting on with it. So the issues and concerns are not homogenous.

Still, there should be some effort to ameliorate as far as reasonably possible health and wellbeing issues.

So then Barry suggests that results from such an appeal-based test would “be binding, irrespective of whether it’s higher or lower than the expected grade”. Well, no. Students are entitled to an appeal of the mark on such tests and a court challenge would affirm this. It would not be in any student’s interest to consent away their right to appeal, a right that is well-established by practice over the last 10 or so years. And if the government attempted to pass a law on this, it would be retrospective to LC students existing rights.

Barry argues that teachers currently predict grades for students applying outside Ireland. The UK is the only country to use predicted grades (and ex ante at that). There are relatively few students that apply, these predictions generally optimistic but more significantly,  offers on foot of these are usually conditional on LC exam performance and in either event are fraught with issues. Here, here, here and here. And here. And here. And….you get the picture.

Barry goes on to say that third level lecturers grade their students for terminal exams. Notwothstanding expertise issues in finding unbiased college-level examiners, these are all ex ante, not ex post assessments.

Appeals to adopt retrospective predicted grades on the basis of the professionalism of teachers is a strawman argument. However, it also accidentally affirms the role of teacher-as-advocate who, to quote Barry “always place their students first”, a role that is not logically consistent with the teacher-as-judge.

Barry’s argument reaches a low point when he says “at a time when fellow citizens don gowns and face masks in life and death situations, asking teachers to exercise professional judgement honourably and impartially should not be viewed as onerous”.

It is not onerousness that is the issue. It is that it is wrong. Barry’s point offers nothing except shame on teachers who don’t buy into a deeply flawed idea.  Yes we are professional – why would he question this – but we are also advocates in a non-standardised education system. Asking us to judge our students in a standardised examination system is like asking a nurse to make the life and death decision usually made by consultants. It is not what we do. Furthermore, we are not intubating students nor delivering medical care to save them from a virus. We are standing up for them. No teacher should made to feel shame for saying that they speak up for their students in an ad hoc system which has the potential to deprive them of a fair chance to access a potentially life-changing place at higher level because in one school they have higher expectations and better inputs than in another. If fair’s fair, then unfair is unfair. We ought not to do anything that we know in advance might compound existing disadvantage.

And I haven’t even mentioned the GDPR nightmare efforts to use predicted grades could produce. What’s on your school record.

But time is pressing.

Tune in for Part 3 tomorrow: A workable solution