Moderna lot 052D22A caught my eye

An epic tale of the struggle faced by data scientists when dealing with obscured and incomplete data sets in pdf format!?

So I made this graph the other day and wrote some words on my Twitter feed. It’s just a fun graph that shows the rate of serious adverse events per 100,000 doses shipped for the 21st earliest and the 21st latest COVID-19 injectable product batches. It’s kind of a stupid idea because there’s no way to draw serious conclusions from the graph (it was generated using VAERS lot data) because there’s no way to know how many of these shipped doses actually went into arms. Nevertheless, it’s totally worth discussing and fun to do! Here’s the original post.

Here’s what I wrote originally in 6 parts on Twitter.

1. I am playing in the data sandbox with the idea of lot variability - ie: some lots are more toxic than others (and thus resulting in higher reporting to #VAERS ) - and made the following plot. The graph shows the SAEs/100,000 doses shipped for the first 21 batches shipped, and the last 21 batches shipped.

2. Now, the thing that bugs me about this (and always has) is that no one has any idea how many of these shipped doses went into arms. I would bet that it would be very few in the case of the latter since people are smartening up. We also have no idea about other confounding factors like reporting bias.

3. I mean, there is a clear phenomenon here: it looks (with the exception of lot 052D22A) that the last 21 lots shipped out weren't really resulting in SAEs being reported to VAERS. Is this because they didn't cause SAEs or because fewer people were reporting SAEs?

4. It is worth nothing that many of these later lots are the bivalent shit and were injected as 4th doses and higher, so one might think that the cumulative 'toxicity' effect would result in higher SAE reporting, not lower.

5. I think that no one took them and that explains the lower rate but then again, what about 052D22A? This is small bivalent Moderna lot with only 24,900 doses, 183 reported AEs to VAERS of which 40 are SAEs. That's a 22% SAE rate as per total AEs. What do you think?

6. The lot numbers are a mess in VAERS - this is not a secret - and although I believe in this idea (because manufacturing issues, DNA, etc), I need hard evidence and I think it can't really be found using #VAERS. It's useful as a 'leaves rustling in the wind' tool, sure, but we need a better system!

Maria Gutschi has written an article that has included this finding and we had a long discussion about this today. During our chat, I noticed that this Moderna bivalent lot 052D22A was not listed in the FDA document she wrote about which was basically a list of Moderna bivalent lots that were passed off as “EUA-ok”. Lot 051D22A was on the list, but 052D22A was not. Here’s a screenshot of the document listing the “ok-ed” batches. CBER signed off on them as “suitable for use”. Ok. But what about 052D22A? Was it simply an oversight that it was left out?

Was it intentionally not checked? Was there a reason it was not checked? Let’s look at that graph again. You can see 051D22A and 052D22A right next to each other. (These batches were shipped are October 11, 2022 and October 13, 2022, respectively.) They are both bivalent Moderna products.

Obviously, one of these things does not belong and in data, this either means there’s a data point that is wrong (like a denominator) or that there’s something actually there, as in, a problem with the lot, in this case. Outliers certainly exist, but they tend to bug me. Which is weird, because I am one myself.

Interestingly, the batch sizes are very different as per the latest December 13, 2023 FOIA-requested Moderna data as per ICAN. Lot 051D22A is listed to have 565,850 shipped doses, while lot 052D22A is listed to have 24,900 shipped doses. That’s a big difference. Maria claims that smaller-sized batches can be indicative of problems occurring in manufacturing. This sounds plausible, but then the question becomes, if they knew there were manufacturing problems, why did they ship this shit?

The total number of adverse events reported to VAERS for lot 051D22A is 190 and for lot 052D22A, 183. Of these, the total numbers of serious adverse events is 38 (20%) and 40 (22%), respectively. So the total number of reports and the rates of SAEs as per total number of AEs are the same for these batches. Other interesting tidbits are that the percentages of reports filed immediately for each lot are 56% and 51%. This makes sense to me since 67% and 64% were on dose 4 or higher, respectively, and based on the hypothesis that there are cumulative damages associated with multiple doses, this is not surprising. The range of adverse events reported went from pneumothorax to seizure, with everything in between.

Here’s the weird thing though. We know that 183 people got the 052D22A products because they reported adverse events. We know that there is under-reporting in VAERS but let’s leave that out for now. We can assume with a modicum of confidence that fewer people received the 052D22A lot because there were simply fewer doses to begin with: ~23 times fewer doses. Since the SAE rates are the same as per total number of AEs, this means that these products are qualitatively the same in terms of potential harms - according to VAERS anyway. But what about when we normalize to dose number as I did in the graph above?

We know that at least 183 people got the 052D22A product. So based on the respective doses-shipped data for 051D22A and 052D22A, there were 161 SAEs/100,000 doses shipped, and only 7 SAEs/100,000 doses shipped. What could explain this difference? The only difference noted here is the difference in batch size, but this doesn’t explain it based on the batch size data we have. We can assume that the same under-reporting factor would apply to both, so any changes in rates according to batch size would simply be proportional because the numerator just scales up. So what it could it be?

I attribute this partially to reporting bias. But the question really remains, what proportion of each batch went into arms and do differences in these proportions explain this graph? Quick answer is no.

Let’s make an assumption: if the 051D22A and 052D22A lots were equally harmful/safe (which they appear to be from VAERS) then we can assume from the SAE rates that the same number of shots went into arms as per each lot. Let’s say 5,000. That would mean that based on their respective doses-shipped data (565,850 and 24,900), only 0.9% of the doses of 051D22A went into arms, while 20% of the doses of 052D22A went into arms. How likely do you think this is?

I think it’s highly unlikely, and that it is far more likely that the opposite is true because where there are more shots, there will be more administrations of shots. Agreed? So if we invert these rates and assign a 20% dose-into-arm to the 051D22A lot and a 0.9% dose-into-arm rate to the 052D22A lot, then this means that 113,170 shots went into arms for the former, and 224 for the latter. If we then apply these ‘actual dose-into-arm’ numbers as new and more accurate denominators, our graph looks like this.

Eww. That’s no good. It’s even more anomalous now.

Let’s make another assumption: if the 051D22A and 052D22A lots were not equally harmful/safe, then first of all, we would have to explain why the SAE rates from VAERS are actually the same. Perhaps the numerators (total SAEs) are different. I can’t think of a reason why there would be a such a discrepancy in under-reporting between these two lots, but it is possible. Perhaps these lots went to different states where there are different rates of AE reporting to VAERS? Let’s assume this is true, and that only SAEs from lot 051D22A are 24 times underreported, for some reason. This would mean that the reported number of serious adverse event for lot 052D22A would be 912 and would make our graph look like this.

This seems implausible as well though, and we’re not into ‘making data try to fit’. We pretty much created two problems instead of just having one. So let’s throw away these assumptions. They were kind of stupid anyway.

What seems far more likely is that there is something wonky about one of the 052D22A data points.

The most likely explanation for this weird yellow bar 052D22A anomaly is that the doses-shipped data is recorded wrong. The data recorded here (23-00086-AP-of-22-02076-from-2022-1014-Moderna-Data-Repulled-2023-0717.xlsx - December 13, 2023) indicates that 24,900 doses were shipped, as was used to produce the original graph.

Meanwhile…

I am writing this like an exciting novel. Did you notice?

OpenVAERS and I started sleuthing into the ICAN files and found something interesting. In the ICAN released data (Combined-Productions-Moderna-Upload-1.xlsx - July 7, 2023), there is no listing for lot 052D22A as part of any of the 13 tabs in the spreadsheet. However, in a separate pdf file (2023-00001-LIT-OS-Supplemental-Response-1.pdf - July 7, 2023), there is a total number of doses shipped for this lot recorded: 810,800, as shown below.

https://icandecide.org/article/exclusive-moderna-lot-and-dose-data-release/

Ok: Questions. Why a pdf? Why no detail as per doses shipped per state? Why no reference to where this number came from? Why the weirdy Terminal font? Why no reference to this dose-shipped size in the later (more recent/updated) files? That’s the weirdest part of this to me: this lot is nowhere to be found in the Combined-Productions-Moderna-Upload file which literally has about a gazillion data points referencing whack-loads of Moderna lots. Did Moderna simply just decide to omit lot shipping data to the degree that it was reduced 16-fold in the later updated file? Why would they do that? It’s odd, to say the least. We are contacting ICAN to try to find out more about this pdf file and if we can substantiate this lot number with more rigor. For now, there’s no way to know which is the correct number.

In any case, if 810,800 is the true number of doses shipped, then our graph looks way more sensible and our anomalous point is gone. This makes me happy.

But I am not happy. I need answers about where this pdf came from, why it looks so weird and why it lists a number that does not carry over to more recent and updated files.

And perhaps the most important question as posed way earlier in this epic novel is: why is this lot not listed in the “EUA-oked” lots as per the FDA document?

Too. Many. Questions.

A grand thank you to Pfizer and Moderna and all of the people responsible for confounding data, botching data, withholding data, releasing data in pdf format, and providing our hardest working lawyers like Aaron Siri with the generous and long-lasting task of FOIA requesting documents - bloody pdf files!? - that you wanted to hide until we were all dead.