Antibody LFA Evaluation

In March 2020, the Mass General and Brigham Center for COVID Innovation (MGBCCI) established the Diagnostics Pillar and several associated Working Groups. The goal of these Working Groups was to identify and evaluate the variety of devices and platforms proposed to diagnose or characterize SARS-CoV-2 infection and response to COVID-19. The Direct to Consumer Working Group focused on technologies meant for non-laboratory use, i.e. marketed directly to consumers and used without involving healthcare providers. These included serological lateral flow assays (LFA), which detect antibodies (IgG and IgM) against SARS-CoV-2, consistent with prior exposure to the virus.

The MGBCCI Diagnostics Accelerator working group embarked on a study evaluating approximately 20 of these LFAs. When tested, these assays were in different stages of approval by the Food and Drug Administration (FDA), which has been granting Emergency Use Authorization (EUA) for those that pass strict FDA evaluation criteria (FDA EUA status of these assays can be found here).

We are reporting MGBCCI data based on standardized laboratory evaluations performed on biobanked plasma samples from confirmed SARS-CoV2-infected patients (based on gold-standard molecular tests) and SARS-CoV2-negative samples collected before the pandemic. The testing evaluated the performance of LFAs in a high complexity laboratory setting and can provide no information on consumer use or suitability as a direct to consumer product.  The testing of the LFAs is intended for MGBCCI evaluation purposes only.  The MGBCCI, associated individuals or institutions, do not endorse any of these products.

One of the goals of the MGB CCI Diagnostics Accelerator is to evaluate COVID-19 diagnostic technologies that may be useful in point-of-care environments.  Here we report the evaluation of 20 lateral flow immunoassays (LFAs) manufactured to detect antibodies specific to SARS-CoV-2 antigens in the blood of individuals exposed to the SARS-CoV-2 virus.

Samples Analysed
Readability & Invalids
Summary Performance
Individual-Level Interpretation
Population-Level Interpretation

Samples Analyzed

We obtained blood plasma from a balanced cohort of 112 individuals for this study, comprising 56 individuals confirmed by PCR as SARS-CoV-2 positive and 56 individuals whose blood was collected in 2018/19 prior to the pandemic. Ten (10) of the 56 pre-pandemic samples were obtained from HIV-positive individuals. To maximise consistency and to aid in the interpretation of the results, the same 112 plasma samples were run on all LFAs.

Readability and Invalids

The table below shows the percent of samples (separately for COVID-negative and COVID-positive individuals) with invalid test results as well as the percent of valid readings that were called differently by multiple laboratory staff.  In general, it is desirable to have a high percentage of valid test results as well as a high degree of agreement in whether or not a band is present between different people reading the test.

Summary Performance

In the table below we report, for each LFA, the sensitivity (% of COVID-exposed individuals for whom antibodies were successfully detected) and specificity (% of COVID-negative individuals for whom no antibodies were detected).  We have reported the performance statistics such that discordant operator readings (i.e. two human operators disagree on the presence/absence of a band on the LFA) as a COVID-negative test result.  This approach sacrifices some sensitivity for greater specificity; by reducing the number of false-positive readings tests are more likely to have better performance in the early stages of a disease outbreak where most individuals have not been exposed (see ‘Population-level Interpretation’ below).”

Individual-level Interpretation

The grid below shows the performance of each LFA in each of the 112 individuals.  Individuals are split left to right into groups representing the 46 COVID-negative/HIV-negative (COVID-), the 10 COVID-negative/HIV-positive (COVID-HIV+), and the 56 COVID-positive/HIV-negative (COVID+).  Colours represent scores computed for IgG (top panel) and IgM (lower panel); where dark red means all operators agreed the test was positive (score=+1), dark blue means all operators agreed the test was negative (score=-1), and intermediate colours (pink and light blue) representing varying degrees of operator disagreement.  Grey represents results reported invalid (i.e. there was a missing positive-control band on the LFA).  We have coded score=0 (complete operator disagreement) as light blue and this counts as a COVID-negative result for the purposes of the sensitivity/specificity values shown in the table above; this approach reduces the number of false-positive readings and is beneficial in the early stages of a disease outbreak where most individuals have not been exposed (see ‘Population-level Interpretation’ below).

Indeterminate: The two operators did not agree, Invalid: The control band was absent, Positive: Two operators read a band, Negative: Two operators did not read a band.

Population-level Interpretation

Because the fraction of people in the general population who have been exposed to COVID-19 (which we call the prevalence) is still low, even tests with high sensitivity and specificity may end up mis-classifying a large number of people.  To address this we created an interactive web app that shows the effect of changing the prevalence (as well as the test performance) on the probability that given a positive test you actually have SARS-CoV-2 antibodies.  We have included data from other LFA studies as well as state and county-level prevalence values for a geographically-refined interpretation of these LFA results. 

https://covid.omics.kitchen