As you suspected, emergency remote instruction during the pandemic...wasn't great.

I hope none of us have to relive the COVID-19 pandemic. Among many difficult challenges, emergency remote instruction (i.e., around March 2020, when suddenly most educators had to become online instructors, overnight, with little to no preparation or resources for doing so) was rough for everybody, but particularly for those students lacking resources and other supports. So, despite being tempting, we really shouldn't use emergency remote instruction as a "test" of the effectiveness of online learning, in general. A true test would require well-trained and well-resourced educators and students, working in environments designed and optimized for online instruction.

But, could I interest you in a randomized, controlled trial of online versus in-person learning during the pandemic? Such a study would be able to control for the general yuckiness of the world at that point, in addition to a wide variety of other factors, while also estimating the effect of learning online v. in-person, for those students, at that time. Welp, that's exactly what Kofoed et al. (2021; apologies if you've seen this study already, and even so, I think it's worth revisiting) did. They were able to randomly assign West Point cadets (they pretty much had to agree to randomization because...they're cadets at West Point) to parallel online and in-person introductory economics courses, and then measure the results.

The findings were eye-opening. Even after controlling for lots of plausible alternative hypotheses, the authors found that online students performed nearly a quarter of a standard deviation worse than cadets who were in-person. The effects were more pronounced for academically at-risk students. In follow-up surveys, the online students reported struggling with concentration and feeling disconnected from the instructor and their peers. And these cadets were getting significant support (e.g., housing at West Point, meals, connections with peers outside of this class, etc.). Imagine how tough emergency remote instruction for people without such supports.

Now, this isn't necessarily an indictment of all online instruction. Online education when designed well and taught with effective online pedagogies, and conducted under normal instead of pandemic conditions, has great promise. But Kofoed et al.'s study illustrates just how tough emergency remote instruction was. I certainly hope we never have something like the pandemic ever again, but in case we do, we had better start preparing educators and students with the resources and tools they need to learn more effectively online.