AI writes passable college papers

A post by Eduref, a company that deals in information about postsecondary education, has gotten some attention for doing a Turing Test for course papers produced by the AI language model, GPT-3.

We hired a panel of professors to create a writing prompt, gave it to a group of recent grads and undergraduate-level writers, and fed it to GPT-3, and had the panel grade the anonymous submissions and complete a follow up survey for thoughts about the writers. AI may not be at world-dominance level yet, but can the latest artificial intelligence get straight A’s in college?

As the saying goes, “C’s get degrees.” Straight A’s in college, however, are far from common, and with AI being far from perfect, GPT-3 performed in line with our freelance writers. While human writers earned a B and D on their research methods paper on COVID-19 vaccine efficacy, GPT-3 earned a solid C. Performing a bit better in U.S. History, humans received a B and C+ on their American exceptionalism paper, while GPT-3 landed directly in the middle with a B-. Even when it came to writing a policy memo for a law class, GPT-3 passed the assignment with a B-, with only one of three students earning a higher grade.

(More coverage at ZDNet and Inside Higher Education.)

Last year, I wrote something for The Conversation, based on my experience of using GPT-2, an earlier version of the same language model, to produce papers for an anthropology course. At the time, GPT-2 was close but not quite close enough.

I concluded:

While computer writing might never be as original, provocative, or insightful as the work of a skilled human, it will quickly become good enough for such writing jobs, and AIs won’t need health insurance or holidays. 
If we teach students to write things a computer can, then we’re training them for jobs a computer can do, for cheaper. 
Educators need to think creatively about the skills we give our students. In this context, we can treat AI as an enemy, or we can embrace it as a partner that helps us learn more, work smarter, and faster.

From all accounts, GPT-3 seems much more capable as is than GPT-2 was. While GPT-3 is not widely available, it won’t be long before it or something like it is. This means we need to rethink what writing assignments are, and what we want them to do.

John Warner, in Inside Higher Ed, suggests a change to how we approach grading is sorely needed:


In this case the problem is in our well-trodden patterns of how we assess student work in the context of school. [GPT-3’s] response is grammatical, it demonstrates some familiarity with the course and it is not wrong in any significant way.

It is also devoid of any signs that a human being wrote it, which, unfortunately does not distinguish it from the kinds of writing students are often asked to do in school contexts, which is rather distressing to consider, but let’s put that aside for the moment.

When confronted with this kind of work, what if we did something differently?

What if we replaced that … sigh … B with a “not complete, try again”?

Because honestly, isn’t that a more appropriate grade than the polite pat on the head that the B signals in this case?

This seems like an opportunity to put something like Labour-Based Grading into wider use.

(Previously (1), (2) on GPT-3.)

Action Button reviews Tokimeki Memorial

Tim Rogers has a six-hour-long Youtube review of the 1990s dating simulation game Tokimeki Memorial.

Tokimeki Memorial is cyberpunk. Tokimeki Memorial is more cyberpunk than Cyberpunk 2077 can possibly ever be. It’s more cyberpunk than Snatcher. It’s more cyberpunk than Shadowrun. It’s more cyberpunk because it’s a genuine, existing, cyberpunk artifact.

Screenshot of the Action Button review of Tokimeki Memorial.

The review, which is more like a documentary miniseries on the game, is incredible for the attention it gives to game design and mechanics, the cultural context of video games in Japan and the US, and the psychology of love and objectification.

It’s incredibly smart and watchable. Academic work should hope to be so fun, accessible, and insightful.

His Doom review is also excellent.

Extreme Anthropology

During my exploration of anthropology in the Directory of Open Access Journals, I came across the Journal of Extreme Anthropology. Just the title entices, but the contents are quite interesting as well.

The journal is run by the Extreme Anthropology Research Network, a group centered in Scandinavia, but with members from all over, that explores the “notion of the ‘extreme’ within contemporary cultural, political and economic environments.”

Its recent special issue on “Security and Morality: Critical Anthropological Perspectives” consists of 11 interesting articles, but one, “Why do we need your research?: The Ethics of Studying Security and the Deilemmas of the Anthropologist-Expert” by Tessa Diphoorn and Erella Grassiani, caught my eye. The authors discuss the double-binds that they face as anthropologists doing research on Israeli and South African policing and private security.

The article focuses on the ethical and methodological challenges of studying “security”—something that is taken for granted as a public good. After all, as the authors point out, who would want to argue for the value of “insecurity”? Diphoorn and Grassiani talk about the misunderstanding they face from funders and other researchers about the nature of their work, the expectations that their informants have of them and which are problematic to meet, and the methodological choices they have to make in order to gain access to situations where some level of suspicion is the norm.

In the process, they ask anthropologists to complicate how we think about “engaged” and “public” anthropology. Diphoorn and Grassiani show that anthropologists face an array of difficult methodological and ethical judgements in dealing with those who are often seen as perpetrating violence for the benefit of the “public.”

Interesting papers from the DOAJ

The Directory of Open Access Journals makes it fairly easy to find articles on many subjects published in smaller journals around the world. My guess is that many of these papers go quite overlooked in North America-centric anthropology, and so I spent a little time digging through articles that had been published this past year in anthropology to see if anything caught my interest. A few did. I’m going to post short blurbs on each of them; they might catch your interest too.

The Ritual Aspects of Ukrainian Beekeeping.” in EtnoAntropologia, a journal of the Società Italiana de Antropologia Culturale. Written by Uliana Movna of the Ethnology Institute of the National Academy of Sciences of Ukraine.

Why I clicked: Bees. I had no idea that bees, their honey, and wax had such cultural importance in the Ukraine.

Excerpt: “As universal ritual symbols, the main products of the life of bees are honey and wax, which, over a long historical period, have “grown into” calendar and family rituals of Ukrainians. During the research, it was revealed the main ritual purpose of honey as a mediator between the bee and man, with other world and souls of the ancestors, the conductor and the amplifier of the processes of transition at the moment of passing by the human the corresponding stages of age and social hierarchy.”

Shit’s getting real in more than one language

Back in March, I wrote a short piece analyzing the COVID-driven toilet paper hoarding phenomenon from an anthropological perspective.

Since then, it’s become my most visited post, and it has resulted in some spin-offs.

Most recently, Gustavo Santaolalla has posted a Spanish translation of the post here.

Before that, Dorjpagma Batsaikhan wrote a Mongolian version.

Way back in April, the Voice of America’s Mandarin language service interviewed me for a segment on toilet paper hoarding, which is on Youtube.

And last, but not least, Henry Alford interviewed me for a story that ended up in the New Yorker.

In the past few months, it hasn’t been this post that gets the most traffic however. My brief tutorial on how to use a virtual avatar in Zoom is the daily number one.