Teaching Writing in the Age of LLMs

LLMs are increasingly being used for various kinds of writing, and their use will continue to expand, but this does not undermine the need for students to learn to write, any more than the existence of calculators makes learning arithmetic redundant. Using such tools before an authentic ability has been developed has numerous negative implications, including being unable to properly assess the quality of the machine's output.

This document explains how to use Googledocs and a Chrome browser plugin to monitor the writing process of students, not just to inhibit the use of ChatGPT or other LLMs, but also to identify elements of an authentic writing and revision process and to help students understand the importance of writing authentically as an aid to clearer thinking.

A student instruction sheet tells students how to set up their Googledoc for shared access. Although the fourth step is specific to the Canvas Course Management System, it could be adapted for use with any CMS, or none, so long as the document links are shared appropriately.

I use the Process Feedback plugin for the Chrome browser. It analyzes all student’s typing and provide summary statistics and graphs. It is more or less a single click installation in Chrome. Access to the writing details is provided through an extra button at the top of each Googledoc.

Process Feedback can be used to detect block pasting of text, retyping from other sources (top to bottom ballistic production with mostly only slight backtracks for typo corrections or synonym changes), and weak revision practices (not a violation of any honor code, just a sign that they aren’t really approaching writing in the right way). You can also see how much time was spent working in the document. Sharing this tool with the students can be helpful because it encourages them to implement good writing practices.

A staged set of assignments building up to the full paper helps students to develop their thinking and learn how to revise in light of feedback. Here is one example of a template that I have used to structure the process. Whether or not you use a template, it is important to tell students that they are to do all their writing in the Googledoc, no exceptions, as evidence of pasting large chunks of text will be considered a failure to complete the assignment.

All of this should be supplemented with an explicit policy about the use of AI. Here is the policy from a recent syllabus of mine:

You may not submit any AI-generated text as your own writing. Some exercises or your paper topic may require you to interact with an AI system and report its responses for the purposes of analysis, but you must clearly attribute any text generated in this way. You may use AI-enhanced tools for basic grammar and wording suggestions without documentation, but if you use generative AI to rewrite your text you must document this by including a transcript of the interaction with the AI. If you are unsure about your usage, ask your instructor!

FAQs

1. Couldn't this all be faked? Yes, but figuring out how to fake an authentic pattern of writing and revising would be more work than just doing the assignment properly. Most students sincerely want to learn to write better but fall back on LLMs when they are in a rush or under other kinds of stress. Staging the assignment as suggested in the template reduces the likelihood of last-minute LLM use.
2. What are the hallmarks of LLM use? Pasting large amounts of text is one indicator, but most students won't do that given that they have been warned not to do so. More common is for students to retype from the LLM output. The hallmarks of this are (i) very short working time, e.g., an entire essay produced in one hour, (ii) linear (ballistic) top-to-bottom production of the essay, with (iii) minimal editing, typically only typo fixes, or perhaps some minor thesaurus based wording changes.
3. What are the signs of weak writing skills or hurried writing? The chief hallmark is that editing of false starts is mostly confined to local (sentence- or paragraph-level) contexts and just one or two editing sessions. Strong writers show signs of global editing over editing sessions spanning multiple days; global editing means that writers will frequently go back several paragraphs or all the way to the beginning of the essay and rewrite the earlier material in light of the ideas they are expressing at the expanding forefront of the essay.
4. Can students edit offline and does that affect the plugin? If a student has set up a document for offline editing per the instructions in the student instruction sheet, then next time they connect their computer to the internet, everything will be synced, including the metadata about the typing used by the plugin.
5. Where can I find other information on using Process Feedback to detect AI use? Yes, see the helpful guide at https://docs.processfeedback.org/identify-ai-usage.html

CC BY-NC 4.0 by Colin Allen <prof.colin.allen@gmail.com> -- You have permission to link to this document or reproduce or adapt it for non-commercial purposes only. Please give appropriate credit, provide a link to the license, and indicate if changes were made.