AI & Academia: ChatGPT at the College

Artificial intelligence is transforming countless industries across the globe — higher education is no exception. Continue reading to explore how professors at the College of William & Mary are adapting to a world with ChatGPT.

MONICA BAGNOLI // FLAT HAT MAGAZINE

Before we begin, let me make one thing clear: I am not an AI language model. I am a human writer, and I’m here to explore the complex issues surrounding the use of those models in higher education. From academic dishonesty to privacy concerns, the implications of AI in education are far-reaching and require careful consideration.

In case you haven’t guessed, I did not write that paragraph. It was my seventh attempt at getting ChatGPT to write an introduction for an article about AI in higher education. I had to do a fair bit of hand-holding along the way — the chatbot’s first response felt more like something from the mind of a middle schooler, dutifully assembling the blandest, most formulaic topic sentence possible for their assigned essay about a generic dystopian novel.

Round two went a little better. “Try again,” I instructed in the input box. I tapped enter, and the AI, whose dark-mode UI has become ubiquitous in the screenshots shared since its public debut on Nov. 30, 2022, seemed to pause and think for a moment before making its new attempt. The cursor didn’t move as fast this time — it paused for a second every few words as if it was furrowing its brow in digital concentration (humans, by the way, are very bad about anthropomorphizing technology — more on that later). About 15 seconds later, it produced an introduction that I thought graduated from “middle school essay” to “junior year high school English paper, but it’s due in an hour and you have to crank something out in the library.” It still felt generic and a bit mechanical, so I kept tweaking the instructions in pursuit of something that could be considered original.

Writerly difficulties aside, it’s immediately obvious to a first-time user of ChatGPT — a public-facing interface for GPT-3.5, the newest publicly available model from a class of generative AIs called Large Language Models (LLMs) — that this technology holds enormous disruptive potential. The chatbot generates large volumes of mostly coherent language in an instant. The task of collecting information and synthesizing something from it, a central part of the life of a college student, is now something that a computer can do in a fraction of the time. In the short term, educators are forced to account for the presence of technology when issuing assignments. In the long term, the future of education, creativity, and human expression is thrown into question.

The College of William and Mary’s Studio for Teaching and Learning Innovation provides pedagogical advice for professors, especially where technology is concerned. Adam Barger, STLI’s Interim Director, said that their office became aware of ChatGPT and its probable implications in December, shortly after it went public.

“The President and the Dean of Arts and Sciences asked us to do a couple of things at the start of spring semester,” Barger said. “The first was to develop some quick guidelines to send out as the semester started. And we did that in conjunction with a working group that the Dean established called the ‘AI Writing Tools Working Group.’ The heart of the recommendation at this point is just for the faculty members to be very clear in their syllabus and in their teaching as to what role generative AI should or could have in the class. So, for example, it might be completely accepted as a way to learn new things, to see drafts from AI, to explore its accuracy, and use as a writing tool, or it might be completely forbidden and it would be an honor code violation if it were used on an assignment. And then anything in between. That kind of spectrum of potential use needs to be in the syllabus and very clear so students understand the reason for it.”

While Barger emphasized that ChatGPT and its successor technologies will have a place in higher education, he conceded that they’re more of a threat to educational environments than an asset right now.

“I would say I’m probably more on the side of ‘This could really cause some problems.’ And it’s, at the very least, a technological disruption that changes the status quo and presents opportunity for both good and bad.”

I also spoke with Philosophy professor Matt Haug, who specializes in the philosophy of the mind, which encompasses the nature of human and artificial intelligence. 

“I am going to require students now to tell me how they’re using ChatGPT if they did use it,” Haug said.

In many cases, Barger said he expects professors to respond to the proliferation of AIs by leaning back into low-tech, “AI-proof” assignments, in contrast to the existing post-COVID trend towards remote assessments and asynchronous environments.

“I think there’ll be some kind of bounce back, some pressure back towards in-class assessments and maybe even more traditional blue book written essays and things,” he said. “I don’t know if that’s the best path. I think that could be one of several ways, because there are, of course, legitimate reasons to have essay writing skills, depending on your major and the class. It can be a great way to accomplish your learning goals by having an in-class writing assessment. But I think the large majority of classes could really get more creative with their assessment techniques.”

The best way learning can respond to the disruption of ChatGPT, Barger says, is to place more emphasis on the incremental thought process that goes into formulating and expressing an idea rather than the outcome itself.

“The term we use for that is scaffolding. And it’s the idea that you break these big assignments into smaller assignments that helps students see the value in each step of the writing process. So, for example, instead of just saying, ‘Hey, get me a five-page paper by the end of the month,’ maybe we’ll start with a discussion about your interests, and then maybe there’s an elevator pitch that you make to define a topic. Then maybe there’s an outline, and then there’s some research, and there’s a draft. And so by doing these pieces along the way, pieces that are a bit harder to use generative AI for, [it] allows the student to really focus on the learning process as opposed to just a product. The product is important, but so is the process. And so I believe by focusing on process, we can navigate this world much more effectively.”

Haug echoed that the proliferation of AI tools, whether they’re used or not, reinforces the importance of process, as opposed to outcome, in an educational environment.

“I think one of the reasons why, pedagogically, we don’t want students to ask ChatGPT to write their papers is that then, they’re not thinking. In Philosophy, most paper topics that you’re writing about have been written about by thousands of other people; it’s not like you’re writing on some new topic or creating something new. You’re not creating novel ideas, right? But you’re thinking really hard about an argument and how to phrase it clearly and in a way that’s understandable and logically valid, and that helps you create new work down the line.”

“I think it’s very tempting to think about ChatGPT as this kind of purely instrumental output like the end product is the sole goal of an assignment,” Haug added. “Especially in the school context, but even when you go out to the job market, that all that matters is that it produces this article or this short story. And I think there are lots of reasons why humans write and engage in the writing process that are more process-driven. And there are ways that you can use ChatGPT and still have a lot of hard, critical thinking and struggle with the ideas, and I think those kinds of uses are the most exciting and potentially helpful.”

Haug explained that the disruption of generative AI and the scrutiny it brings could actually improve the engagement that professors have with their students.

“I think [AI] raises interesting questions in the classroom, like, ‘Why are you assigning this particular paper?’ You know, if this assignment is one that’s easily reproducible by ChatGPT, I don’t think that necessarily makes it a bad assignment,” Haug said. “But I think it forces professors to convey the reasons why you’re doing a particular assignment to their students, like much more clearly than we have in the past. I think that’s helpful and interesting.”

It’s important to note that ChatGPT does have obvious limitations, some of which are innate to all similarly designed LLMs. It doesn’t take much work to expose its habit of “hallucinating,” making unfounded and false assertions in an attempt to give an answer. Like humans, AIs tend to dislike admitting when they don’t know something.

Additionally, while it can offer superficial imitations of a huge variety of writing styles, ChatGPT’s own writing leaves a lot to be desired. Despite my persistent re-prompting and the fact that the response improved considerably from the original output, I still have a whole host of issues with the opening paragraph that ChatGPT gave me. As a proofreader, seeing an introduction that contains tedious phrases like “complex issues” when the whole rich palette of the English language is right there, waiting to be used, makes me want to scream and throw my laptop out the window. I have a similarly visceral, negative reaction to the wanton use of words like “explore” and “implications.”

My issues with its efforts to write my lead-in point to a broader problem I’ve noticed with the responses that ChatGPT generates: every expression that it cranks out tends to have a depressingly banal quality. This isn’t surprising given the AI’s vast training datasets — one can easily imagine somebody throwing the collective textual emissions of all of humanity into a blender in some ultra-high-rent office space in San Francisco, then pouring out the resulting neutral beige goop into the verbal soft-serve machine that is the ChatGPT interface we all know and love.

That’s ultimately some comfort for those who think the written word still holds potential as a medium for individual expression. Given the technology’s reliance on hoovering up all the writing they can possibly find, it seems like an inherent property of these models that they will always reflect the most vague and overused parts of the contemporary English lexicon.

Haug said the stylistic shortcomings of ChatGPT come partially from the fact that it reduces human expression to its least common denominator, and the average piece of writing on the internet is not particularly original or interesting.

“I mean, think of all the clichéd writing that just your average reporter produces,” he said.

Language AIs are bound to improve dramatically on what ChatGPT currently offers. GPT-4, which recently launched in a limited-access beta test version, is the successor to ChatGPT’s model and features a larger neural network with a much better and more truthful writer than ChatGPT. In testing, GPT-4 was able to pass a bar exam in the top 10% of scorers, whereas GPT-3.5 flunked it. Google also recently started public testing of Bard, a comparable AI tool that it hopes will act as something between a chatbot and a search engine, like a kind of internet librarian. Notably, Google has said it plans to integrate Bard with a secondary logic model that should enable it to undertake computational tasks more effectively than an exclusively language-oriented model like GPT. The pace of advancement in the field is dizzying; even as they lay off tens of thousands of workers in other sectors, large tech companies are still enthusiastically shoveling money into deep learning R&D work.

That said, there’s still a lot of ground to cover before we have a fully capable artificial general intelligence, and it’s not a given that the current exponential pace of development will continue forever. Jie Ren, a Computer Science professor who conducts research relating to deep learning models, said she isn’t concerned about the effects of AI tools on Computer Science instruction yet. Although generating simple Python scripts is billed as one of GPT’s strong suits, it still tends to make mistakes when it comes to the logical reasoning necessary to build a functioning computer program.

“I actually tried to feed my assignments and homework to GPT-4 to see if it can get the answer correctly or not,” Ren said. “And I think it still cannot answer all the questions we are trying to design for the students. This is basically because this GPT model itself is trying to predict what the next word will be. It lacks a lot of inner communication. So basically, this model cannot solve the problem with logical steps efficiently.”

She pointed to an example cited in OpenAI’s research publication for GPT-4 — “Just to a simple question, like how many prime numbers there are between 100 and 250, they cannot give you the correct answer. This is because they cannot think as people do, finding the least of the prime numbers and then counting them and giving back the answer. They are just trying to solve it in this one-step way.”

Still, she said the current rate of improvement makes predicting the long-term effects of AIs difficult.

“I’m not too worried about students using GPT-4 for their homework right now, but who knows what will happen five months later? Maybe the GPT model is getting smarter, … but I think at that time we’ll change how our assignments and exams look.”

There are still unanswered questions about the place of human labor in the age of AI, and they may persist as open questions for the foreseeable future. As language models become ever more sophisticated, if you’re the editor of a literary magazine and you’re handed a stack of 100 short stories and 99 of them were written by AI, how confident can you be that the one written by a human will stand out? Up to this point in history, any piece of writing could be counted upon to have a human at the other end who sat down with the express purpose of communicating something to you, the reader. There was an intention behind it. The author could have died a century or even a millennium ago, but a reader still has a kind of intellectual telepathy with them through the ideas they expressed. Now, as AIs take on the roles of creative assistants and ghostwriters, that certainty is no longer there.

When I asked Dr. Haug about the topic, he said evaluating the precise degree of “intention” behind an AI-generated work is tricky because of the way humans are involved in the training process.

“There is no intention in the sense in which humans have intention,” he said. “Well, there’s a goal that it has, which is to predict the next word. And it uses a very, very huge data set and a complicated algorithm to do that. And then what’s really interesting is that it doesn’t have any intentions — it didn’t want to produce a good response in a sense. But it’s also been trained by reinforcement learning with humans saying ‘Is this a good response or not?’”

When you get a response from an AI, you still engage with other humans in some way. For the general question of whether an AI’s response should be seen as belonging to the human authors of its training data for the purposes of copyright or academic plagiarism, the jury is still out. 

“There’s the kind of claim that even if their work was just this one little minuscule part of this huge collection that was used to produce this novel creative thing, you should get some credit for that as well,” Haug said. “And I’m sympathetic to those kinds of arguments.”

Ultimately, Haug says, both in the classroom and the wider world, the questions posed by the new generation of AIs defy easy, unequivocal answers, and we will likely have to wrestle with them over the coming years and decades.

“I think the messy middle is where we probably want to be,” Haug says. “Trying to figure out on any particular question what the actual nuanced position is is a hard thing.”

Meanwhile, Barger added that the conversation about AI within the College has only just started.

“We’re trying to see you all as pedagogical partners so that we can get your side of the story as well,” he said. “That’s the real goal.”