Investigating AI Accuracy in the First-Year Composition Course

Type of Presentation

Individual paper/presentation

Conference Strand

Ethics in Information

Target Audience

Higher Education

Second Target Audience

K-12

Relevance

Getting students to use generative AI in creative ways can show them the benefits and drawbacks of relying on such technology as sources of credible information. If instructors want to students learn how to critically evaluate AI tools as potential sources of information, those students should test out the tools for themselves and evaluate the outputs.

Proposal

As first-year composition instructors navigate the swell of changes brought about by generative AI tools, one area of concern is the use of these tools as sources of information. What many instructors already know about these tools, namely that they fabricate deceptively plausible information and sources, is not something students may fully understand until they are forced to spot it for themselves. In the fall of 2023, I began working to teach my own students how to do this within the framework of a multipart assignment that ultimately reveals the students' own perceptions about the accuracy of generative AI. The assignment parts are as follows:

Step 1: Students write a traditional essay which stakes out a position on a topic of their choosing and attempts to support the position with credible information from a source or sources.

Step 2: Students prompt generative AI to write a similar essay that takes the opposite or alternate position as the student's essay.

Step 3: Students analyze and critique the generative AI essay and write up their own evaluations of the computer-generated text, including assessments of the sources cited and specific information presented by the generative AI system.

From working closely with the students through this assignment process and from reading their analysis of generative AI, I learned much about their perceptions of generative AI and its reliability as pre-writing assistant, as content reviewer, as editor, and as a source of information.

This presentation will briefly survey the current reality of generative AI accuracy and compare that to what I have learned about my own students’ perceptions based on their experimentation with ChatGPT.

Presentation Description

This presentation is organized into three sections: 1. Overview of generative AI accuracy 2. A model for teaching students about generative AI accuracy (and ethical use) 3. Student perceptions about the accuracy of generative AI systems and implications

Keywords

Generative AI, research, first-year writing

Publication Type and Release Option

Presentation (Open Access)

Share

COinS
 
Apr 19th, 4:15 PM Apr 19th, 5:00 PM

Investigating AI Accuracy in the First-Year Composition Course

As first-year composition instructors navigate the swell of changes brought about by generative AI tools, one area of concern is the use of these tools as sources of information. What many instructors already know about these tools, namely that they fabricate deceptively plausible information and sources, is not something students may fully understand until they are forced to spot it for themselves. In the fall of 2023, I began working to teach my own students how to do this within the framework of a multipart assignment that ultimately reveals the students' own perceptions about the accuracy of generative AI. The assignment parts are as follows:

Step 1: Students write a traditional essay which stakes out a position on a topic of their choosing and attempts to support the position with credible information from a source or sources.

Step 2: Students prompt generative AI to write a similar essay that takes the opposite or alternate position as the student's essay.

Step 3: Students analyze and critique the generative AI essay and write up their own evaluations of the computer-generated text, including assessments of the sources cited and specific information presented by the generative AI system.

From working closely with the students through this assignment process and from reading their analysis of generative AI, I learned much about their perceptions of generative AI and its reliability as pre-writing assistant, as content reviewer, as editor, and as a source of information.

This presentation will briefly survey the current reality of generative AI accuracy and compare that to what I have learned about my own students’ perceptions based on their experimentation with ChatGPT.