Testing my GPT to grade essays: an example

Russ Roberts' piece on the media and Israel

I fed this essay by Russ Roberts to the grader GPT that I created. It came back with

The op-ed essay you've shared offers an extensive exploration of media coverage, especially in relation to the Israeli-Palestinian conflict. The essay scrutinizes the biases of different media outlets and the impact of these biases on public perception. Here's the evaluation:

1. Clarity of Argument: The essay is clear in its proposition that media outlets are biased, influenced by the competitive nature of digital journalism and the ideological leanings of their audiences. The examples provided (e.g., coverage by The New York Times, Reuters, etc.) effectively illustrate this point. The discussion of Arnold Kling's "Three Languages of Politics" and the lens through which liberals and conservatives view conflicts like the Israeli-Palestinian one enriches the argument.

2. Fairness of Argument: The essay attempts to present a balanced view by acknowledging biases on both sides of the political spectrum. However, it leans more towards criticizing left-leaning media and their portrayal of the Israeli-Palestinian conflict. The essay could benefit from more examples or analysis of right-leaning media biases to maintain a more neutral stance.

3. Use of Evidence: The essay uses specific examples from prominent media outlets to support its claims. However, some statements, particularly those regarding the portrayal of specific events, lack detailed references or verification, which could strengthen the argument's credibility.

4. Bias and Representation: The essay suggests a personal bias towards the Israeli perspective, evident in certain sections where the author expresses personal views and experiences. While this personal perspective adds depth, it also introduces a subjective element that could be perceived as undermining the essay's neutrality.

5. Logical Fallacies: There are no blatant logical fallacies; the arguments are mostly structured logically. The essay does, however, engage in some generalizations about media consumers and journalists, which could be interpreted as an oversimplification.

6. Respectful Engagement: The essay maintains a respectful tone throughout and avoids inflammatory language. It encourages understanding different perspectives, which is commendable in a discussion that often becomes heated and polarized.

Overall, the essay is informative and thought-provoking, with a strong focus on media bias and its implications. However, to enhance its objectivity and neutrality, a more balanced representation of biases across the political spectrum would be beneficial.

Grade: B+

This grade reflects the essay's strengths in clarity and engagement with the topic, while also acknowledging areas where balance and neutrality could be improved.

I found this feedback mildly annoying. I did not establish “balance and neutrality” as criteria for grading. I did emphasize treating the other side with respect and due consideration of its strong points. I probably need to modify my instructions in order to clarify this point.

I myself thought that, if anything, Russ bent over backwards too far to give consideration to the anti-Israel viewpoint. Many people suspect that ChatGPT has been trained to be notably leftist in its views. I wonder if it will only play the “balance and neutrality” card against conservatives. If I do see signs of left-wing bias, I will have to modify instructions to counteract that.

What I ask of you is to suggest essays to grade. You can put links in the comments. If the essay is behind a paywall (e.g. WaPo or NYT or Atlantic or Commentary), in addition to leaving a comment please share the entire essay with me at arnoldsk@us.net. Thank you.