New Study Reminds Us That ChatGPT Does Not Really *Understand* What You Want It To Do

Amid chatter about ChatGPT’s reportedly degrading performance, a new study found that recent open-sourced large language models (LLMs), including OpenAI’s GPT-3, perform “surprisingly better” on datasets released before the training data creation date than on datasets released after. 

The University of California, Santa Cruz paper by Changmao Li and Jeffrey Flanigan suggests that it isn’t that ChatGPT’s performance is degrading because new tasks are different from what the models are trained on, but that we forget that these models, especially the groundbreaking GPT-3, performed astoundingly well because they were trained with massive amounts of data, with a vast amount of examples of what is asked of them, and not particularly because they understand the tasks per se. 

As writing teacher and AI in education specialist Anna Mills puts it, it’s like “it has studied advance copies of lots of tests,” however, “when you give it new tests (tasks with no examples in its training data), it performs worse.”

The paper emphasizes that LLMs use a retrieval-based approach that mimics intelligence, as tech entrepreneur Chomba Bupe points out.

OpenAI may be having trouble catching up. Claims of getting “lazy” (which many have equated to “degrading”) have been plaguing OpenAI’s paid model, GPT-4, in recent weeks. 

The company, through its X account, explains that training a chat model “is not a clean industrial process,” and that it’s “less like updating a website with a new feature and more an artisanal multi-person effort to plan, create, and evaluate a new chat model with new behavior!”


Information for this story was found via X, and the sources and companies mentioned. The author has no securities or affiliations related to the organizations discussed. Not a recommendation to buy or sell. Always do additional research and consult a professional before purchasing a security. The author holds no licenses.

Leave a Reply

Video Articles

Endeavour Mining Q1 Earnings: Cash Flow Is King

G Mining Oko West Feasibility: Move Fast, Break.. Nothing?

New Gold Q1 Earnings: What’s Behind The Market’s Surprising Reaction?

Recommended

Giant Mining Encounters Native Copper As Hole MHB-34 Hits 563 Metres Depth

Verses Hits Commercialization Stage With Genius AI Platform

Related News

OpenAI in Talks for Massive Funding Round, Valuation Could Top $100 Billion

OpenAI, the company behind the popular AI chatbot ChatGPT, is reportedly in discussions to raise...

Friday, August 30, 2024, 02:54:00 PM

OpenAI’s Alarming Shift Via Defense Contractor Partnership Raises Serious Questions

OpenAI, the company behind ChatGPT, has taken a consequential step into the defense sector by...

Monday, December 9, 2024, 10:36:00 AM

Are Hard-Working Content-Farmers being replaced by AI?

And what effect could the monoculture they create have on our information supply? Our last...

Wednesday, May 10, 2023, 03:03:00 PM

Sam Altman Is Already In Discussions To Return To OpenAI

He may be down, but its not quite certain if he’s actually out. The Verge...

Sunday, November 19, 2023, 07:25:00 AM

Elon Musk, Other Tech Leaders, Scientists Sign Petition to Pause AI Development, Establish Safety Protocols

An open letter has been released to call for developers of AI technology “to immediately...

Thursday, March 30, 2023, 03:40:00 PM