Over half of ChatGPT answers to programming questions contain misinformation, yet the majority of developers are still keen to use AI tools and report both personal satisfaction and increased productivity from using them.

Findings about the adoption of AI tools and developers perceptions of them come from a recent survey by Stack Overflow which asked 1,700 developers across its community about their use of code assistants and the overall effect these tools have on their productivity.

The survey reveals that 76% of participants are already using or plan to use AI code assistants. The roles that use these tools the most amongst professional developers are academic researchers (87%), AI developers (76%), frontend developers (75%) and data scientists (67%).

Asked which code assistants developers use ChatGPT clearly  dominates with a massive 84% of respondents nominating it. GitHub Copilot is in second place with 49% using it.  Visual Studio IntelliCode came in third with 11%, followed by Codeium (7%) and Google Gemini, formerly Duet (4%). Eight products are used between 3% and 1% of survey participants and a further 4 by fewer than 1%.

Notice that the top two bars of this chart have been truncated and that the scale applies only from Visual Studio Intellicode.

In terms of ease of use, the top scores were awarded to Codeium (84%), GitHub Copilot (76%) and ChatGPT (61%) and the same three in order came out with the highest satisfaction scores (86%, 72%, 65% respectively) and highest increase in productivity (x2.5, x2.3, x2.2 respectively).

In the Stack Overflow survey 38% of developers report code assistants provide inaccurate information half of the time or more. This corresponds well with a finding by researchers from Purdue University presented last month at the Computer-Human Interaction conference that 52 percent of programming answers generated by ChatGPT contain misinformation.

The study addressed the question Is Stack Overflow Obsolete? by investigating how well ChatGPT performs in answering programming questions compared to Stack Overflow. Specifically the researchers took 517 programming questions asked on Stack Overflow and examined the correctness, conciseness and consistency of ChatGPT answers. They found that 52 percent of ChatGPT answers contain misinformation, 77 percent of the answers are more verbose than human answers, and 78 percent of the answers suffer from different degrees of inconsistency to human answers.

Despite these shortcomings, by conducting a small user study with 12 participants, they found that ChatGPT answers were preferred 35% of the time due to their comprehensiveness and their use of well-articulated language style and they overlooked the misinformation in the ChatGPT answers 39% of the time. which led the Purdue researchers to state:

This implies the need to counter misinformation in ChatGPT answers to programming questions and raise awareness of the risks associated with seemingly correct answers.

As we reported recently, see OpenAI Enriched By Stack Overflow, Stack Overflow is playing its part in this endeavour by partnering with OpenAI and giving it access to its entire repository of programming questions and answers. Another way to improve ChatGPT answers is to raise the quality of prompts ensuring that they contain clear information and also indicate the risk of a wrong or incomplete answer. 

How can it be that accuracy is so unimportant to us that we ever prefer the answers of an AI agent to a human?

More Information

Stack Overflow code assistant survey results

Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions

Related Articles

OpenAI Enriched By Stack Overflow

AI Helps Generate Buggy Code

Stack Overflow Traffic Slumps As Devs Turn to ChatGPT

Copilot Chat Improves Confidence and Enjoyment

GitHub Copilot Provides Productivity Boost

The Impact of Copilot Chat

Tell A Chatbot "Take a deep breath ..." For Better Answers

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.