Here an example of a fully correct almost verbatim ChatGPT4 answer (almost unedited, only with the output of the code included to show that the code works & was verifiably correct) :
Ridge Regression in R where coefficients are penalized toward numbers other than zero
This answer was left, though it received some unfair criticism in the comments section & was not upvoted relative to another less detailed answer to the question of the OP that did not have reproducible R code. I should note that as a domain expert I could immediately verify this answer to be correct. I also disclosed its use, as I should have. This was the output after a single prompt - the exact question asked in this case, plus two more prompts to ask for some extra details.
For another answer I had the OpenGL+SDL code correctly produced by ChatGPT4. This also just took three prompts - two where I pasted in some demo code of the nara and rdyncall packages (as they are not in the ChatGPT training set - they are only available on github) & then the actual question. The only bug in the original code was that the image came out upside down, but with the 4th prompt ChatGPT4 corrected that. I then manually added the benchmarks & wrote out the rest of the answer. This answer was deleted repeatedly by moderators, which is still not resolved, but in the meantime I reposted a somewhat edited answer:
Performant 2D SDL or OpenGL graphics in R for fast display of raster image using rdyncall package and SDL/OpenGL calls. Here I had originally disclosed the use of ChatGPT4 & how I had verified the correctness of the code (by showing the benchmarks & correct output), but doing so invited the wrath of moderators who then promptly deleted the answer (3 times no less). This now forced me to slightly rewrite the answer & some bits in the code to evade the answer being deleted. A very bad outcome if you ask me. This question I had given 4 or 5 bounties before over the past 5 years and it received a high number of votes, but it never received an adequate answer - the other answer that is there I would count as a "hallucinated human-written answer" as it hallucinates that solution would solve my problem & would work (it should rather just be a poor suggestion, written as a comment at best). Yet, it is still more highly rated than my answer because moderators and users keep on downvoting my correct answer merely out of antipathy for GPT (even if in that case I wrote most of the answer myself AND it's my own question). Before, at one stage someone also posted a GPT3 or ChatGPT3.5 produced answer that was not working - this I detected as such & I downvoted that answer and it was taken down (if as the OP I would have had the power to do this I would have done this myself, that would decrease the burden on moderators significantly). Of note: the accuracy of ChatGPT4 for coding and maths (together with the Wolfram Alpha plugin) has become way better than for the free ChatGPT3.5. An entirely different kettle of fish... See here for some recent maths benchmarks: https://arxiv.org/abs/2306.08997.
For this answer I used GPT to translate some R code to Rcpp, making the code run 15x faster than my own pure R code (I am an Rcpp beginner, so that was useful, but I could of course confirm that the code was correct & working): Faster way to calculate the Hessian / Fisher Information Matrix of a nnet::multinom multinomial regression in R using Rcpp & Kronecker products (again I disclosed its use). The rest of the answer was manually written. Here some Rcpp expert would in principle have been able to write an answer, but in reality it is unlikely that anyone would ever have bothered to put in the required time.
In all 3 cases it would be dead easy for any moderator and the OP to verify that what is posted makes sense as reproducible code plus output or benchmarks are presented, which makes it clear that the code was checked. It would be much less work for any moderator to check that than to try to infer if someone had used ChatGPT4 or not if they wouldn't have disclosed it. So for programming related questions at least I don't see any problem. Things might be different for answers requiring particular factual knowledge & authoritative sources. There I wouldn't recommend ChatGPT just yet, even though the ScholarAI and web browser plugins can now also give sources, and pasting in a couple of references from Google Scholar manually of course would hardly be any work.
A fourth case, not posted here on Stack Overflow but on Twitter, but instructive nevertheless, was a case where I gave ChatGPT4 with the Wolfram Alpha plugin a verbal description of a biological problem (that had not been analysed yet in the available academic literature). It then correctly wrote down the corresponding differential equation system (using its LLM logic) and then passed this to Wolfram Alpha / Mathematica - there it made a slight mistake though not including a given constraint (S+P=1 in this example), which caused Wolfram Alpha to say the system was unsolvable. It then proceeded solving the differential equation system itself for the stable point, but made a slight mistake there when rearranging terms (much like we could make a mistake when working with pencil and paper). In this case, even though it didn't quite give the correct final result, it was easy to see for me where it went wrong & correct the mistake. It also successfully made various modifications to the model if I presented it with additional biological details that had to go in into the model & made correct modifications to the differential equation system. I will include the final model in an article that I plan to submit shortly.
Here the input and part of the output:
For fun I also tried to see how good ChatGPT4 could be at rating Stack Overflow answers & I would say pretty good. For this question & the answers posted it gives good and accurate comments on each of the answers & it would give each of the answers a 6/10, while my answer it gave a 9/10. Based on my expert knowledge here that would seem about right. The replies are pasted below. ChatGPT4's own answer was also pretty good (certainly if edited a tiny bit, e.g. just retaining the best suggestion in terms of method - almost as good as my answer there right now). The one going on about transformations I would probably give a lower score, as that's kind of irrelevant here. Probably ChatGPT4 based moderation would be too expensive still to do at scale, but smaller, cheaper AI bots could probably be trained & do a lot of the moderation work on SO eventually if desired, if human moderator volunteers would not be able to keep up. And the original posters should in my mind also get more privileges to delete wrong or totally irrelevant answers. The OP should be best placed and have the highest incentive to evaluate the correctness of any answer & whether the proposed solutions actually solve his/her problem or not. This should not be the task of moderators. The nonworking RGL solution to my question e.g. I had always wanted to delete myself, but I can't unfortunately.
ChatGPT4 is also pretty good at redacting existing answers and adding more details, e.g. reproducible code or even authoritative sources. For example, taking the most accurate existing answers that lacks in sufficient detail, and asking ChatGPT4 to add essential details & authoritative sources gets you this - much better than what is currently there (which is saying little more than "pick up a book on ICA"), though still not perfect (e.g. quadratic programming problems have linear constraints too, just quadratic loss) :
These examples are drawn from my experience working daily with ChatGPT4 over the past months for work (in academia, my background is evolutionary biology & biostatistics). I could post 100s more examples, but I believe these examples are representative of the current quality of ChatGPT4.
I would like to see some formal benchmarks of the accuracy of ChatGPT4 relative to the accuracy of an average answer or the best accepted answer on Stack Overflow done in a fully blind way (this doesn't cut it): just give ChatGPT4 500 SO questions from after Sept. 2021 (GPT training date cutoff), get answers from ChatGPT4 & SO & have experts rate them blind. My subjective feeling, mainly working on statistical questions and programming in R, is that ChatGPT4 is on a par with an average Stack Overflow answer in terms of quality, but perhaps not even too far off from the quality of best accepted answers, at least for the field in which I work (can't tell for other fields). For me it doesn't bullshit any more than real humans, even domain experts. For the example above, the answer of a user named Carl is in fact misguided & irrelevant, but an unsuspecting visitor might have a hard time inferring this. ChatGPT4 also admits sometimes not knowing something about a given topic, that it doesn't have sufficient information, that you would better consult a domain expert, or will tell you if a particular suggestion is plain wrong. Sure, it's sometimes wrong or makes some small mistakes. So do humans. Many domain experts can sound convincing but quite frequently they are dead wrong too. With ChatGPT4 now scoring 155 on a regular IQ test and giving answers to university entrance exams that would be sufficient to get admitted to Stanford or MIT, making it land in the top10%, it's clear it's become pretty good.
Most academic journals, like PNAS, did not opt for a blanket ban on generative AI, but allow responsible use. All that a blanket ban will achieve is that users will stop disclosing the use of ChatGPT, making it hard or impossible to police (except to pick out the most blatant abuse), or that users will stop visiting SO altogether as ChatGPT can solve their problems faster, which could risk causing traffic to SO to decline further. And answers that can't be solved assisted by ChatGPT will most of the time also not end up receiving a human generated answer. In fact, right now only 8 out of my own 20 questions on Cross Validated were ever answered - 3 of which by myself - that's only a success rate of 25% to get useful help from others that solved my original problem. If I paste the 12 unanswered questions into ChatGPT4 I would count about 83% of the responses as useful & largely or entirely correct (i.e. I would have given them a checkmark, whereas the current answers are either beating about the bush, not solving my original problem or are plain wrong & for some there is no answer at all). And this is unredacted ChatGPT answers - which should not be allowed to be posted anyway - but with a bit of redaction (e.g. pasting in some authorative sources etc) such answers clearly would have been better than anything posted right now. As this concerns unanswered questions, posted a long time ago, this should be a representative, non-cherry picked set of examples. From this, it is clear that more complex problems that ChatGPT4 cannot solve likely would never be answered on SO either (e.g. because they would still require significant theoretical work to solve). In my mind, there is also a disconnect between the results above and the statements in the replies to the ban on generative AI announcement or articles like this one. Most of the time those answers and comments seem outdated & are not in line with the current quality of generative AI or published benchmarks.
To be clear: I do appreciate that some users abuse the system by posting unverified copy & paste answers from ChatGPT (including from the free, lower quality versions) and that this should not be allowed & countered. But the examples of obvious abuse that ggorlen cites in this thread were (1) almost always detected by the OP, who should have the privilege to delete such answers, which would in that way not cause any extra burden on moderators and (2) would be obvious to detect automatically & send a warning or kick out of the system, without moderators having to do this (typically new users that suddenly start posting loads of low quality posts in high succession & never get back to any comment or ever get their answers upvoted & post nonworking code). The examples cited are also just that: examples of clear abuse that would be easy enough to pick out. It says nothing about the % of posts where ChatGPT assisted in formulating an answer (or GPT Bing output, which for many will now be the default search engine) but where it was not disclosed, because they would be impossible to tell apart from any other good answer. And that's ultimately all that counts: is the answer any good? So what I strongly object to is that this whole issue of GPT on SO has devolved into some semi-religious anti-AI witchhunt. I agree with Philippe that this does run the risk of many more users just leaving the platform, which would be a great shame, as having an active user base is essential to keep this plaform going.
