-427

For posts following this one, see:


Today, we released a blog post from Stack Overflow CEO Prashanth Chandrasekar about AI and its intersection with communities. Because of the timely nature and the content of it, I’m reproducing it here for comments or questions from community members. You can find the original blog post here.

Senior members of the company staff will be reviewing everything here and will respond to questions, though it may be 24 hours or so before answers are ready, just because of the amount of interest that this is likely to get. Please leave any questions that you have below as individual answers.

Community is the future of AI

By Prashanth Chandrasekar

Throughout history, great thinkers have made predictions about how new technology would reshape the way in which humans work and live. With every paradigm shift, some jobs grow, some change, and some are lost. John Maynard Keynes wrote in 1930 that new technology meant humans would be working 30 hours a week or less, and that the main challenge would be what to do with all our free time. So far, predictions of this nature haven’t exactly come true. As new technology empowers us, we push ourselves to new heights and reach for previously unattainable goals.

Over nearly 15 years, Stack Overflow has built the largest online community for coders to exchange knowledge, a place where anyone with an internet connection can ask or answer questions, free of charge, and learn from their peers. Stack Overflow for Teams, our enterprise SaaS product, is trusted by over 15,000 organizations to serve as their internal knowledge bases. With the recent advent of dramatically improved artificial intelligence, many industries are wondering how technologies like ChatGPT will change their business. For software development, the answer seems more immediate than most. Even before the latest wave of AI, a third of the code being written on popular code repositories was authored by an AI assistant. 

Today, sophisticated chatbots, built on top of cutting edge large language models (LLM), can write functional code for a website based on nothing more than a photo of a rough sketch drawn on a napkin. They can answer complex queries about how to build apps, help users to debug errors, and translate between different languages and frameworks in minutes. At Stack Overflow, we’ve had to sit down and ask ourselves some hard questions. What role do we have in the software community when users can ask a chatbot for help as easily as they can another person? How can our business adapt so that we continue to empower technologists to learn, share, and grow?

It’s worth reflecting on an important property of technological progress. The Jevons Paradox shows us that, as innovation allows us to do more, we settle on a new normal, moving the goal posts for what we expect of people and organizations, then competing to see who can find new ways to pull ahead of the pack. For knowledge work, as the cost of an action diminishes, we often do more of it. Abstracting away repetitive or tedious tasks frees technologists up to make new discoveries or progress innovation.

If new AI systems make it possible to create software simply by chatting with a computer, my prediction is that, far from the job of programmer disappearing, we’ll end up with millions of new software developers, as workers from fields like finance, education, and art begin making use of AI-powered tools that were previously inaccessible to them. We are enthusiastic about welcoming this next generation of developers and technologists, providing them with a community and with solutions, just as we have for the last 15 years. We’ve got a dedicated team working on adding GenAI to Stack Overflow and Stack Overflow for Teams and will have some exciting news to share this summer.

I’ve heard similar sentiments expressed recently by Microsoft founder Bill Gates, by Geoff Hinton, the godfather of the neural network approach that produced today’s AI revolution, and by Stephen Wolfram, a pioneer across computer science and mathematics. Each sees in today’s AI the potential for the loss of certain jobs, yes, but also, if history is a guide, a future in which a great variety of more highly skilled work becomes available to an even larger group of people. Just as tractors made farmers more productive, we believe these new generative AI tools are something all developers will need to use if they want to remain competitive. Given that, we want to help democratize knowledge about these new AI technologies, ensuring that they are accessible to all, so that no developers are left behind. 

I talk to developers of varying experience levels all of the time, and I’ve been hearing anecdotes of novice programmers building simple web apps with the help of AI. Most of these stories, however, don’t begin and end with an AI prompt. Rather, the AI provides a starting point and some initial momentum, and the human does additional research and learning to finish the job. The AI can debug some errors, but is stymied by others. It can suggest a good backend service, but often can’t solve all the points of friction that arise when integrating different services. And of course, when a problem is the result not of instructions from a machine, but human error, the best answers come from other people who have experienced the same issues. 

For more experienced programmers, AI will be an amplifier of their existing skill, making them more ambitious in their projects. The result, as Jevons would predict, is that they spend more time with AI, but also more time creating new ideas, researching new topics, and asking new questions that had not occurred to them before. They feel empowered to reach farther beyond their traditional skillset and to push the boundaries in terms of the kind of work they want to take on.

We are excited about what we can bring to the fast moving arena of generative AI. One problem with modern LLM systems is that they will provide incorrect answers with the same confidence as correct ones, and will “hallucinate” facts and figures if they feel it fits the pattern of the answer a user seeks. Grounding our responses in the knowledge base of over 50 million asked and answered questions on Stack Overflow (and proprietary knowledge within Stack Overflow for Teams) helps users to understand the provenance of the code they hope to use. We want to help coders stay in the flow state, allowing them to create with the latest tools with the confidence that they will be able to document and understand the provenance, source, and context of the code being generated. 

Community and reputation will also continue to be core to our efforts. If AI models are powerful because they were trained on open source or publicly available code, we want to craft models that reward the users who contribute and keep the knowledge base we all rely on open and growing, ensuring we remain the top destination for knowledge on new technologies in the future.

AI systems are, at their core, built upon the vast wealth of human knowledge and experiences. They learn by training on data - for example open-source code and Stack Overflow Q&A. It is precisely this symbiotic relationship between humans and AI that ensures the ongoing relevance of community-driven platforms like Stack Overflow. Allowing AI models to train on the data developers have created over the years, but not sharing the data and learnings from those models with the public in return, would lead to a tragedy of the commons. It might be in the self-interest of each developer to simply turn to the AI for a quick answer, but unless we all continue contributing knowledge back to a shared, public platform, we risk a world in which knowledge is centralized inside the black box of AI models that require users to pay in order to access their services. 

As the AI landscape continues to evolve, the need for communities that can nurture, inform, and challenge these technologies becomes paramount. These platforms will not only offer the necessary guidance to refine AI algorithms and models but also serve as a space for healthy debate and exchange of ideas, fostering the spirit of innovation and pushing the boundaries of what AI can accomplish.

Our thesis on community as the center of a safe, productive, and open future for AI also offers some exciting prospects for our business. Stack Overflow for Teams, our enterprise, private version of Stack Overflow, helps to power a community-driven knowledge base inside of 15K+ organizations like Box, Microsoft, and Liberty Mutual. Decades of institutional knowledge, shaped and curated by subject matter experts and experienced teams, allows the employees at these organizations to more easily collaborate, improving productivity and trust. 

Incorporating generative AI technologies into the organizations using Stack Overflow for Teams will allow us to layer a conversational interface on top of this wealth of information. We believe this could lead to tremendous productivity gains: from new hires being able to onboard more quickly, to speed up developer workflows, as users are able to quickly ask questions and retrieve answers tapping into the company’s history, documentation and Q&A.

The example above is just one of many possible applications of GenAI to our Stack Overflow public platform and Stack Overflow for Teams, and they have energized everyone at our company. We’ll be working closely with our customers and community to find the right approach to this burgeoning new field and I’ve tasked a dedicated team to work full time on such GenAI applications. I’ll continue to share updates through channels such as my quarterly CEO blog, but I’ll be back in touch soon to announce something big on this topic. In the meantime, thank you to our community and customers for continuing to help us on our mission to empower the world to develop technology through collective knowledge.

55
  • 109
    Anyhow, my personal feeling is that AI is a huge wave washing the world these days. And it's not a good thing. It's going to cause enormous damage in the long run, not end of the world as some say, but damage that will take lots of years to fix. So while I understand the desire to jump on the worldwide AI wagon, I'm sad SE will also take part in it. :( Commented Apr 17, 2023 at 15:18
  • 111
    How will the LLM be trained to ignore all of the out of date and plain wrong answers on the SE network? Commented Apr 17, 2023 at 15:33
  • 24
    If you have questions, to ensure that they are seen and responded to, please add them as an answer, not just a comment.
    – Philippe StaffMod
    Commented Apr 17, 2023 at 15:44
  • 46
    You've probably heard this a lot of times before, but you'll get fewer downvotes than people unhappy with this, due to this site having its own pool of reputation. Commented Apr 17, 2023 at 16:40
  • 220
    So let me get this straight; we've deleted tens of thousands of CGPT answers just to have another CGPT clone introduced to the site itself? You could at least take us out to dinner before you screw us that hard. CGPT already exists; what's the point in introducing a technology we've so far distanced ourselves from precisely over quality concerns, and when either the same tech or a better tech exists elsewhere on the internet?
    – Zoe
    Commented Apr 17, 2023 at 17:41
  • 38
    @JourneymanGeek Based on the wording, it doesn't really matter where I point it out. If I write an answer, I'm going to have to expand it because answers are more serious than comments. However, this isn't a feedback post. Note how the question explicitly asks for questions that'll be responded to; this isn't a feedback post, it just looks like one. This is an early announcement, and no amount of negative feedback will change anything, because this isn't a feedback post. I'm not going to spend more time on something that will almost guaranteed end up being ignored or at the very least not
    – Zoe
    Commented Apr 18, 2023 at 0:06
  • 71
    This seems very much in the same vein as the cryptocurrency craze of 2020/2021, just with GPT: everyone is doing it, so let's do it too! Stack Exchange does have a massive Q&A count in its 50 million Q&A... the problem is that some 30 million (if not more) are... low quality or duplicates. Stack Exchange content is probably a great corpus of data for training big models, for sure, but I'd caution Prashanth (if he ever visits Q&A and reads comments) to not overestimate the content the company is working with here (lest we end up with yet another gen AI suffering from accuracy problems.
    – TylerH
    Commented Apr 18, 2023 at 16:00
  • 21
    I mean, i get that, but that seems a bit too aspirational to me, given what the company has promised and delivered previously. ¯\_(ツ)_/¯ It's more likely we'll just get clippy.
    – Kevin B
    Commented Apr 18, 2023 at 16:39
  • 9
    @Trilarion It shouldn't be so hard for a company like Stack Overflow to beat Google at making a better in-site search for its own content than what Google can do. SO has access to more data and better domain knowledge of the content, and should have no problem attracting enough talent to develop such a thing, given its overall reputation as a company.
    – TylerH
    Commented Apr 18, 2023 at 18:00
  • 9
    @Braiam It should fairly obvious how a company focused on broad internet searching has different concerns than a company focused on searching its own system. If Google wanted to dedicate a team to making a site search for questions on SO, for example, then yes they could probably out-compete even SO's own best concerted effort. But they aren't doing that, because it's not their job or concern, and much of what is useful for searching SO (all the parameters and keywords one can use for searching) are simply not supported in Google search because they only apply to SE sites.
    – TylerH
    Commented Apr 18, 2023 at 19:45
  • 19
    @Braiam Also, no, Google's primary goal has been ads, not search, since the acquisition of DoubleClick in 2007. You can tell that this is true given that search has remained fundamentally unchanged since around that time, except for the fact that ads and promoted content have continued to increase in presence in the Google search space. Google had the best product around that time, as a matter of hard effort, and has been able to keep and expand their domain presence by simply outspending any competition thanks to their ad money.
    – TylerH
    Commented Apr 18, 2023 at 19:46
  • 35
    10 years from now it's going to be really interesting to see how this post and the comments in reaction to it have aged.
    – Mentalist
    Commented Apr 19, 2023 at 4:20
  • 33
    Is anybody else a bit concerned that the CEO himself couldn't be bothered to post this with his own account? I mean I hope this gentelmen actually believe in the core product and understands how it works.
    – ouflak
    Commented Apr 21, 2023 at 15:28
  • 22
    @Philippe You can never win all of meta - but I do think tailoring the messaging to the audience helps. If I may be blunt - Prasanth writes very much like a management/business sort, and its almost a different language, as much as there's quirks to communication here for an 'outsider', or when technical sorts communicate and a non technical sort is trying to keep up. Commented Apr 22, 2023 at 3:17
  • 21
    The one piece of actionable information in this CEO's post was "I’ve tasked a dedicated team to work full time on such GenAI applications [and I'll back to you in 3 months]". Everything else was exactly the kind of ineffective mushy regurgitated keywords and calculated copycat know-it-all posing that could be effectively emulated by the current version of ChatGPT. --- Not even considering the question "how do ChatGPT and human "intelligence" differ? Intellectually as dead and un-curious as a doornail. Commented Apr 22, 2023 at 7:47

39 Answers 39

424

I'm not quite as excited as Prashanth to be participating in training AI, I participate here because I like teaching people, and I'm not particularly impressed by the job ChatGPT is currently doing of that, despite all the existing material it's been trained on.

In particular, I've recently seen a new flavor of question here on sites like Biology.SE that fits the format: "ChatGPT told me XYZ, can you explain this to me?" This pattern isn't new, "My professor/teacher/textbook told me XYZ and I'm confused" is pretty common. The difference is, no one should be learning from ChatGPT right now. It's not capable of understanding, it's a method of generating carefully crafted BS that appears to be knowledgeable without actually having knowledge.

From generating citations to completely fake papers to thinking Romania is land-locked to lifting plagiarized text, right now one of the biggest challenges to everyone else is filtering and fact-checking this BS. While versions of AI-assisted code writing have been around for quite awhile, the latest iterations are something quite different and I fear the hype will drive expectations for productivity that are not achievable without creating massive problems and security risks where the people paying the consequences will not be the same as the people collecting the benefits.

I don't really get what "Community is the future of AI" is supposed to mean, but for now, I fear it means that Our Community is not being treated respectfully by purveyors and users of recent AI technology releases.

19
  • 104
    I don't understand why people expect a language model to be able to reason about. It's not intelligent by any stretch.
    – Braiam
    Commented Apr 17, 2023 at 17:14
  • 60
    @Braiam I think it's largely because advocates for the technology keep telling them they should and how they're missing out if they don't use this new technology for everything. Commented Apr 17, 2023 at 17:18
  • 26
    @Braiam It's misinformation; or, "fake news", if you wanna call it that. People uneducated in the field are told by people they believe have the knowledge, that AI is "these things", and then they believe it; it's as simple as that. Commented Apr 17, 2023 at 18:31
  • 21
    If I recall correctly, what was risible - & telling - was ChatGPT's claiming Romania is a land-locked country bordered by the Black Sea. Commented Apr 17, 2023 at 21:32
  • 10
    @Scortchi-ReinstateMonica That's right, thank you - I had trouble finding a recounting of this particular example to link to. Basically, the example showed that ChatGPT 'knew' what sorts of words belong in a paragraph talking about land-locked countries: if I recall, this included how it presents hardships and results in a strong-willed people, and also it knew to discuss borders of land-locked countries, which for Romania included a body of water, but has no actual concept of the meaning of "land-locked". Commented Apr 17, 2023 at 21:51
  • 3
    Re "creating massive problems and security risks": This has (allegedly) already happened (Samsung semiconductor leak). Commented Apr 17, 2023 at 21:53
  • 6
    @Makoto I suppose Joseph is responding to my suggestion that people shouldn't use ChatGPT for learning, which I stand by. It seems like he merely wants to write code that doesn't error, which I suppose you can do with a trial and error sequence of ask ChatGPT, try, and then maybe ask on SO if it fails. But, I'd argue that's not really the same thing as learning, and while it might be okay for, say, modifying a plot where the result is evident, you're likely to make errors in data analysis if you try to "learn" statistics from ChatGPT, because those sorts of errors don't stop program execution. Commented Apr 17, 2023 at 21:54
  • 11
    @BryanKrause: Yes, ChatGPT's proclivity for confabulation ("hallucination") is rightly notorious, but its proclivity for self-contradiction is equally worth underlining. For non-sequiturs too: I've been told that snails don't eat mice because they are from different taxonomic groups. Commented Apr 17, 2023 at 23:03
  • 21
    @Scortchi-ReinstateMonica Indeed, but these are all symptoms of the same underlying issue that ChatGPT and similar models don't know anything except the statistics of words, and while this turns out to be sufficient to generate a somewhat believable mirage it isn't a substitute. It's a cute trick when you know what the right answer is, dangerous when you don't. Commented Apr 17, 2023 at 23:13
  • 9
    Here's an example (from a month ago) of ChatGPT getting mixed up with simple numerical facts. From scottaaronson.blog/?p=7094#comment-1947593 It first gives some correct info about Fibonacci numbers, but then it says: "Two consecutive natural numbers that have the same parity are 8 and 9, both of which are odd"
    – PM 2Ring
    Commented Apr 18, 2023 at 3:41
  • 37
    All the companies eager to hump the latest buzzword aren't necessarily using ChatGPT but a far inferior AI though. So while your concerns about ChatGPT are valid, rest assured that nothing equally bad will reach SO, but more likely something hundred times worse. As seen on 100% of all sites using a "support" AI bot to "help" users. Some 20 years after MS Clippy, these bots have not evolved at all.
    – Lundin
    Commented Apr 18, 2023 at 10:44
  • @Lundin: Yes, an example is the horrible, horrible moderation bots on Quora (of unspecified IQ). For instance, on the whole, they don't know any tech words. They think all nouns are common nouns. Commented Apr 19, 2023 at 12:19
  • I was almost certain that there was a network trained on StackOverflow (among other things) already or at least dataset containing some of StackOverflow content, but I can't find it right now. I do hope LLMs could help with boilerplate code etc., but so far everything I run locally performed poorly (maybe it's because consumer hardware only allows running cut down models or open source models are not as good as the closed source ones or maybe LLMs have a long way to go still).
    – jahu
    Commented May 4, 2023 at 8:14
  • @jahu ChatGPT/GPT3 is trained on a broader data set that includes StackOverflow/StackExchange. I don't know that it's specifically known that they used that part of the data set but it's at least broadly assumed that they did. Commented May 4, 2023 at 13:16
  • @BryanKrause Since (non-programmers) claim that it can write source code, it pretty much have to be trained on SO or otherwise the sources are too diverse and unreliable. (SO is also pretty unreliable since the average post quality is poor) Also if SO was to allow answers generated by AI, then it couldn't learn programming any longer, because it would start learning from itself and get stuck in a recursive loop of stupidity turning dumber with each iteration.
    – Lundin
    Commented May 4, 2023 at 13:40
337

This entire announcement makes me question the CEO's understanding of what his company's platforms actually are. There's a reason AI has been banned on several of the sites. We're actively fighting it over on Stack Overflow. This worries me. Don't get me wrong; I love AI/ML; I find it highly interesting, and I do look forward to the development of it into the future, but it has no place on SE.

Honestly, my first impression was: "wtf?" This seems so detached from the reality at the SE sites, which is highly worrisome, for someone that should know the reality, and is in charge of their future.

Please do not add GenAI to Stack Overflow. You're free to experiment with it, but don't pollute Stack Overflow with it. We're not interested in fact-checking AI content; we are interested in generating the content ourselves.

layer a conversational interface on top of this wealth of information

But why...? This is a distraction. These sites exist to easily let us find answers to the questions we seek, not be bombarded with noise. What purpose does this serve?



I find this post very demotivating. I’m not sure if there’s any point in continuing to moderate and curate Stack Overflow. You have made me insecure about my efforts to SE. Are they lasting? Will my work be undone? I don’t know. And a simple «no» in response, is not enough to cure that lack of trust I have.

Other problems I take issue with:

  • MSE is a second class citizen to the blog
  • The CEO doesn't post this himself on MSE
  • You don't link to the MSE discussion from the blog
  • The MSE discussion is not visible in the sidebar on any of the other sites and Meta sites. As such, it's invisible, because there's no link to it from the blog.
  • My comment on the blog linking to this MSE discussion takes forever to be accepted; many people will have missed it by the time it finally gets put there. Also, it's not my job to do that; it's yours. Resolved one day after being featured.
11
  • 2
    @Andreasdetestscensorship ask chatgpt to remove all the marketing speaks :)
    – Braiam
    Commented Apr 17, 2023 at 17:15
  • 4
    @Braiam I was under the impression that ChatGPT was designed to add them, not remove them. Commented Apr 17, 2023 at 17:19
  • NotTheDroids seems to be able to achieve that meta.stackexchange.com/a/388405/213575
    – Braiam
    Commented Apr 17, 2023 at 17:20
  • @Braiam Not a great attempt, but sure, I guess it worked somewhat. With the advent of generative AI, I guess we'll see more of it. We'll have to use a disregarding AI to strip it out. Commented Apr 17, 2023 at 17:28
  • 132
    This is gonna sound hella pessimistic, but...the CEO has been pretty far out of touch for a while now. Even the commitment to read and participate on Meta once a quarter seems to have become little more than a to-do item than something that is really meant to build that understanding.
    – Makoto
    Commented Apr 17, 2023 at 21:18
  • 13
    @Makoto I think you meant «realistic», not «pessimistic». :/ Commented Apr 17, 2023 at 21:52
  • 2
    "I’m not sure if there’s any point in continuing to moderate and curate Stack Overflow" I wouldn't take this that far. Seems like an overreaction to me. It's not like they're going to delete posts you've written, or undo moderation activities you've done. Though it might mean more content to moderate. But who knows? They haven't given us much detail yet.
    – starball
    Commented Apr 18, 2023 at 0:21
  • 19
    @starball "Though it might mean more content to moderate." And that's the essence. Flooding the site with this content is undoing the effort we put into keeping the sites clean and tidy. Commented Apr 18, 2023 at 0:22
  • 4
    eh. if AI-generated content gets added in to the Q&A pages, there's a good chance that I'll userscript it out of my UI (like I've already done with the blogposts and the new related questions section under unanswered questions). something something if I can't see it, it doesn't exist.
    – starball
    Commented Apr 18, 2023 at 0:24
  • "The CEO doesn't post this himself on MSE" see also Phillipe's comment: "The truth of the matter is, that's my job. I'm here to do what I do well, and he's here to do what he does well. His time is better spent on tasks that only he can do. That's why I don't write code, for instance: there are people on staff who are way better at it than I am. Same applies here."
    – starball
    Commented Apr 22, 2023 at 20:32
  • 5
    CEOs do not generally "know the reality" or have an "understanding of what [their] company's platforms actually are". In that, they are similar to LLMs. They just produce BS that sounds intelligible enough to attract gullible customers and investors. Commented May 17, 2023 at 10:13
242

Do not even think about incorporating something like ChatGPT into this site, or anywhere on the network.

To do so would be a slap in the face of all of the real contributors and their hard work and diligence to ensure that the content that is being produced comes from actual subject matter experts and those knowledgeable in the field.

The attitude from the tech industry is that AI is this panacea that will solve problems or make things easier, and I have the impression that this perspective is not changed within Stack Overflow. This worries me; y'all seem eager to solve a problem that y'all have had next to no success in quantifying.

One cannot claim to be supportive of creative minds or individuals when these AI models are generated and built on top of so much unlicensed, unprivileged data. (This is why I have reservations on things like GitHub Copilot, for instance.) It really doesn't take a lot of snooping around to make this observation.

I will absolutely not consent to having my data processed in pursuit of any AI integration, and I'll be looking at what my options are to prevent that in the coming weeks, should you move forward with this approach.

20
  • 5
    "The attitude from the tech industry is that AI is this panacea that will solve problems or make things easier", I mean, it will allow me to write BS that is asked of me. Stuff that I know will have zero impact on the business but everyone loves to ask others.
    – Braiam
    Commented Apr 17, 2023 at 17:16
  • 56
    The CC BY-SA license all your contributions have been labelled with, permits SE to use your contributions as they see fit. Legally, you already consented to have your data used in the pursuit of AI integration, and you cannot revoke this license, so you can't prevent it. Commented Apr 17, 2023 at 17:23
  • 29
    @Andreasdetestscensorship: I guess the option I'd have is to cease further contributions. There's too many ethical matters at stake for me on this one.
    – Makoto
    Commented Apr 17, 2023 at 17:31
  • 2
    @Makoto How believable and trustworthy is the idea of ceasing contributions? We still remember the highly turbulent years. I logged off for 2,5 years, because of it, but just came back 1-2 months ago. And now there's this. I'm not sure if I have faith in our ability to make these demands, and following up on them. Commented Apr 17, 2023 at 17:34
  • 14
    @Andreasdetestscensorship Not as they see fit. As the terms of the license permit. Commented Apr 17, 2023 at 18:11
  • 4
    @BryanKrause "You are free to: Share — copy and redistribute the material in any medium or format; Adapt — remix, transform, and build upon the material for any purpose, even commercially. Under the following terms: [attribution, same-license]". Seems pretty free to me. Commented Apr 17, 2023 at 18:25
  • 20
    @Andreasdetestscensorship Attribution and same-license are extremely important terms that limit and restrict many uses. Commented Apr 17, 2023 at 18:27
  • 19
    @Andreasdetestscensorship what do you mean? If the AI uses text I have written in its output or even to get to its output, then the license requires it to attribute me. I don't see how that is even possible here, which would make it impossible to train AI using SE content.
    – terdon
    Commented Apr 17, 2023 at 18:49
  • 4
    @terdon While collecting source data, their automated process compiles a list of all contributors. The link to this list is attached to all output data of the model trained with this source data. Commented Apr 17, 2023 at 18:52
  • 13
    @terdon all of the big LLMs are almost certainly trained in part on Stack Overflow content. I don't think the legal issues here are settled at all, but in any case the problem is already present with the current models, the SE model would be nothing new in this regard. Commented Apr 17, 2023 at 18:52
  • 10
    @MadScientist exactly. I am just pointing out that using it within SE doesn't magically remove the need for attribution. Andreas, I don't see how that can actually meet the requirement to reference the specific source by name. But IANAL, and I suspect you're not either, so let's wait and see what kind of monstrosity SE try to foist on us and deal with it then. I bet you it will not have a way of clearly indicating that the text it generates came from that post, but maybe I'll be wrong.
    – terdon
    Commented Apr 17, 2023 at 18:56
  • 4
    "The attitude from the tech industry" Rather, the attitude of the marketing departments of the tech industry. And from there CEOs can listen to their marketing department or their engineering department. Even the AI itself, if asked, is far more skeptical towards the use of AI than the marketeers. So it would seem to be that we can outsource the decision-making about whether or not to implement AI in a product from the CEO to an AI and thereby get better results.
    – Lundin
    Commented Apr 18, 2023 at 10:50
  • 9
    @Lundin except that is not AI. It's a language model. We've had this problem before. We call self driving instead of driving assistance, autoaim instead of aim helper; over promise and under deliver. While much of the crap that is asked today would be able to generate an adequate response by language model, and therefore less crap being asked, is a good thing™, not all of it can be generated in a way that is useful.
    – Braiam
    Commented Apr 18, 2023 at 18:38
  • 3
    @Braiam: Of course ChatGPT is AI. Just like AlphaGo and millions more. It's not an AGI (Artificial General Intelligence) though. The confusion is somewhat understandable in the case of ChatGPT, because it's a multi-domain AI with no obvious domain boundaries. But the field of AI has been quite well defined for literally decades, and Machine Learning has been at the core of AI equally long. Commented Apr 20, 2023 at 7:36
  • 4
    @Braiam: "Industry trying to retcon"?! Industry wasn't even been involved seriously in the early days of AI. The AGI or "strong AI" concept can be traced back at least to Searle's "Chinese Room" paper from 1980. Commented May 21, 2023 at 22:40
176

Please provide an official interpretation of the CEO's wording for regular Meta Stack Exchange users.

The above post, as it was originally intended for a different audience, uses a communication style that might be magnificent for the target audience and the publishing platform, but the "storytelling methodology" has resulted in the writer including a lot of stuff that looks to be irrelevant for this context and space.

AFAIK, most of Meta Stack Exchange's regular users aren't CEOs, aren't top decision makers handling advertising budgets, aren't finance market stock buyers, aren't top-notch tech influencers. Most users are "regular" folks interested in having one or more healthy Q&A communities. For many of these communities, Stack Overflow is the template for the Q&A model and it informs their initial and ongoing operation. It's clear that without Stack Overflow the other sites that support these communities can't exist, that is not being questioned.

While many of the current communities were proposed and supported by Stack Overflow users from their conception to their launch, each community has developed their own culture and workings. This observation is relevant as most communities are being affected by ChatGPT and have reviewed the possibility of banning generated text content. Many of them have done this.

What are the official implications and repercussions of the current CEO's words for the Stack Exchange communities? I'm not asking that you write down the roadmap or describe the features that are coming as it's clear that you will do that when you are ready. I'm not asking you to add content, but rather communicate the same content in a way that is easier to understand for the regular audience of this space.

15
  • 54
    I certainly don't struggle to read through the marketing "mud", but what a waste of time it is. Commented Apr 17, 2023 at 18:42
  • 60
    I wholeheartedly agree with the criticism about the writing style. Why do people write things in such a roundabout way if they could straightforward present what all this is about in an as concise as possible manner? It always gives the impression that it wants to look like more than it really is in the end. What a pity. I feel like I really need an AI to understand what some people are saying. Commented Apr 17, 2023 at 19:54
  • 16
    @Trilarion Obviously because the intention is to not be straightforward. If they get straight to the point, people will have a definitive answer, and take a stand. This writing style keeps everyone on the edge, hoping it isn’t as bad as it really is. It’s manipulative. Commented Apr 17, 2023 at 21:59
  • 48
    Apparently they accidentally posted the unedited ChatCEO output. PS Some hallucinations: "we want to craft models that reward the users who contribute" ?? "It is precisely this symbiotic relationship between humans and AI that ensures the ongoing relevance of community-driven platforms like Stack Overflow." ??
    – philipxy
    Commented Apr 18, 2023 at 2:33
  • 5
    This! I really want to know if this blog is "I'm modern and with the AI times and a brilliant tech innovator CEO visionary invest now and gimme lots of money" or "I actually want to do stuff with LLMs". I really hope it's the first
    – Erik A
    Commented Apr 18, 2023 at 9:10
  • 17
    You can always vote to close as ‘needs details or clarity’… Commented Apr 18, 2023 at 11:48
  • 27
    My best effort at a non-marketing version: "Stack Overflow (the company) has spun up a new team of employees dedicated to training a LLM (and performing product research in general) based on network Q&A content with the goal of increasing user satisfaction as well as bringing in more users to the network. We will share more about what we have been working on some time this summer."
    – TylerH
    Commented Apr 18, 2023 at 18:06
  • 1
    @Trilarion a round about = 'red herring'...
    – bonCodigo
    Commented Apr 21, 2023 at 10:47
  • 2
    @Trilarion the writing style communicates membership in a social group. This is true in politics, academia, gangs, neighborhoods, and so on. "We can identify you as one of us by the way you say the thing you say." Commented Apr 25, 2023 at 14:05
  • 1
    @Michael The writing style is simple to emulate "The AI of the future constitutes an endless community and bridges the gap between humans to evolve around advances in computing power as well as an ever growing training set, a perfect symbiosis that I will bring forward by dedicating teams on it, perfecting our technology but also increasing the benefit for everyone in this great new world." See. I'm more worried about there not being enough content hidden and the additional effort required to decipher the blog post, not really about the social class. Commented Apr 25, 2023 at 15:56
  • 1
    @Trilarion Oh, yes. I understand the problem, and I agree. I was only addressing the "why" question. (And, now that I think a bit longer... what are the odds that an LLM "polished" the prose in question?) Commented Apr 25, 2023 at 16:32
  • 7
    Maybe the CEO's statement was written by ChatGPT? It kinda has that style.
    – FuzzyChef
    Commented Apr 25, 2023 at 19:59
  • @FuzzyChef I feel unappreciated.
    – philipxy
    Commented Apr 28, 2023 at 23:02
  • 2
    I have trouble making out any concrete information in this massive wall of text in the blog post. It reads like ChatGPT output. "Hey ChatGPT, generate me 5000 words that sound important based on the following 10 bullet points."
    – usr
    Commented May 14, 2023 at 17:50
  • I guess we users now have the option of using ChatGPT to summarize this and back-convert to actual meaning.
    – usr
    Commented May 14, 2023 at 17:51
160

I’ve tasked a dedicated team to work full time on such GenAI applications.

Let's get things straight. The company has the budget to waste on "GenAI applications" (whatever that is supposed to mean), but doesn't for:

  • keeping Traducir open-source (in case anyone happens to not know why it's important: it powers community-lead efforts for providing translation strings for localized versions of Stack Overflow);
  • fixing issues and improving the public Stack Exchange API (I am not even counting MSE and MSO requests, just the Stack Apps ones);
  • fixing issues reported on MSO and MSE, as well as implementing feature requests (MSO, MSE);

The list is far from being exhaustive. In light of the following statement:

Community and reputation will also continue to be core to our efforts.

How does syphoning limited resources from the enormous list of problems and improvements that can be implemented across the network to something that is considered wasted effort at best by many members of the said community (the general tone of the discussion under the post is proof enough of that) correlate with the claim that the community is at the core of the company's efforts?

5
  • 2
    community is core to SE's efforts BUT SE is determined to replace that community with AI... Makes perfect sense...
    – jwenting
    Commented Apr 23, 2023 at 13:28
  • 2
    The difference is that AI is something that can be directly monetized. It is easier to allocate resources to projects that have a more easily visible return on investment.
    – Stevoisiak
    Commented Apr 25, 2023 at 18:42
  • 4
    related: Today's actions by Stack Overflow
    – starball
    Commented May 11, 2023 at 0:06
  • 1
    Totally agree. They never had time to implement things like this and many others the community has been asking for years. Instead, they want to do another thing that nobody asked for...
    – hkotsubo
    Commented May 17, 2023 at 0:23
  • 1
    @hkotsubo yeah, well, that's what we get for the grave mistake of founders to make the company a for-profit one... Making things for the public platform ain't going to earn them nothing apart from our gratitude, and the latter is worth exactly 0$. Since they are also an asset for Prosus, they'll jump through as many hoops to make profit as they are asked to. Admittedly, there's been a period of improvement in terms of trying to address our feature requests, but since the recent mass layoffs I am seeing a turn the old behavior. Commented May 17, 2023 at 23:19
125

Apparently, even before this initiative picked up steam, the result of this has been, and to use the exact term I used before - "a large negative impact to staffing and resources". There was a significant reduction in staffing - including folks working on public Q&A.

These would a have direct impact on quite a few requests from the community, as well as the company's ability to support the community.

Each sees in today’s AI the potential for the loss of certain jobs

Well looks like that's happened already

To me - this means a reversion to pulling resources from the public Q&A sites to fund potential projects or ones that were failing, and a clear failure to support the community.

As much as the new shiny things - we need to make sure that fundamental needs of these community - support both in terms of community and development needs are met.

As things stand there has been an immediate failure in the company's commitment to the community.


Assuming that this isn't SO/SOFT only - what does this actually mean for the communities - and what are the benefits for us?

There are fundamental parts of the network that have not really gotten attention - including some relatively 'trivial' feature requests over design changes (like better access to meta and chat) , and reimplementation of features that have been ignored, or put on the back burner due to lack of resources (notifications), or other priorities.

We've also have a significant amount of workload from folks 'naively' using ChatGPT and other such LLM/ML tools. I'm not purely sceptical of these tools in context, but I'm wondering looking at the costs of other organisations doing this - with massive GPGPU farms and such. It is incredibly expensive - and well, when features and requests that directly benefit the community now are delayed, I'd ask if this is the right call - and would be concerned that resources that could be put to more direct use is diverted to a moonshot.

I've been here long enough to see the rather painful effects that the company getting a bet like this wrong can have. Pivoting to the new hot technology while it’s hot makes sense for a startup, but without a solid idea of what the benefits are is concerning.

So practically, what are the benefits to the communities, and what are the guarantees that if this fails, that there wouldn't be a large negative impact to staffing and resources as has happened in the past?

Critically - is there a plan to turn 'ideas' into a revenue source and clear goals for this team to meet, as opposed to a sense of FOMO?

At Stack Overflow, we’ve had to sit down and ask ourselves some hard questions. What role do we have in the software community when users can ask a chatbot for help as easily as they can another person?

As an active user of the site? Experience, and the synthesis of knowledge. Humans make links and put their collected knowledge together. I've even occasionally gotten bespoke answers to unusual issues because someone found it interesting. ChatGPT writes like a 13-year-old trying to do a literature review.

We are excited about what we can bring to the fast moving arena of generative AI. One problem with modern LLM systems is that they will provide incorrect answers with the same confidence as correct ones, and will “hallucinate” facts and figures if they feel it fits the pattern of the answer a user seeks.

Yes, we noticed. We've even banned the use of many of these tools on many of our sites. They're basically emitting entirely plausible garbage, and that's worse than complete garbage.

. Grounding our responses in the knowledge base of over 50 million asked and answered questions on Stack Overflow (and proprietary knowledge within Stack Overflow for Teams) helps users to understand the provenance of the code they hope to use. We want to help coders stay in the flow state, allowing them to create with the latest tools with the confidence that they will be able to document and understand the provenance, source, and context of the code being generated.

While you have a more focused corpus of information, it doesn't necessarily mean that the generative AI/LLM models you use will actually understand what you're doing. Maybe this might lead to better search, but I feel like outside the lowest hanging fruit, unless you've got something up your sleeve OpenAI and Google doesn't have, it’s hopelessly optimistic.

I'd also question how feeding information into an LLM model works with our licence. Would GenAI-processed results still be under the CC-BY-SA licence and how would you properly attribute the original source of the question?

Incorporating generative AI technologies into the organizations using Stack Overflow for Teams will allow us to layer a conversational interface on top of this wealth of information.

My previous job was with a vendor for the local government. Outside the 'data cannot leave the country', the very idea of feeding our data to an external source would freak my former boss out. There's a news article of information leaking as well at Samsung, including confidential source code. This might not be as good an idea as it seems.


RESPONSE by Philippe:

The goal is to leverage the latest technology to add value for the Stack Overflow community. We are working closely with community members throughout this process to experiment with GenAI to build solutions to solve for historical pain points with the site. We would love to explore the following areas with community members and will remain open to feedback as we test and learn about these subjects:

  • Guided question asking to improve question quality
  • Tooling for the site and content moderation tooling to reduce the load on moderators
  • Content recommendations to improve discovery and lower duplication
  • Personalization of search experiences to improve content discovery
  • Trends and insights on emerging topics in the community to highlight new knowledge contributions and subject matter experts

For example, historically, community members have manually provided feedback to newer community members asking questions on the site. Leveraging GenAI to give newer community members real-time feedback on asking questions that are appropriate on Stack Overflow might reduce the load on community members.

We will also work with the Moderators and power users on the site to test and refine the application of these AI solutions going forward: we are committed to building, updating, and curating high-quality knowledge on our site and ensuring these voices are an important part of the feedback cycle as we test these newer solutions to existing community challenges. -- Philippe

3
  • 3
    @Philippe On the whole topic of giving new contributors feedback on post improvements and following community guidelines, this is my PSA that reading SO's on-topic page, the general dont-askpage, the generic how-to-ask page, SO's minimal-reproducible-example page, and the general closed-questions page takes only approximately 30 minutes. It's really not that much to expect and will server the reader forever. I'd be much more enthusiastic about work on encouraging new askers to read those Help Center pages.
    – starball
    Commented Apr 20, 2023 at 2:30
  • 5
    I've got two questions that machine learning could try to answer automatically to help the Q&A quality. Based on (question text), what existing question(s) is the input a duplicate of (LLM model very needed, generative one not so much) and what sections of the site help is the post author unfamiliar with?
    – Ben Voigt
    Commented Apr 23, 2023 at 1:36
  • 8
    The first time someone comes to Stack Overflow, they won't know how useful the community will be to them. It would never have been worthwhile for someone to spend 30 minutes reading about how to ask a good question on Yahoo Answers back in the day. A first-time user won't know the difference between a crappy forum and this site, so they almost inevitably won't spend half an hour reading documentation. I agree with Ben Voigt - finding existing posts more effectively would be valuable if it works, but generating text is not the important part.
    – user7868
    Commented Apr 24, 2023 at 1:26
119

It is precisely this symbiotic relationship between humans and AI that ensures the ongoing relevance of community-driven platforms like Stack Overflow.

This is obviously not correct!

How can somebody who understands anything about Stack Overflow think this is correct? The relevance of Stack Overflow has nothing to do with AI generated content. AI generated content is not even allowed there, and the community overwhelmingly supports that ban.

Did anyone check this announcement before it was published? Is there anybody at the company who can tell the CEO he is totally wrong about this?

11
  • Well, as I understand it, AI was allowed before ChatGPT per the GitHub Copilot meta SO question. However, when that was written, any reasonable SO user would have understood these answers to be a tiny and controllable fraction of the overall answers pool, so your main point is still correct. Commented Apr 19, 2023 at 19:11
  • he wants to replace human contributors with AI bots, because he can (or thinks he can) control the bots better than the humans...
    – jwenting
    Commented Apr 23, 2023 at 13:32
  • 2
    @jwenting how many end-of-the-world stories have been written on the premise that AI could be controlled better than humans? Commented Apr 23, 2023 at 16:45
  • "...the community overwhelmingly supports that ban" - I'm afraid this is wishful thinking. We do not actually know how many community members lean towards restricting AI use in any way, let alone banning it altogether. Commented May 4, 2023 at 8:11
  • 20
    @მამუკაჯიბლაძე Yeah we actually do. meta.stackoverflow.com/questions/421831/…. 3719 for, 79 against. 98% of the community supports the ban and this is even enough of data to be quite certain statistically. When was the last time you asked any group of people about their opinion of anything and got 98% consensus in return?
    – Lundin
    Commented May 4, 2023 at 13:54
  • @Lundin Sure, on stackoverflow. On such sites (mathoverflow another example) rubbish is easily detectable, so AI would only be a hindrance there. But on sites like philosophy or rpg it could well be welcome, and might even outperform most of the human contributors. Commented May 5, 2023 at 4:05
  • 4
    @მამუკაჯიბლაძე The CEO's quote and my answer both say Stack Overflow, not all Stack Exchange sites.
    – kaya3
    Commented May 5, 2023 at 4:06
  • 3
    For what it's worth, RPG SE's community also overwhelmingly supports that site's ban on ChatGPT-generated content, too.
    – kaya3
    Commented May 5, 2023 at 4:22
  • @kaya3 Sorry, did not notice it! Commented May 5, 2023 at 7:52
  • 1
    @მამუკაჯიბლაძე A huge number of network sites have come out strongly against allowing generative content, including RPG gaming, theology, culture, politics and other non-tech sites. If you think LLMs might somehow better suited to philosophy than humans you haven't explored what a markov chain might decide is morally defensible.
    – Caleb
    Commented Jun 10, 2023 at 10:56
  • @Caleb What I had in mind is not so much any moral issues but rather that if an LLM can successfully stand a Turing test on one of the SE sites, this (a) somehow makes actual value of that site questionable and (b) means that generative content would not do much harm to that site and might even improve it to some extent. Commented Jun 10, 2023 at 18:34
111

Currently about 60 sites in the Stack Exchange network ban the use of ChatGPT in them.

Now that Stack Exchange it taking a leap towards AI, does it mean it will also override those bans and allow ChatGPT (and other AI generation programs) posts all around the network?

(Personally I really hope that they'll let the bans stay, but have to ask.)


RESPONSE by Philippe:

The existing bans on ChatGPT content are intended to do a few things:

  1. prevent plagiarism on the sites, since ChatGPT does not attribute content.
  2. prevent a tidal wave of content that LOOKS like it might be correct but isn’t.

There remains a need for both of those things, and we will support those SE sites that choose to continue to ban answers that are created by generative AI programs like ChatGPT.

Internally the work that we are doing toward integrating AI solutions is geared toward using versions of this technology to support the community’s efforts and goals. We are committed to building solutions that benefit the community, with the quality of the content on our sites always at the front of mind. We are committed to building trust in the collective knowledge of the community on the site. This means we will highlight where knowledge came from and prohibit the creation of knowledge that looks superficially solid but isn’t accurate (such as using AI to create answers that have not been reviewed, edited or verified by a community member). We also will not be using AI to create unattributed answers. -- Philippe

28
  • 4
    Okay, first and foremost let me not knee-jerk to this. This is a legitimate question.
    – Makoto
    Commented Apr 17, 2023 at 16:52
  • 38
    But if they decide to override those bans, I'm going to hit the door. There'll be no sense in trying to save what's left. Worse, it'd be damn near impossible to do so.
    – Makoto
    Commented Apr 17, 2023 at 16:52
  • 3
    just seems like a silly question to me. The bans that are in place are banning a particular abuse of chatgpt that isn't allowed anyway. generative ai existing on the site won't suddenly make plagiarizing content allowed.
    – Kevin B
    Commented Apr 17, 2023 at 16:55
  • 26
    @KevinB: Given that the type of clientele a ban on ChatGPT is meant to prevent, said clientele seeing/reading "AI on Stack Overflow is OK now" isn't going to exactly take the time to really parse out what that means in specifics...
    – Makoto
    Commented Apr 17, 2023 at 16:56
  • 3
    @KevinB I beg to differ. When a company officially leading AI projects, having half their sites, including the main one, ban it doesn't look good. And in business, what matters most is appearance. I can totally see this happening. Commented Apr 17, 2023 at 16:58
  • 1
    @Makoto You're welcome to stay in the chat rooms, though. And I'm sure you'll continue finding answers here, until they're overgrown by the jungle created by AI. Commented Apr 17, 2023 at 17:55
  • 2
    @Andreasdetestscensorship I don't think SE will actually use AI to generate posts. More likely something more subtle, e.g. using it to determine questions quality, finding things out of place (voting rings) and more like this. It would be useful, yes, but like I said in the answer, the point is they add water to the wave that will drown the world, even if those water are useful in a way. Commented Apr 17, 2023 at 18:00
  • 5
    @ShadowTheSpringWizard How does generative AI help with that? Commented Apr 17, 2023 at 18:04
  • 4
    @ShadowTheSpringWizard Oh, no. More posts about blockchains and the next successful crypto wallet thing. Commented Apr 17, 2023 at 18:23
  • 17
    "Currently 89 sites in the Stack Exchange network ban the use of ChatGPT in them." ... I wonder if the CEO was even aware of this fact when he wrote this blog post.
    – Travis J
    Commented Apr 18, 2023 at 5:29
  • 6
    @Philippe thanks for clearing my doubts. First time I see response edited into a post, thinking about it, that's good, and no real alternative for long response like that. Cheers! (Though it does confuse users who want to vote, will they vote for what I ask in the answer, or for your response?) Commented Apr 18, 2023 at 16:07
  • 5
    @ShadowTheSpringWizard - I agree. Can't come up with a better way to do it though. Open to suggestions!
    – Philippe StaffMod
    Commented Apr 18, 2023 at 16:56
  • 2
    Yeah, but I assure you that won't get built today, so working with what we've got....
    – Philippe StaffMod
    Commented Apr 18, 2023 at 16:59
  • 2
    @Philippe You could use the comments for feedback: "Upvote this comment if you like the response, downvote this comment if you don't like the response". (yes, this is a joke) Commented Apr 18, 2023 at 18:09
  • 3
    @starball - to be quite honest, I'm not sure. I know that it's being thought through though, and at some point it'll cross my desk, usually fairly early in the design process, and I'll be able to weigh in (either myself or through a proxy, one of my community managers), and this is an opinion that we will represent. I would suspect that we're probably looking at some sort of rep threshold, but I have very little upon which to base that speculation.
    – Philippe StaffMod
    Commented Apr 24, 2023 at 4:53
94

I am not here to debug ChatGPT.

I am not here to debug generative LLMs in general.

1
  • 23
    Or contribute to their corpus
    – miken32
    Commented Apr 24, 2023 at 22:11
83

Disclaimer: I am personally fully against AI-generated answers/solutions on SE. While this answer is somewhat harsh, I do feel the criticisms within it are fair.


... John Maynard Keynes wrote in 1930 that new technology meant humans would be working 30 hours a week or less, and that the main challenge would be what to do with all our free time. ...

Can we get a citation for that "30 hours a week or less"? NPR and The Guardian both say 15 hours, not 30. Additionally, assuming you are referring to this document written by Keynes, it says "Three-hour shifts or a fifteen-hour week may put off the problem for a great while." (page five).

... Even before the latest wave of AI, a third of the code being written on popular code repositories was authored by an AI assistant.

Again, according to what? I haven't looked, but what specifically does "popular code repositories" actually mean? And how is this (un-named) source determining how much of the code is AI-assisted? Is it some statistic from a product like GitHub Copilot? Also, when exactly do you consider the latest wave of AI to have started? When GH Copilot was released? Or when ChatGPT was released? Or some other point in time? You make a claim, but it is largely unclear and contains no source at all.

Today, sophisticated chatbots, built on top of cutting edge large language models (LLM), can write functional code for a website based on nothing more than a photo of a rough sketch drawn on a napkin. They can answer complex queries about how to build apps, help users to debug errors, and translate between different languages and frameworks in minutes. At Stack Overflow, we’ve had to sit down and ask ourselves some hard questions. What role do we have in the software community when users can ask a chatbot for help as easily as they can another person? How can our business adapt so that we continue to empower technologists to learn, share, and grow?

That's more fair. I've seen the project about generating the website - I agree, it's pretty interesting. At the same time, though, AI-generated code from chatbots has many limitations, perhaps most notably that it is often flat-out wrong, even if it sounds confident/correct. This is a large part of why SO and other SE sites have banned ChatGPT-generated answers.

It’s worth reflecting on an important property of technological progress. The Jevons Paradox shows us that, as innovation allows us to do more, we settle on a new normal, moving the goal posts for what we expect of people and organizations, then competing to see who can find new ways to pull ahead of the pack. For knowledge work, as the cost of an action diminishes, we often do more of it. Abstracting away repetitive or tedious tasks frees technologists up to make new discoveries or progress innovation.

Fair enough, that's reasonable.

If new AI systems make it possible to create software simply by chatting with a computer, my prediction is that, far from the job of programmer disappearing, we’ll end up with millions of new software developers, as workers from fields like finance, education, and art begin making use of AI-powered tools that were previously inaccessible to them.

That’s quite the run-on sentence.

We are enthusiastic about welcoming this next generation of developers and technologists, providing them with a community and with solutions, just as we have for the last 15 years. We’ve got a dedicated team working on adding GenAI to Stack Overflow and Stack Overflow for Teams and will have some exciting news to share this summer.

At risk of being harsh/rude, WHY?! Please no. A very large amount of work and effort has been put into finding and removing AI-content on SO, and the rest of the network (when it is banned on a site). ChatGPT answers are often wrong and not wanted. I cannot speak for others, but I certainly do not welcome that, nor am I excited for that in any way. Harsh, but that feels like you are flat-out oblivious to the fact that (overall) the community does not want ChatGPT answers (per the vote-counts on the ChatGPT ban announcement on MSO a.k.a. the absolute highest scoring post on MSO).

[image caption] Community members and AI must work together to share knowledge and solve problems

No, we certainly don't. SO and SE has been solving problems and sharing knowledge for many years without ChatGPT. It is certainly not a "must".

I’m not alone in thinking AI might lead to an explosion of new developers. I’ve heard similar sentiments expressed recently by ... Bill Gates, ... Geoff Hinton, ... and by Stephen Wolfram ... Each sees in today’s AI the potential for the loss of certain jobs, yes, but also, if history is a guide, a future in which a great variety of more highly skilled work becomes available to an even larger group of people. Just as tractors made farmers more productive, we believe these new generative AI tools are something all developers will need to use if they want to remain competitive. Given that, we want to help democratize knowledge about these new AI technologies, ensuring that they are accessible to all, so that no developers are left behind.

Sure, some developers are excited about it. I also find it interesting. But when you say "we want to help democratize knowledge about these new AI technologies", does that mean you want to share information about those technologies? I don't have an issue with that, and doing so is good (assuming it is on-topic, of course)... but actually adding AI into SE is different. Can you clarify which you're referring to?

I talk to developers of varying experience levels all of the time, and I’ve been hearing anecdotes of novice programmers building simple web apps with the help of AI. Most of these stories, however, don’t begin and end with an AI prompt. Rather, the AI provides a starting point and some initial momentum, and the human does additional research and learning to finish the job. The AI can debug some errors, but is stymied by others. It can suggest a good backend service, but often can’t solve all the points of friction that arise when integrating different services. And of course, when a problem is the result not of instructions from a machine, but human error, the best answers come from other people who have experienced the same issues.

Yes. Exactly. Sometimes, it does it correctly. Sometimes, it can't. The best answers do come from other people, not chatbots.

For more experienced programmers, AI will be an amplifier of their existing skill, making them more ambitious in their projects. The result, as Jevons would predict, is that they spend more time with AI, but also more time creating new ideas, researching new topics, and asking new questions that had not occurred to them before. They feel empowered to reach farther beyond their traditional skillset and to push the boundaries in terms of the kind of work they want to take on.

I'm not convinced of your first sentence, but that's just my personal viewpoint, though.

One problem with modern LLM systems is that they will provide incorrect answers with the same confidence as correct ones, and will “hallucinate” facts and figures if they feel it fits the pattern of the answer a user seeks. Grounding our responses in the knowledge base of ... questions on Stack Overflow (and ... knowledge within Stack Overflow for Teams) helps users to understand the provenance of the code they hope to use. ...

That's a good goal... but, how exactly will you "ground ... responses in the knowledge base of 50 million asked and answered questions"? Specifically, how will you ensure only correct answers? This is way easier said than done - if it was practically do-able, it would have likely already been done.

Community and reputation will also continue to be core to our efforts. If AI models are powerful because they were trained on open source or publicly available code, we want to craft models that reward the users who contribute and keep the knowledge base we all rely on open and growing, ensuring we remain the top destination for knowledge on new technologies in the future.

Yep, the community is very important. But could you elaborate on this a bit? Like, community members will get rewarded when a chatbot answer based on theirs is marked helpful? What exactly does this mean?

AI systems are, at their core, built upon the vast wealth of human knowledge and experiences. They learn by training on data – for example open-source code and Stack Overflow Q&A. It is precisely this symbiotic relationship between humans and AI that ensures the ongoing relevance of community-driven platforms like Stack Overflow. Allowing AI models to train on the data developers have created over the years, but not sharing the data and learnings from those models with the public in return, would lead to a tragedy of the commons. It might be in the self-interest of each developer to simply turn to the AI for a quick answer, but unless we all continue contributing knowledge back to a shared, public platform, we risk a world in which knowledge is centralized inside the black box of AI models that require users to pay in order to access their services.

I'm still not convinced of any benefit from adding AI content into SO, even if it is free. Also, what "symbiotic" relationship? The one in which SO flat-out banned it? Not exactly what I’d call symbiotic at all

As the AI landscape continues to evolve, the need for communities that can nurture, inform, and challenge these technologies becomes paramount. These platforms will not only offer the necessary guidance to refine AI algorithms and models but also serve as a space for healthy debate and exchange of ideas, fostering the spirit of innovation and pushing the boundaries of what AI can accomplish.

Yes. Innovation is good, as is progress. Productive discussion of those technologies is also good.

The next paragraph is little more than an ad for SO Teams, so I’m omitting it.

Incorporating generative AI technologies into the organizations using Stack Overflow for Teams will allow us to layer a conversational interface on top of this wealth of information. We believe this could lead to tremendous productivity gains: from new hires being able to onboard more quickly, to speed up developer workflows, as users are able to quickly ask questions and retrieve answers tapping into the company’s history, documentation and Q&A.

Would it be limited for SO for Teams? Or would it go to actual SO? Not trying to be overly harsh, but can you elaborate on what information a chatbot would be able to give that wouldn't be accessible by searching?

The example above is just one of many possible applications of GenAI to our Stack Overflow public platform and Stack Overflow for Teams, and they have energized everyone at our company. We’ll be working closely with our customers and community to find the right approach to this burgeoning new field and I’ve tasked a dedicated team to work full time on such GenAI applications. I’ll continue to share updates through channels such as my quarterly CEO blog, but I’ll be back in touch soon to announce something big on this topic. In the meantime, thank you to our community and customers for continuing to help us on our mission to empower the world to develop technology through collective knowledge.

NO please.

I do NOT want generative AI in the public SO. Also, is everyone at the company actually excited about this? I suppose maybe, but it seems like an exaggeration. Not going to name any names, but what about the previous statement (from a staff member, on MSE), that SE would "work internally progresses on identifying these posts and making our systems more resilient to issues like this in the future" (referring to ChatGPT-answers). That answer made me think SE was going to help communities deal with AI-answers. Can you clarify if that answer is still true?

Also, my answers on SE are intended for humans, not for training a machine learning model with.

As someone that has personally put in a large amount of effort and time into dealing with AI-generated answers on SE, I'm extremely disappointed to hear that you may incorporate generative AI models into SO.

4
  • 27
    "a third of the code being written on popular code repositories was authored by an AI assistant" Another problem with that is that amount of code is not a good quality metric. The two-thirds of code not being authored by an AI assistant are probably the more interesting parts. Commented Apr 17, 2023 at 20:48
  • 3
    for lazy readers like me, would you be okay with narrowing down the quoted sections? For some of the quoted sections, it's hard to tell when skimming what part of the quoted section you're responding to. If you want to keep it for context, that's fine though.
    – starball
    Commented Apr 17, 2023 at 20:57
  • 1
    @starball I've narrowed it down some, once I'm at my laptop again I'll see if I can trim the quotes any further (without removing the meaning). But yep, the quotes weren’t clear about what specifically I was referring to
    – cocomac
    Commented Apr 17, 2023 at 22:01
  • 2
    The third of code quote I saw by GitHub a while back.
    – Seggan
    Commented Apr 24, 2023 at 1:56
77

After noticing some worrisome exchanges about "SE is going to use generated content" in the chat rooms, I came here to verify those rumors and to determine concretely what the actual "plan" was.

Yet after reading the "announcement" multiple times... I don't yet know what the plan is. I can't see what you are talking about. I can't see what your ideas are. I can't see what you are going to do. Nor do other users apparently since most answers start with a "if you mean to...".

This announcement could be summarized as a haiku and no actual meaning would be lost.

GenAI is future
Many cool guys say that too
Let's jump the wagon

All I can see is that (as Zoe pointed out in a comment) this actually feels more like an "early announcement" than a "we want your feedback on our plan" post. If anything, because if you wanted actual feedback we would at least need to know WHAT the plan actually is.

Oh, yes, you then say that "soon" you will be telling us what the "big plan" is, but that would only mean that we can't have any question now, right? Also, does this "soon" mean "after we already committed to it, you will get to know when we have already decided and your feedback won't matter anymore"? What is the point of asking us to "leave any questions that you have below" NOW if we actually have to GUESS YOUR INTENT because you didn't tell us what this is all about???

tripleee speculated about three different scenarios in their post. For starters... I guess it would be nice to know if any of these actually fit what your are planning to do.

5
  • 15
    What we got: "I’ll continue to share updates through channels such as my quarterly CEO blog, but I’ll be back in touch soon to announce something big on this topic." it's a 🌈🧚✨ s u r p r i s e ✨🦄🍀!!
    – starball
    Commented Apr 18, 2023 at 8:15
  • 9
    « What is the point of asking question NOW». Well, because it’s not a question, but a statement. They don’t seem to want feedback. Commented Apr 18, 2023 at 8:19
  • 10
    In the absence of solid information, ChatGPT hallucinates. In the absence of solid information, MSE hallucinates. ;)
    – PM 2Ring
    Commented Apr 19, 2023 at 0:37
  • 3
    @PM2Ring: But even with the solid information of our feedback, MSE keeps hallucinating. Commented Apr 20, 2023 at 7:45
  • "this actually feels more like an "early announcement" than a "we want your feedback on our plan" post" 1st time? Commented Jun 12, 2023 at 18:04
61

Please do not integrate this algorithm into the site.

It plagiarizes

The algorithm will reproduce whole sections of one to one content without citation nor warning. It then strings these sections together with transitional words such as however, additionally, or furthermore.

It cannot generate

Aside from the issue of reproducing existing works without citing them, the algorithm cannot actually generate legitimate content which fails at its own primary goal. Instead, any time the algorithm generates content, it simply goes with the most likely to occur series of content; as opposed to the correct series of content. This is dangerous at worst, and "hallucinating" at best.

It acts as an authority

The algorithm presents all of these flaws, the issues with citation and of lying hallucinating, as if they are canon; as if each statement, code, or demonstration were 100% accurate and trustworthy. Integrating something which presents content in such a manner is negligent for a place aimed at being a repository of knowledge.


There is no need to integrate an algorithm which has these traits. It is dangerous, negligent, and ethically questionable.

20
  • 6
    You are assuming that they will include the current version of ChatGPT as-is, which is likely not the case. There is a lot of ongoing research into how to solve the issues you have brought up. Perplexity AI comes to mind, which cites its sources and is thus (in theory) no worse than human-generated content. Commented Apr 18, 2023 at 3:47
  • 12
    GPT is a lossy compressor: the compression ratio for GPT-3 is ~50, which is the ratio of the size of the training data to the size of the parameters. (We don't have the numbers for vanilla ChatGPT or GPT-4). I agree that it doesn't generate, it stochastically re-generates. And it only chooses the most likely token when its temperature parameter is zero (and there's only 1 most likely token). But the temperature is usually set to a non-zero value, which allows it to produce more "interesting" sequences.
    – PM 2Ring
    Commented Apr 18, 2023 at 3:56
  • 11
    @TheGuywithTheHat - It is all based on the same algorithm though, the transformer algorithm: "a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely". This algorithm leans more towards simply producing existing language as opposed to learning concepts and applying them; it is also entirely the basis of today's "chatbots" which all suffer the same drawbacks.
    – Travis J
    Commented Apr 18, 2023 at 4:01
  • 6
    @TravisJ I am aware of how they work, my career is in deep learning and I have used GPT as early as 2019. A transformer, while technically an "algorithm", is not the algorithm. A transformer or transformer-based model is only a single part of the entire system that creates and uses the LLM. While practically all LLMs are currently based on transformers, differences in how they are trained and used can give vastly different results. It is entirely feasible for Stack Overflow to develop a cross between ChatGPT and Google, with most of the benefits and few of the drawbacks of each of them. Commented Apr 18, 2023 at 6:30
  • 2
    @TheGuywithTheHat - The training will only effect what it can reproduce, and the use will only effect the style of reproduction. However, it is still just that, reproduction. There is no creation, no error checking, and that is by design. Too many corners were cut to create this in a narrow scope, and tweaking minor aspects of the design won't alter the core functionality. It needs to be able to dog food its own productions, and that gets back into more complicated Neural Networks and away from this Transformer algorithm.
    – Travis J
    Commented Apr 18, 2023 at 6:39
  • 5
    There are a lot of assumptions in your comment. I think that MSE comments are not the place for an in-depth discussion. "it is still just [reproduction]" -- Does that matter? "There is no creation" -- It depends on how you define creation, and the model's output is still equally useful regardless of how you define the word. "no error checking" -- ChatGPT does not currently have error checking, but there is no fundamental reason that error checking cannot be built into a system with an LLM. "and that is by design" -- for current ChatGPT, yes, but not necessarily for other/future LLMs. Commented Apr 18, 2023 at 7:55
  • 4
    Assumptions? Perhaps your bias, unintended or not, may be at play here. It absolutely does matter when it is purely reproduction but does not cite the source. That is the core issue of plagiarism. I feel like I am just repeating my points. Without error checking, combining multiple aspects of fact for a wild assertion (its version of producing, not necessarily creating) is a very large problem with the outputs being given. Again, yes, this is all by design.
    – Travis J
    Commented Apr 18, 2023 at 16:22
  • 2
    @TravisJ Let me summarize: You are correct about the current state of ChatGPT. You are not correct about LLM systems in general. Commented Apr 18, 2023 at 17:59
  • 3
    @TravisJ GPT is absolutely a real neural net; there's no fundamental reason that transformers are any less "real" than other types of NNs. Additionally, even GPT-3 uses more than just a basic attention layer. Additionally, OpenAI hasn't released the model architecture for GPT-4, so we don't know what is used in the latest model. Additionally, even if we assume that the model is "dumb", there's no reason that it can't be included as part of a larger system that does do explicit error checking, citing sources, and other "smart" things. Commented Apr 18, 2023 at 19:09
  • 3
    I don't know why you striked out the word "lying". Because that's exactly what ChatGPT has been trained to do: It was trained to get the thumbs up from non-*experts, and generating an answer that *appears to be knowledgeable while being totally fabricated, is an excellent way to get those thumbs up from the non-experts. And that turned ChatGPT into a quite eloquent impostor. Commented Apr 19, 2023 at 8:14
  • 3
    @JonathanReez - 100% confident. Yes, I have tried it. It is wholly inaccurate for things of a delicate nature. It gets simple things wrong, for example the list of basketball players who have scored more than 50 games; in this list it shows the totals not only incorrectly but in an incorrect order. It gets complex things wrong, such as fuel to air ratios; it cannot comprehend what a fuel to air ratio is, and instead will use the most common or the highest page ranked response. However, that breaks down in exact situations.
    – Travis J
    Commented Apr 26, 2023 at 15:56
  • 3
    @JonathanReez - Simply put, it cannot math. Anything that requires more than a simple math function will break down, and unit conversion is often off by orders of magnitude. It just doesn't know what it is looking at, someone literally took out the logic circuit for these implementations and replaced it with a statistic weighting circuit. Best guess works well when you are looking up definitions for basic information in a somewhat unfamiliar area. It breaks down when exact answers are obvious from expert perspectives.
    – Travis J
    Commented Apr 26, 2023 at 15:59
  • 2
    @JonathanReez Chat-GPT cannot write code. The internal grammar that it forms from training is only reproducible, there is no composition or comprehension. In general "RTFM" questions have been off topic for a long time at Stack Overflow, and as a result the lookup type of response that might fit just doesn't really exist as a niche. If you were to ask one of these implementations to create something unique, it would fail, albeit while looking like it succeeded... and insisting that it had succeeded. For example, creating html structure comes out unbalanced, and it will flat out lie about svg.
    – Travis J
    Commented Apr 26, 2023 at 16:08
  • 1
    As a result, of all of this, GPT-4 would more than likely fail to live up to competing in any sense on Stack Overflow, except in the space where it misleads people.
    – Travis J
    Commented Apr 26, 2023 at 16:09
  • 3
    I already answered that question, Stack Overflow will outperform 100% of the time. I will give a heavy however to this though. However, due to excessive content "curation" at Stack Overflow over the past 3 years, content has begun to become stale as the question asking rates plummet and the answer/question rate declines. There is a possible edge case where the GPT lookup finds some overlap in providing new definitions for areas which have become stale. This is not a facet of the chat bot so much as a failure of Stack Overflow to move towards content creation.
    – Travis J
    Commented Apr 26, 2023 at 16:14
56

Note that I've flagged ~800 GPT answers on Stack Overflow. I was honestly hoping for a bit more information about how Stack Exchange as a company (rather than the individual sites) plans to handle the deluge of unverified AI answers that all too often turn out to be hallucinations.

Instead, I ended up not quite sure what was being announced, even after reading it through twice. I do turn to ChatGPT myself for its ability to re-word and summarize information, so I decided to ask it to summarize the post to see if there was something I missed:

Prashanth Chandrasekar, CEO of Stack Overflow, believes that artificial intelligence (AI) will not replace programmers but rather will create millions of new software developers from fields like finance, education and art who previously didn't have access to AI-powered tools. Sophisticated chatbots, built on top of cutting-edge large language models, can already write functional code and answer complex queries about how to build apps, debug errors, and translate between different languages and frameworks in minutes. While AI provides a starting point and some initial momentum, humans do the additional research and learning to finish the job. AI will amplify existing programming skills and help programmers spend more time creating new ideas, researching new topics, and asking new questions that had not occurred to them before. Stack Overflow plans to add GenAI to its products, democratize knowledge about new AI technologies, and provide a community and solutions to the next generation of developers and technologists. (emphasis added)

This was really in line with my personal reading of the post. Since everything but the last sentence seems to me to be rather commonly accepted, I guess the "announcement" can be boiled down to the final sentence, which still leaves things quite open.

13
  • 4
    seems like it did a rather poor job of summarizing the post.
    – Kevin B
    Commented Apr 17, 2023 at 16:11
  • 14
    @KevinB It's pretty much what I got out of the post after reading it several times. Did you have a different takeaway? Commented Apr 17, 2023 at 16:13
  • 15
    "<Marketing Soup> we're adding GenAI to SO and SOfT <Marketing Soup>"
    – Kevin B
    Commented Apr 17, 2023 at 16:15
  • 7
    @KevinB Ah, so we did pretty much have the same takeaway, it seems ;-) Commented Apr 17, 2023 at 16:16
  • 1
    Just curious, how did you find all those 800 answers? You have some script? Commented Apr 17, 2023 at 16:36
  • 10
    @ShadowTheSpringWizard There's several techniques for identifying CGPT (including watching bountied questions, or searching for recurring phrases), but it does help that there's an absurd quantity. 800 posts was the daily volume at one point (and that wasn't even close to the peak volume, though I don't remember the numbers)
    – Zoe
    Commented Apr 17, 2023 at 17:56
  • 1
    @Zoe huh, and I thought I was spending lots of time on SE... :D Commented Apr 17, 2023 at 18:03
  • 2
    @ShadowTheSpringWizard To be clear, the numbers are for Stack Overflow only. I don't know what the numbers were network-wide. If you had visited and browsed new posts on SO daily, you too would've found a lot of CGPT, especially in December
    – Zoe
    Commented Apr 17, 2023 at 18:06
  • 7
    @ShadowTheSpringWizard While I'm aware of scripts to help detect, I use a far more "manual" method that I honestly would prefer stay "loosely guarded". At this point, I'm of the belief that we have some users (a small minority of the total GPT answers) that are actively working to evade detection, and I'd like to not give them any more ideas on how to do so ;-) Commented Apr 17, 2023 at 18:17
  • 5
    I'll throw out one of the more "obvious" methods, though, in hopes of at least slowing it down. When I do spot an answer that I might suspect is GPT-based, it's a dead giveaway if the user posted a detailed answer just 3 minutes after their previous (also GPT) answer :-). E.g. 20 long answers in 2 hours is pretty suspect. Commented Apr 17, 2023 at 18:22
  • 7
    Thanks for the ChatGPT summary, I guess. ;) But I'm still not clear on what "Community is the future of AI" is supposed to mean...
    – PM 2Ring
    Commented Apr 18, 2023 at 3:45
  • @NotTheDr01ds I see. One thing though, if one changes AI generated answer so it becomes something else, it might become valid. But that's really case-by-case thing. Commented Apr 18, 2023 at 7:41
  • ChatGPT is not well known for being concise.
    – NeilG
    Commented May 19, 2023 at 12:42
50

Incorporating generative AI technologies into the organizations using Stack Overflow for Teams will allow us to layer a conversational interface on top of this wealth of information. We believe this could lead to tremendous productivity gains.

I think it'd be more beneficial and productive to work on better search functionality, and better possible-duplicate detection in the Ask Question module

Generative AI is working with past information- probably taking from our past Q&A.

  1. If it can answer a new question, I'd have suspicions about the new-ness of the question.
  2. If it can't (likely when faced with a new question), then you should just leave it to your community of experts who actually know what they're talking about.

In either case, what's important on the asker's side is to search, and what would be more useful to them (and anyone else looking for possible duplicates of a question) is better search functionality.

I'm confused about whether this is coming to SO/SE or just SO for Teams

Incorporating generative AI technologies into the organizations using Stack Overflow for Teams will allow us to layer a conversational interface on top of this wealth of information.

This paragraph talks about integrating AI into SO for Teams and doesn't mention SO, but the next paragraph mentions both:

The example above is just one of many possible applications of GenAI to our Stack Overflow public platform and Stack Overflow for Teams.

So which is it? Are you just adding AI to SO for Teams? Or also to SO (/ the rest of the public Stack Exchange network)? I don't care much about SO for Teams. But if AI is coming to SO or the rest of the Stack Exchange sites, I really hope it's put in a completely separate area in each network site than regular Q&A. For example, I don't want to see AI-written answers under question posts.

Either that, or you've got to (not really, but it sure would be nice if you'd) talk with us and think real hard about

  • How to make sure bad AI-generated content is easy for people to vote down (I'm thinking of rep thresholds)

  • How that content is going to age over time. What would you do when the language model updates? What would you do with already-written answers with older language models?

  • What moderation burden is this going to put on the community and its moderators?


For the rest of my thoughts, see also my answers to


various other commentary:

We are excited about what we can bring to the fast moving arena of generative AI

The community has already been bringing it since inception. We write content licensed under CC-BY-SA. The language models are using our community-written content (whether the way they use it is allowed under CC-BY-SA, I don't know. I'm not a lawyer).

AI systems are, at their core, built upon the vast wealth of human knowledge and experiences. They learn by training on data - for example open-source code and Stack Overflow Q&A. It is precisely this symbiotic relationship between humans and AI that ensures the ongoing relevance of community-driven platforms like Stack Overflow.

So yes. Agreed :) and with the rest of that paragraph. Except that I wouldn't say "symbiotic". Maybe "parasitic" (with Stack Exchange being the host organism). Even when people "contribute" AI-generated answers to questions here, it's often a hallucination (and against the site rules because of the hallucinations and other reasons like moderation and curation burden).

Our thesis on community as the center of a safe, productive, and open future for AI also offers some exciting prospects for our business

Most of what was written up to that point was pretty grounded / concrete (at least- I thought so). But I don't really get this statement. It seems a bit wishy-washy to me. I suppose community is important in that that's where voting and editing comes in (curation), but I'd say the bigger source of value in what we do here is freely contribute expertise. The experts are (in my current view) the starting point of what makes Stack Exchange valuable. I don't think it's true to say that all the experts here are here for community. They might be attracted to here because of the standard of quality and name-recognition of the large network sites though (not sure what either of those things has to do directly with community). Or they might be here for magic internet points and/or to alleviate an inferiority complex. Regardless of what is central to the future of AI, what is really central to Stack Exchange (which is what I care about) is fact-based Q&A. To say community is central to Stack Exchange (not that anyone has said that here) could be misleading, since we're not a discussion platform and we don't do chit-chat (except in chat subdomain sites).

Side note: Another problem I've seen with ChatGPT answers on SO is that they often have a bunch of "buffer info"- info that may be useful, but really isn't necessary in the context of the question. One of the selling-points of Stack Exchange (see the tour page) is that we avoid noise.

10
  • 1
    "I don't want to see AI-written answers under question posts." Many of us fully agree with you. However, the CEO didn't quite say that AI would be used to generate answers. OTOH, he didn't tell us exactly what it will be used for, either.
    – PM 2Ring
    Commented Apr 18, 2023 at 4:12
  • 16
    IMHO, there is room for LLM-based tools on the network. Eg, a search tool with a conversational interface that can help find related questions and dupes. A more intelligent wizard to guide question creation. A tool that can rephrase questions and answers to eliminate typos and awkward grammar, although of course there's a risk that the LLM will alter the meaning, which may be difficult for the author to detect, especially if English isn't their native language, or they're using unfamiliar jargon.
    – PM 2Ring
    Commented Apr 18, 2023 at 4:12
  • 2
    LLMs are capable none of these things. All it could do is gaslight visitors.
    – chx
    Commented Apr 18, 2023 at 6:38
  • 4
    @chx LLMs can do useful processing of language, but you do need to prompt them appropriately, and be aware of their limitations. It also helps if you have some idea of how they work, and realise that they're primarily operating on syntax, with only a very tenuous grasp of semantics. I said more about that here. Unfortunately, many people get misled by GPT output, mistakingly thinking that it knows what it's talking about.
    – PM 2Ring
    Commented Apr 18, 2023 at 7:18
  • 2
    @PM2Ring most importantly, if you have no idea what the answer is, then the generated content can be helpful or dangerous. Thus you also need to be at least generally aware of what the answer is. At least for technical topics. You don't need to if you're trying to generate ideas for a story or something.
    – VLAZ
    Commented Apr 18, 2023 at 7:20
  • 2
    LLMs can do useful processing of language <= absolutely can not. They can provide a word which is most likely to follow previous words. Nothing about language there. Finding meaning in the word salad it spouts is apophenia, nothing more.
    – chx
    Commented Apr 18, 2023 at 7:26
  • 5
    @VLAZ Sure. Using GPT to actually answer technical questions is madness. It'd be safer to get financial advice from the Wolf of Wall Street. ;) And if I were a programming teacher, I'd forbid my students from looking at the code it regurgitates. I learned to be a good coder by studying the code of excellent programmers. ChatGPT has read a lot of code, both good and bad, but since it doesn't really know what it's doing, we have to classify its behaviour as cargo-cult coding.
    – PM 2Ring
    Commented Apr 18, 2023 at 7:30
  • 7
    @chx Here's a crude analogy. ChatGPT is like a person doing a jigsaw puzzle. You give it a prompt of a few joined pieces and it adds more pieces, one by one. The picture on the jigsaw consists of a bunch of words written in English, and the jigsaw pieces are cut so that when you join them together the result will be syntactically correct English. However, the guy doing the puzzle doesn't understand English. When we see the result, it looks impressive because we understand English, but ChatGPT just knows how to stick bits of jigsaw together.
    – PM 2Ring
    Commented Apr 18, 2023 at 7:34
  • 3
    FWIW, I posted that analogy in this chat conversation: chat.stackexchange.com/transcript/message/63357211#63357211
    – PM 2Ring
    Commented Apr 18, 2023 at 7:36
  • If you go back to the origins of Stack Overflow, one of the founding principles is the reduction of noise. It was the founders firm belief that noise is what eventually killed every good information site, and they were determined to reduce it by any means necessary. Answers with AI-generated noise would be the beginning of the end, if they're right. Commented Apr 23, 2023 at 19:11
43

Quite simply, if people want to use some kind of AI assistance to answer their questions quickly, then they can use those.

If they want an actual human, often expert, to answer their questions, they will log into Stack Overflow.

I don't understand what the benefits of mushing these two approaches together would be. Stack Overflow's unique selling point is that it (should be at least) expert humans answering questions using their personal experience. Why risk killing this to add something people can do on another site1?

1 I know the answer is $$$

2
  • 7
    Re "what the benefits of mushing these two approaches together would be": Perhaps they want the users back that were lost to ChatGPT (users who just want an answer ASAP, any answer, no matter the quality, human or AI-generated). They would count in the KPIs.. Commented Apr 18, 2023 at 22:31
  • 1
    From a brand perspective, it makes a lot of sense for the question "Where should I go to solve my programming problem?" to always be "Stack Overflow." It's not like it's completely impossible that this could turn out to be a useful and productive initiative. We just don't know yet (but I'm not holding my breath).
    – tripleee
    Commented Apr 19, 2023 at 17:41
36

Welcome to the enshittification of Stack Overflow!

No one at the helm is reading or cares about this page.

Make no mistake, within less than a year new visitors will get a first answer or at least a question rewording suggestion provided by a ChatGPT-alike.

13
  • 16
    Jon Skeet only wrote the first 1000 answers on C# himself. Afterwards he secretly trained a LLM on himself and let it run. Commented Apr 17, 2023 at 20:45
  • 1
    "new visitors will get a first answer or at least a question rewording suggestion provided by a ChatGPT-alike" This is not inherently a bad thing. It could end up being bad if SO doesn't implement it in a good way, but I don't think we should assume that yet. Commented Apr 17, 2023 at 20:48
  • 2
    That is an interesting, if quite long, analysis. For example, "Attention is like cryptocurrency: a worthless token that is only valuable to the extent that you can trick or coerce someone into parting with "fiat" currency in exchange for it." and "This is just what Twitter has done as part of its march to enshittification: thanks to its "monetization" changes, the majority of people who follow you will never see the things you post. I have ~500k followers on Twitter and my threads used to routinely get hundreds of thousands or even millions of reads. Today, it's hundreds..." Commented Apr 17, 2023 at 22:56
  • 8
    @TheGuywithTheHat It is 100% to be devastatingly bad. How many of these transitions have we suffered through to think it will be otherwise? Slashdot, digg, twitter, even Google, the list is endless.
    – chx
    Commented Apr 18, 2023 at 5:36
  • 1
    @chx Can you explain how either of those would be bad? In the first case, the user at least gets an initial answer. (They can follow up if it fails, and we can all laugh at the dumb AI.) In the second case, the community gets a better-formulated question. Of course SO might fumble this by instead treating moderators as unpaid filters for AI content, but that's not what your specific examples are about.
    – lofidevops
    Commented Apr 18, 2023 at 18:03
  • 6
    I commented on other answers multiple times: problem is, ChatGPT answers inherently have zero information value, it's a stochastic parrot but apophenia makes people think there is something, they are essentially gaslit. The outcome will be a massive influx of garbage and as time goes on, harder and harder to filter garbage.
    – chx
    Commented Apr 18, 2023 at 18:12
  • 1
    @chx Ok, so your concern is that AI generated garbage will in fact lead to more low quality questions along the lines of "AI told me to rm -rf and now it's broken"? I think that's a fair concern. I am hoping they will use asker-driven gamification rather than unpaid labor to solve it, I guess we'll see. The enshittification article is great btw. I am hoping the economics aren't the same here 😨
    – lofidevops
    Commented Apr 18, 2023 at 18:16
  • Also, apophenia! I think you've just explained humans to me.
    – lofidevops
    Commented Apr 18, 2023 at 18:18
  • 2
    @chx I disagree that the value of ChatGPT output is strictly zero. Yes, it's unreliable, and many people are mistaking its language competence as general intelligence, eg this OP who is convinced that ChatGPT is a pair programming partner, and not just a rubber duck that can speak. However, I also believe that LLM can be a functional component of a process if used properly, with awareness of its flaws and limitations.
    – PM 2Ring
    Commented Apr 19, 2023 at 1:58
  • @lofidevops no, that's your human thinking. I am thinking more of questions and answers written in garbagese, if you get what I mean. Just word salad. Answers containing code snippets which look valid but are not. And so on.
    – chx
    Commented Apr 19, 2023 at 2:22
  • @Trilarion How do you know this about Jon Skeet? Commented May 10, 2023 at 10:57
  • 1
    @SunderamDubey Sorry, I don't know this. I only hallucinated that. I guess that many questions are quite similar to each other and that one could indeed train an AI in a certain tag to basically repeat what has been said and kind of adapt to slightly different questions. But this was more meant in a friendly joking way. I rather guess that Jon Skeet writes all his answers anew every day. Commented May 10, 2023 at 12:25
  • @lofidevops I see you're not familiar with Quora.
    – mchid
    Commented Jul 13, 2023 at 14:20
32

The GenAI hype-train manages to (just about) clear the bar for usefulness set by its illustrious recent predecessors (web3.0 and metaverse). But regardless of its own merits, it's important to recognise that it is currently a hype-train.

Public interest is high, the news both tech and non-tech are excitedly reporting about it left and right and the average Joe is lapping that up. But this level of public giddiness isn't sustainable - there will be something else new and exciting that will come along soon and capture the public's interest and generative AI will continue but it won't be grabbing every third headline any more.

This isn't a new phenomenon - humans have been getting bored with the latest and greatest for a very long time, just ask NASA how quickly the general public lost interest in the moon-landings.

So if SO Inc is wanting to get in on this area - are they prepared for what happens in 3, 6, or 9 months when it's not headline news any more? Is this something that is genuinely in the best interests of the company and the network and would add real value without that hype? Because if the answer is no - then I'd question the wisdom of throwing any real resources at this, and instead cash in on the hype by writing a few blog posts, have some people discuss it on the podcast or whatever. Sure you probably lower your ceilings on potential rewards doing that - but you avoid throwing millions at something and having it turn out to be a boondoggle.

1
32

I must be a very unlucky person. These things that seem to sprout like flowers in other people's IT gardens just don't happen to me. Every question I ask the AI comes with errors at best, that's when the AI decides to be creative and starts spewing a bunch of information taken from I don't know where.

I'm not saying that AI is useless or anything like that. Far from it, I like AI and I like the promise of social progress that it brings to humanity but what I mean is that it's premature for us to have this type of integration in the community that is the foundation of activities directly and indirectly related to Information Technologies.

Asked about the possibility of its participation in the ChatGPT communities, ChatGPT responded:

My participation in the ChatGPT communities is limited to providing responses to your questions and engaging in conversation with you. As a language model, I am not capable of actively participating in the same way as humans do. However, I am here to assist you in any way I can and provide you with the information and insights you need.

I am convinced that the training of AI is currently not robust enough to meet the needs of our communities. An error made with the endorsement of AI has the potential to spread virally and be incorporated into various systems used around the world.

I believe that any AI system must be tested, improved, specialized, and retested before it is made available to our public, and that its responses should be finally curated by members of the Stack Exchange community.

Without being polite, I feel like they're saying to me:

-Okay, thank you for these years of unpaid work but we don't need you anymore. From now on, we'll have a little system that will work for free just like you, but with one difference, the system it's a bit imprecise and doesn't complain when it sees the mistakes we've made and in fact, it doesn't even see them.

5
  • it must be my mistake but I edited it and the system republished the answer Commented Apr 17, 2023 at 17:50
  • Don't feel guilty, this is something that commonly happens. Apparently you deleted the original post. While there is no rule about how handle this cases, some time is better to keep the oldest post instead of the newest.
    – Rubén
    Commented Apr 17, 2023 at 17:53
  • 12
    "I am convinced that the training of AI is currently not robust enough to meet the needs of our communities." It's not just the training. It's the design itself of the AI that limits it. No amount of training will fix the issues inherent in the model. Commented Apr 17, 2023 at 18:55
  • 3
    To be fair, any specialized AI creation from Stackoverflow Inc. would likely take some time to be fully operational. By then, the concern about the reliability of such automated systems could already have been lowered. Starting to work on that now would allow StackOverflow to gain important experience and get a stake in that market (whenever it will come to its full potential). Commented Apr 17, 2023 at 19:57
  • 15
    Re "I must be a very unlucky person": No, it only takes about 10 minutes to realise ChatGPT is a pathological liar when it comes to factual things. Or it gives very bad advice. Do keep the transcripts as evidence to prove that this is actually the case. But it can be put to good use (e.g., to generate ideas) Commented Apr 17, 2023 at 22:33
32

Please, no. AI is not remotely close to any real "artificial intelligence" (in the literal sense) these days, and it does not belong as a contributor on SO/SE sites in any shape or form at the current time, unless you mean to recast the sites as training grounds for future generations of GPTs. If that's the choice, I expect such a change of direction to alienate and drive away many of the longtime good-faith contributors, myself included.

IMHO the very suggestion of integrating "generative AI technologies" shows a huge disconnect between sweet dreams of (future) AIs helping make everything better vs. the reality of (current) AIs being mistakenly trusted to provide the answers, which are often wrong, but may also be deceivingly plausible many times.

We've seen this on Math SE already, where users have been quoting complete nonsense written by AIs with much nonchalance and self-assurance, then wondering whether they were "missing something".

(A random example from the last hour is here, which was deleted since, but still visible to 10k+ rep users. The AI confidently stated that "two of these n numbers must have the same parity, which means that one of them must divide the other". It takes very little math knowledge to realize how hilariously nonsensical that assertion is. Of course, humans can and do make silly mistakes, too. Difference, however, is that most of us refrain from posting every one of them out of a sense of propriety, and, anyway, we would not be able to generate them at the rate of hundreds per minute even if we wanted to for whatever trolling reason.)

To end on the same note I started off with, AI is not remotely close to any real "artificial intelligence" (in the literal sense) these days. The gap can be closed in one of two ways. Either keep making AI smarter until it gets close. Or, dumb down the collective "intelligence" until it matches today's GPTs. Unfortunately, SO/SE embracing generative AI now looks like it's leaning towards the latter. Please, no.

3
  • 4
    just curious- did you see the link that got added to the top of the question post?
    – starball
    Commented Apr 20, 2023 at 6:28
  • 3
    @starball Yes, but that's narrowly framed to "better question titles". The scope of the post here is far wider. Also, one example does not make a rule. Just curious, did you see the link in my answer?
    – dxiv
    Commented Apr 20, 2023 at 6:39
  • 1
    can't view it (no 10k on Mathematics). I saw the quote though.
    – starball
    Commented Apr 20, 2023 at 6:43
24

Let's get something straight: this is just another big-tech (or, in this case, big-tech wannabe) jumping on the bandwagon. And jumping on the bandwagon usually doesn't lead to good long-term decisions.

A CEO, who as far as we know has never posted a question on Stack Overflow, should not be the one making the call on whether or not we like AI on our sites. I have no problem with Stack Overflow having a CEO or making a profit, but corporate has absolutely no standing to say how to better run our sites. The community's decision to ban ChatGPT has become the most upvoted meta post ever by far. Why does this one hotshot tech CEO get to override all of our voices?

Even if we ignore all the practical obstacles other posts have pointed out, the very idea that the CEO would put out a statement that clearly contradicts the sentiment of everyone on this network is a straight-up insult to the whole network. Not to mention this guy has repeatedly bragged about how "democratic" Stack Overflow is. For god's sake, just let the people who know the site run the site.

7
  • 8
    "Why does this one [...] CEO get to override all of our voices?" because... they're the CEO? not that that fact enthralls me, but it is what it is.
    – starball
    Commented Apr 21, 2023 at 4:25
  • Related on the point of long-term, see also motosubatsu's answer post (my answer post also briefly touches on that)
    – starball
    Commented Apr 21, 2023 at 4:28
  • 6
    I had a hard long think about this - and it feels like its more about the person - than the issue at hand. And as tempting as it is to want to lay the blame at the feet of one person - both good and bad is generally a team effort. Also looking at the actual deliverables, its less "chatGPT" and more integrating these tools into the normal workflow of the site. The blog is terrible but I don't feel that - looking at the link at the head of the question now, it actually represents what SE wants to do very well. Commented Apr 21, 2023 at 5:43
  • "just let the people who know the site run the site" This site is privately owned so this is simply impossible. But you could help doing something like this. There are community driven Q&A variants out there in the wild or you could help building one to your liking. However with StackOverflow, what you envision, will likely not happen. Commented Apr 21, 2023 at 6:57
  • 2
    I'd argue that the skills needed to use the site, the skills needed to run the site and the skills needed to run the company are not the same. While I do push for more hiring in the community where the opportunity presents itself - and feel the company can do better, I'd rather judge the CEO by the choices they make than how much reputation they have here Commented Apr 23, 2023 at 1:01
  • SO Inc has de facto & de jure "standing" because they own the site. They might not have "competence". Please get what you are saying correct & clear.
    – philipxy
    Commented Apr 29, 2023 at 11:53
  • 1
    "should not be the one making the call on whether or not we like AI on our sites." - We need to abandon this proprietary software and start over, so that "our sites" are actually ours, rather than being owned, controlled and operated by a company with incentives misaligned to ours. Wikipedia can only be what it is because of the non-profit Wikimedia Foundation (and even then, there are a lot of problems). Commented May 3, 2023 at 1:54
22

I agree with the general consensus that ChatGPT in its current state can be dangerous when used incorrectly (which is especially likely with inexperienced coders). However, when used correctly, it could be helpful.

Grounding our responses in the knowledge base of over 50 million asked and answered questions on Stack Overflow (and proprietary knowledge within Stack Overflow for Teams) helps users to understand the provenance of the code they hope to use.

This is the key point that I think a lot of people here are missing. LLMs could point users to existing answers, which doesn't give it an opportunity to hallucinate information. They can also easily solve simple issues and provide tailored guidance for question askers.

Many ChatGPT answers are not easy to verify, but that doesn't mean that all of them are like that. If I make a simple mistake such as forgetting a return statement, ChatGPT can see that and point it out to me. I see the potential for a system where an LLM can respond to particularly simple questions before they are even posted, reducing the flood of low-quality questions.

There are currently LLM solutions that aim for trustworthy answers that cite sources (perplexity.ai comes to mind). These are of course only as good as the training data, but that training data is by definition as reliable as the answers given by humans. In this way, it could be used as a smarter search engine that is customized for Stack Overflow.

Another safe way to integrate LLMs into SO is by providing question-asking guidance to new users. It could be possible for an LLM to say, for example, "Hey, it looks like you are encountering an error, but you haven't included a specific error message in your question. If you haven't already, could you copy-paste the error you're getting directly into your question?"

Overall I don't think it will be easy to use LLMs/GenAI in a non-disruptive and safe way, but I am cautiously optimistic that it will be possible to do so. One thing to keep in mind is the self-driving car evaluation problem: do we aim for perfection, or do we just aim for being better than the average humans who are currently doing the task? Perfection is ideal, but better-than-human performance is still worth using.

16
  • 8
    "It could be possible for an LLM to say, for example, "Hey, it looks like you are encountering an error, but you haven't included a specific error message in your question. If you haven't already, could you copy-paste the error you're getting directly into your question?"" Sounds like a very useful application. But I'm not sure it is on top of the priority list. Commented Apr 17, 2023 at 20:01
  • 4
    @Trilarion Sure, that's why I said multiple times that there are good and bad ways that GenAI could be used, but never speculated on exactly how SO will end up using it. Commented Apr 17, 2023 at 20:13
  • 4
    Given all this garbage is capable of is spitting out the next words most likely to follow, @Trilarion's thinking is just wishful thinking. There is nothing and as long as there is not a fundamental change in AI, there will be nothing. Seeing any value or meaning in the output is apophenia. Understandable but still.
    – chx
    Commented Apr 18, 2023 at 6:37
  • 1
    @chx It doesn't matter if the text on your screen was written by a human or an AI, the content is still equally correct or incorrect regardless of its source. If we could train an AI to correctly predict the exact word that I am going to use next, that word is then equivalent to that of a human. There are currently shortcomings with the reliability of LLMs, but they are not insurmountable. Commented Apr 18, 2023 at 7:41
  • 3
    @chx I mostly agree, but not fully. It's not totally apophenia / Eliza effect. The network does manage to catch some conceptual / semantic structure as it's learning syntactic structures. GPT-4 is even more impressive than ChatGPT (3.5); OTOH, it's more dangerous because it can be harder to detect when it's uttering nonsense.
    – PM 2Ring
    Commented Apr 18, 2023 at 7:52
  • 4
    It doesn't matter if the text on your screen was written by a human or an AI, the content is still equally correct or incorrect regardless of its source....A Human make errors, and I would be surprised if you never saw any error on any answers on SO/SE before. But at the very least, a Human know how to understand and discern. Based on what you're saying, a simple markov chain model is basically at the same level intellectually as everyone on SO. Commented Apr 18, 2023 at 9:21
  • 1
    "it can be harder to detect when it's uttering nonsense" It is not. What it always utters is neither nonsense nor correct: it has zero information value. Putting information value to it , judging whether it's correct is incorrect is utterly pointless. The best thing you can do is ignore. Might as well listen to your parrot at home, that's what this is: a stochastic parrot.
    – chx
    Commented Apr 18, 2023 at 9:21
  • 1
    @NordTheStarWizard "a simple markov chain model is basically at the same level" -- This is a massive straw man. LLMs are way way way more complex than a markov chain. You correctly point out that humans make mistakes, so there's no need for perfection for LLMs to surpass humans in Q&A, especially if we confine it to only simple questions. There already exist AIs that can provide sources for their claims, and AIs that you can ask to double-check their own work. There's no reason that we can't make an AI that will search additional sources if it is not yet confident enough in its answer. Commented Apr 18, 2023 at 9:39
  • 1
    @chx Does SO answers written by humans have value? What if we search google and get snippets of those answers? What if we make an AI that takes the most relevant snippets and concatenates them into a single paragraph? What if the AI then processes the snippets to make them flow smoothly from one to the other? What if we then streamline the process so that the AI does those last two steps simultaneously? It seems like you are arguing that the original answers have value, but the answer produced by the AI has no value. At what point does it lose value? Commented Apr 18, 2023 at 9:57
  • 1
    You never mentioned complexity being an issue, though? You only said, "It doesn't matter if the text on your screen was written by a human or an AI, the content is still equally correct or incorrect regardless of its source.". Given even non-complex algorithms such as the "AI" in games, or even markov chain could count here. It does not matter if it's made by a Human, so this is just me taking literally what you said. Also, I never said that LLMs making "mistakes" (which would count as giving them some form of intelligence, which they do not) would be on the same level as Humans. Commented Apr 18, 2023 at 10:03
  • (cont2) We understand and can infer context without always needing to look things up or having a large dataset. Context understanding is only relevant to Humans. chatbot might give the impression it "understand", or even infer context based on prompt and the statistical properties of it's dataset, but that isn't the same thing. Are you truly sure statistical theory/properties are being used in Humans brain? Commented Apr 18, 2023 at 10:20
  • 1
    @NordTheStarWizard If you want to argue semantics, let's say that I said "LLMs are at a way higher level than markov chains." However, I never claimed that markov chains could not generate a correct answer. I think that they could generate one, it's just unlikely. In comparison, an LLM is much more likely to generate a correct answer, and I see no reason that the probability of an LLM giving a correct answer couldn't exceed the probability of a human giving a correct answer. My point about mistakes is that it doesn't matter whether an AI can correct its mistakes if it makes very few of them. Commented Apr 18, 2023 at 10:29
  • 1
    @NordTheStarWizard I am also not claiming that LLM definitely will exceed humans, I'm merely claiming that it is within the realm of possibility. We should not yet assume that it is impossible, and should instead try to see if it is possible. I am not sure that "statistical theory/properties are being used in Humans brain," but I am not sure that that is not the case. If it is, then an AI would be able to emulate it. Even if it is not, then it could still be possible for an AI to simulate it. It is difficult to prove a negative, so for now we should make an attempt to prove the positive. Commented Apr 18, 2023 at 10:36
  • 16
    This pretty much aligns with my thoughts on it: perhaps the biggest chore on SO for the past decade or so has been pointing folks to answers that already exist - a task a LLM could in theory do well enough. Not as well as a SME, but perhaps significantly better than a random person using search, and more importantly tirelessly and without complaint. And using computers for soul-deadening chores is always a good use of their time.
    – Shog9
    Commented Apr 18, 2023 at 13:33
  • 2
    @Shog9: Yes, a LLM could be very good at duplicate detection and also determining which question-asking-advice is most applicable to a particular post. Neither of those involve any generative component. Let it identify the most relevant existing knowledge and then not paraphrase it, but link and reproduce it (attribution included).
    – Ben Voigt
    Commented Apr 23, 2023 at 2:04
18

Let me try to synthesize an analysis of some possible interpretations and my (and I expect, the broader community's) attitude to these variations.

  • "Yikes, no!"

    Interpretation: "We will start polluting the site with ChatGPT content based on its current capabilities, and with our usual rigorous testing and interaction with the broader user community" (that's sarcasm for "quickly, surprisingly, with bad timing, and without proper thought or coordination").

  • "Mmmmmmaybe..."

    Interpretation: "We want to jump on the current hype bandwagon and spend months on an expensive project which will probably ultimately be cancelled. But if it's successful, we expect to see some sort of next-gen AI which actually doesn't try to answer when it doesn't know, which could really boost the site's capabilities and your user experience. For the time being, though, don't be nervous; this is just something we put out to hopefully instill some new market confidence in our stock evaluation." (The last sentence only illustrates how bad I am at market-speak, I'm afraid. I hope you get the idea.)

  • "Yes! You're on the right track!"

    Interpretation: "Our focus is on reducing the number of badly articulated beginner questions by answering them before they are actually published on the site. We have a proof of concept for a new question model which shows some promise already."

9
  • 4
    Related to "Yes!": The Guy With The Hat's post, PM 2Ring's comment
    – starball
    Commented Apr 18, 2023 at 6:03
  • 1
    We should be reducing the number of badly articulated beginner questions by refusing to publish them on the site until they have been reviewed by humans from a more restricted pool (at least, "logged-in users" rather than the general public; but probably ones meeting some reputation threshold). Commented Apr 18, 2023 at 11:22
  • 2
    Given how divergent those interpretations are, does the post actually tell us anything substantial? Commented Apr 18, 2023 at 11:44
  • @KarlKnechtel Not sure if that’s scalable. The review queues are already quite full. I just responded to a complete beginner question (badly written too) from somebody with 7000 reputation, so we can’t guard by reputation either (unless you’re prepared to let a lot slip through). Commented Apr 18, 2023 at 12:48
  • 5
    @user3840170 see my post above, that is EXACTLY my point. The original post tell us nothing, yet actually ask us for feedback and questions about that very same "nothing". That is why tripleee had to write down three totally divergent scenarios only to be able to post something significant.
    – SPArcheon
    Commented Apr 18, 2023 at 13:14
  • 3
    I presume the third one is entirely sarcastic. It will never happen given the current KPIs of the company (quantity) (or be amended with "...or the number of users helped by the Stack Overflow Generative AI Help Wizard (SOGAHW)"). Commented Apr 18, 2023 at 13:31
  • 1
    I mean, they did try some ML with their suggested questions experiment in questions. It went about as well as you'd expect.
    – Makoto
    Commented Apr 18, 2023 at 22:46
  • 1
    @Andreasdetestscensorship My position is that the large majority of already existing questions on Stack Overflow do not meet standards already, so I do not care in the slightest about whether policies for admitting new questions "scale". Just making those closures is on the order of person-centuries of technical debt, without even trying to take into account any administrative overhead. The only new questions at this point should be the artificial canonicals needed to clean up the existing mess, frankly. Commented May 3, 2023 at 2:02
  • 1
    @KarlKnechtel Effectively turning off the site is going to reduce its appeal, though. But it's true that we are increasingly unlikely to find any gold amongst all the gravel that new users post.
    – tripleee
    Commented May 3, 2023 at 4:14
16

There are several things that large language models and their ilk can be useful for in the context of Stack Exchange. Let me list some of them:

  • Recommending titles for the questions. Humans are very bad summarizing content in terms that are relevant. I'm sure anyone can dig up several examples and discussions about this. Language models are good at crunching big texts and summarizing it.
  • In the same vein, it could be used for making suggestions about the content to add information. The prompt would be something like "what information this question lacks that there are comments in similar questions asking for?". Basically, offloading the work of readers to the machine so it reduces the friction in the amount of comments left on the question, the need to close questions because they lack information, and the user perceiving user interactions as hostile.
  • Summarizing duplicates. We've tried this before by suggesting related questions titles with not much success (measured by the number of users that either alter their question or decides not to ask it). Instead of that, something akin to "of these related questions, summarize the 3 top positively scored answers, and include a summary of the question itself, asking if it's what they are trying to do". Obviously, improving the way to detect duplicates would help here, to make sure that semantic equivalence is maintained.

I would be very sad if those three branches of research aren't explored.

15

This is less about the content of the blog post and more about what SE should do from preventing this from backfiring. I'm a little surprised that nobody's explicitly put this out yet, considering that there are...28 answers and counting, but please: listen to the community.

Yes, the CEO has some sort of vision about wedding SE with AI in some sort of warped matrimony. And perhaps there's merit to it, or at least to parts of it (In my opinion, there isn't, but what do I know?). But the overwhelming consensus here in the answers, comments, and votes is that no, we don't want it, certainly not what's been implied in this vague-but-foreboding blog post.

SE's biggest failures have been when they've just steamrolled ahead with a plan without seeking input from the community, or when they get input from the community and proceed to ignore it. But some of their best moments are when they ask for our feedback, acknowledge our feedback, and respond to it by, at the very minimum, factoring it into the decision-making.

So whatever you do, please do it gradually, please do it thoughtfully, and please let us have a voice in some of the decisions. And please do it having fully considered the effects that this would have on the community.

1
14

Things I look forward to (then a question)

1. Finding citeable answers faster with ML guidance

With a ML-driven chatbot, or similar interface, I can imagine it will be easier to find human-written answers without actually posting a question. The image below is a screenshot from an ML-driven search engine (not shown are the citations below the answer, probably the most valuable part):

An ML-driven search engine answers the question "What is a src layout in Python?"

2. Writing better questions with ML guidance

I can easily imagine the Ask Wizard evolving to help newcomers and old-timers alike improve the quality of their questions. Including:

  • answering them without posting a question (see above)
  • helping them find resources that might help them answer their own question

(Obviously I could use ML tools to do these today, no integration necessary.)

3. Generate hypothetical answers I can test, approve and then publish

I'm most skeptical of this one, but with the right gamification it might be possible. (Harsh penalties for anyone publishing nonsense.)

4. Continue to have the Stack Overflow data dump publicly available under BY-SA 4.0

(I assume this still exists, I sometimes miss the news.) This public data dump of the community's hard work is a social asset. I would love to see SO release it as an LLM model compatible with Dolly 2.0 or whatever the open format ends up being. Failing that, keep the data openly and publicly available so that others can try is a must.

(Obviously this doesn't apply to Stack Overflow for Teams data.)

My question

Am I being overly optimistic to think that these are the kinds of ideas Stack Overflow is considering, rather than generating work for volunteer moderators?

Follow-up

Journeyman Geek points to Phillipe's response in his answer (emphasis mine):

... We are working closely with community members throughout this process to experiment with GenAI to build solutions to solve for historical pain points with the site. ... Leveraging GenAI to give newer community members real-time feedback on asking questions that are appropriate on Stack Overflow might reduce the load on community members.

And in Phillipe's post "An example of a generative AI tool: Creating better question titles":

We know that title creation is difficult for new askers, and this [ML powered process would be] a low-friction way to make that process easier for both askers and moderators alike.

So I think Stack Overflow are indeed planning labor-reducing tools. Whether they implement my brilliant ideas is another story ;)

3
  • 9
    “Am I being overly optimistic to think that these are the kinds of ideas Stack Overflow is considering, rather than generating work for volunteer moderators?” — Yes. Commented Apr 18, 2023 at 18:43
  • 6
    I mean... i wouldn't be surprised if these kinds of things are what they're prophesizing, I would however be surprised if they were able to deliver.
    – Kevin B
    Commented Apr 18, 2023 at 18:49
  • 5
    If you see my answer - I think Phillipe covers some of that in his edit Commented Apr 18, 2023 at 22:52
14

And just who is supposed to review and moderate all of the crap AI content posted by users farming for no-effort reputation, in order to weed out the worryingly low percentage of correct answers from the incorrect ones? Perhaps the CEO, who as far as I'm aware has never actually curated a single post on the site, is somewhat unaware of the (free) labor shortage issue he has in this regards. We're already hurting for reviewers. It often takes a subject matter familiar/expert to spot that an AI-generated post is indeed factually completely wrong. It can easily be argued that we're already overwhelmed on some reviewing fronts (and that this was perhaps the case even before the flood of incorrect AI-generated content started coming in).

Also, what about Stack Overflow's reputation? As word gets out that more and more answers on the site are just plain wrong despite being so deceptively well formatted and well written, this site's value will tank and it may not ever recover. I don't know how you get your reputation back from that disaster.

Any good executive always has the long-term interests of the company as priority. I think the CEO needs to explain how allowing a bulge of untested, far-too-often incorrect, information on this site will benefit the community or the corporation in the long term or even the short term.

2
  • 5
    What's the connection between the CEOs plans for generative AI tools and "crap AI content posted by users farming for no-effort reputation"? Admittedly, that blog article is a bit vague, but the follow-up responses from Phillipe clearly state that the goal is to provide tools that help to preserve (and possibly improve) the quality of content on the site(s), not to compromise it. There is no plan to permit inept use of ChatGPT etc by clueless rep farmers.
    – PM 2Ring
    Commented Apr 21, 2023 at 17:32
  • 2
    Re "this was perhaps the case even before the flood of incorrect AI-generated content": Yes, many late answers on Stack Overflow are completely bogus (the result of blindly using a search engine and blindly pasting the result into the answer box, without understanding any of it). It just isn't widely recognised (and more proof is needed). Thus code dumps and plagiarised content. Commented Apr 28, 2023 at 18:32
10

If new AI systems make it possible to create software simply by chatting with a computer, my prediction is that, far from the job of programmer disappearing, we’ll end up with millions of new software developers, as workers from fields like finance, education, and art begin making use of AI-powered tools that were previously inaccessible to them.

Tell me you have never trained a junior developer without telling me you have never trained a junior developer. Even then, a junior developer is someone who (hopefully) wants to be writing software, nevermind individuals with little-to-no technical aptitude. This pie-in-the-sky fantasy that non-technical people will be able to chat with an LLM only about their domain knowledge and end up with a usable, novel, or deployable application is misguided at best.

Prashanth implies that people from non-technical backgrounds will start using "AI" to write code. To make my point as concise as possible: LLMs will not enable millions of non-developers to suddenly be capable of writing software. Reason I bring up junior devs, which are far more likely to have either an aptitude or interest in software development, still require a great deal of training to understand delivering software beginning to end. LLMs may prove helpful in bringing juniors up to speed and augmenting people already in the field, not workers that may prefer painting.

0
9
+100

I have just one simple question, yes or no, but would like to get honest answer even if not pleasant.

Were the recent layoffs part of this future project(s) the blog post mentions? It would make sense, as such projects need lots of money, and business can't afford leaking too much money without compensating it.

8

Since there's a few complaints over the writing style of the blog post - so I made an attempt to 'translate' this to something a little more accessible. This is a 'classic' community wiki - I may miss things or misinterpret things, so feel free to add, subtract or correct my interpretation of this. I feel rather than a literal interpretation between the 'business'/'management' writing style - annotations make more sense.

I'm quoting, but not strictly following the paragraph structure Prasanth had originally, for ease of reference. There's also some bits where I feel like it’s more about the 'intent' than the exact wording - these are in the form of 'editorial' notes.

It would be nice to whip this into shape as a full blown, stand-alone document, but there's a bit more work for that.

Throughout history, great thinkers have made predictions about how new technology would reshape the way in which humans work and live. With every paradigm shift, some jobs grow, some change, and some are lost. John Maynard Keynes wrote in 1930 that new technology meant humans would be working 30 hours a week or less, and that the main challenge would be what to do with all our free time. So far, predictions of this nature haven’t exactly come true. As new technology empowers us, we push ourselves to new heights and reach for previously unattainable goals.

A bit of rough background of how the effects of technology was expected to affect work, and where such predictions have fallen short, ending with how technology could enable reaching more difficult goals.

Over nearly 15 years, Stack Overflow has built the largest online community for coders to exchange knowledge, a place where anyone with an internet connection can ask or answer questions, free of charge, and learn from their peers. Stack Overflow for Teams, our enterprise SaaS product, is trusted by over 15,000 organizations to serve as their internal knowledge bases.

(ed: A brief introduction to Stack Overflow and Stack Overflow for Teams)

With the recent advent of dramatically improved artificial intelligence, many industries are wondering how technologies like ChatGPT will change their business. For software development, the answer seems more immediate than most. Even before the latest wave of AI, a third of the code being written on popular code repositories was authored by an AI assistant.

Today, sophisticated chatbots, built on top of cutting edge large language models (LLM), can write functional code for a website based on nothing more than a photo of a rough sketch drawn on a napkin. They can answer complex queries about how to build apps, help users to debug errors, and translate between different languages and frameworks in minutes.

Essentially the thesis here - that LLM/AI tools have had an effect on software development, with significant amounts of code written by AI assistance. There's also chatbots that can translate various human inputs to code and answer questions.

At Stack Overflow, we’ve had to sit down and ask ourselves some hard questions. What role do we have in the software community when users can ask a chatbot for help as easily as they can another person? How can our business adapt so that we continue to empower technologists to learn, share, and grow?

This seems to reflect some self doubt - with Stack Overflow questioning if it has a place where a chatbot can answer questions, and if change is needed for the business to remain relevant

It’s worth reflecting on an important property of technological progress. The Jevons Paradox shows us that, as innovation allows us to do more, we settle on a new normal, moving the goal posts for what we expect of people and organizations, then competing to see who can find new ways to pull ahead of the pack. For knowledge work, as the cost of an action diminishes, we often do more of it. Abstracting away repetitive or tedious tasks frees technologists up to make new discoveries or progress innovation.

Jevon's Paradox - essentially, as work gets easier, there's more of it, not less in this case

If new AI systems make it possible to create software simply by chatting with a computer, my prediction is that, far from the job of programmer disappearing, we’ll end up with millions of new software developers, as workers from fields like finance, education, and art begin making use of AI-powered tools that were previously inaccessible to them.

These tools won't remove the need for software developers, but make it easier for folks who are not 'traditional' software developers to use these tools to perform tasks

We are enthusiastic about welcoming this next generation of developers and technologists, providing them with a community and with solutions, just as we have for the last 15 years. We’ve got a dedicated team working on adding GenAI to Stack Overflow and Stack Overflow for Teams and will have some exciting news to share this summer.

(Nothing to add here for the moment)

I’ve heard similar sentiments expressed recently by Microsoft founder Bill Gates, by Geoff Hinton, the godfather of the neural network approach that produced today’s AI revolution, and by Stephen Wolfram, a pioneer across computer science and mathematics. Each sees in today’s AI the potential for the loss of certain jobs, yes, but also, if history is a guide, a future in which a great variety of more highly skilled work becomes available to an even larger group of people. Just as tractors made farmers more productive, we believe these new generative AI tools are something all developers will need to use if they want to remain competitive. Given that, we want to help democratize knowledge about these new AI technologies, ensuring that they are accessible to all, so that no developers are left behind.

Fairly influential tech leaders believe that while AI could cause certain jobs to become obsolete, there's 'new' types of skilled work that is accessible to more people. There's a belief that generative AI tools are essential, and accessible to anyone who needs them.

I talk to developers of varying experience levels all of the time, and I’ve been hearing anecdotes of novice programmers building simple web apps with the help of AI. Most of these stories, however, don’t begin and end with an AI prompt. Rather, the AI provides a starting point and some initial momentum, and the human does additional research and learning to finish the job. The AI can debug some errors, but is stymied by others. It can suggest a good backend service, but often can’t solve all the points of friction that arise when integrating different services. And of course, when a problem is the result not of instructions from a machine, but human error, the best answers come from other people who have experienced the same issues.

AI tools can help someone get started on a project, but when it comes to errors, and more complicated problems it’s sometimes not very good. It’s still not a replacement for human experience - when people have had the same problems, they know how to resolve the problem better.

For more experienced programmers, AI will be an amplifier of their existing skill, making them more ambitious in their projects. The result, as Jevons would predict, is that they spend more time with AI, but also more time creating new ideas, researching new topics, and asking new questions that had not occurred to them before. They feel empowered to reach farther beyond their traditional skillset and to push the boundaries in terms of the kind of work they want to take on.

Rather than replacing more experienced developers - it might allow them to do more innovative things, and expand their abilities.

We are excited about what we can bring to the fast moving arena of generative AI. One problem with modern LLM systems is that they will provide incorrect answers with the same confidence as correct ones, and will “hallucinate” facts and figures if they feel it fits the pattern of the answer a user seeks. Grounding our responses in the knowledge base of over 50 million asked and answered questions on Stack Overflow (and proprietary knowledge within Stack Overflow for Teams) helps users to understand the provenance of the code they hope to use. We want to help coders stay in the flow state, allowing them to create with the latest tools with the confidence that they will be able to document and understand the provenance, source, and context of the code being generated.

Current LLM systems don't really differentiate between whether answers are correct or wrong and will make things up that look like they answer the questions a user has. By basing our responses on the data on Stack Overflow and Stack Overflow for Teams, they know what the model they are using is trained on. This would allow them to use these tools know what they were trained on (ed: "Flow State" is the current marketing slogan Stack Overflow seems fond of).

Community and reputation will also continue to be core to our efforts. If AI models are powerful because they were trained on open source or publicly available code, we want to craft models that reward the users who contribute and keep the knowledge base we all rely on open and growing, ensuring we remain the top destination for knowledge on new technologies in the future.

(ed: I'm not actually sure if models here means LLM model, or models of doing things, and it does feel like it’s talking about integrating)

AI systems are, at their core, built upon the vast wealth of human knowledge and experiences. They learn by training on data - for example open-source code and Stack Overflow Q&A. It is precisely this symbiotic relationship between humans and AI that ensures the ongoing relevance of community-driven platforms like Stack Overflow. Allowing AI models to train on the data developers have created over the years, but not sharing the data and learnings from those models with the public in return, would lead to a tragedy of the commons. It might be in the self-interest of each developer to simply turn to the AI for a quick answer, but unless we all continue contributing knowledge back to a shared, public platform, we risk a world in which knowledge is centralized inside the black box of AI models that require users to pay in order to access their services.

AI systems are only as good as the data they are using, and open source code and Stack Overflow Q&A are examples of this. That AI learns from humans and humans can use AI is why community platforms will be relevant. However if AI learns from data, and developers use AI for quick answers, without knowledge contributed back, knowledge will end up behind a paywall.

As the AI landscape continues to evolve, the need for communities that can nurture, inform, and challenge these technologies becomes paramount. These platforms will not only offer the necessary guidance to refine AI algorithms and models, but also serve as a space for healthy debate and exchange of ideas, fostering the spirit of innovation and pushing the boundaries of what AI can accomplish.

(ed: Not much to say here)

Our thesis on community as the center of a safe, productive, and open future for AI also offers some exciting prospects for our business. Stack Overflow for Teams, our enterprise, private version of Stack Overflow, helps to power a community-driven knowledge base inside of 15K+ organizations like Box, Microsoft, and Liberty Mutual. Decades of institutional knowledge, shaped and curated by subject matter experts and experienced teams, allows the employees at these organizations to more easily collaborate, improving productivity and trust.

(ed: It's a pitch for Stack Overflow for Teams)

Incorporating generative AI technologies into the organizations using Stack Overflow for Teams will allow us to layer a conversational interface on top of this wealth of information. We believe this could lead to tremendous productivity gains: from new hires being able to onboard more quickly, to speed up developer workflows, as users are able to quickly ask questions and retrieve answers tapping into the company’s history, documentation and Q&A.

(ed: This does feel like they are talking about a chatbot over SO data)

The example above is just one of many possible applications of GenAI to our Stack Overflow public platform and Stack Overflow for Teams, and they have energized everyone at our company. We’ll be working closely with our customers and community to find the right approach to this burgeoning new field and I’ve tasked a dedicated team to work full time on such GenAI applications. I’ll continue to share updates through channels such as my quarterly CEO blog, but I’ll be back in touch soon to announce something big on this topic. In the meantime, thank you to our community and customers for continuing to help us on our mission to empower the world to develop technology through collective knowledge.

People are apparently excited about this, and there's a specific full-time team for Generated AI applications. There's some big news coming soon.

4
  • 1
    Journeyman, you have too much time and I'm not sure interpreting other's works is so useful. Probably not many people will read it anyway. Commented Apr 25, 2023 at 15:58
  • 4
    There's multiple comments saying its hard to read, and at least 2 chatgpt translations. It also didn't take that long though the post in question was written over multiple short sessions. Commented Apr 25, 2023 at 23:34
  • 2
    I appreciate it! Especially the paragraph-by-paragraph breakdown
    – GammaGames
    Commented Apr 28, 2023 at 14:32
  • 1
    Re "Generated AU": Do you mean "generative AI"? Commented Apr 28, 2023 at 18:14
7

Perhaps GenAI would be useful if it a) cited all the sources used to pull its answer together, and b) every upvote or 'thanks that worked' tick on the GenAI text resulted in two upvotes on the sources cited.

Edit: Useful and correct answers are still the overarching goal. If AI can provide answers tailored to seekers individual circumstance, drawing from the library of experienced people's contributions, without invoking yet more effort from those people, well, that's a good situation. An awesome one even.

If however AI answers all float to the top, and people's efforts and contributions are quietly pushhed off to the background, that's a horrible situation.

This suggestion is only a way of incorporating generated text. The central idea is ensuring that people remain the forefront of the Stack Overflow community, that's the important part.

13
  • 3
    why two upvotes?
    – starball
    Commented Apr 19, 2023 at 5:21
  • 2
    @starball I guess to give more credit to humans than to the AI (which would get a single upvote). However, I don't really see the GenAI boosting "Look I have 100Million rep and just because I plagiarized all the answers here." Commented Apr 19, 2023 at 7:51
  • 1
    I'd only give at most 1 upvote. I can't understand why 2 upvotes would make sense.
    – starball
    Commented Apr 19, 2023 at 7:53
  • 4
    @starball the numbers I gave are just off the top of my head. They're place holders for the central idea of rewarding the people who gave the text generator something to work with. Commented Apr 21, 2023 at 0:40
  • Would you also be in favor of banning human answers on StackOverflow that don't provide a reference? If not, why does it matter if an AI does that? Commented Apr 25, 2023 at 4:51
  • 1
    @JonathanReez I think it's bad form for humans to not reference sources. Banning no cites from people? I wouldn't go that far. Downvote works fine for that. AI are not human, they shouldn't be treated the same. Commented Apr 26, 2023 at 5:24
  • 1
    @mattwilkie but humans often can't reference the source for coding answers. I.e. you might not necessarily remember where you've read about a particular hash function implementation. Or you might not remember where you've learned about how to change the Windows settings to solve a problem on SuperUser.SE. Should all such answers be downvoted? You're making the assumption that humans always cite their sources but that's easy to invalidate with a 5 minute review of StackOverflow answers. Commented Apr 26, 2023 at 14:22
  • @mattwilkie and I'm confused as to why AI needs to be held to a higher standard. Shouldn't we be blind to what the origin of the answer is? Commented Apr 26, 2023 at 14:38
  • 1
    @JonathanReez it's because lack of citation is prevalent that I used 'bad form' rather than 'ban' or some other systemic approach. We should encourage a culture where referencing sources is normal, and incentivized. AI needs to be held to higher standard in this context because it can answer thousands of questions in the time it takes a person to merely read a single question. If the AI answers everything, as they very well might, then people will stop contributing, because there's nothing left in it for them. Upvotes and the associated community recognition are the only reward we have. Commented Apr 26, 2023 at 19:31
  • The AI shouldn't get any upvotes at all. None at all.
    – Benjamin R
    Commented May 4, 2023 at 18:19
  • @BenjaminR I considered that first, but then realised that useful and correct answers are still the overarching goal. If AI can provide answers tailored to seekers individual circumstance, drawing from a library of experienced people, without invoking yet more effort from those people, well, that's a good situation. If however AI answers and the library of experts is ignored and disenfranchised, that's a horrible situation. Commented May 4, 2023 at 22:06
  • @mattwilkie Since a GPT style AI in this context is just synthesising its response from actual SO/SE users, it shouldn't get any public reward. Internally, yes, it absolutely needs those metrics to improve the quality of its answers, but it should be seen for what it is: a facilitator and aide, not as a true "Answerer", because it is not. It is just synthesising the learning, expertise, wisdom, and voluntary help of actual humans who gave answers. This is the most correct way of viewing the situation.
    – Benjamin R
    Commented May 6, 2023 at 14:03
  • I think it'd be pretty important that it gets publicly visible rewards, actually. We have to consider that questions may get answers from both humans and AI. Consumers of Stack Exchange sites need a way to gauge what's the best answer, regardless of who or what wrote it. If an AI has the accepted answer (perhaps it answered first and was already accepted before humans got around to it) with 30 upvotes, but later a human comes along and answers and gets 200 upvotes, we'd want to see this. If the AI didn't publicly get reputation/votes, this comparison would be challenging.
    – Seth Falco
    Commented May 9, 2023 at 10:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .