Given how the feature is going to look and how we believe that people will use it, are there reasons you would be skeptical of the results of the experiment?
Frankly, Philippe, I am extremely skeptical of the experiment for a whole list of reasons. For the most part, they've been outlined well enough in feedback other users have already provided (here, here, here, here, here, and here), so I'll try to be brief. First of all, the stated benefits of the feature look rather "empty" and unsubstantiated:
First, question askers spend less time crafting the perfect title for their questions, and instead can focus on the content of the question.
The problem here is that users have never, in general, spent any significant amount of time "crafting the perfect title". Has the company conducted any research as to whether users are slowed down by having to come up with a title, or is this just a vacuously true statement? As a damning evidence to the contrary, let's take a quick look at the slice of newest question titles on Stack Overflow:
In case this isn't obvious from the screenshot, all of the titles shown (you can repeat this experiment a thousand times, and the output will not change) are either: borderline incomprehensible, non-descriptive, and sometimes outright gibberish. And all of them show a distinct lack of care, thought, and research put into them.
Users who can carefully craft question titles are few and far between, and they are generally capable of making one without any assistance of an LLM tool.
Second, question reviewers are able to better understand the content of the question, making it easier to suggest edits or improve the post.
The concern here is that it also is unsubstantiated. Surely, the titles may start to look better and show higher proficiency in English, but can you show that the usage of an LLM tool will solve the fundamental issues at hand leading to bad titles: lack of experience, general incompetence, severe gaps in knowledge, inability to identify problems, etc.?
Unless the tool is able to address all those, what you will get in the end is plausible-looking misleading nonsense that will certainly not make it easier to understand questions or suggest edits to posts. In fact, as many others have already mentioned, it might lead to more curators skipping over well-written titles due to them looking superficially fine.
Finally, end users of Stack Overflow can more easily understand if the question is relevant to their needs.
The same concern about the second point applies here: can the company substantiate the claim at least in theory since a better title is no more useful for understanding whether the question is relevant than a worse one if it is still nonsensical or misleading?
In fact, this is one of the reasons there is a temporary ban on AI-generated content on Stack Overflow (and many other sites of the network): the content tends to look good while being either incorrect, misleading, nonsensical, or outright harmful.
As for concerns regarding the "five core questions":
Do users’ questions perform better when they’ve received title-drafting assistance?
If you will only be looking at whether title-assisted questions get more upvotes and less downvotes, you risk misinterpreting improved reception due to the post looking better as the post being of higher quality. You need to investigate a whole complex of factors, including, most importantly, how often are the posts closed as needing improvement.
Are titles written with assistance edited more or less often by community members?
Here you risk mistaking a reduction in edits due to the title looking superficially better for the post requiring less improvement. On the one hand, as also pointed by others, editors might start skipping posts with well-written titles. On the other hand, if the posts start to get more edits, you need to look at what edits are made qualitatively, as an uptick in edits can mean the generated titles are even worse than the manual ones just as much as being easier to understand and thus edit.
Do readers and answerers interact differently with questions that received title-drafting assistance? For example, via reviews or comments.
Again, this depends on how you are going to approach the analysis. If it is going to be the usual "there was an X increase in number of Y", you risk severely misinterpret an uptick for an improvement (which might just as well indicate that the tool makes content less understandable requiring more back-and-forth).
A downtick in a purely quantitative analysis can also either mean that the content requires less interaction to parse or that the title makes it harder to engage with the post in equal measure.
Do better titles lead to a reduction in users – particularly new users – abandoning their questions? Does it increase the rate at which new users come back to the site in the future?
The question abandonment criterion is highly prone to misinterpretation too: more users abandoning drafts might be due to the UX accompanying the change being subpar and / or confusing. A reduction in abandoned drafts might mean that the velocity of posting a question is simply greater due to less effort required from the user.
It does not take into account that greater velocity might actually be a bad thing precisely because the users do not need to spend effort to formulate their title. It also does not consider that a greater number of abandoned drafts can mean that users come up with a solution or realise their mistake while trying to come up with a title on their own.
How many users will accept the titles that are recommended to them?
Finally, this criterion is unlikely to provide any useful information at all. Question askers, as a general rule, come from a position of lack of knowledge. More often than not they are incapable of determining where the problem even is (and the "quality" of existing titles attests to that), let alone judge whether the generated title is actually better and not better looking.
It is highly likely that users, and especially new users, will choose one provided to them solely on the basis of it being well-formed. I also pose that the ones who will choose to write their own title will likely be ones who already know what they are doing (and thus find the suggested titles insufficient / incorrect / missing the point).
Unless you can account for that, I highly doubt this will be a useful data point.
To summarize the section about the core questions (the benefit ones can be reduced to a single concern: are those just empty statement, or can you actually show the usage of the tool, in principle, can provide such benefits), please, when you eventually mark the experiment as a success, do not just say:
- "the performance criterion was satisfied as there posts started to get more upvotes and less downvotes";
- "the editing criterion was satisfied due to posts receiving less edits";
- "the interaction criterion was satisfied because there have been more comments / answers on those posts";
- "the abandonment criterion was satisfied as more users have seen their posts to completion";
- "the acceptance criterion was satisfied due to a lot of users accepting suggested titles".
Those are all very flawed metrics to use, and I worry you'll just miss the forest for the trees with this initiative.
