June 2023 Data Dump is missing

Few things have made me more livid than what we've just learned through the answer given by AMtwo.

The data dump is one of the main reasons why I'm still participating to this day and has been part of the core values of this community. Please restore it, or consult with us to establish some equivalent way to protect and export our contributions as a whole.

I would like to bring the following pieces of history to the foreground. This is an excerpt of Joel Spolsky speaking on the Stack Overflow podcast #84

Oh, expropriation of community content that... We created Stack Overflow to be against it. If there's anything that's more in the DNA of Stack Overflow than that, I don't know what it is. That's one of our most core things. You can see this all over the place in the design of Stack Overflow.

First of all, from day one, we use the CC-wiki license. And it's basically a license, it says that we don't own the content that's on there, which is why we make those database dumps that are available.

Because we wanted to make sure that if no matter what happens, literally no matter who we sell to, or raise money from, or turn the site over to, and even if they take Stack Overflow, and make it an evil site where you have to pay to look at things and there's pop-up ads and pop-under ads, and you know, dancing chariots of fire that cross the screen and punch the monkey, and, man, I can take so many evil things anyway. And it just becomes a big gigantic spam site.

Doesn't matter because just take the latest CC-wiki download that we provided and go start your own site saying, you know what, this is gonna be the clean version. And I think a lot of people will follow you. We very, very deliberately built Stack Overflow in a way that there wouldn't be any chance of locking and we're pretty much doing the same thing with Stack Exchange.

Also, on the blog, written by Jeff Atwood in 2009:

The community has selflessly provided all this content in the spirit of sharing and helping each other. In that very same spirit, we are happy to return the favor by providing a database dump of public data.

Friday evening

That's it, and the dump is uploaded. Many thanks to the stackers involved, especially Aaron Bertrand, who babysat the thing for days.

Update as of Friday morning (2023-06-16 12:29:39Z)

We're at 40%, average upload speed is 0.25 Mbps :(

This is not likely to complete today.

Update as of Thursday evening (2023-06-16 00:52:06Z)

We're pleased to say that all but one of the files are completed, ahead of the promised schedule. However, the remaining one file (32 gigs) is uploading extremely slowly due to some network slowness. Unfortunately, I think it is likely that the remaining file will not be uploaded by end of day Friday as we initially hoped. We have folks who are continuing to nursemaid this incredibly slow upload along, but it is very clear that our initial plan is at risk at this point. We will get it up as soon as we can.

(We were up to over 50%, but unfortunately the job was in our NY data center, which experienced a blip today and the job was wiped out in the process. It has been restarted, but at 0.50 Mbps, it's slow going. I wish I could project a completion date at this point, but I hesitate to do so because I don't want to miss another stated target.)

I would also like to apologize to the community here for the lack of clarity in my initial announcement here. Although several people proofed it, we missed the lack of the word "permanently" in it, which caused confusion and frustration. That was not my intent, obviously. I thank those who counseled patience and calm while we resolved the situation, and I thank Catija for actually managing that resolution. In short, we tried, but our editing of my initial draft was insufficient, and I own that. It was my writing, and it was not clear. I'll try to be sure that doesn't happen again.

Update as of Wednesday evening (2023-06-15 02:48:09Z):

By way of status, we began uploading the dumps yesterday. For some reason, the Internet Archive is internet throttled right now (seriously so, we're getting around 0.25 Mbps) so it's taking MUCH longer than anticipated. Long enough that I seriously suggested writing to disk and FedEx.

However, I still anticipate that we will meet the promised delivery of end of day Friday.

Posted Tuesday 2023-06-13 22:32:13Z:

Much has been written lately of the company’s decision to pause the distribution of the anonymized data dump that has historically been posted.

Our intention was never to stop posting the data dump permanently, only to pause it while we begin to collect more information on how it was being used and by whom - especially in light of the rise of large language models (LLMs) and questions around how generative AI models are handling attribution. However, it’s clear that many individual users (academics, researchers, etc) have an immediate need to access updated versions of the dumps. So we are re-enabling the automatic data dumps (and uploading the one that’s about a week overdue). We believe that this can happen by end of the day Friday. We will continue to work toward the creation of certain guardrails (for large AI/LLM companies) for both the dumps and the API, but again - we have no intention of restricting/charging community members or other responsible users of the dumps or the API from accessing them.

As part of this project, API users should be on the lookout for a very brief survey that will be coming out (it will be announced here and on stackapps.com) that asks about the features that you most use/would like to see in the API or data dumps moving forward so that we can plan for those, as well as collect general input.

In the meantime, the data dumps will be re-enabled by end of day Friday. We will communicate here when that has been completed or if there are any delays. We will also post here prior to making any future changes to the dumps or distribution of the dumps.

We are looking for ways to gate access to the Dump, APIs, and SEDE, that will allow individuals access to the data while preventing misuse by organizations looking to profit from the work of our community.

That ship has sailed. OpenAI's currently publicly available models (at time of writing) famously have a knowledge cut-off date of 2021, and few people care. Stack Exchange data is gathered, and fed into these models, via the Common Crawl project. Nobody is using Stack Exchange Data Dumps to train AI models.

Even if you were willing to violate the CC BY-SA licenses (of various versions) under which we contributed all that we have contributed, and release the data dumps under some restrictive license, it would achieve nothing, because these companies are already not respecting the restrictions of the licenses.

Rightly or wrongly (imo, wrongly), All Rights Reserved is being interpreted as legal permission to perform bulk statistical analyses that launder away any hope of attribution, then justify the actions by the fact that attribution becomes technically impossible. Our CC BY-SA license doesn't stand a chance. If Prosus wants a return on its investment in Stack Overflow, it should consider putting its weight behind getting the "SA" part of the license enforced against the likes of OpenAI. (I'd happily waive 20% of my part of the settlement from a collective-action lawsuit, if you want to profit more directly, and I'm happy to negotiate something higher.)

What you're doing now is just… destroying the network. For no good reason. I've stopped recommending people to Stack Overflow – when normally I would be teaching entire classes full of students how to use it. I am now actively avoiding contributing Q&A, which is a shame, because I've got 7 or 8 Q&A pairs tucked away in my drafts folder, meant for Stack Overflow, that I may never have the motivation to polish up now.

What you hope to achieve here is unachievable. Ask your lawyer. Heck, ask on Law Stack Exchange. What you're doing is not helping, and you do not know that you do not know that.

Side note: SEDE is already gated behind a Google ReCAPTCHA for not-logged-in users. That's to prevent people taking down your servers with a trillion high-intensity unauthenticated requests, but it serves this purpose quite competently, too. (If you want to make it slightly stricter, gate un-ReCAPTCHA'd SEDE behind the 15-reputation barrier used for upvoting: we have solid empirical evidence that that's high enough to make abuse detectable and manageable.)

To highlight the significance that the data dump may have held for me and others:

I signed up to Stack Overflow in order to contribute to a free library of knowledge for everyone, knowing that the content license meant that I give my knowledge to everyone in the world (even AI companies if they can abide by the license but just to everyone, the license doesn't differentiate uses, it's all the same to it). I liked that a lot and wanted that. Specifically, I never wanted to lock my knowledge in only one location. That's why I saw the quarterly data dumps not only as an addon but as an essential core part of this service. And this understanding worked for many years.

But now the company is not the same anymore than it used to be (with Jeff Atwood at the helm) and they cancelled the insurance (the data dumps) in order to stop some competition.

That doesn't help me. I want that everyone can use my knowledge equally, only limited by the license requirements (attribution, whatever that means nowadays). While legally nothing has changed (although the company was also really bad in this regard in the past), practically the data dumps were the only independent and accessible storage of the knowledge. I wanted the company to compete on a technological level with better features, not by locking in the content more.

Without the data dumps I trust the company exactly as much as zero. There is no compromise possible for me. I cannot continue under these circumstances.

And I hope I never have to read any of these blog posts about community at the center ever again.

Stack Overflow senior leadership is working on a strategy to protect be able to solely monetize Stack Overflow data from being misused by charging companies building LLMs extortionate amounts of money. While working on this strategy, we decided to stop the dump until we could put guardrails in place. Once again ignoring the community instead of being proactive and informing them regarding this.

We are working on setting up the infrastructure to do this correctly in the age of LLMs --- where we continue to be open and share the data with our developer community but work to set up a formal framework for large AI companies that want to leverage the data.

We're quickly scrambling because this would be an amazing opportunity for us to make some more money off your backs.

We are looking for ways to gate access to the Dump, APIs, and SEDE, that will allow individuals access to the data while preventing misuse by organizations looking to profit from the work of our community. We are working to design and implement appropriate safeguards and still sorting out the details and timelines.

We're probably going to degrade the experience for people within the community who used to use these dumps because we need to somehow get companies to pay us to use this data in a more friendly way.

As for this this line:

We will provide regular updates on our progress to this group.

Group?

Do they mean group as in group of users interested in the dumps? Or, are they that detached with the community that they call these sites groups like you have WhatsApp groups?

Nevertheless, I can't believe this was actually unironically written when they didn't even bother to tell the community that this was happening.

I guess this is what comes from having employees that no longer interact with the community.

Anyone who seriously trusts SO after all they've done deserves to be fooled.

"imagine if stackexchange went from free to paid..." All of the content posted by users on Stack Exchange network is licensed under CC-BY-SA. If SE ever blocks free access to the content, anyone can host it elsewhere. Pulling EE-like shenanigans is not practical in SE.

Source

@JCL1178: I agree except the data is open source. If EE dies, the data is gone, especially if the past happens again.

Source

^{Technically speaking, this is an answer, you might not like it or think that it's not a great answer but it doesn't make it any less of an answer. So, if you happen to feel particularly strongly about it, feel free to downvote but I personally don't think it deserves deleting. But hey, what do I know?}

¯\_(ツ)_/¯

The resumption of these dumps is very welcome, and I'm glad the company has decided to listen to the community on this issue.

That said, there's something in Phillipe's answer which doesn't sound right to me:

Our intention was never to stop posting the data dump, only to begin to collect more information on how it was being used and by whom

Both parts of this statement apparently contradict other statements by SE, Inc. staff (former and current).

According to AMtwo's answer here, "The job that uploads the data dump to Archive.org was disabled on 28 March, and marked to not be re-enabled without approval of senior leadership." This is inconsistent with the claim that the intention was never to stop posting the data dump; it was stopped in March, and the stopping was clearly intentional.

Also, according to Jody Bailey's answer here, SE, Inc. is "working on a strategy to protect Stack Overflow data from being misused by companies building LLMs. While working on this strategy, we decided to stop the dump until we could put guardrails in place." This is inconsistent with the claim that the pause in availability of the dump was "only" so that SE, Inc. could begin to collect information about usage.

If it's true that the intention was only ever to begin collecting information about usage, why was this intention not mentioned earlier, and why was a different intention mentioned instead?

Our intention was never to stop posting the data dump

Okay that sounds real nice of you... But uh- why was there zero transparency in all of this? It was revealed by ex-staff, and then actual staff jumped in to stop the sh*tstorm.

As part of this project, API users should be on the lookout for a very brief survey [...] about the features that you most use/would like to see in the API or data dumps moving forward so that we can plan for those, as well as collect general input.

Why a survey though? You have this?

• is:q [feature-request] [api] -[status-completed] -[status-declined] -[status-bydesign] score:1.. closed:no
• is:q [feature-request] [data-dump] -[status-completed] -[status-declined] -[status-bydesign] score:1.. closed:no
• is:q [feature-request] [data-explorer] -[status-completed] -[status-declined] -[status-bydesign] score:1.. closed:no

I want to re-iterate what others have said in the comments. The fact that my contributions to Stack Overflow are licensed under CC BY-SA is a significant motivating factor for why I contribute in the first place. I want the knowledge I share here to be accessible and freely usable by everyone, including LLMs, and such use is already allowed by the current license so it's unclear what "misuse" Stack Exchange Inc. is trying to prevent here.

In the past, sites have even gone so far as to mirror user content from Stack Overflow wholesale, and that was always considered okay as long as the sites provided attribution as required by the license and weren't doing anything illegal.

Has this changed? The wording in Stack Exchange's official answer to this question has me concerned. If "organizations looking to profit from the work of our community" is now considered a problem, I suppose the next logical step is to remove the CC BY-SA license and replace it with CC BY-NC-SA, or stop releasing user contributions under an open license entirely? If that's the path things are headed down now, I will be significantly less motivated to contribute to Stack Exchange sites in the future. I want to contribute to an open repository of knowledge that anyone can use, not a proprietary database controlled entirely by Stack Exchange, Inc.

In an answer to an identical question Philippe today announced that the data dumps will be reinstated again, and the June data dump should be available soon.

Much has been written lately of the company’s decision to pause the distribution of the anonymized data dump that has historically been posted.

Our intention was never to stop posting the data dump, only to begin to collect more information on how it was being used and by whom - especially in light of the rise of LLMs and questions around how genAI models are handling attribution. However, it’s clear that many individual users (academics, researchers, etc) have an immediate need to access updated versions of the dumps. So we are re-enabling the automatic data dumps (and uploading the one that’s about a week overdue). We believe that this can happen by end of the day Friday. We will continue to work toward the creation of certain guardrails (for large AI/LLM companies) for both the dumps and the API, but again - we have no intention of restricting/charging community members or other responsible users of the dumps or the API from accessing them.

As part of this project, API users should be on the lookout for a very brief survey that will be coming out (announced here and on stackapps.com) that asks about the features that you most use/would like to see in the API or data dumps moving forward so that we can plan for those, as well as collect general input.

In the meantime, the data dumps will be re-enabled by end of day Friday. We will communicate here when that has been completed or if there are any delays. We will also post here prior to making any future changes to the dumps or distribution of the dumps.

My take on it: The immediate danger to data dumps not being available anymore is averted, but I find it hard to believe that it was not the intention to "stop posting the data dump". If that wasn't the case, why not announce that back when the decision was made, and why announce a reversal of the stop only after the community writes extensively about it?

This reads rather like damage mitigation, and my trust in the company not doing it again anytime soon is still really close to or at zero.

This is just my personal opinion, but stopping the data dumps even if it caused only a delay in the end is still a bad sign. One doesn't need to stop a service in order to research it, and the company should know very well how important the data dumps are to the community. At the very least, this shows a certain disinterest in the feature.

Disclaimer: sorry, the current format is due to a merge, and originally this was a reply to the linked company post.

So, let's look at the official reply.

Much has been written lately of the company’s decision to pause the distribution of the anonymized data dump that has historically been posted.

Personally I have nothing to say here, but I can see foresee some users pointing out that the chosen wording seems to imply that it was the userbase specifically that said too much about the issue and hallucinated conspiracies that did not exist (before this message the company limited all interaction to just one post). Therefore I accept that this could feel a little condescending to some, but in my opinion I would just ignore it.

Our intention was never to stop posting the data dump, only to begin to collect more information on how it was being used and by whom - especially in light of the rise of LLMs and questions around how genAI models are handling attribution.

This is in direct contrast with the information the community got before from the company CTO that said

We are looking for ways to gate access to the Dump, APIs, and SEDE, that will allow individuals access to the data while preventing misuse by organizations looking to profit from the work of our community

It is a similar, but different message. It is also worth noticing that if the intent was indeed "begin to collect more information on how it was being used"

• I don't see how you could do that while the dump was not accessible anymore.
• I don't see why you should do that without any warning.

But wait! Maybe that was the test? Getting data about how long it would take to the community to notice? If that was your plan, great! Now we know that someone in the community is using the dump and will notice when it goes missing.

That said, there is also another logical association some will probably make. This happened about at the same time the company announced to the press some plan about the monetization of the data dump.

However, it’s clear that many individual users (academics, researchers, etc) have an immediate need to access updated versions of the dumps. So we are re-enabling the automatic data dumps (and uploading the one that’s about a week overdue).

Again, I won't argue here, but I'll let you know that this reads a tad like "But since you folk can't wait five minutes [...]". The problem is not the wait; the problem is that the stop came out of nowhere since you "forgot" to announce it and even if you probably are not legally bound to provide said data, users still expect to be warned if the implicit agreement changes.

We believe that this can happen by end of the day Friday. We will continue to work toward the creation of certain guardrails (for large AI/LLM companies) for both the dumps and the API, but again - we have no intention of restricting/charging community members or other responsible users of the dumps or the API from accessing them.

I fear you... can't? Either you provide the dump "as is" and nothing can stop anyone to repost it even if the original download required a login or you try to "watermark" it in some way that would workaround the current license and make it a derivative work that would not be reproducible, but in that case... That is not the dump anymore.

As part of this project, API users should be on the lookout for a very brief survey that will be coming out (announced here and on stackapps.com) that asks about the features that you most use/would like to see in the API or data dumps moving forward so that we can plan for those, as well as collect general input.

In the meantime, the data dumps will be re-enabled by end of day Friday. We will communicate here when that has been completed or if there are any delays. We will also post here prior to making any future changes to the dumps or distribution of the dumps.

Fine, I guess. Why wasn't this the original way to go?

Conclusions

This announcement leaves me with some mixed feelings.
On the front, this seems like a "victory" for rationality—the dump has been restored, everything is fine now and you even promised to work on a solution together (albeit in the form of a survey).

Yet, in the current circumstances of the ongoing strike, I can't help but feel like this has been "staged" in some way as a part of a bigger strategy to show that the company did its part to "reach out" to the moderators on strike and now they should do their part to unless they are "bad actor with an agenda to prolong a conflict that the company was trying to solve rapidly".

The decision to shut down the dump seems to precede the strike and that was confirmed by a neutral third party—an ex-employee, so the things should not be related. But at the same time, this victory only restores the initial status quo without any real gain for the community. With this stunt, the company has managed to "do its part toward an agreement" without actually changing anything, and now it feels only expected that the moderators should concede, "do their part" on something too. To put it bluntly, while it makes little sense if we assume all the related information the community got are accurate, all manages to feel very strategical and in a way even convenient for the company.

To answer the question of "will Stack Exchange eventually charge for API or data dump use", this can be found in the WIRED article from April 20th: Stack Overflow Will Charge AI Giants for Training Data

It has such passages as:

Stack Overflow’s decision to seek compensation from companies tapping its data, part of a broader generative AI strategy, has not been previously reported. It follows an announcement by Reddit this week that it will begin charging some AI developers to access its own content starting in June.

and contains information on company direction from the CEO:

“Community platforms that fuel LLMs absolutely should be compensated for their contributions so that companies like us can reinvest back into our communities to continue to make them thrive,” Stack Overflow’s Chandrasekar says. “We're very supportive of Reddit’s approach.”

Chandrasekar described the potential additional revenue as vital to ensuring Stack Overflow can keep attracting users and maintaining high-quality information. He argues that will also help future chatbots, which need “to be trained on something that's progressing knowledge forward. They need new knowledge to be created.” But fencing off valuable data also could deter some AI training and slow improvement of LLMs, which are a threat to any service that people turn to for information and conversation. Chandrasekar says proper licensing will only help accelerate development of high-quality LLMs.