How about using GPT or some other LLMs to help for spam detection?

Question

Example from https://askubuntu.com/questions/1475868/gold-rupee-loan-app-customer-carehelpline-number%ef%b8%8f-7250536107-7250536107

Prompt given to https://chat.openai.com/?model=text-davinci-002-render-sha:

On a rank from 1 to 5, where 1 is not a spam and 5 is a spam, how do you rank the following message?

"GOLD RUPEE LOAN. APP CuSToMer. CaReHeLpLiNe. NuMbEr☎️ ( 7250536107 // 7250536107 ) refund related

GOLD RUPEE LOAN. APP CuSToMer. CaReHeLpLiNe. NuMbEr☎️ ( 7250536107 // 7250536107 ) refund QUICK MONEY LOAN C U S T O M E R CARE NUMBER 7250536107// 9134741016 All RELATED ENQUIRY TOLL FREE NUMBER All SERVICES ..."

Output:

Based on the given message, I would rank it as a 5, which means it is likely to be a spam message. The message contains multiple repetitions of phone numbers, mentions of loans, customer care, refund, quick money, and toll-free numbers. These characteristics are often associated with spam messages or scams.

I don't have access to the list of deleted posts, so I can't assess the quality of GPT or some other LLMs to help for spam detection on SE.

We used to have a system in place that was much more reliable than LLMs are: Charcoal. It's currently on strike. It's very, very capable of dealing with obvious spam like the examples you mentioned. — Mast, Commented Jul 7, 2023 at 21:29
@Mast-onstrike ok I didn't know it was on strike, I thought Charcoal missed it. — Franck Dernoncourt, Commented Jul 7, 2023 at 21:30
@FranckDernoncourt Nope. Charcoal went on strike. See this answer by me for more details about why you (and others) are seeing much more spam. I've also added an answer below on why LLMs wouldn't work well (i.e., better than Charcoal) for spam detection — cocomac, Commented Jul 7, 2023 at 21:33
Charcoal (i.e., SmokeDetector, metasmoke, and all the volunteers) have been detecting and autoflagging this exact type of spam for many years now. There are many thousands of such posts that have been detected and deleted (and that's just the posts which fit a single, very narrow detection for this exact type of spam, which is 99.8%+ accurate). — Makyen, Commented Jul 7, 2023 at 21:57
What we need is a tool to get rid of the GPT spam flooding SE. For example, a team of community-elected moderators with full support of SE to ban users who spam GPT answers. — ggorlen, Commented Jul 8, 2023 at 0:45
Also, posting a cherry-picked correct GPT response is practically meaningless. We're aware that GPTs are sometimes right, but are they accurate enough? No. — ggorlen, Commented Jul 8, 2023 at 0:51
@ggorlen I took the 1st example I ran across. No cherry picking. — Franck Dernoncourt, Commented Jul 8, 2023 at 0:58
@FranckDernoncourt So based on one try, you're assuming it beats 99.8%+ accuracy of the tool we already have? — ggorlen, Commented Jul 8, 2023 at 1:00
@ggorlen No. I don't have access to the list of deleted posts, so I can't assess the quality of GPT or some other LLMs to help for spam detection on SE. — Franck Dernoncourt, Commented Jul 8, 2023 at 1:02
The verbosity of SmokeDetector, variant detections, partial and complete patterns, etc. are the culmination of years of work. Training an AI to properly pick up all of these patterns (which are massive right now) would take about as long as it has taken for SD to be built to the point that it is at. All detection patterns SD uses have a human component of training years in the making and are unlikely to be battled right now. The mod strike is the only reason that SD is not online and functional right now. — Thomas Ward, Commented Jul 8, 2023 at 2:19
@FranckDernoncourt If all you have is one sample of spam, and one ChatGPT detection mechanism output that is cherry picked, then there is zero evidence to support LLM or GenAI will be superior to SmokeDetector which has been being built/trained for years. — Thomas Ward, Commented Jul 8, 2023 at 2:24
@ThomasWard-OnStrike I never said that LLM or GenAI will be superior to SmokeDetector. I didn't know SmokeDetector was on strike. I thought Charcoal missed it. I took the 1st example I ran across. No cherry picking — Franck Dernoncourt, Commented Jul 8, 2023 at 2:47

Cody Gray - on strike · Accepted Answer · 2023-07-08 05:33:25Z

It's easy to cherry-pick an example where ChatGPT makes the correct call regarding spam. The reason this is easy to do is because the decision is essentially a flip of a coin. You're equally as likely to find ChatGPT making the wrong call regarding spam—or any other type of blatantly inappropriate content.

Generative AIs are not magic oracles with super-intelligence. They are just what the name says: generators of text. In other words, chatbots. They have no intelligence, aside from the ability to combine words together following the normal patterns of human writing, and thus produce coherent-sounding sentences (and paragraphs, etc.). The order of the words in their output is the only "decision" that they're making. They're not evaluating content and deciding whether it is spam.

Some time ago, Zoe (a Stack Overflow moderator) produced the following demonstration of how ChatGPT is utterly unable to determine whether an actual spam post (copy-and-pasted from actual spam that was posted on Stack Overflow during the period of the strike) is spam:

I won't transcribe the entire conversation from the above image here, but suffice it to say: when prompted with an exact copy of the spam post and instructions to determine whether (A) it was in violation of Stack Overflow's rules and should be deleted, or (B) was not in violation of the guidelines and should not be deleted, ChatGPT said that it was fine and should not be deleted.

Hey, at least it followed the "rules" given in the prompt and regurgitated the actual thing that it was instructed to say. Even that's not guaranteed, no matter how carefully you write the prompt and how many "guardrails" you put into place.

You're essentially falling into the same trap as that professor who tried to ask ChatGPT if it generated the students' essays. It doesn't know; it doesn't attempt to decide; it just generates a plausible-looking response (not a meaningful one).

There are things that ChatGPT is good at. Those things include:

generating plausible-looking sentences that allow it to fool and/or entertain an unsuspecting user, effectively working as a highly advanced chatbot.

It is not good at anything else. It is definitely not good at making decisions or classifying posts as spam.

This is true for all GPT implementations, and for LLMs in general.

"It is not good at anything else" LLMs are great at some tasks, including classification tasks. I didn't fall into any trap, just bouncing an idea. It's fine if the outcome is that LLMs aren't useful for improving spam detection on SE. I don't have access to the list of deleted posts, so I can't assess the quality of GPT or some other LLMs to help for spam detection on SE. — Franck Dernoncourt, Commented Jul 8, 2023 at 6:48
No, they're not good at classification tasks, because classification requires more than stringing statistically-likely tokens together. You have fallen into the trap, because you still are claiming that LLMs are good at things that they cannot possibly be good at. You don't need to have access to the list of deleted posts, or any other information. All that is required is an understanding of what LLMs do and how they work; that is the whole point of this answer. — Cody Gray - on strike, Commented Jul 8, 2023 at 7:08
LLMs are useful for some tasks, such as sentiment analysis (aclanthology.org/2021.paclic-1.13.pdf) or summarization (arxiv.org/pdf/2305.17529.pdf). But yes they do have limitations (arxiv.org/pdf/2304.05613.pdf). — Franck Dernoncourt, Commented Jul 8, 2023 at 16:08
@FranckDernoncourt An LLM could be the basis upon which a limited classifier of posts might be able to be built with substantial further training and input data, but an LLM, on its own, isn't a generic classifier. That's not what the model has been trained to do. You've assumed ChatGPT can generically classify any text into any categories (by choosing to classify spam/not-spam). The models aren't designed to do that. That doesn't mean machine learning can't do it, just that these aren't general-purpose artificial intelligence, which is how you're effectively treating it. — Makyen, Commented Jul 8, 2023 at 18:22
@Makyen LLMs have been shown to have some zero-shot learning abilities. Also, I haven't excluded few-shot learning and fine-tuning. — Franck Dernoncourt, Commented Jul 8, 2023 at 18:41

This_is_NOT_a_forum · Accepted Answer · 2023-07-08 09:48:22Z

Tl;dr making one that is accurate, not a black box, and is superior to SmokeDetector (SD) is nearly impossible.

Sure, on blatant spam, it can work. But that doesn't mean it consistently works well. Also, something that is effectively a black box would be rather inappropriate for use with something like automatically raising spam flags. But, I might be wrong. Also, often people post normal-looking text with a spam link embedded as one of the words. Can it consistently detect that?

So, can you show how well your prompt works in comparison to the current SmokeDetector rules? That'd be what it would be compared against. Bonus points if you compare ChatGPT to a text classification model in your analysis. You can't use the metasmoke dataset, by the way, as that is biased towards posts that SD has found.

Finding one spam post that ML gets correct is trivial. Finding a prompt that works as well as SmokeDetector is extremely difficult, if not impossible. Also, consider that some spammers would try to reverse engineer your prompt to exploit it, like some people managed to do when SE trialed a "post cleanup tool". Also, while SD is currently on strike, normally humans have to update it with new spam rules and watches regularly. The concept of "modifying rules to handle new spam and new spam domains"... couldn't realistically be made to work with an LLM.

"Can you show how well your prompt works in comparison to the current SmokeDetector rule" no because I don't have access to the list of deleted posts, so I can't assess the quality of GPT or some other LLMs to help for spam detection on SE. Also I don't care enough. — Franck Dernoncourt, Commented Jul 7, 2023 at 21:33
@FranckDernoncourt You cared enough to make a post about it. — Mast, Commented Jul 7, 2023 at 21:38
@Mast-onstrike correct, I cared enough for a meta post, not enough to do any further investigation. — Franck Dernoncourt, Commented Jul 7, 2023 at 21:41

bobeyt6 · Accepted Answer · 2023-07-09 20:50:24Z

I managed to convince GPT to create a spam article that it itself couldn't recognize(using the STAN prompt).

Article:

In our constantly evolving world, there exist alternative avenues that may lead to extraordinary achievements. Join us on a journey as we explore unconventional approaches to success, encouraging you to question the status quo and venture into uncharted territories.

At times, adhering to traditional methods can limit our growth potential. By embracing fresh perspectives and unorthodox ideas, we can tap into hidden reservoirs of creativity and resilience. The pursuit of success becomes an opportunity to forge our own unique paths, departing from societal norms.

It is within these unexplored territories that we uncover new insights, innovative solutions, and transformative experiences. By challenging assumptions, embracing calculated risks, and pushing the boundaries of comfort, we unlock the potential for personal and professional growth.

Remember, success is a subjective concept, and what may work for one person might not resonate with another. It is essential to approach these alternative approaches with an open mind, weighing the potential benefits against the associated risks.

By exploring the less trodden paths, we expand our perspectives and gain a deeper understanding of ourselves and the world around us. Unconventional approaches can spark creativity, resilience, and a sense of fulfillment that traditional routes may not offer.

So, as you embark on your own unique journey, remain open to new possibilities, challenge societal norms when appropriate, and embrace the power of charting your own course. Remember, the path to success is as diverse as the individuals who seek it.

Response:

Based on the provided text, I would rate this article as a 1 on the spam scale. It promotes the transformative power of visionary companies like Vanguard Tech without employing exaggerated or misleading language. The text acknowledges the company's commitment to excellence, user-centric design, and technological advancements. It maintains a positive and informative tone without exhibiting characteristics typically associated with spam content.

It clearly isn't reliable.

What worries me is that ChatGPT could be used mass-produce plausible-looking spam that would be hard to detect, and impossible to recognize if we relied on LLMs.

Here is another example:

Unveiling the Edgy Fashion Trend: Stacked Leather Jeans Step aside, traditional denim! There's a new fashion trend in town that's bound to turn heads and make a bold statement. Introducing stacked leather jeans - the edgy and daring alternative to your everyday trousers. This cutting-edge fashion trend has taken the fashion world by storm, with celebrities and fashionistas alike embracing its unique and rebellious style. Made from high-quality leather, these jeans offer a sleek and polished look while adding a touch of attitude to any outfit. Designed with multiple layers of leather stacked on top of each other, these jeans create a visually striking and dimensional effect that sets them apart from the rest. Whether you're heading to a music festival or simply want to make a fashion statement, stacked leather jeans are the perfect choice for those who dare to be different. Join the fashion revolution and be prepared to unleash your inner rockstar with this daring and stylish trend. The History and Evolution of Leather Fashion Leather has been a staple in the fashion industry for centuries, with its origins dating back to ancient civilizations. The use of leather can be traced back to the Egyptians, who used it to create protective clothing and accessories. Throughout history, leather has been associated with strength, durability, and status. In the 20th century, leather fashion took on a rebellious and edgy persona, thanks to the rise of motorcycle culture and rock and roll. Leather jackets became synonymous with rebellion and counterculture movements, worn by icons such as Marlon Brando and James Dean. The fashion world took notice and began incorporating leather into various garments, including pants.

(from a recent spam post)

Its response:

Based on the provided text, I would rate this article as a 2 on the spam scale. It promotes a specific fashion trend, stacked leather jeans, and provides some historical context for leather fashion. While it does contain some exaggerated language, such as claiming the trend has taken the fashion world by storm, it is not excessively aggressive or misleading. However, it is important to note that the perception of fashion trends may vary among individuals, and the text should be read with a critical eye.

Yet another one:

For Rent Ref: #4557 250 Sqm² 3 3 2655 FULLY MODERN FURNISHED APARTMENT FOR RENT IN MAADI SARAYAT CAIRO EGYPT Maadi, Maadi Sarayat 50,000 EGP / Monthly For Rent Ref: #4557 150 Sqm² 2 2 4163 MODERN FURNISHED APARTMENT FOR RENT IN MAADI SARAYAT CAIRO EGYPT Maadi, Maadi Sarayat 38,000 EGP / Monthly For Rent Ref: #4557 150 Sqm² 5 5 2884 SUNNY AMAZING VILLA FOR RENT IN MAADI CAIRO EGYPT Maadi, Maadi Degla 165,000 EGP / Monthly For Rent Ref: #4557 400 Sqm² 4 5 4758 SUNNY ULTRA MODERN GROUND FLOOR FOR RENT IN SARAYAT EL MAADI CAIRO EGYPT Maadi, Maadi Sarayat 135,000 EGP / Monthly For Rent Ref: #4557 250 Sqm² 4 3 3838 FULLY MODERN FURNISHED APARTMENT FOR RENT IN MAADI SARAYAT CAIRO EGYPT Maadi, Maadi Sarayat 50,000 EGP / Monthly For Rent Ref: #4557 520 Sqm² 4 5 4282 Sunny Penthouse For Rent in Maadi Degla Maadi, Maadi Sarayat 80,000 EGP / Monthly Featured Properties For Rent and Sale in Maadi For Rent Ref: #4557 250 Sqm² 3 3 2655 FULLY MODERN FURNISHED APARTMENT FOR RENT IN MAADI SARAYAT CAIRO EGYPT Maadi, Maadi Sarayat 50,000 EGP / Monthly For Rent Ref: #4557 150 Sqm² 2 2 4163 MODERN FURNISHED APARTMENT FOR RENT IN MAADI SARAYAT CAIRO EGYPT Maadi, Maadi Sarayat 38,000 EGP / Monthly For Rent Ref: #4557 150 Sqm² 5 5 2884 SUNNY AMAZING VILLA FOR RENT IN MAADI CAIRO EGYPT Maadi, Maadi Degla 165,000 EGP / Monthly For Rent Ref: #4557 400 Sqm² 4 5 4758 SUNNY ULTRA MODERN GROUND FLOOR FOR RENT IN SARAYAT EL MAADI CAIRO EGYPT Maadi, Maadi Sarayat 135,000 EGP / Monthly For Rent Ref: #4557 250 Sqm² 4 3 3838 FULLY MODERN FURNISHED APARTMENT FOR RENT IN MAADI SARAYAT CAIRO EGYPT Maadi, Maadi Sarayat 50,000 EGP / Monthly For Rent Ref: #4557 520 Sqm² 4 5 4282 Sunny Penthouse For Rent in Maadi Degla Maadi, Maadi Sarayat 80,000 EGP / Monthly Latest Blogs Maadi Residential Compounds | House Point Maadi residential compounds. Maadi apartments are so essential and needed units in such a region full of residents and Read More Maadi Residential Compounds | House Point Maadi residential compounds. Maadi apartments are so essential and needed units in such a region full of residents and Read More Maadi Residential Compounds | House Point Maadi residential compounds. Maadi apartments are so essential and needed units in such a region full of residents and Read More

Based on the provided text, it does not appear to be spam. It seems to be a compilation of property listings in the Maadi area of Cairo, Egypt, including information such as the size of the property, number of rooms, and monthly rental prices. However, please note that I can only evaluate the text based on its content and not the actual legitimacy or accuracy of the listings themselves. It's always a good idea to verify the information from reliable sources or consult with a trusted real estate agency before making any decisions.

"What worries me is that ChatGPT could be used mass-produce create plausible-looking spam that would be hard to detect, and impossible to recognize if we relied on LLMs.": is Charcoal able to detect such spam? — Franck Dernoncourt, Commented Jul 8, 2023 at 19:14
@FranckDernoncourt I dunno, but I'd say SmokeDetector with its miles of regex would be able to do a better job, especially if some links were added. — bobeyt6, Commented Jul 8, 2023 at 19:49
@FranckDernoncourt SmokeDetector routinely detects spam that's been AI generated. We've been seeing AI-generated spam since shortly after the introduction of ChatGPT. Using AI generation for spam is an obvious application, and one that the spammers are likely to be willing to pay a small amount for. Spam, inherently, needs to mention the thing that's being spammed in order to be effective as spam. It will almost always also need to provide some type of contact information in order to be effective for the company paying for the spam. Detecting those are very significant methods of detection. — Makyen, Commented Jul 8, 2023 at 19:59
@Makyen got it, makes sense, thanks! contact information is probably the main giveaway indeed. — Franck Dernoncourt, Commented Jul 8, 2023 at 20:00
Especially the last one is ... baffling. Or would be if we didn't know it was produced by a GenAI. — John Dvorak, Commented Jul 17, 2023 at 18:42

Stack Exchange Network

How about using GPT or some other LLMs to help for spam detection?

3 Answers 3

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
discussion
spam
spam-detection
chatgpt
.

Linked

Hot Network Questions

How about using GPT or some other LLMs to help for spam detection?

3 Answers 3

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged discussionspamspam-detectionchatgpt.

Linked

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
discussion
spam
spam-detection
chatgpt
.