Best prompt and model for fact-checking a text (disinformation/fake-news detection)

Question

Given a short text <text_to_check>, I want the LLM to check whether there are some facts stated in the text which are NOT true. So I want to detect 'disinformation' / 'fake news'. And the LLM should report which parts of the text are not true.

How would the "best" prompt look like for this task ?

And what is the best 'compact' LLama-2 based model for it ? I suppose some kind of instrucion-following LLM. The LLM shall run on a mobile device with <= 8 GB RAM, so the largest model I can afford is ~ 13B (with 4-bit quantization in llama.cpp framework).

Looking at Alpaca Leaderboard (https://tatsu-lab.github.io/alpaca_eval/), the best 13B models there are XWinLM (not sure if supported by llama.cpp), OpenChat V3.1 and WizardLM 13B V1.2. So I suppose I will use one of those models

Answer 1

How would the "best" prompt look like for this task ?

Here is a paper published today by Meta on having the LLM fact checks some text via some plan verification (in this case, its own answers, but their strategy could also apply to OP's case): Chain-of-Verification Reduces Hallucination in Large Language Models. Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, Jason Weston

Generation of plausible yet incorrect factual information, termed hallucination, is an unsolved issue in large language models. We study the ability of language models to deliberate on the responses they give in order to correct their mistakes. We develop the Chain-of-Verification (CoVe) method whereby the model first (i) drafts an initial response; then (ii) plans verification questions to fact-check its draft; (iii) answers those questions independently so the answers are not biased by other responses; and (iv) generates its final verified response. In experiments, we show CoVe decreases hallucinations across a variety of tasks, from list-based questions from Wikidata, closed book MultiSpanQA and longform text generation.