Removing duplicate paragraphs with Edit Pad Pro or Notepad++

Question

I have a .docx file that contains mcqs which are in the format as shown below. The problem is there are many duplicate mcqs and I would therefore like to know if a regex can be created to detect all duplicate mcqs.

I have Edit Pad Pro 7,Notepad++,powergrep and sublime text. and all the regex that I have used until now deleted duplicates on a line by line basis, thereby deleting options from other questions even though the questions don't match.

So basically what I am saying is I need a regex that can delete all the duplicate mcqs only if the whole mcq matches, not individul lines or sentences.

I am a novice with respect to regex, so please excuse any inadequacies.

Lichen planus occurs most frequently on the?
A.  buccal mucosa.
B.  tongue.
C.  floor of the mouth.
D.  gingiva.

In the absence of “Hanks balanced salt solution”, what is the most appropriate media to transport an avulsed tooth?
A.  Saliva.
B.  Milk.
C.  Saline.
D.  Tap water.

Which of the following is the most likely cause of osteoporosis, glaucoma, hypertension and peptic ulcers in a 65 year old with Crohn’s disease?
A.  Uncontrolled diabetes.
B.  Systemic corticosteroid therapy.
C.  Chronic renal failure.
D.  Prolonged NSAID therapy.
E.  Malabsorption syndrome.

Lichen planus occurs most frequently on the?
A. buccal mucosa.
B. tongue.
C. floor of the mouth.
D. gingiva.

expected result

Lichen planus occurs most frequently on the?
A.  buccal mucosa.
B.  tongue.
C.  floor of the mouth.
D.  gingiva.

In the absence of “Hanks balanced salt solution”, what is the most appropriate media to transport an avulsed?
A.  Saliva.
B.  Milk.
C.  Saline.
D.  Tap water.

Which of the following is the most likely cause of osteoporosis, glaucoma, hypertension and peptic ulcers in a 65 year old with Crohn’s disease?
A.  Uncontrolled diabetes.
B.  Systemic corticosteroid therapy.
C.  Chronic renal failure.
D.  Prolonged NSAID therapy.
E.  Malabsorption syndrome.

Answer 1

• Ctrl+H
• Find what: (([^?]+\?\R(?:.+\.\R)+)[\s\S]+?)\2
• Replace with: $1
• check Wrap around
• check Regular expression
• DO NOT CHECK . matches newline
• Replace all

Explanation:

(           : start group 1
  (         : start group 2
    [^?]+   : 1 or more any character that is not "?"
    \?      : a question mark
    \R      : any kind of line break
    (?:     : start non capture group
      .+    : 1 or more any character but newline
      \.    : a dot
      \R    : any kind of line break
    )+      : end group, must appear 1 or more times
  )         : end group 2
  [\s\S]+?  : 1 or more any character, not greedy
)           : end group 1
\2          : another occurrence of group 2

Replacement:

$1          : content of group 1

Answer 2

Technically in the given input there are no duplicates as ‘A. buccal mucosa.’ and ‘A. buccal mucosa.’ differ in number of spaces after 'A.'.

However, the intuition suggests that such cases should be somehow spotted.

As you mentioned in the comment that you are using https://regex101.com/ I will use this webpage to do the matches and replaces.
I selected flavor: javascript and set two flags in the regular expression section: g (global) and s (single line).

I will use 3 patterns to deal with this string.

The first pattern searches for identical question_and_answer occurences. If there are discrepancies between them, those won't be treated as duplicates.
If there are more than one duplicate all of them will be captured.

(?<=^|\n)([^\n]+)(\n)(\D{1}\.\s+[^^]+?)(\n{2})(?=[^^]*\1\n\3)

Input (TEST STRING):

Question 1?
A. SomeA1.
B. SomeB1.

Question 1?
A.     SomeA1.
B.     SomeB1.

Question 1?
A. SomeA1.
B. SomeB1.

Output (SUBSTITUTION): // one duplicate removed

Question 1?
A.     SomeA1.
B.     SomeB1.

Question 1?
A. SomeA1.
B. SomeB1.

If we just want to find duplicates based on the questions then this pattern should work but should only be used for information only.

(?<=^|\n)([^\n]+)(\n)(\D{1}\.\s+[^^]+?)(\n{2})(?=[^^]*\1)

Input (TEST STRING):

Question 1?
A. SomeA1.
B. SomeB1.

Question 1?
A.     SomeA1.
B.     SomeB1.

Question 1?
A. SomeA1.
B. SomeB1.

Output (SUBSTITUTION): // it looks ok, but it's a trick and must be used with caution

Question 1?
A. SomeA1.
B. SomeB1.

Ideally if we know what kind of deviations we might find in the data we can clean the data before the first pattern is applied, like in the following example where multiple spaces are replaced with just one space.

Find: (?<=\n)(\D{1}\.)(\s+)([^^]+?\n)
Substitution: \1 \3

Input (TEST STRING):

Lichen planus occurs most frequently on the?
A.  buccal mucosa.
B.             tongue.
C.                        floor of the mouth.
D.  gingiva.

Output (SUBSTITUTION):

Lichen planus occurs most frequently on the?
A. buccal mucosa.
B. tongue.
C. floor of the mouth.
D. gingiva.