First, let me clear immediately a misconception: avesse capito and avevi detto are not at the same tense: the first is at the "congiuntivo trapassato", but the second is at the "indicativo trapassato prossimo" (notice that the auxiliary verb is at the "indicativo imperfetto"). The "congiuntivo trapassato" of dire would be avessi detto. Neither of them is at the "congiuntivo imperfetto": that would be capisse and dicessi respectively
That said, let's have a look at the rules about tenses in the subordinates.
The period here can be analized as
Dubitavo (main clause)
|
che Marta avesse capito ciò (subordinata oggettiva)
|
che le avevi detto (subordinata relativa)
The tense of the main clause is at the "indicativo imperfetto", and in the first subordinate clause we need to express a sense of anteriority using the subjunctive mood (why we're using the subjunctive mood is a whole other topic althogher). There is a handy table in Serianni's Italiano about this (XIV.55), but the upshot is that to express anteriority with respect to a main clause set in the past we need to use the "congiuntivo trapassato", that is avesse capito.
Now, let's take a look at the second subordinate clause. This is a relative clause ("subordinata relativa"), because che here is a pronoun and not a conjunction (one quick way of recognizing this is to see if you could translate it with who, whom, which, what in English). In this case (when explicating a property of an object) these subordinates are usually at the indicative, as this is. The tense of choice is the indicativo trapassato prossimo, which indicates a further anteriority with respect to the clause it is referring to ("che Marta avesse capito ciò").
In particular here the author wants to emphasize that the act of you saying comes before the act of Marta understanding. If the speaker wanted to express them as simultaneous, they would have used the indicativo imperfetto (that is "...che le dicevi").
