What is the oldest instance of magical familiars being used by witches in fiction, and also called familiars, or a cognate? For example, an English novel having familiars and calling them 'familiars', or a French novel having familiars, and calling then 'familier' (whatever the plural of that is...)
The Online Etymology Dictionary has the first use of Familiar to mean a spirit being from 1580s, but that's for things like the Malleus Maleficarum, I think?
If an illustration of a dubious event counts as fiction, then this late 1500s sketch could be the first reference to familiars:
The author is unknown, but there is a description here:
An image of a witch and her familiar spirits taken from a publication that dealt with the witch trials of Elizabeth Stile, Mother Dutten, Mother Devell and Mother Margaret in Windsor, 1579.
Whether the term "familiar" would have been used by the author at the time is unclear. However, the sketch was made some time after 1579 and in 1584 we find what is likely the first appearance of the term "familiars" in this context within the English corpus. This appears in the book Scot's Discovery of Witchcraft, edited by Reginald Scot. (This could be earlier than Malleus Maleficarum.)
The term appears in the very long subtitle of the book:
Proving the Common Opinions of Witches Contracting with Divels, Spirits, Or Familiars, and Their Power to Kill, Torment, and Consume the Bodies of Men, Women, and Children, Or Other Creatures by Diseases Or Otherwise, Their Flying in the Air...
This is less than half of the subtitle, by the way.
This reference, however, is not a work of fiction (at least, it wasn't intended to be a work of fiction — beliefs were a little different back then).
I believe we should refer to the French corpus for the first clear appearance in fiction. The first usage seems to be in the 1857 poetry collection Les fleurs du mal (The Flowers of Evil) by Charles Baudelaire.
The French poet Charles Baudelaire, a cat fancier, believed in familiar spirits:
It is the familiar spirit of the place;
It judges, presides, inspires Everything in its empire; It is perhaps a fairy or a god? When my eyes, drawn like a magnet To this cat that I love...
(Source, in English)
There are of course earlier instances of the concept in both languages, in folklore and tales, etc. However, I have focused on the appearance of the actual term, as requested by the OP.
The OED gives 1583 as the first instance of this noun usage in English, by Arthur Golding, in a translation from Jean Calvin’s French:
1583 A. Golding, tr. J. Calvin Serm. on Deuteronomie cviii. 661, A Sorcerer, or a charmer, or [he] that asketh Counsell at spirites that are called familiars [Fr. esprits familiers].
As a translation, this comes directly from the French esprit familier; the analogous familiar spirit is already attested in English from 1545, with roughly the same meaning:
a 1545 T. Lanquet Epitome of Chron. (1559) iii. f. 233v, [He] confessed, that he dyd it by the mocion of a familyar spirite.
and the OED traces these in turn back to Latin spiritus familiaris in this same sense, from the 15th century, though without further details.
One wonders: why did Golding translate esprits familiers as familiars, rather than the more direct and already established familiar spirits? I am speculating here, but he might well have had in mind another now-archaic noun sense of familiar, the OED’s sense 1a:
1a. A member of a person's household or family, esp. a servant of a person of high rank. Hence in wider use: a subordinate. Now hist. and rare.
This dates back to 1250, and seems to have still been in standard currency in the 16th century; so perhaps this was what Golding had in mind when he translated esprits familiers as familiars.
(The OED also suggests a connection with familiar devil (1464); but their 1464 citation for familiar devil seems to have a rather different meaning, not obviously connected to the “witch’s spirit servant” sense, and its first citation in this sense is in 1599, later than the familiar spirit examples above.)
