(covers information from several alternate timelines)
Madam, adapted from the French madame or "my lady" and abbreviated as ma'am, was a polite or formal title used to address a woman, often one of stature or higher ranking.
In 19th century Earth, the El-Aurian visitor Guinan was addressed by her peers, such as Samuel Clemens, as "Madam Guinan." (TNG: "Time's Arrow", "Time's Arrow, Part II")
While visiting an alternate 2337, Spock, while posing as a cousin named Selek, referred to his mother Amanda as "my lady Amanda." (TAS: "Yesteryear")
Spock acknowledged Azetbur as "madam", and following her being Klingon Chancellor, she was address by the Federation President, and later Captain Kirk, as "Madam Chancellor". (Star Trek VI: The Undiscovered Country) Jean-Luc Picard also acknowledged Kahlest as "madam". (TNG: "Sins of The Father") Whereas also, the term "my lady" was also commonly used in the Klingon Empire for women of high statue, such as to Grilka by Tumek; to Sirella by her husband, General Martok, and by Jadzia Dax. (DS9: "Looking for par'Mach in All the Wrong Places", "You Are Cordially Invited")
In Starfleet parlance, junior officers as well as enlisted officers frequently addressed superior officers as "ma'am". (TOS: "The Cage", "Court Martial", Star Trek VI: The Undiscovered Country, et al.) When Harry Kim first joined the bridge crew of USS Voyager, he initially addressed Captain Kathryn Janeway as "sir", which despite Starfleet protocol, she didn't care to be addressed as. He apologized, addressing her as "ma'am", which she acknowledged was "acceptable in a crunch, but I prefer Captain." Some time later, he would address her as "ma'am", and she told him that "It's not crunch time yet, Mister Kim. I'll let you know when." (VOY: "Caretaker") In 2375, after Kim had succumb to olan'vora, which he had transmitted from Derran Tal, and was instructed by Captain Janeway to stop seeing her, he responded by stating that he had "served on this ship for five years and said, "yes, ma'am" to every one of your orders. But not this time." (VOY: "The Disease")
The Bajoran arbiter Els Renora was addressed as "Madam Arbiter" in 2369. (DS9: "Dax")
Later that year, after being given the duty of escorting several visiting Federation ambassadors around Deep Space 9, Julian Bashir mistakenly addressed Ambassador Taxco, simply as "Ambassador". She corrected him mid-sentence that it was "Madam Ambassador", a formality later not overlooked by Commander Benjamin Sisko. Later, after Bashir had saved the lives of the ambassador and her colleagues, the sufficiently impressed ambassador responded to Bashir's final "Madam Ambassador", stating "Please, Julian. Call me Taxco." (DS9: "The Forsaken")
When the Trill initiate Arjin trained with Jadzia Dax, he frequently referred to her as "ma'am," up to the point she finally told him, "Arjin, if you truly want to become a Trill host someday, you'll never call me ma'am again." He responded instead addressing her by her rank, Lieutenant, but she insisted he call her "Jadzia". (DS9: "Playing God")
In Cardassian court the presiding head of the Supreme Tribunal, Makbar, was addressed as "Madam Archon" in 2370. (DS9: "Tribunal")
In 2371, in Janeway's Gothic holonovel, Janeway Lambda One, one of the primary children characters, Henry Burleigh introduced himself and his sister, Beatrice Flora, to Janeway's character Mrs. Davenport, noting that "You will address us as my Lord and my Lady." (VOY: "Learning Curve")
During Captain Janeway's first encounter with Q in 2372, he addressed her as "Madam Captain," which she did not take kindly, going so far as to add, "And please don't call me Madam Captain." (VOY: "Death Wish")
T'Rina and Laira Rillak were addressed by Captain Saru and Captain Michael Burnham as "Madam President."