TL;DR: It Was Done, But Timing Wasn't Helpful
There are at least two technically successful, but economically unsuccessful, 100% (or close enough) clones:
Neither made any substantial impact.
It Wasn't Task and Time to be Done
For one, the Mac ROMs are a whole OS in 64 KiB on ROM. That's eight times the size of the PC-BIOS. Heck, DOS 1.x was only 9 KiB, even combines just a fraction of MacOS.
Next it seems as if the question is based on an oversimplified view of the development involved. It was way more nuanced and took years until the 100% clones became a thing.
Last but not least, it needed a killer application to make buying 100% clones good idea - for the PC it was Lotus 1-2-3 (*1).
Clones Didn't Start the x86 Business.
As is well recorded, in the PC/MS-DOS world, Compaq and Phoenix (among others) successfully "clean-room" implemented their own versions of the IBM BIOS, thus opening the door to the PC clone market.
It's important to consider that the IBM PC was neither the first, nor the unchallenged and instant success as it seems in hindsight. It's real success goes hand in hand with the availability of 100% clones, which in turn only started after x86 PCs established MS-DOS as a major OS.
While MS produced PC-DOS at first only for IBM, CP/M86 (*2) and 86-DOS were already available for 8086/88 based computers. Major competition for IBM, like the Sirius 1, existed before the IBM-PC was introduced (*3). In addition MS started supplying their MS-DOS for companies like Lifeboat and Sirius with their non-clones. Other companies like Eagle and CDP followed with similar machines.
The PC clone market did not start until MS-DOS became a major OS first. It was more than two years after the first 86-DOS and 1.5 years after the IBM-PC that Compaq brought out the first 100% clone. In fact, Compaq's plans were a huge gamble, as until then MS-DOS applications were usually rather clean products with none or only limited machine dependency. They were just extremely lucky that at about the same time Lotus 1-2-3 became available (*4), and having a 100% clone simplified software logistics.
Size Matters
[Wikipedia] seems to suggest that the main reason nobody was able to "clean-room" the Mac ROMs was because they were much larger and more complex. The implication being that the effort involved in reverse engineering and creating the clean-room implementation could not have been done in a commercially viable manner.
The Mac ROMs aren't just 8 KiB of rather simple I/O routines but a full fledged OS in 64 KiB - almost 4 times the combination of BIOS (8 KiB) and DOS (9 KiB). It's safe to assume that complexity does grow exponentially with size. So while redoing the BIOS is a job a single developer, even a hobbyist (*5), can do, recreating MacOS is a way larger task. And all of that for a machine specific OS with little sales so far?
It helps to have a look at the numbers. In 1984, when the Mac was released, Apple sold a whooping 250,000 units, but IBM a good 4 times that. IBM's PC revenue in 1984 alone, 4 bln USD, was twice that of Apple as a whole - which at the time and for years to follow was all dependent on Apple II sales(*6). The Mac did sell, but not enough to even fully cover its costs, despite the high price.
Where could a potential clone maker see a market?
How close to the truth is this?
Kinda. As shown the Mac market was comparably small, thus any investment must return a high per unit return - unlike the PC market where volume was king. This combined with high investment is exactly the firewall Apple created (*7).
All (somewhat) successful solutions thus worked around it by using Apple parts, particular in two ways at opposite ends of the market:
High end suppliers found their sub niche by taking Apple components and integrating them with additional features like more RAM, faster CPU and better screens(*8), while.
Low end competition offered bare bone add-on solutions for existing computers where users had to add their own set of genuine Mac ROMS (*9).
It Was Done Anyway
In particular, did anyone ever actually try to create a clean-room implementation,
There were two major attempts:
In Unitron's case it was a real clean room redevelopment, completely rewriting the whole ROM. It may be the special case of a large but protected market - Brazil - that enabled the effort.
NuTek's idea was different. They created a mostly compatible hardware (chipset) and a compatible OS using Motif as GUI base, that way saving a lot of coding effort. That way all software using official OS entry points and/or direct hardware access would run fine. Software using ROM internals might not. The approach can be compared to running DR-DOS on a hardware compatible PC clone.
and if so what was Apple's reaction?
In case of Unitron Apple used a lot of US political power to pressure the Brazilian government into stopping the project (*10). In contrast NuTek did make it to market with two of their own clones as did several of their customers (*11). Except it mostly fizzled as it came out right at a time when Sculley switched to include low priced offerings like the LC series and the Classic Macs. A move that was continued by Spindler. With Macs at more competitive pricing, clones had a hard time to gather any notable market share.
*1 - As always, the guys handling the money spend it first to please themself.
*2 - IBM not only intended CP/M-86 as the OS for the PC, but also started selling it
*3 - In fact, IBM delayed the market entry for the PC in major European markets (UK and Germany) for almost two years, as they could not imagine gaining any market share against the Sirius.
*4 - Compaq's hard push with the Flight Simulator was pure marketing genius. Wile it was complex to show any advantage of their clone with business software, running a game is something even the least informed 'journalist' could do and write about.
*5 - Nicely shown by Greg's fine GlaBIOS, developed in less than a year (first functional version) as an after work hobby project.
*6 - In 1984 Apple IIe sales were a good 1 million, about as many as IBM sold PCs. In hindsight the most successful II year ever - after that the PC took over.
*7 - Rather involuntarily as it's not hard to imagine that they would have loved to be a leader in volume :))
*8 - Such machines ranged from Portables like the MacMobile all the way to nifty constructs like Colby's Walkmac and Classmate.
*9 - Quite popular with the Atari ST and offers like Magic Sac. The solution consisted of cartridge for the ROMs and a loader patching hardware related functions to fit the ST hardware. Converting an Atari into a Mac could be done for less than 200 USD ... plus ROMs. Using a Stacy resulted in a portable Mac before the Mac Portable at a fraction of the price.
*10 - May have not needed that much as Brazil was in a deep economic crisis after a military dictatorship - so quite dependent on good will from major trade partners.
*11 - NuTek saw themself as a chipset maker, selling their chips to clone makers. A business model quite successfully demonstrated at the time by PC chipset makers.
