Cannot expect Open source binaries to be signed as those are free.
There is no contradiction here: signing is free too. I don't think this helps though, because someone could maliciously build from modified sources and then sign the binary. So signing only works if you trust the original builder and want to make sure that the binary is indeed what they have provided and not an altered version.
It sounds like you're looking for reproducible builds. For this to work the project has to be based on a tech stack and toolchain that support reproducible builds. If it's not, then you're out of luck. If it is, you can build it yourself and compare results. You're right that this may not be practical for large projects. Unfortunately security oftentimes comes at the cost of convenience.
It's worth noting that ultimately you're also trusting that the toolchain isn't malicious. Same for the hardware you're building on. You will have to trust someone, unless you're willing to build the hardware yourself and reverse engineer entire toolchain.