I am expanding on my homelab server, and have decided to use a small publicly hosted server as a simple proxy. My homelab will operate Mailcow and I want to use only port 587 for secure SMTP submission, but I want to make sure my domain or IP doesn't appear broken, off, or somehow anomalous to say - a deliverability scan or blacklist/greylist crawler or email client/etc.
If you're using a proxy for MX, and if the homelab host itself doesn't appear in DNS, then it doesn't matter what you do in the latter because the homelab host does not "represent" your domain – only the proxy, or whatever appears in your DNS MX records, is relevant.
The communications between your homelab and your proxy can use literally anything, whether it's SMTP with TLS, or cleartext SMTP, or dial-up UUCP; those are all internal matters and outside the scope of what a "deliverability scan" knows about.
I don't want to look like the server does not have SMTP available - but rather that secure SMTP is required on port 587.
This is completely useless for the purpose of TLS because the two ports work identically in the way TLS is set up. Both of them accept cleartext SMTP that is later upgraded via "STARTTLS" commands – so if your goal is to redirect to 587 purely for the sake of enabling STARTTLS usage, you can already enable the same STARTTLS on port 25.
(In general, there is no point in explicitly redirecting clients from port A to port B where they would receive the exact same service, you might as well just offer the service directly on port A.)
In fact, most SMTP MX servers out there already support STARTTLS on port 25 in the exact same way as they do on port 587 (Gmail has done a good job at "encouraging" domains to get rid of the red lock icon that it now shows for TLS-incapable MX servers).
(The difference between 25 and 587 is not so much about TLS as it is about authentication requirements, i.e. the purpose of separation is that port 587 is supposed to require user authentication and is strictly for injecting outbound messages (i.e. relaying allowed), while port 25 does not offer authentication and is supposed to remain strictly for inbound messages.)
Most domains generally do not require TLS on port 25 just yet, but that is because they still want to accept messages from MX clients that are incapable of TLS, not because port 25 would make that inherently impossible. You, however, can easily enforce the requirement on your own server, e.g. in Postfix all you need to do is copy the "smtpd_tls_security_level" setting in master.cf.
Similarly, most domains do not require TLS for outbound deliveries either – so even if you enforce TLS on your side, you could still be impersonated by a server that just pretends to have a bad TLS setup. If you want to tell other domains (those who are running modern setups, at least) that they should expect TLS to succeed when delivering to your MX servers (still on port 25), you can set up MTA-STS for that purpose – it is a bit like HSTS for HTTP.
Finally, if the TLS (or port) requirement is supposed to be for your internal homelab-to-proxy communications, then no redirects are needed – you can directly specify the port number in the relevant configuration, e.g. in the homelab's "smarthost address".
So I will have port 25 explicitly banned(reject) by IPtables to prevent resource drain from fraudulent access attempts, but I don't want to look like the server does not have SMTP available
For the purpose of deliverability from other domains, this exactly means that your server does not have SMTP available.
As mentioned above, the split between 25 and 587 is not about TLS at all – it is about having different authentication and relaying policies. Ports 465 and 587 are specifically for message submission by your authenticated clients and will not be used by external mail-exchange servers.
There is no way for you to indicate that a mail-exchange server should use a different port when talking to you; when they want to deliver you a message via MX records, it's over port 25 or nothing.
are there statuses that can go with the reject to explain it like HTTP error codes or anything like that?
For SMTP – there are (indeed SMTP status codes are the origin of HTTP codes), but none that would say "you need to use on a different port".
(Again, because that's functionally useless; the port number doesn't define what you can do on it.)
If you block the port via iptables so that clients cannot establish a connection at all – no, there's no ICMP error code that would redirect clients to another destination. (The existing "Redirect" packet has a different purpose and can only point to another gateway.)
maybe a simple TCP status service that just handles stateful rejections? so I can leave that port closed in IPtables, but not appear just down on SMTP.
No, this indeed does not make any sense. If the port is "closed in iptables" then that literally means there is no way for any TCP client to make a connection to anything on that port, whether it's a real SMTP service or just a "status service", so it will appear to be down for SMTP clients.
