The answer to this question is interesting, and it's that not serving HTTP doesn't actually help. The attacker HTTPS contemplates controls whether victims see SYN+ACK packets in response to their 80/tcp SYNs. TCP itself isn't authenticated. So you need something "sticky" in the browser to remind it not to try 80/tcp, and thus risk being bamboozled by a MITM attacker.
> The attacker HTTPS contemplates controls whether victims see SYN+ACK packets in response to their 80/tcp SYNs.
This informationally dense and adventurously worded sentence is the kind that you can only understand if you already understand it, it feels like. I certainly can't unpack it without getting my hiking gear on. Not this rainy morning, though, may the transport layer gods forgive me.
if an attacker is in the position to try to MITM TLS, they're in the position to just serve whatever they want on port 80 even if your server isn't doing that.