It works for ssh because you generally know the parties you are trying to connect to ahead of time and have ways to communicate with them (to confirm keys etc).
You do not have the same relationship with 99.99% of the servers you will be connecting to over https. It does not work.
Actually I don't visit webservers with a uniform random distribution. I come back to some frequently (eg. my bank, HN, BBC News, ...) Maybe that's just me though.
It works for ssh because you generally know the parties you are trying to connect to ahead of time and have ways to communicate with them (to confirm keys etc).
You do not have the same relationship with 99.99% of the servers you will be connecting to over https. It does not work.