"Could be known to an attacker" is not the same as "Is known to an attacker". Certainly, the more determined your attacker is, the more those two converge, and thus you should treat them as indistinguishable in defender against determined attackers. But by Github publishing all users' public SSH keys, they've made the "could be known" into "known" for even casual attackers.