There are limits to its adaptability, especially for processing-heavy tasks like vision for which the brain has specialized hardware. There is a world of difference between interpreting 144 electrodes to pick out just a couple of bits of information, and seeing.

In particular, if the brain were infinitely plastic then you could just wire the output from a HD video camera to some random spot on the brain and wait for the brain to figure it out.