At my job, we do A LOT of UI testing using selenium. We mainly use the FireFoxDriver, and i did some basic tests comparing it to phantomjs, and phantomjs is much faster!
If we use the plain HtmlDriver than it might be as fast, but that doesn't use WebKit, nor has good support for C#... :(