Considering jQuery is available on Google's CDN, and lots of websites just fetch it from there, it's not necessarily true that using a smaller library will actually translate into slower load times in most cases. Similarly, if you concatenate all your javascript together, that gets rid of the main culprit where slow page loads are concerned, which is latency, not file size.
I ended up porting from jQuery to Zepto for my mobile PhoneGap app, where the files are stored locally. Porting to Zepto resulted in performance gains due to the browser taking less time to actually parse the JavaScript. Here's a blog post I wrote with more info and performance comparison: http://blog.pamelafox.org/2011/11/porting-from-jquery-to-zep...
That's not the whole point, even if it's in your cache it takes time to evaluate. Jquery and jqueryui together are taking 300ms in my chrome - blocking the page painting
Imagine loading that on a mobile device, where something like that could easily take a few seconds. That's another reason why you really don't want to use jQuery on mobile.
