I worked somewhere where we did this, though there they even could search the Internet. It has the benefits you describe, but the one downside is that if someone gets onto a wrong track (solving the wrong problem, getting hung up on some immaterial detail, etc) then no one is there to help them along (most candidates, despite urging, seem not to ask), so you have to be more careful with the 'test' materials or you won't get a lot of signal variation for differing skill levels. Before one overhaul we had a very bimodal (got it easily or didn't get it at all) distribution.

It's a bit of an art to come up with tasks that let you accurately rank candidates.