It's a good example of the problem though. Any drive that relies on reaction mass will take too long to cross the distance to even nearby stars. The rocket equation kills the plan unless you're willing to make an absolutely ridiculous starship.
So you need a reactionless drive, which thus far is entirely science fiction.
This is even before you start thinking about micrometeorites in the way (which long transits increase the danger from), and simply maintaining complex machines in perfect working order for decades or centuries in the harsh vacuum of interstellar space with no external power source (no solar panels).
And that's before you start thinking about how we don't even know how to colonize an alien world, and unless you're willing to wait centuries to get intelligence on the planets you're flying in blind.
IMHO the answer to the Fermi Paradox is the most depressing one. Interstellar travel is too difficult and expensive. By the time it's feasible you are already well past the point of needing it. You can build endless orbital colonies and mine your home system and it's nearly unlimited resources forever. There is no need to embark on a colossally expensive boondoggle to colonize a distant solar system when everything you need is at home.
Yes, it's hard at our current tech level. But not nearly hard enough to explain the Fermi thing. I mean, take your scenario of solar system development: it'd be natural to exploit the Kuiper Belt and then the Oort, and the Oort blends into interstellar travel, as Freeman Dyson pointed out -- still quick on a geological timescale.
The problem with thinking about it on geologic timescales is that humans (and civilizations!) don't exist on geologic timescales.
It's going to be a tough sell to tell people "You need to pay an extra tax for your entire life so we can fund construction of an interstellar ship that your great grandchildren might witness the launch of, but won't arrive at its destination for a thousand years after that."
It's hard to imagine anything that could motivate the human race to do that short of the Vogon constructor fleet announcing that it will arrive within a couple of centuries, ready or not.
I meant https://en.wikipedia.org/wiki/Colonization_of_trans-Neptunia... where they're colonizing those spaces as a place to live -- supposing civilization has already spread through the solar system, we're not talking about needing some kind of super Manhattan project to spread a little farther out. You just need people who feel crowded in the already-full habitats.
Of course this is speculation, but it's one chink in the case for the cost of travel stopping life from spreading. Personally I expect fancier higher-tech faster means in the actual future, like Forward's Star Wisp scheme.
There is precedent: look at the funding and construction of European cathedrals. Often times the people who started on then where dead by the time they were finished
