Also, you don't HAVE to compile everything from scratch during installs. Gentoo has had Gentoo Reference Platform (GRP) Packages available for quite some time now. They are a snapshot of prebuilt packages people can use during install rather than build their own. The only downside is that the packages aren't maintained and are only created during every release. Which is about twice a year. So you would have to keep up on the security issues and build new packages as they come out. However, you don't have to emerge the entire world. `glsa-check -p affected` from app-portage/gentoolkit will emerge those packages with known security holes.
If you do come across some dependence issues or broken packages, say a package is linked to a specific library version. `revdep-rebuild -av`, also from app-portage/gentoolkit, scans all your binaries for missing libraries, figures out which package the binary belongs, and reemerge those ebuilds. No more broken packages...
I would have to agree that on a slower server, building from source is less than ideal. However, you can always do it during off-peak hours. If you kill the emerge process, you can always start it back up where it left off with the `emerge --resume`.
Also, you can use ccache, which is a compiler cache utility. http://gentoo-wiki.com/Ccache
"It uses the GCC -E switch and a hash to detect when a compilation can be satisfied from cache. The effect is that packages frequently compile 5-10 times faster than they would otherwise.
The first time that you emerge a package after setting up ccache, there will be a very slight increase in its compilation time, but thereafter (when the compilation is cached), you will notice significant reductions."
Gentoo "Best Practices" http://gentoo-wiki.com/HOWTO_Maintain_Gentoo_-_%22Best_Practices%22