Old news for Apache/Linux folks, but I’ve spent way more time with IIS than Apache. Mod_rewrite is very cool. So a combination of search engine friendly urls, wget, and Lucene could allow for full text search of a dynamic site without spidering. Nice.