storing a large index; searching it

2 questions:

i need to crawl a large site, and basiclly create a hash like this:

key = url, value = page contents

i later need to search this hash for urls.

i am concerned that if the website im parsing is 2gb, i will run out of memory, not to metion the fact that i dont want to use more then 256 megs of ram.

if i store this hash to disk, i will still need to load it to search it.

is there a solution for this that does not involve interacting with databases?

2nd question:

has anyone here used lucene in the past and comfortable with it?

[598 byte] By [mkoryaka] at [2007-11-26 13:10:45]