Where can one get large list of urls?
Verisign has there “TLD Zone Access Program”, where one can get all the .com, .net and .edu domains. See http://www.verisign.com/information-services/naming-services/com-net-registry/page_001052.html and http://www.verisign.com/information-services/naming-services/com-net-registry/page_001051.html
Dmoz has a RDF dumb off there database of about 5 mill urls here http://rdf.dmoz.org/
Anyone knowing about any other sources?