Hello,
I am building a searchengine with php. Nearly finished.
Now, I need to build the web crawler.
Since most websites have xml sitemap for web crawlers to use to find their site links, I prefer to build an xml site map crawler than a general http crawler.
I am not having much luck finding a php tutorial on it. Using these keywords on Google:
"sitemap crawler"+"php tutorial" OR "php", -"sitemap generator"
If you know of any tutorial or free code then let me know.
The simpler the code, the better.
I will keep you updated on this thread to what I have found so far. If you find any better than mine then drop me a line here.
Thanks