How to write a crawler?

manpreet Best Answer 2 years ago

_x000D_ You'll be reinventing the wheel, to be sure. But here's the basics: A list of unvisited URLs - seed this with one or more starting pages A list of visited URLs - so you don't go around in circles A set of rules for URLs you're not interested in - so you don't index the whole Internet Put these in persistent storage, so you can stop and start the crawler without losing state. Algorithm is: while(list of unvisited URLs is not empty) { take URL from list remove it from the unvisited list and add it to the visited list fetch content record whatever it is you want to about the content if content is HTML { parse out URLs from links foreach URL { if it matches your rules and it's not already in either the visited or unvisited list add it to the unvisited list } } }