Best technology(s) for web crawling [closed]

manpreet Best Answer 2 years ago

2ND EDIT: I guess the suggested library must be the most efficient and best in the world without room for improvement as nobody even attempted to answer the questions regarding the technology, only down voted as if I hadn't already had a library for this.

SO is very clear in their down voting rules:

Instead of voting down: If the post is spammy or offensive, flag it. If the question is duplicate or off-topic, flag it for moderator attention. If something is wrong, please leave a comment or edit the post to correct it.

EDIT: Not sure why down voted, however I got one of the answers I wanted.

What would be the best technology, language, et cetera for creating a web crawler (In terms of finding the actual URIs/URLs inside of the HTML)?

Things I have considered and tried: - C# Substring methods (String manipulation) - Regex - Xslt transformation / XPath

Is there some sort of standard for this? Are there already libraries for this?

Would also like to be able to include IP addresses