Creating a bot/crawler

manpreet Best Answer 2 years ago

_x000D_ There are two parts to this. Get the page from the remote site. Read the page into a class that you can parse. For the first part, if you are planning on running this client-side, you are likely to run into cross-site issues, in that your page, served from server X, cannot request pages from server Y, unless the correct headers are set. See: CORS with Dart, how do I get it to work? and Dart application and cross domain policy or the site in question needs to be returning the correct CORS headers. Assuming that you can actually get the pages from the remote site client-side, you can use HttpRequest to retrieve the actual content: // snippet of code... new HttpRequest.get("http://www.example.com", (req) { // process the req.responseText }); You can also use HttpRequest.getWithCredentials. If the site has some custom login, then you will probably problems (as you will likely be having to Http POST the username and password from your site into their server - This is when the second part comes in. You can process your HTML using the DocumentFragment.html(...) constructor, which gives you a nodes collection that you can iterate and recurse through. The example below shows this for a static block of html, but you could use the data returned from the HttpRequest above. import 'dart:html'; void main() { var d = new DocumentFragment.html(""" Foo """); // print the content of the top level nods d.nodes.forEach((node) => print(node.text)); // prints "Foo" // real-world - use recursion to go down the hierarchy. } I'm guessing (not having written a spider before) that you'd be wanting to pull out specific tags at specific locations / depths to sum as your results, and also add urls in hyperlinks to a queue that your bot will navigate into.