home recent changes unmerged conflicts (0)
Projects/Lanvalia

Page source:
Lanvalia is a small [[wp:Web Crawler|Web Crawler]] for research purposes (data
mining). It is programmed in [[http://www.ruby-lang.org|Ruby]] using the
excellent [[http://www.sqlite.org|sqlite]] and
[[http://github.com/whymirror/hpricot|Hpricot]] libraries.

== User-Agents:
* Lanvalia/0.2-dev (+http://soultcer.net/lanvalia/; Ruby/1.8.6)
* Lanvalia/'''version''' (+http://soultcer.com/lanvalia/; robots.txt: Lanvalia)

== Blocking
Lanvalia obeys the [[wp:Robots Exclusion Standard|robots.txt exlusion
standard]] (only robots.txt file, no meta tags).

=== Example robots.txt entry
User-Agent: Lanvalia

Disallow: /

== Problems
If the bot is misbehaving, please [[Contact|contact]] me (don't forget to
include access log or at least time, ip and url).

# vim:tw=78