Larbin

Larbin is an HTTP Web crawler that can fetch more than 5 million pages a day on a standard PC (pentium II 300, 128 Mo SDRAM and a 10 Mbit ethernet card, with a good network). Larbin uses standard libraries, plus adns. The program is multithreaded but prefers using select instead of a lot of threads (for efficiency purposes). The advantage of Larbin over wget or ht://dig is that it is much faster (because it opens a lot of connexions at a time) and very easy to customize).

Common uses include: a crawler for a standard search engine, a crawler for a specialized search engine (xml, images, mp3...), and to provide statistics about servers or page contents).

Last updated 24 Jun, 2008


User level: Submit a level

User Rating:

Homepage

License(s) :

GPLv2

Rate it!

 

About

Leadership
Related Projects

Wget, ht://Dig

Versions

2.6.3

2.6.3 stable released 2002-07-16

User Community and Support

User guide included

General Resources
Support Resources

Development

Developer Resources
Bug Tracking Resources
 

Please send comments on these web pages to bug-directory@fsf.org, send other questions to info@fsf.org.

Copyright © 2000 - 2009 Free Software Foundation, Inc., 51 Franklin Street, 5th Floor, Boston, MA 02110-1301, USA

The copyright licensing notice below applies to this text. Any software described in this text has its own copyright notice and license, which can usually be found in the distribution itself.

Permission is granted to copy, distribute, and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts.