Heritrix icon

Heritrix For Mac

  n/a
GPL   

Open source, extensible, web-scale, and archival-quality web crawler. #Web crawler  #Website archiver  #Website fetcher  #Crawl  #Archive  #Fetch  

Description

changelog

Free Download

Heritrix is an open source flexible, robust, extensible, and scalable web crawler capable of fetching, archiving, and analyzing the full diversity and breadth of internet-accessible content.

Heritrix (sometimes spelled heretrix, or misspelled or mis-said as heratrix/heritix/ heretix/heratix) is an archaic word for heiress (woman who inherits).

What's new in Heritrix 3.2.0:

  • Minor version number bumped because there are some changes that are not backward-compatible. For one, heritrix now requires java 1.6. It's also been a while since the last release, and there some some significant new features, and lots of other changes.
  • URL-agnostic deduplication (HER-2022) - Deduplication for identical content, even at different urls. Configuration is documented here: Duplication Reduction Processors
  • Automatic form login (HER-2031) - With new processors ExtractorHTMLForms and FormLoginProcessor, heritrix can detect typical html login forms and submit a configured username and password to them. Configuration is documented at http://builds.archive.org/javadoc/heritrix-3.x-snapshot/org/archive/modules/forms/ExtractorHTMLForms.html
  • Option to forget all but latest checkpoint (HER-2051) - Enable periodic checkpointing without worrying that it will overflow the disk. When enabled, earlier checkpointed logs are rolled up into new checkpointed log, earlier bdb checkpoint dirs are deleted and earlier checkpoint dirs are deleted. Related is HER-2056, option not to rollover warcs on checkpoint.
Read the full changelog
User Comments
This enables Disqus, Inc. to process some of your data. Disqus privacy policy

Heritrix 3.2.0

add to watchlist add to download basket send us an update REPORT
  runs on:
Mac OS X (PPC & Intel)
  file size:
32.9 MB
  filename:
heritrix-3.2.0-dist.zip
  main category:
Internet Utilities
  developer:
  visit homepage