Open source, extensible, web-scale, and archival-quality web crawler. #Web crawler #Website archiver #Website fetcher #Crawl #Archive #Fetch
Heritrix is an open source flexible, robust, extensible, and scalable web crawler capable of fetching, archiving, and analyzing the full diversity and breadth of internet-accessible content.
Heritrix (sometimes spelled heretrix, or misspelled or mis-said as heratrix/heritix/ heretix/heratix) is an archaic word for heiress (woman who inherits).
What's new in Heritrix 3.2.0:
- Minor version number bumped because there are some changes that are not backward-compatible. For one, heritrix now requires java 1.6. It's also been a while since the last release, and there some some significant new features, and lots of other changes.
- URL-agnostic deduplication (HER-2022) - Deduplication for identical content, even at different urls. Configuration is documented here: Duplication Reduction Processors
- Automatic form login (HER-2031) - With new processors ExtractorHTMLForms and FormLoginProcessor, heritrix can detect typical html login forms and submit a configured username and password to them. Configuration is documented at http://builds.archive.org/javadoc/heritrix-3.x-snapshot/org/archive/modules/forms/ExtractorHTMLForms.html
- Option to forget all but latest checkpoint (HER-2051) - Enable periodic checkpointing without worrying that it will overflow the disk. When enabled, earlier checkpointed logs are rolled up into new checkpointed log, earlier bdb checkpoint dirs are deleted and earlier checkpoint dirs are deleted. Related is HER-2056, option not to rollover warcs on checkpoint.
Heritrix 3.2.0
add to watchlist add to download basket send us an update REPORT- runs on:
- Mac OS X (PPC & Intel)
- file size:
- 32.9 MB
- filename:
- heritrix-3.2.0-dist.zip
- main category:
- Internet Utilities
- developer:
- visit homepage
IrfanView
With support for a long list of plugins, this minimalistic utility helps you view images, as well as edit and convert them using a built-in batch mode
Microsoft Teams
Effortlessly chat, collaborate on projects, and transfer files within a business-like environment by employing this Microsoft-vetted application
calibre
Effortlessly keep your e-book library thoroughly organized with the help of the numerous features offered by this efficient and capable manager
Windows Sandbox Launcher
Set up the Windows Sandbox parameters to your specific requirements, with this dedicated launcher that features advanced parametrization
Zoom Client
The official desktop client for Zoom, the popular video conferencing and collaboration tool used by millions of people worldwide
4k Video Downloader
Export your favorite YouTube videos and playlists with this intuitive, lightweight program, built to facilitate downloading clips from the popular website
Context Menu Manager
Customize Windows’ original right-click context menu using this free, portable and open-source utility meant to enhance your workflow
ShareX
Capture your screen, create GIFs, and record videos through this versatile solution that includes various other amenities: an OCR scanner, image uploader, URL shortener, and much more
7-Zip
An intuitive application with a very good compression ratio that can help you not only create and extract archives, but also test them for errors
Bitdefender Antivirus Free
Feather-light and free antivirus solution from renowned developer that keeps the PC protected at all times from malware without requiring user configuration
% discount
ShareX
- ShareX
- 7-Zip
- Bitdefender Antivirus Free
- IrfanView
- Microsoft Teams
- calibre
- Windows Sandbox Launcher
- Zoom Client
- 4k Video Downloader
- Context Menu Manager
essentials
Click to load comments
This enables Disqus, Inc. to process some of your data. Disqus privacy policy