Search results

Throttle requests to a domain by total bandwidth in a specified period of time

enhancement

fetch

parse

Status: Open.

#19 In ScottMansfield/widow;

· ScottMansfield opened on Sep 22, 2015

All outbound requests should have a User-Agent attached to them

enhancement

fetch

Minor

parse

Status: Open.

#18 In ScottMansfield/widow;

· ScottMansfield opened on Jun 18, 2015

Check for robots meta tag while parsing a page

enhancement

Medium

parse

Status: Open.

#15 In ScottMansfield/widow;

· ScottMansfield opened on May 26, 2015

Add support for @import in CSS files and inline <style> contents

bug

enhancement

parse

Status: Open.

#13 In ScottMansfield/widow;

· ScottMansfield opened on May 26, 2015

mailto:, phone:, etc break parsing

bug

enhancement

Major

parse

Status: Open.

#11 In ScottMansfield/widow;

· ScottMansfield opened on May 19, 2015

Implment rate-limiting on a per-host basis

enhancement

fetch

Major

Status: Open.

#10 In ScottMansfield/widow;

· ScottMansfield opened on May 12, 2015

Add links by content type to the main page data

enhancement

parse

Status: Open.

#9 In ScottMansfield/widow;

· ScottMansfield opened on May 12, 2015

Have a better story around local caching independent of the crawling stages

enhancement

fetch

index

Major

parse

Status: Open.

#6 In ScottMansfield/widow;

· ScottMansfield opened on May 8, 2015

Investigate more accurate timing of website response

enhancement

fetch

Minor

Status: Open.

#5 In ScottMansfield/widow;

· ScottMansfield opened on May 8, 2015

Support for sitemap.xml

fetch

Major

parse

Status: Open.

#4 In ScottMansfield/widow;

· ScottMansfield opened on May 8, 2015

Add support for If-Modified-Since and ETag headers

enhancement

fetch

Major

parse

Status: Open.

#3 In ScottMansfield/widow;

· ScottMansfield opened on May 8, 2015

Add support for robots.txt for any website

bug

enhancement

fetch

Major

parse

Status: Open.

#2 In ScottMansfield/widow;

· ScottMansfield opened on May 8, 2015