Skip to content
@toimik

Toimik

Open source projects for an upcoming web search engine
  • Singapore

Pinned Loading

  1. WarcProtocol WarcProtocol Public

    Parser for WARC (aka WebArchive) files

    C# 11 3

  2. CommonCrawl CommonCrawl Public

    Common Crawl's processing tools

    C# 9

  3. UrlNormalization UrlNormalization Public

    URL normalizer to canonicalize (standardize) the text representation of a URL to determine if differently-formatted URLs are identical

    C# 4

  4. SitemapsProtocol SitemapsProtocol Public

    Parsers for sitemap / sitemap index (aka Sitemaps Protocol)

    C#

  5. RobotsProtocol RobotsProtocol Public

    Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag

    C#

Repositories

Showing 7 of 7 repositories
  • CommonCrawl Public

    Common Crawl's processing tools

    toimik/CommonCrawl’s past year of commit activity
    C# 9 Apache-2.0 0 0 0 Updated Oct 15, 2024
  • SitemapsProtocol Public

    Parsers for sitemap / sitemap index (aka Sitemaps Protocol)

    toimik/SitemapsProtocol’s past year of commit activity
    C# 0 Apache-2.0 0 0 0 Updated Jul 15, 2024
  • WarcProtocol Public

    Parser for WARC (aka WebArchive) files

    toimik/WarcProtocol’s past year of commit activity
    C# 11 Apache-2.0 3 0 0 Updated Jul 9, 2024
  • Wikimedia Public

    Wikimedia Downloads' processing tools

    toimik/Wikimedia’s past year of commit activity
    C# 0 Apache-2.0 0 0 0 Updated May 2, 2024
  • UrlNormalization Public

    URL normalizer to canonicalize (standardize) the text representation of a URL to determine if differently-formatted URLs are identical

    toimik/UrlNormalization’s past year of commit activity
    C# 4 Apache-2.0 0 0 0 Updated May 2, 2024
  • RobotsProtocol Public

    Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag

    toimik/RobotsProtocol’s past year of commit activity
    C# 0 Apache-2.0 0 0 0 Updated May 2, 2024
  • IpAddressEnumeration Public

    IP address enumerators

    toimik/IpAddressEnumeration’s past year of commit activity
    C# 0 Apache-2.0 0 0 0 Updated May 2, 2024

Top languages

Loading…

Most used topics

Loading…