Skip to content

[Feature Request] Don't throw away already fetched data on timeout #4

@dector

Description

@dector

Severity: minor.

STR:

podman run --rm -v $(pwd):/workspace ghcr.io/vladkampov/markdocify:latest https://developer.android.com/

AR:
After 10 minutes getting

level=warning msg="Scraping timed out - consider increasing scraping_timeout in config" reason=scraping_timeout_exceeded timeout=10m0s
Error: scraping failed: context deadline exceeded
Usage:
  markdocify [URL] [flags]

Flags:
      --concurrency int   Number of concurrent workers (for URL mode) (default 3)
  -c, --config string     Configuration file path
  -d, --depth int         Maximum crawl depth (for URL mode) (default 8)
  -h, --help              help for markdocify
  -o, --output string     Output file path
  -v, --version           version for markdocify

Error: scraping failed: context deadline exceeded

and there is no (even partial) artefacts present.

ER:
There are some artefacts in workdir/temp-workdir present so user don't need to do scrapping everything again.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions