Skip to content

Releases: thiswillbeyourgithub/wdoc

Release 4.0.1

07 Oct 23:02
@thiswillbeyourgithub thiswillbeyourgithub
4.0.1
fb49e60
Compare
Choose a tag to compare
Release 4.0.1 Latest
Latest

What's new

What's new

This release focuses on langfuse v3 compatibility and improved error handling.

🐛 Fixes

  • Langfuse v3 compatibility

    • [89f5132] Update callback import for langfuse v3
    • [07257e0] Use langfuse opentelemetry for v3

  • Document loading robustness

    • [3039bcf] Prevent crash when no documents remain after transform_documents
    • [101c7f7] Add assertion to verify documents were found

📝 Documentation

  • [56866d1] Add warning for using youtube audio backend instead of whisper or deepgram

🔧 Maintenance

  • [fb49e60] Bump version 4.0.0 → 4.0.1

Commits details since the last release

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

wdoc/utils/misc.py

wdoc/utils/loaders/youtube.py

wdoc/utils/misc.py

wdoc/utils/loaders/init.py

wdoc/utils/loaders/init.py

Contributors

thiswillbeyourgithub
Assets 2
Loading

Release 4.0.0

05 Oct 16:36
@thiswillbeyourgithub thiswillbeyourgithub
4.0.0
e1548c4
Compare
Choose a tag to compare
Release 4.0.0

What's new

What's new

This release focuses on major performance improvements through lazy loading and deferred imports, extensive code refactoring for better maintainability, and improved testing infrastructure.

⚡ Performance

🔧 Fixes

♻️ Refactoring

  • Modularized loaders: Split monolithic loader file into separate modules [df1a0ad, d3ed873, f0a3fce, b249068, 984a8d3, def441f, fb421cc]
    • Created dedicated files for PDF, Anki, URL, audio, HTML, and other loaders
    • Enabled lazy loading of loader modules [7fc5fad]
  • Extracted task-specific functions to separate modules:
    • Moved parse_doc to utils/tasks/parse.py [1c7c6e4]
    • Moved query/search retrieval logic to task modules [7982051, c2e6142]
    • Moved evaluate_doc_chain to shared_query_search.py [8965c48]
    • Extracted query splitting logic to shared utility [4bb54a5]
    • Moved source_replace to query.py [0ce5f4f]
    • Moved autoincrease_top_k to query.py [38e82b4]
  • Split search and query task methods with better type hints [1d94644, 824f395, 319b8eb]
  • Moved debug_exceptions to logger module [99cc99f]
  • Moved VectorStore filtering code to filters.py [de4ce57]
  • Added wdocSummary dataclass for type hinting [9fc51c0, 92f5c47]
  • Added lazy caching for all_texts property [79b1661, 7b45948]
  • Removed obsolete import_tricks.py [5116616]

🧪 Testing

  • Improved test cleanup and temp folder removal [a768642, 35ef63e, c149f5d, 913378a]
  • Better verbose output in cost tests [342ad3f]
  • Use Mistral for OpenRouter API tests (zero data retention) [8f511dd]
  • Added shell-based CLI test script for more reliable testing [cc74a84, 4170567]
  • Added check for wdoc[full] installation [7cb9a3c]
  • Updated Ollama embedding test to use embeddingsgemma [4d47631]
  • Improved test assertions with more info [3d0f947]

📦 Dependencies

  • Bumped langchain version [98fd2cb]
  • Bumped litellm version [7aa2ce1]
  • Bumped langfuse version (litellm bug fix) [fc16e5e]
  • Updated general dependencies [616457c]
  • Added unstructured to required dependencies [c98d0e9]
  • Added bumpver to dev packages [54be0e2]

✨ Features

  • Added wdoc[full] installation option for all optional dependencies [6321942]
  • Added beartype runtime type validation for numpy arrays [691dbff]
  • Prioritize throughput and Groq when using OpenRouter [f049846]
  • Enable lazy loading of imports by default [7c2e397]

📝 Documentation

  • Updated default models to latest Gemini in README and help [761ddd1, 0868086, 78e562f]
  • Clarified that binary embeddings are not always better [fb611c4]
  • Added link explaining fixed cache of LLM issue [fdc3c64]
  • Improved docstrings for summarization functions [a06f570]
  • Added docstring for VectorStore filtering [2bd8dcb]

🎨 Code Quality

🔖 Version

  • Bumped version 3.3.1 → 4.0.0 [e1548c4]

Commits details since the last release

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

setup.py

README.md

tests/run_all_tests.sh

tests/run_all_tests.sh

tests/run_all_tests.sh

tests/test_wdoc.py

tests/test_wdoc.py

tests/test_wdoc.py

tests/test_cli.sh

wdoc/wdoc.py

tests/run_all_tests.sh

wdoc/utils/tasks/query.py
wdoc/wdoc.py

wdoc/utils/llm.py
wdoc/utils/retrievers.py
wdoc/utils/tasks/query.py
wdoc/utils/tasks/summarize.py

wdoc/main.py
wdoc/wdoc.py

setup.py

setup.py

setup.py

wdoc/utils/misc.py

wdoc/utils/customs/litellm_embeddings.py
wdoc/utils/embeddings.py
wdoc/utils/llm.py
wdoc/utils/loaders/shared_audio.py
wdoc/utils/misc.py
wdoc/utils/retrievers.py
wdoc/utils/tasks/query.py
wdoc/utils/tasks/summarize.py
wdoc/wdoc.py

wdoc/init.py

wdoc/utils/loaders/shared_audio.py
wdoc/utils/misc.py

wdoc/utils/retrievers.py

wdoc/main.py
wdoc/utils/customs/binary_faiss_vectorstore.py
wdoc/utils/embeddings.py
wdoc/utils/env.py
wdoc/utils/interact.py
wdoc/utils/llm.py
wdoc/utils/misc.py
wdoc/utils/prompts.py
wdoc/utils/tasks/parse.py
wdoc/utils/tasks/query.py
wdoc/utils/tasks/search.py

Read more

Contributors

thiswillbeyourgithub
Loading

Release 3.3.1

26 Jul 08:24
@thiswillbeyourgithub thiswillbeyourgithub
3.3.1
0690df9
Compare
Choose a tag to compare
Release 3.3.1

What's new

This release focuses on improving code quality through comprehensive type hint fixes and enhanced testing infrastructure.

🔧 Fixes

  • Type Hints: Comprehensive type hint improvements across the codebase

  • Model Compatibility: Fixed issue where some models consider <answer> as implying </think> ([09684bb])

  • Langchain Integration: Fixed callable_chain compatibility by creating runnables without decorators ([0c89cac])

✨ Enhancements

  • Type Checking: Replaced manual type checking with import hook system ([56b353a], [6b3ddab])
  • Logging: Reduced verbosity of litellm logging ([9a4a69c])
  • Search: Added duplicate check for DuckDuckGo search results ([f68c8a4])

🧪 Tests

  • Added comprehensive test for DuckDuckGo search functionality ([7dbd3c2])
  • Fixed existing CLI tests ([781f6d6])

📦 Version

  • Bumped version from 3.3.0 to 3.3.1 ([0690df9])

Commits details since the last release

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

tests/test_wdoc.py

tests/test_cli.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/batch_file_loader.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/wdoc.py

wdoc/utils/loaders.py

  • [e65abad] by @thiswillbeyourgithub, 20 hours ago:
    Revert "fix: typehint of load_one_doc"
    This reverts commit f0037b54ac5ce317442e672f12e1da266b58c5c1.

wdoc/utils/loaders.py

wdoc/utils/loaders.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/tasks/query.py

wdoc/wdoc.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/tasks/query.py

wdoc/utils/misc.py

wdoc/utils/customs/callable_runnable.py
wdoc/utils/misc.py
wdoc/utils/tasks/query.py
wdoc/wdoc.py

wdoc/main.py
wdoc/utils/batch_file_loader.py
wdoc/utils/embeddings.py
wdoc/utils/interact.py
wdoc/utils/llm.py
wdoc/utils/loaders.py
wdoc/utils/logger.py
wdoc/utils/misc.py
wdoc/utils/prompts.py
wdoc/utils/retrievers.py
wdoc/utils/tasks/query.py
wdoc/utils/tasks/summarize.py
wdoc/utils/typechecker.py
wdoc/wdoc.py

wdoc/init.py
wdoc/utils/customs/callable_runnable.py
wdoc/utils/misc.py
wdoc/utils/tasks/query.py
wdoc/utils/typechecker.py
wdoc/wdoc.py

wdoc/utils/customs/binary_faiss_vectorstore.py

wdoc/utils/prompts.py

wdoc/utils/prompts.py

Contributors

thiswillbeyourgithub
Loading

Release 3.3.0

19 Jul 12:44
@thiswillbeyourgithub thiswillbeyourgithub
3.3.0
6435133
Compare
Choose a tag to compare
Release 3.3.0

What's new

This release focuses on adding DuckDuckGo web search capabilities and introducing binary embeddings support for more efficient vector storage.

✨ New Features

DuckDuckGo Web Search Integration

  • [372fe57] Add DuckDuckGo search support with URL extraction and metadata
  • [273195e] Support wdoc wdb "your query" shorthand for web search
  • [03bfe08] Add DuckDuckGo search tests and documentation

Binary Embeddings Support

  • [c528bad] Add support for binary embeddings with 8x memory reduction
  • [8f65197] Enable FAISS vectorstore compression by default
  • [37ebd97] Create CompressedFAISS subclass with zlib compression

🐛 Bug Fixes

Core Functionality

  • [0d72efd] Fix wrong decorator used for load_one_doc
  • [edcf671] Fix ddg_region type (str not int)
  • [66ab177] Fix type hints for ddg_safesearch and loading_failure
  • [957936c] Use keyword arguments instead of fire when calling wdoc

Testing Environment

  • [d3de58e] Fix piped input/output handling in pytest environment
  • [42ff516] Prevent pipe usage in pytest environment
  • [c78dc0b] Add pytest environment detection

🧪 Testing Improvements

  • [1b09996] Fix the run_all_test script
  • [8ed1d0c] Add comprehensive DuckDuckGo search functionality tests
  • [b184177] Split CLI tests into separate test_cli.py file
  • [9d7fe9c] Split parsing tests into separate test_parsing.py file
  • [12b012d] Move vector store tests to dedicated test file

📚 Documentation

  • [d7d6b04] Explain how to run tests in README
  • [dc15001] Clarify how to disable parallel processing
  • [df4b79f] Document debug mode's effect on loading_failure default
  • [1832299] Add shell examples for DuckDuckGo usage

🔧 Enhancements

CLI/UX Improvements

  • [7e994a6] Rename parse_file function to parse_doc
  • [4aa247e] Re-ask for input when empty query provided in CLI
  • [57d5d5f] Fix Fire's pager issue in CLI

Performance THISISANAMPERSAND Reliability

  • [68d4c75] Bump LiteLLM to latest version for improved startup time
  • [ab9c5e9] Add parallel processing option for Whisper audio splits
  • [6b13044] Add loop counter and crash protection for recursive file processing

🔄 Version Update

  • [6435133] Bump version from 3.2.5 → 3.3.0

Commits details since the last release

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

tests/run_all_tests.sh

README.md

tests/test_cli.py

wdoc/utils/loaders.py

wdoc/docs/help.md

wdoc/utils/batch_file_loader.py

wdoc/utils/misc.py

wdoc/docs/help.md

wdoc/utils/misc.py

wdoc/docs/help.md

tests/test_cli.py

tests/test_cli.py

tests/run_all_tests.sh

wdoc/utils/env.py
wdoc/utils/misc.py

tests/conftest.py

wdoc/utils/misc.py

tests/test_wdoc.py
wdoc/utils/env.py
wdoc/utils/misc.py

tests/test_cli.py

tests/test_cli.py

tests/test_cli.py

wdoc/main.py

wdoc/main.py

wdoc/main.py

wdoc/main.py

wdoc/utils/loaders.py

wdoc/utils/loaders.py

README.md
tests/test_cli.py
wdoc/docs/examples.md

tests/run_all_tests.sh

tests/run_all_tests.sh

tests/test_cli.py

tests/test_cli.py

tests/test_cli.py

tests/test_cli.py

wdoc/docs/examples.md

wdoc/docs/examples.md

Read more

Contributors

thiswillbeyourgithub
Loading

Release 3.2.5

17 May 18:35
@thiswillbeyourgithub thiswillbeyourgithub
3.2.5
6740ea2
Compare
Choose a tag to compare
Release 3.2.5

What's new

This release brings several improvements to command-line argument handling and filetype detection, along with key bug fixes and build process enhancements.

✨ Features

  • CLI & Filetype Detection:
    • Enhanced handling of multiple implicit CLI arguments by utilizing a filetype detector to infer appropriate actions ([ab76610]).
    • Introduced a specific exception for clearer error reporting when filetypes cannot be inferred, including a new error for an undetectable filetype ([520f4ce], [39af223]).
  • Build Process:
    • Integrated sphinx-apidoc into the ReadTheDocs build process via a pre-build job in .readthedocs.yaml ([cc86c7b]).

🐛 Fixes

  • Corrected an issue with sys.argv handling that led to duplicated arguments ([e7cf185]).
  • Updated litellm dependency to resolve crashes experienced on Windows environments ([cfff0ac]), see #20.

🛠️ Improvements & Refactoring

  • Filetype Detection Internals:
    • Refactored filetype detection logic into a dedicated function for better modularity ([b453748]).
    • Added debug logging to the filetype detector to aid in troubleshooting ([05966c6]).
  • Code Quality:
    • Improved documentation by adding docstrings to custom exception classes ([8e6ca1a]).

Chores

  • Version bumped to 3.2.5 ([82b7f81]).

Commits details since the last release

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

wdoc/main.py

setup.py

.readthedocs.yaml

wdoc/main.py

wdoc/utils/batch_file_loader.py

wdoc/utils/batch_file_loader.py

wdoc/utils/errors.py

wdoc/utils/errors.py

wdoc/utils/batch_file_loader.py

Contributors

thiswillbeyourgithub
Loading

Release 3.2.4

15 May 14:55
@thiswillbeyourgithub thiswillbeyourgithub
3.2.4
ed7a9c7
Compare
Choose a tag to compare
Release 3.2.4

What's new

This release primarily focuses on significant documentation enhancements, crucial bug fixes for stability and build processes, and introduces updated dependencies and tokenization.

✨ New Features

  • Upgraded default token estimation to use gpt-4o-mini tokenizer, replacing gpt-3.5-turbo ([6d41817]).
  • Integrated the latest yt-dlp for YouTube downloads ([ab207b4]).
  • Environment variable documentation is now automatically added to the EnvDataclass class __doc__ ([ed9dd38]).

🐛 Bug Fixes

  • Resolved a crash on ReadTheDocs caused by missing yt-dlp dependency ([f5068a3]).
  • Fixed an issue where accessing env.__class__ on ReadTheDocs could cause a crash ([4e180f0]).
  • Corrected relative import paths in wdoc that were preventing Sphinx API documentation builds ([ade5930]).
  • Fixed issues with the Sphinx API command in the FAQ section of the README ([38008aa], [ff093a2]).
  • Ensured collapsible bars in documentation function correctly ([3cef833]).

📚 Documentation & Refinements

  • Extensive updates and fixes to Sphinx documentation generation and content:
    • Addressed outdated Sphinx documentation files ([90bde99]).
    • Improved API autodoc parameters for clearer documentation ([243de66]).
    • Excluded private and special members from documentation ([7abedd4]).
    • Added Sphinx command to FAQ in README ([1e6602e]) and removed private members from it ([11ae11b]).
    • Updated copyright year to 2025 ([bd7e3c5]).
  • Streamlined documentation structure and configuration:
    • Removed unused make files (Makefile, make.bat) for documentation ([07b0a7d]).
    • Removed unused argument for theme flyout display ([17bc5e6]).
    • Removed unused templates path ([6bffa20]) and CSS ([712df08]).
    • Removed duplicate README from the documentation source ([2b93162]).
    • Added a documentation table to the main index ([1dfe2b3]).

⚙️ Build & Chores

  • Bumped version to 3.2.4 ([ed7a9c7]).

Commits details since the last release

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

setup.py

docs/source/conf.py

wdoc/utils/env.py

docs/source/conf.py

wdoc/docs/help.md
wdoc/utils/env.py

docs/Makefile
docs/make.bat

docs/source/conf.py

README.md

README.md

docs/source/wdoc.rst
docs/source/wdoc.utils.batch_file_loader.rst
docs/source/wdoc.utils.customs.compressed_embeddings_cache.rst
docs/source/wdoc.utils.customs.fix_llm_caching.rst
docs/source/wdoc.utils.customs.rst
docs/source/wdoc.utils.embeddings.rst
docs/source/wdoc.utils.env.rst
docs/source/wdoc.utils.errors.rst
docs/source/wdoc.utils.flags.rst
docs/source/wdoc.utils.import_tricks.rst
docs/source/wdoc.utils.interact.rst
docs/source/wdoc.utils.llm.rst
docs/source/wdoc.utils.loaders.rst
docs/source/wdoc.utils.logger.rst
docs/source/wdoc.utils.misc.rst
docs/source/wdoc.utils.prompts.rst
docs/source/wdoc.utils.retrievers.rst
docs/source/wdoc.utils.rst
docs/source/wdoc.utils.tasks.query.rst
docs/source/wdoc.utils.tasks.rst
docs/source/wdoc.utils.tasks.summarize.rst
docs/source/wdoc.utils.typechecker.rst
docs/source/wdoc.wdoc.rst

README.md

wdoc/init.py
wdoc/main.py
wdoc/utils/init.py
wdoc/utils/batch_file_loader.py
wdoc/utils/customs/init.py
wdoc/utils/embeddings.py
wdoc/utils/env.py
wdoc/utils/import_tricks.py
wdoc/utils/interact.py
wdoc/utils/llm.py
wdoc/utils/loaders.py
wdoc/utils/logger.py
wdoc/utils/misc.py
wdoc/utils/prompts.py
wdoc/utils/retrievers.py
wdoc/utils/tasks/init.py
wdoc/utils/tasks/query.py
wdoc/utils/tasks/summarize.py
wdoc/utils/typechecker.py
wdoc/wdoc.py

README.md

docs/source/conf.py

docs/source/conf.py

docs/source/index.rst

docs/source/conf.py

docs/source/_static/custom.css
docs/source/conf.py

docs/source/index.rst

  • [6d41817] by @thiswillbeyourgithub, 25 hours ago:
    new: use gpt-4o-mini tokenizer by default to estimate tokens
    previously we used the ageing gpt-3.5-turbo

Signed-off-by: thiswillbeyourgithub 26625900+thiswillbeyourgithub@users.noreply.github.com

wdoc/docs/help.md
wdoc/utils/misc.py

setup.py

Contributors

thiswillbeyourgithub
Loading

Release 3.2.3

13 May 16:35
@thiswillbeyourgithub thiswillbeyourgithub
3.2.3
f62a232
Compare
Choose a tag to compare
Release 3.2.3

What's new

This release primarily focuses on enhancing context management for embedding models, improving debugging utilities, and updating documentation for better clarity. It also includes several important bug fixes and feature additions.

✨ Features

  • Introduced a new environment variable WDOC_MAX_EMBED_CONTEXT to allow capping the context size for embedding models ([d9e200f8])
    • Documentation for this new variable has been added ([a2408fd0])
  • Enhanced debugging by ensuring debug prints are always active when md_printer is used. This helps in retrieving LLM answers from logs if they weren't saved to a file ([69db1916])
  • Added the current date to summary metadata and headers to help reduce potential LLM hallucinations ([64ca4665])

🐛 Fixes

  • Text Splitting & Context Handling:
    • Addressed an issue where large language models have more context than embedding models by setting a max_tokens limit for the text splitter ([dac6802d])
    • Fixed an edge case where the wdoc max chunk setting could be ignored ([196b3a00])
    • Corrected an old variable name within the text splitting logic ([767bc754])
  • Updated the default model to gemini 2.5 preview to reflect its renaming on OpenRouter ([22978609])
  • Improved the mechanism for ignoring initial "breathing" or placeholder lines in summaries ([4dbcf158])

📚 Documentation

  • Clarity and Enhancements:
    • Clarified the usage of save and load functionalities ([9d9642d4]) and specifically advised against using them simultaneously ([5270c350])
    • Made multiple clarifications to the README for better understanding ([9284ff54], [cb4cb519], [f677e5a2], [39e0da55])
    • Updated Ollama examples to recommend snowflake-arctic-embed2 instead of bge-m3 ([d045702b])
    • Added documentation for the WDOC_MAX_EMBED_CONTEXT environment variable ([a2408fd0])
  • Removed a documentation file (summary_rag.md) that was not yet ready for release ([6d20c220])

⚙️ Chore & Maintenance

  • Version bumped to 3.2.3 (following an earlier bump to 3.2.2 [[71ac503c]]) ([f62a2322])
  • README Updates:
    • Updated TODO items ([8f2cbfd7], [5d090421])
    • Added a PyPI badge for better project visibility ([60ef4112])

Commits details since the last release

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

summary_rag.md

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

README.md

  • [69db191] by @thiswillbeyourgithub, 40 minutes ago:
    new: now debug print is used anyway when md_printer is used
    this is to make you able to go to the logs to fetch and answer form the
    LLM if you have forgotten to store it to a file

Signed-off-by: thiswillbeyourgithub 26625900+thiswillbeyourgithub@users.noreply.github.com

wdoc/utils/logger.py
wdoc/wdoc.py

wdoc/docs/help.md

wdoc/utils/env.py
wdoc/utils/misc.py

wdoc/utils/misc.py

wdoc/utils/misc.py

wdoc/utils/misc.py

  • [2297860] by @thiswillbeyourgithub, 86 minutes ago:
    fix: set default model to gemini 2.5 preview without date timestamp
    openrouter renamed that model apparently

Signed-off-by: thiswillbeyourgithub 26625900+thiswillbeyourgithub@users.noreply.github.com

README.md
wdoc/utils/env.py

wdoc/docs/help.md

wdoc/docs/help.md

wdoc/docs/examples.md

README.md

README.md

README.md

README.md

README.md

README.md

wdoc/wdoc.py

wdoc/utils/tasks/summarize.py

Contributors

thiswillbeyourgithub
Loading

Release 3.2.1

03 May 17:08
@thiswillbeyourgithub thiswillbeyourgithub
3.2.1
03aeab2
Compare
Choose a tag to compare
Release 3.2.1

What's new

This small patch release primarily focuses on integrating OpenRouter for model pricing/metadata and refining cost calculations.

✨ Features

  • Set default models to use OpenRouter ([915699c]).
  • Fetch model prices and metadata automatically from OpenRouter, improving reliability ([7f840b7]).

🐛 Fixes & Enhancements

  • Much improved price calculation and handling:
    • Reworked computation to account for internal thinking costs ([c0b90d8]).
    • Enhanced method for retrieving model prices, supported parameters and maximum context size ([a17b41c]).
    • Clearer error message when model prices are missing ([2b29a9d]).
  • Updated litellm dependency ([179b589]).

🧪 Tests

  • API integration tests now fail faster if an underlying API call fails ([9a0c856]).

Commits details since the last release

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

README.md
wdoc/utils/env.py

wdoc/utils/llm.py
wdoc/utils/misc.py
wdoc/utils/tasks/summarize.py
wdoc/wdoc.py

wdoc/utils/misc.py
wdoc/wdoc.py

tests/run_all_tests.sh

wdoc/utils/misc.py
wdoc/wdoc.py

wdoc/utils/misc.py

setup.py

Contributors

thiswillbeyourgithub
Loading

Release 3.2.0

03 May 14:05
@thiswillbeyourgithub thiswillbeyourgithub
3.2.0
7d69d79
Compare
Choose a tag to compare
Release 3.2.0

What's new

This release focuses on improving the command-line interface (especially handling piped input/output), enhancing language model interactions (switching defaults, better caching, Ollama support), and increasing overall stability through numerous bug fixes and testing improvements.

✨ Features

  • Added arguments to set specific keyword arguments (kwargs) for language models (--model_kwargs, --query_eval_model_kwargs) ([1392553]).
  • Introduced WDOC_LLM_REQUEST_TIMEOUT environment variable for LLM request timeouts (default 600s), useful for Ollama ([ec3c0c5]).
  • Switched default models from Claude Sonnet/Haiku to Gemini 1.5 Pro/Flash ([82ef10d]).
  • Unified LLM handling to primarily use ChatLiteLLM, removing direct ChatOpenAI usage ([30a0f0c]).
  • Enabled cost tracking for queries, storing the cost in the output ([e7753af]).
  • Added automatic download of nltk punkt tokenizer during post-installation ([44f5bf8]).
  • Overhauled Command Line Interface (CLI) argument parsing for wdoc and wdoc parse using fire ([7c51ed2], [2f4748d]).
  • Removed the --pipe argument, relying on automatic stdin/stdout detection ([b03e79a], [2e6c1dd], [838f164]).
  • Removed the separate wdoc_parse_file entry point; use wdoc parse instead ([2e878d2]).
  • Added a new script media_url_finder.py ([beaf8fa]).

🐛 Fixes

  • LLM PLACEHOLDER Caching:
    • Resolved issues with LLM caching, including invalidation when kwargs change and LangChain's SQLite cache ([cb785da], [3e3e753]).
    • Fixed edge cases in thinking block parsing for models like Gemini and updated tags (<thinking> -> <think>) ([e111bdb], [d0ae21a], [ca9245b], [99ed332]).
    • Corrected underflow errors in cost calculation due to low LLM prices ([3f18f5d], [95a1984]).
    • Addressed issues specific to Ollama: API key requirement relaxation, price assumption (zero), litellm naming (ollama_chat -> ollama), and context window estimation ([d2f92a3], [5784b25], [43c6340], [c3c15e1]).
    • Fixed handling of testing/testing models and associated parameters ([b995197], [91b5e67], [7cf840c], [9a7b95b]).
    • Fixed query_retrievers parsing ([02d7412]).
    • Pinned litellm version for stability ([1b17c78]).
  • CLI PLACEHOLDER Piping:
    • Improved detection and handling of piped input/output ([2e6c1dd], [509626a], [db2fa0f]).
    • Fixed crashes and hangs when using pipes, especially with long inputs or specific test commands ([f59f34b], [414de8d], [b95b125], [826e7aa], [b6f7fd7], [177be6b]).
    • Corrected argument parsing issues affecting the --help command ([c909337]).
    • Ensured logs are not colorized and Markdown rendering is disabled when outputting to a pipe ([f1d63cd], [fe2665c]).
    • Fixed issues where debug prints or warnings were incorrectly suppressed or handled ([64fcd60], [a7724ff]).
  • General:

🧪 Testing

  • Improved test setup for caching, using separate directories and disabling cache where necessary ([9104f86], [89f4859], [085a87e], [6935fe7]).
  • Added tests for OpenRouter/default models, piping functionality, summary/query tasks with testing models, and environment variable handling ([06e35b0], [bbb8371], [caae34c], [cb9d237], [eaafafd], [1f835eb]).
  • Refactored pipe tests to use subprocess explicitly and fixed related issues (stderr redirection, pytest capture, shell usage) ([38a3571], [7f3249a], [573acf9]). (Note: Some pipe tests were later commented out ([45cf419])).

⚡ Enhancements

  • Reworked logic for detecting and modifying model parameters based on the task ([564c4f9]).
  • Improved load_media function to handle online media more robustly by finding and clicking appropriate buttons ([049c9cb], [67772f8], [c5828d3]).
  • Added checks to prevent exceeding total token limits during summarization ([9bdcabc]).
  • Refined logging levels and Markdown printing logic ([edfec82], [4ca394c], [895a60f]).

📚 Documentation

⚙️ Build PLACEHOLDER Chore

Commits details since the last release

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

  • [30a0f0c] by @thiswillbeyourgithub, 24 minutes ago:
    new: stop using both ChatOpenAI and ChatLiteLLM
    ChatLiteLLM seems to now work reliably

Signed-off-by: thiswillbeyourgithub 26625900+thiswillbeyourgithub@users.noreply.github.com

tests/test_wdoc.py
wdoc/docs/help.md
wdoc/utils/llm.py
wdoc/utils/retrievers.py
wdoc/utils/tasks/query.py

tests/conftest.py

wdoc/utils/misc.py

.gitignore

tests/test_wdoc.py
wdoc/utils/misc.py

wdoc/utils/tasks/summarize.py
wdoc/wdoc.py

wdoc/wdoc.py

wdoc/utils/llm.py

wdoc/wdoc.py

wdoc/wdoc.py

tests/test_wdoc.py

wdoc/utils/customs/fix_llm_caching.py

wdoc/docs/help.md
wdoc/utils/llm.py

wdoc/docs/help.md

tests/test_wdoc.py

wdoc/wdoc.py

  • [d0ae21a] by @thiswillbeyourgithub, 2 hours ago:
    fix: reworked and improved how thinking_answer_parser works
    some weak models could fail despite usable results

Signed-off-by: thiswillbeyourgithub 26625900+thiswillbeyourgithub@users.noreply.github.com

wdoc/utils/misc.py

Signed-off-by: thiswillbeyourgithub 26625900+thiswillbeyourgithub@users.noreply.github.com

wdoc/utils/misc.py

wdoc/utils/misc.py
wdoc/utils/prompts.py
wdoc/utils/tasks/query.py

tests/test_wdoc.py
...

Read more

Contributors

thiswillbeyourgithub
Loading

Release 3.1.0

18 Apr 13:19
@thiswillbeyourgithub thiswillbeyourgithub
3.1.0
e93dcad
Compare
Choose a tag to compare
Release 3.1.0

What's new

This release primarily focuses on enhancing logging capabilities and fixing issues related to piping behavior.

Version bump to 3.1.0 ([e93dcad6]).

✨ New Features

  • Logging:
    • Always display the default log location ([2fe2c431]).
    • Set log level to debug for log files and critical when used in a pipe ([130058a1]).

🚀 Enhancements

  • Logging:
    • Improved log format ([61465aff], [dc06ccfd]).
    • Increased probability of early logger initialization ([01f01ac7]).
    • Clearer error messages from python-magic ([c846dafa]).

🐛 Fixes

  • Piping:
    • Resolved confusion between input and output during piping ([e175b7d5]).
    • Corrected initialization of is_piped variable ([e4532d30]).
  • Logging & Environment:
    • Fixed default handler issue in logger ([43c859dd]).
    • Prevented potential crash related to environment variable handling ([d3b1e2bc]).

🧹 Minor Changes

  • Removed unused imports ([f3c05962]).
  • Adjusted test imports structure ([69738119]).
  • Removed commented code ([86b51102]).
  • Removed unused disable_md_printing argument ([b3af430e]).

✅ Testing

  • Added test for exception handling ([dfbfad54]).
  • Added environment variable tests ([0fba8a13]).

Commits details since the last release

bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py

wdoc/utils/batch_file_loader.py
wdoc/utils/env.py
wdoc/utils/loaders.py
wdoc/utils/logger.py
wdoc/utils/misc.py
wdoc/wdoc.py

wdoc/docs/help.md
wdoc/wdoc.py

wdoc/utils/logger.py

wdoc/utils/logger.py

wdoc/utils/logger.py

wdoc/utils/env.py
wdoc/utils/misc.py

wdoc/init.py
wdoc/main.py

wdoc/utils/logger.py

wdoc/utils/logger.py

wdoc/utils/logger.py

wdoc/utils/logger.py

wdoc/utils/env.py
wdoc/utils/logger.py

tests/test_wdoc.py

tests/test_wdoc.py

tests/test_wdoc.py

wdoc/utils/env.py

wdoc/utils/loaders.py

Contributors

thiswillbeyourgithub
Loading