Chunkers, Loaders vs Tasks and Pipelines vs Engines

[KingLeoJr]

Hello,
I hope you are fine.

Could you clarify below queries, please?

• How Chunkers, and Loaders are different to any other Task? Can they not be Tasks instead?
• [If yes to above then] How Engines are different to Pipelines? Are they not preconfigured version of Pipeline?
• Are Drivers for WebSearching and Web Scrapping optional? As they might not be needed for each type of Web Searching Tool.
• What is the best place to understand available Griptape Cloud Assistants?
• Tasks goes with any Structures in Griptape, is this correct understanding? Also, what is Structure in Structures [Other than Workflow, Pipeline and Agent]?

[collindutter]

Hey @KingLeoJr, great questions! We acknowledge that some of these concepts aren’t as well-documented as they should be, but we’re actively working to improve that. I hope to address these gaps in #1607. In the meantime, I’ll do my best to clarify.

How are Chunkers and Loaders different from Tasks?

Chunkers and Loaders serve distinct roles compared to Tasks. While Tasks provide a generalized interface for executing units of work, Chunkers and Loaders are more specialized.

• Loaders are responsible for retrieving data from a source and parsing it into an Artifact. For example:

  • ImageLoader reads an image file from disk and converts it into an ImageArtifact.
  • WebLoader scrapes a webpage and extracts its body content into a TextArtifact.

• Chunkers process text by breaking it into smaller segments that fit within the context window of an LLM. This is particularly useful in Retrieval-Augmented Generation (RAG), where chunking helps ensure that data stored in a vector database maintains strong semantic coherence.

Can they not be Tasks instead?

Loaders and Chunkers are sometimes used by Tasks, but they are unlikely to become Tasks themselves. They function as lower-level utilities designed for pre-processing or as components within higher-level abstractions like Tasks.

How are Engines different from Pipelines? Aren’t they just preconfigured Pipelines?

Engines in Griptape encapsulate common LLM patterns, such as RAG and evaluations, and provide them as ready-to-use modules. Thinking of them as prebuilt Pipelines is a useful analogy. Future versions of Griptape may further embrace this idea by structuring all Engines as pre-configured Structures, but for now, they remain distinct abstractions that can be used either directly or within a Structure.

Are Drivers for Web Searching and Web Scraping optional?

No, if you’re using WebSearchTool or WebScraperTool, Drivers are required. Specifically:

• The WebScraperTool relies on a web_loader: WebLoader, which includes a default Driver:
  

  griptape/griptape/loaders/web_loader.py

  Lines 17 to 20 in 762958f

  	web_scraper_driver: BaseWebScraperDriver = field(
  	default=Factory(lambda: TrafilaturaWebScraperDriver()),
  	kw_only=True,
  	)

Drivers enable these tools to function; without them, the tools would not do much at all.

Where can I learn more about available Griptape Cloud Assistants?

We are actively improving our documentation in this area. For now, the best resources are:

• What are Assistants?
• Assistant Task
• Assistant Drivers for Griptape Cloud

Do Tasks work with any Structure in Griptape?

Yes, that’s correct! Tasks are the fundamental unit of work, and Structures provide the framework for organizing them. However, a recent change is making Tasks more powerful on their own. This update might eventually remove the need for Agents and Pipelines, as Tasks could be run sequentially without additional abstractions.

What does "Structure in Structures" mean?

This refers to Structures that invoke other Structures. Currently, this can be achieved using StructureRunTask and StructureRunTool. Here are some relevant resources:

• Structure Run Task
• Structure Run Drivers
• Multi-Agent Workflow Example

Hope this helps, please don't hesitate to reach out with more questions!

[KingLeoJr]

Thank you for the reply collindutter

After going through Griptape docs, few design choices did not make sense to me, that's why I thought of sending this message here.

Loaders, Chunkers: as you said they seem like utilities than exclusive component/entity. I thought it might make more sense if they are accessed as utilities[or treat them as Task] rather than defining an exclusive entity to avoid introducing unnecessary complexity [I wanted to include Tokenizers as well but it seems to be a mix of utility and config.]
Going by the same principle of simplicity, Engines seems like an implemented Pipeline. [Thinking of it as a Pipeline can save from introducing an additional entity/component.]
There are numerous ways to scrape or search web which does not necessarily need an integration with any 3rd party tool. e.g if we are using something like exa.ai then it might make sense to use a driver, however, if we are using something like DuckDuckGo then not sure why there is a need for a driver.
I was already concerned that Task is designed to be accessed at ANY type of Structure and now I believe this might be heading in a direction which might not necessarily resonate with me.
I will go through these links later. At the moment, I am wondering if there is a Pipeline, do we need to introduce additional Entity [StructureRunTask]

I really appreciate your response. Thank you.