diff --git a/README.md b/README.md index 651b31894..898e70616 100644 --- a/README.md +++ b/README.md @@ -21,10 +21,10 @@ ### 🔨 Build Reliable & Scalable GenAI Apps -- **Swap LLMs anytime** – Switch between [100+ LLMs via LiteLLM](https://ragbits.deepsense.ai/how-to/llms/use_llms/) or run [local models](https://ragbits.deepsense.ai/how-to/llms/use_local_llms/). -- **Type-safe LLM calls** – Use Python generics to [enforce strict type safety](https://ragbits.deepsense.ai/how-to/prompts/use_prompting/#how-to-configure-prompts-output-data-type) in model interactions. -- **Bring your own vector store** – Connect to [Qdrant](https://ragbits.deepsense.ai/api_reference/core/vector-stores/#ragbits.core.vector_stores.qdrant.QdrantVectorStore), [PgVector](https://ragbits.deepsense.ai/api_reference/core/vector-stores/#ragbits.core.vector_stores.pgvector.PgVectorStore), and more with built-in support. -- **Developer tools included** – [Manage vector stores](https://ragbits.deepsense.ai/cli/main/#ragbits-vector-store), query pipelines, and [test prompts from your terminal](https://ragbits.deepsense.ai/quickstart/quickstart1_prompts/#testing-the-prompt-from-the-cli). +- **Swap LLMs anytime** – Switch between [100+ LLMs via LiteLLM](https://ragbits.deepsense.ai/stable/how-to/llms/use_llms/) or run [local models](https://ragbits.deepsense.ai/stable/how-to/llms/use_local_llms/). +- **Type-safe LLM calls** – Use Python generics to [enforce strict type safety](https://ragbits.deepsense.ai/stable/how-to/prompts/use_prompting/#how-to-configure-prompts-output-data-type) in model interactions. +- **Bring your own vector store** – Connect to [Qdrant](https://ragbits.deepsense.ai/stable/api_reference/core/vector-stores/#ragbits.core.vector_stores.qdrant.QdrantVectorStore), [PgVector](https://ragbits.deepsense.ai/stable/api_reference/core/vector-stores/#ragbits.core.vector_stores.pgvector.PgVectorStore), and more with built-in support. +- **Developer tools included** – [Manage vector stores](https://ragbits.deepsense.ai/stable/cli/main/#ragbits-vector-store), query pipelines, and [test prompts from your terminal](https://ragbits.deepsense.ai/stable/quickstart/quickstart1_prompts/#testing-the-prompt-from-the-cli). - **Modular installation** – Install only what you need, reducing dependencies and improving performance. ### 📚 Fast & Flexible RAG Processing @@ -32,20 +32,20 @@ - **Ingest 20+ formats** – Process PDFs, HTML, spreadsheets, presentations, and more. Process data using [Docling](https://github.com/docling-project/docling), [Unstructured](https://github.com/Unstructured-IO/unstructured) or create a custom parser. - **Handle complex data** – Extract tables, images, and structured content with built-in VLMs support. - **Connect to any data source** – Use prebuilt connectors for S3, GCS, Azure, or implement your own. -- **Scale ingestion** – Process large datasets quickly with [Ray-based parallel processing](https://ragbits.deepsense.ai/how-to/document_search/distributed_ingestion/#how-to-ingest-documents-in-a-distributed-fashion). +- **Scale ingestion** – Process large datasets quickly with [Ray-based parallel processing](https://ragbits.deepsense.ai/stable/how-to/document_search/distributed_ingestion/#how-to-ingest-documents-in-a-distributed-fashion). ### 🤖 Build Multi-Agent Workflows with Ease -- **Multi-agent coordination** – Create teams of specialized agents with role-based collaboration using [A2A protocol](https://ragbits.deepsense.ai/tutorials/agents) for interoperability. -- **Real-time data integration** – Leverage [Model Context Protocol (MCP)](https://ragbits.deepsense.ai/how-to/agents/provide_mcp_tools) for live web access, database queries, and API integrations. -- **Conversation state management** – Maintain context across interactions with [automatic history tracking](https://ragbits.deepsense.ai/how-to/agents/define_and_use_agents/#conversation-history). +- **Multi-agent coordination** – Create teams of specialized agents with role-based collaboration using [A2A protocol](https://ragbits.deepsense.ai/stable/tutorials/agents) for interoperability. +- **Real-time data integration** – Leverage [Model Context Protocol (MCP)](https://ragbits.deepsense.ai/stable/how-to/agents/provide_mcp_tools) for live web access, database queries, and API integrations. +- **Conversation state management** – Maintain context across interactions with [automatic history tracking](https://ragbits.deepsense.ai/stable/how-to/agents/define_and_use_agents/#conversation-history). ### 🚀 Deploy & Monitor with Confidence -- **Real-time observability** – Track performance with [OpenTelemetry](https://ragbits.deepsense.ai/how-to/project/use_tracing/#opentelemetry-trace-handler) and [CLI insights](https://ragbits.deepsense.ai/how-to/project/use_tracing/#cli-trace-handler). -- **Built-in testing** – Validate prompts [with promptfoo](https://ragbits.deepsense.ai/how-to/prompts/promptfoo/) before deployment. +- **Real-time observability** – Track performance with [OpenTelemetry](https://ragbits.deepsense.ai/stable/how-to/project/use_tracing/#opentelemetry-trace-handler) and [CLI insights](https://ragbits.deepsense.ai/stable/how-to/project/use_tracing/#cli-trace-handler). +- **Built-in testing** – Validate prompts [with promptfoo](https://ragbits.deepsense.ai/stable/how-to/prompts/promptfoo/) before deployment. - **Auto-optimization** – Continuously evaluate and refine model performance. -- **Chat UI** – Deploy [chatbot interface](https://ragbits.deepsense.ai/how-to/chatbots/api/) with API, persistance and user feedback. +- **Chat UI** – Deploy [chatbot interface](https://ragbits.deepsense.ai/stable/how-to/chatbots/api/) with API, persistance and user feedback. ## Installation @@ -292,10 +292,10 @@ Explore `create-ragbits-app` repo [here](https://github.com/deepsense-ai/create- ## Documentation -- [Tutorials](https://ragbits.deepsense.ai/tutorials/intro) - Get started with Ragbits in a few minutes -- [How-to](https://ragbits.deepsense.ai/how-to/prompts/use_prompting) - Learn how to use Ragbits in your projects -- [CLI](https://ragbits.deepsense.ai/cli/main) - Learn how to run Ragbits in your terminal -- [API reference](https://ragbits.deepsense.ai/api_reference/core/prompt) - Explore the underlying Ragbits API +- [Tutorials](https://ragbits.deepsense.ai/stable/tutorials/intro) - Get started with Ragbits in a few minutes +- [How-to](https://ragbits.deepsense.ai/stable/how-to/prompts/use_prompting) - Learn how to use Ragbits in your projects +- [CLI](https://ragbits.deepsense.ai/stable/cli/main) - Learn how to run Ragbits in your terminal +- [API reference](https://ragbits.deepsense.ai/stable/api_reference/core/prompt) - Explore the underlying Ragbits API ## Contributing diff --git a/docs/index.md b/docs/index.md index 9e7f54f8e..b834edaf1 100644 --- a/docs/index.md +++ b/docs/index.md @@ -41,10 +41,10 @@ hide: ### 🔨 Build Reliable & Scalable GenAI Apps -- **Swap LLMs anytime** – Switch between [100+ LLMs via LiteLLM](https://ragbits.deepsense.ai/how-to/llms/use_llms/) or run [local models](https://ragbits.deepsense.ai/how-to/llms/use_local_llms/)). -- **Type-safe LLM calls** – Use Python generics to [enforce strict type safety](https://ragbits.deepsense.ai/how-to/prompts/use_prompting/#how-to-configure-prompts-output-data-type) in model interactions. -- **Bring your own vector store** – Connect to [Qdrant](https://ragbits.deepsense.ai/api_reference/core/vector-stores/#ragbits.core.vector_stores.qdrant.QdrantVectorStore), [PgVector](https://ragbits.deepsense.ai/api_reference/core/vector-stores/#ragbits.core.vector_stores.pgvector.PgVectorStore), and more with built-in support. -- **Developer tools included** – [Manage vector stores](https://ragbits.deepsense.ai/cli/main/#ragbits-vector-store), query pipelines, and [test prompts from your terminal](https://ragbits.deepsense.ai/quickstart/quickstart1_prompts/#testing-the-prompt-from-the-cli). +- **Swap LLMs anytime** – Switch between [100+ LLMs via LiteLLM](https://ragbits.deepsense.ai/stable/how-to/llms/use_llms/) or run [local models](https://ragbits.deepsense.ai/stable/how-to/llms/use_local_llms/)). +- **Type-safe LLM calls** – Use Python generics to [enforce strict type safety](https://ragbits.deepsense.ai/stable/how-to/prompts/use_prompting/#how-to-configure-prompts-output-data-type) in model interactions. +- **Bring your own vector store** – Connect to [Qdrant](https://ragbits.deepsense.ai/stable/api_reference/core/vector-stores/#ragbits.core.vector_stores.qdrant.QdrantVectorStore), [PgVector](https://ragbits.deepsense.ai/stable/api_reference/core/vector-stores/#ragbits.core.vector_stores.pgvector.PgVectorStore), and more with built-in support. +- **Developer tools included** – [Manage vector stores](https://ragbits.deepsense.ai/stable/cli/main/#ragbits-vector-store), query pipelines, and [test prompts from your terminal](https://ragbits.deepsense.ai/stable/quickstart/quickstart1_prompts/#testing-the-prompt-from-the-cli). - **Modular installation** – Install only what you need, reducing dependencies and improving performance. ### 📚 Fast & Flexible RAG Processing @@ -52,20 +52,20 @@ hide: - **Ingest 20+ formats** – Process PDFs, HTML, spreadsheets, presentations, and more. Process data using [Docling](https://github.com/docling-project/docling), [Unstructured](https://github.com/Unstructured-IO/unstructured) or create a custom parser. - **Handle complex data** – Extract tables, images, and structured content with built-in VLMs support. - **Connect to any data source** – Use prebuilt connectors for S3, GCS, Azure, or implement your own. -- **Scale ingestion** – Process large datasets quickly with [Ray-based parallel processing](https://ragbits.deepsense.ai/how-to/document_search/distributed_ingestion/#how-to-ingest-documents-in-a-distributed-fashion). +- **Scale ingestion** – Process large datasets quickly with [Ray-based parallel processing](https://ragbits.deepsense.ai/stable/how-to/document_search/distributed_ingestion/#how-to-ingest-documents-in-a-distributed-fashion). ### 🤖 Build Multi-Agent Workflows with Ease -- **Multi-agent coordination** – Create teams of specialized agents with role-based collaboration using [A2A protocol](https://ragbits.deepsense.ai/tutorials/agents/) for interoperability. -- **Real-time data integration** – Leverage [Model Context Protocol (MCP)](https://ragbits.deepsense.ai/how-to/provide_mcp_tools/) for live web access, database queries, and API integrations. -- **Conversation state management** – Maintain context across interactions with [automatic history tracking](https://ragbits.deepsense.ai/how-to/agents/define_and_use_agents/#conversation-history/). +- **Multi-agent coordination** – Create teams of specialized agents with role-based collaboration using [A2A protocol](https://ragbits.deepsense.ai/stable/tutorials/agents/) for interoperability. +- **Real-time data integration** – Leverage [Model Context Protocol (MCP)](https://ragbits.deepsense.ai/stable/how-to/provide_mcp_tools/) for live web access, database queries, and API integrations. +- **Conversation state management** – Maintain context across interactions with [automatic history tracking](https://ragbits.deepsense.ai/stable/how-to/agents/define_and_use_agents/#conversation-history/). ### 🚀 Deploy & Monitor with Confidence -- **Real-time observability** – Track performance with [OpenTelemetry](https://ragbits.deepsense.ai/how-to/project/use_tracing/#opentelemetry-trace-handler) and [CLI insights](https://ragbits.deepsense.ai/how-to/project/use_tracing/#cli-trace-handler). -- **Built-in testing** – Validate prompts [with promptfoo](https://ragbits.deepsense.ai/how-to/prompts/promptfoo/) before deployment. +- **Real-time observability** – Track performance with [OpenTelemetry](https://ragbits.deepsense.ai/stable/how-to/project/use_tracing/#opentelemetry-trace-handler) and [CLI insights](https://ragbits.deepsense.ai/stable/how-to/project/use_tracing/#cli-trace-handler). +- **Built-in testing** – Validate prompts [with promptfoo](https://ragbits.deepsense.ai/stable/how-to/prompts/promptfoo/) before deployment. - **Auto-optimization** – Continuously evaluate and refine model performance. -- **Chat UI** – Deploy [chatbot interface](https://ragbits.deepsense.ai/how-to/chatbots/api/) with API, persistance and user feedback. +- **Chat UI** – Deploy [chatbot interface](https://ragbits.deepsense.ai/stable/how-to/chatbots/api/) with API, persistance and user feedback. ## Installation diff --git a/packages/ragbits-core/src/ragbits/core/sources/asana.py b/packages/ragbits-core/src/ragbits/core/sources/asana.py new file mode 100644 index 000000000..f0e072803 --- /dev/null +++ b/packages/ragbits-core/src/ragbits/core/sources/asana.py @@ -0,0 +1,409 @@ +from __future__ import annotations + +import os +from contextlib import suppress +from pathlib import Path +from typing import Any, ClassVar, List, Union, Optional +from typing_extensions import Self +from collections.abc import Iterable +from io import TextIOWrapper + +from ragbits.core.audit.traces import trace, traceable +from ragbits.core.sources.base import Source, get_local_storage_dir +from ragbits.core.sources.exceptions import SourceConnectionError, SourceError +from ragbits.core.utils.decorators import requires_dependencies + +with suppress(ImportError): + from asana import ApiClient, Configuration, TasksApi, ProjectsApi, WorkspacesApi + from asana.rest import ApiException + +_OPT_FIELDS = [ + "actual_time_minutes", + "approval_status", + "assignee", + "assignee.name", + "assignee_section", + "assignee_section.name", + "assignee_status", + "completed", + "completed_at", + "completed_by", + "completed_by.name", + "created_at", + "created_by", + "custom_fields", + "custom_fields.asana_created_field", + "custom_fields.created_by", + "custom_fields.created_by.name", + "custom_fields.currency_code", + "custom_fields.custom_label", + "custom_fields.custom_label_position", + "custom_fields.date_value", + "custom_fields.date_value.date", + "custom_fields.date_value.date_time", + "custom_fields.description", + "custom_fields.display_value", + "custom_fields.enabled", + "custom_fields.enum_options", + "custom_fields.enum_options.color", + "custom_fields.enum_options.enabled", + "custom_fields.enum_options.name", + "custom_fields.enum_value", + "custom_fields.enum_value.color", + "custom_fields.enum_value.enabled", + "custom_fields.enum_value.name", + "custom_fields.format", + "custom_fields.has_notifications_enabled", + "custom_fields.is_formula_field", + "custom_fields.is_global_to_workspace", + "custom_fields.is_value_read_only", + "custom_fields.multi_enum_values", + "custom_fields.multi_enum_values.color", + "custom_fields.multi_enum_values.enabled", + "custom_fields.multi_enum_values.name", + "custom_fields.name", + "custom_fields.number_value", + "custom_fields.people_value", + "custom_fields.people_value.name", + "custom_fields.precision", + "custom_fields.resource_subtype", + "custom_fields.text_value", + "custom_fields.type", + "dependencies", + "dependents", + "due_at", + "due_on", + "external", + "external.data", + "followers", + "followers.name", + "hearted", + "hearts", + "hearts.user", + "hearts.user.name", + "html_notes", + "is_rendered_as_separator", + "liked", + "likes", + "likes.user", + "likes.user.name", + "memberships", + "memberships.project", + "memberships.project.name", + "memberships.section", + "memberships.section.name", + "modified_at", + "name", + "notes", + "num_hearts", + "num_likes", + "num_subtasks", + "offset", + "parent", + "parent.created_by", + "parent.name", + "parent.resource_subtype", + "path", + "permalink_url", + "projects", + "projects.name", + "resource_subtype", + "start_at", + "start_on", + "tags", + "tags.name", + "uri", + "workspace", + "workspace.name", +] + +_SIMPLE_OPT_FIELDS = [ + "name", + "notes", + "assignee", + "assignee.name", + "completed", + "created_at", + "modified_at", + "due_on", + "start_on", + "start_at", + "tags", + "tags.name", + "projects", + "projects.name", + "custom_fields", + "custom_fields.name", + "custom_fields.display_value", + "custom_fields.text_value", + "dependencies", + "dependencies.name", + "num_subtasks", + "permalink_url", + "uri", + "workspace", + "workspace.name" +] + +_COMPLEX_OPT_FIELDS = [field for field in _OPT_FIELDS if field not in _SIMPLE_OPT_FIELDS] + +class AsanaSource(Source): + """ + Source for data stored in the Asana. + """ + + _asana_client: ClassVar["ApiClient" | None] = None + _asana_access_token: ClassVar[str | None] = None + _asana_tasks_api: ClassVar["TasksApi" | None] = None + _asana_configuration: ClassVar["Configuration" | None] = None + _asana_configuration_settings: ClassVar[dict[str, Any] | None] = None + _asana_fields: ClassVar[List[str] | None] = _SIMPLE_OPT_FIELDS + + project_id: str + + protocol: ClassVar[str] = "asana" + + @property + def id(self) -> str: + """Get the source identifier.""" + return f"asana://{self.project_id}" + + @classmethod + @requires_dependencies(["asana"], "asana") + async def list_sources(cls, workspace_id: str | None = None, *args: Any, **kwargs: Any) -> Iterable[Self]: + """ + List all Asana projects as sources. + + Args: + workspace_id: Optional workspace ID. If not provided, will use the first available workspace. + + Returns: + Iterable of AsanaSource instances for each project. + """ + # Get workspace if not provided + if not workspace_id: + workspaces_api = WorkspacesApi(cls._get_client()) + workspaces = list(workspaces_api.get_workspaces({})) + if not workspaces: + return [] + workspace_id = workspaces[0]['gid'] + + # Get all projects from the workspace + projects_api = ProjectsApi(cls._get_client()) + projects = list(projects_api.get_projects({"workspace": workspace_id})) + + # Convert projects to AsanaSource instances + sources = [] + for project in projects: + sources.append(cls(project_id=project['gid'])) + + return sources + + @classmethod + async def from_uri(cls, path: str) -> Iterable[Self]: + """ + Create Source instances from a URI path. + + Expected format: asana://project_id + """ + if not path.startswith("asana://"): + raise ValueError(f"Invalid Asana URI: {path}. Expected format: asana://project_id") + + project_id = path[8:] # Remove "asana://" prefix + if not project_id: + raise ValueError("Project ID is required in Asana URI") + + return [cls(project_id=project_id)] + + @classmethod + def set_access_token(cls, access_token: str) -> None: + """ + Set the access token for the Asana client. + """ + cls._asana_access_token = access_token + + @classmethod + def set_fields(cls, fields: List[str]) -> None: + """sets the return fields from asana that provide task info""" + cls._asana_fields = fields + + @classmethod + @requires_dependencies(["asana"], "asana") + def _get_client(cls) -> "ApiClient": + """ + Get the Asana client. + """ + if cls._asana_client is None: + configuration = Configuration() + configuration.access_token = cls._asana_access_token + cls._asana_client = ApiClient(configuration) + return cls._asana_client + + @classmethod + @requires_dependencies(["asana"], "asana") + def _get_tasks_api(cls) -> "TasksApi": + """ + Get the Asana tasks API. + """ + if cls._asana_tasks_api is None: + cls._asana_tasks_api = TasksApi(cls._asana_client) + return cls._asana_tasks_api + + def _set_configuration_settings(cls, settings: dict[str, Any]) -> None: + """ + Set the Asana configuration settings. + """ + cls._asana_configuration_settings = settings + + @classmethod + @requires_dependencies(["asana"], "asana") + def _get_configuration(cls) -> "Configuration": + """ + Get the Asana configuration. + """ + if cls._asana_configuration is None: + try: + cls._asana_configuration = Configuration() # need this to be empty first + for entry, value in cls._asana_configuration_settings.items(): + setattr(cls._asana_configuration, entry, value) + except Exception as e: + raise Exception(f"Error setting Asana configuration settings: {e}") + return cls._asana_configuration + + @traceable + @requires_dependencies(["asana"], "asana") + async def fetch(self) -> Path: + """ + Fetch the Asana source and dump tasks to text files. + + Returns: + The local path to the directory containing the dumped task files. + + Raises: + SourceConnectionError: If there is an error connecting to Asana API. + SourceError: If an error occurs during task fetching or file writing. + """ + local_dir = get_local_storage_dir() + file_local_dir = local_dir + file_local_dir.mkdir(parents=True, exist_ok=True) + + with trace(project_id=self.project_id) as outputs: + try: + # Filter tasks by project ID + tasks = list(self._get_tasks_api().get_tasks( + opts={ + "project": self.project_id, + "opt_fields": ",".join(self._asana_fields) + } + )) + + # Dump tasks to text files + self._dump_tasks_to_files(tasks, file_local_dir) + + outputs.path = file_local_dir + return file_local_dir + + except ApiException as e: + with trace("asana_api_error") as error_outputs: + error_outputs.error = f"Asana API error: {e}" + raise SourceConnectionError() from e + except Exception as e: + with trace("asana_fetch_error") as error_outputs: + error_outputs.error = f"Error fetching Asana tasks: {e}" + raise SourceError(f"Error fetching Asana tasks: {e}") from e + + @staticmethod + def _write_task_information(f: TextIOWrapper, task: Any) -> None: + """ + Write task information to a file. + + Handles basic info and just writes to a file. + """ + f.write(f"ASANA TASK INFORMATION\n") + # Basic task information + f.write(f"Task ID: {task.get('gid', 'N/A')}\n") + f.write(f"Name: {task.get('name', 'N/A')}\n") + f.write(f"Notes: {task.get('notes', 'N/A')}\n") + f.write(f"Status: {'Completed' if task.get('completed', False) else 'Incomplete'}\n") + f.write(f"Created: {task.get('created_at', 'N/A')}\n") + f.write(f"Modified: {task.get('modified_at', 'N/A')}\n") + f.write(f"Due Date: {task.get('due_on', 'N/A')}\n") + f.write(f"Start Date: {task.get('start_on', 'N/A')}\n") + + # Assignee information + assignee = task.get('assignee') + if assignee: + f.write(f"Assignee: {assignee.get('name', 'N/A')}\n") + else: + f.write("Assignee: Unassigned\n") + + # Project information + projects = task.get('projects', []) + if projects: + f.write(f"Projects: {', '.join([p.get('name', 'N/A') for p in projects])}\n") + else: + f.write("Projects: None\n") + + # Tags + tags = task.get('tags', []) + if tags: + f.write(f"Tags: {', '.join([tag.get('name', 'N/A') for tag in tags])}\n") + else: + f.write("Tags: None\n") + + # Custom fields + custom_fields = task.get('custom_fields', []) + if custom_fields: + f.write(f"\nCUSTOM FIELDS:\n") + for field in custom_fields: + field_name = field.get('name', 'Unknown Field') + field_value = field.get('display_value', field.get('text_value', 'N/A')) + f.write(f"{field_name}: {field_value}\n") + + # Dependencies + dependencies = task.get('dependencies', []) + if dependencies: + f.write(f"\nDEPENDENCIES:\n") + for dep in dependencies: + dep_name = dep.get('name', 'Unknown Task') + f.write(f"- {dep_name}\n") + + # Subtasks count + num_subtasks = task.get('num_subtasks', 0) + f.write(f"\nSubtasks: {num_subtasks}\n") + + # Permalink + permalink = task.get('permalink_url') + if permalink: + f.write(f"Permalink: {permalink}\n") + + + def _dump_tasks_to_files(self, tasks: Iterable[Any], output_dir: Path) -> None: + """ + Dump tasks and their information to local text files. + + Args: + tasks: Iterable of Asana task objects + output_dir: Directory to save the text files + """ + tasks_dir = output_dir / "asana_tasks" + tasks_dir.mkdir(parents=True, exist_ok=True) + + with trace(project_id=self.project_id) as outputs: + for i, task in enumerate(tasks): + # Create a safe filename from task name or use index + task_name = task.get('name', f'task_{i}') + safe_filename = "".join(c for c in task_name if c.isalnum() or c in (' ', '-', '_')).rstrip() + safe_filename = safe_filename.replace(' ', '_')[:50] # Limit length + + if not safe_filename: + safe_filename = f"task_{i}" + + task_file = tasks_dir / f"{safe_filename}_{i}.txt" + + # Write task information to file + with open(task_file, 'w', encoding='utf-8') as f: + self._write_task_information(f, task) + + outputs.path = tasks_dir + outputs.message = f"Wrote {len(list(tasks))} tasks to {tasks_dir}" \ No newline at end of file diff --git a/packages/ragbits-core/tests/unit/sources/test_asana.py b/packages/ragbits-core/tests/unit/sources/test_asana.py new file mode 100644 index 000000000..306caf9ee --- /dev/null +++ b/packages/ragbits-core/tests/unit/sources/test_asana.py @@ -0,0 +1,169 @@ +import os +from pathlib import Path + +import pytest + +from ragbits.core.sources.asana import AsanaSource + + +@pytest.fixture(autouse=True) +def setup_local_storage_dir(tmp_path: Path): + """Set up a temporary local storage directory for tests.""" + original_local_storage_dir = os.environ.get("LOCAL_STORAGE_DIR") + os.environ["LOCAL_STORAGE_DIR"] = str(tmp_path) + yield + if original_local_storage_dir is not None: + os.environ["LOCAL_STORAGE_DIR"] = original_local_storage_dir + else: + del os.environ["LOCAL_STORAGE_DIR"] + + +class TestAsanaSource: + """Test cases for AsanaSource with real API calls.""" + + def test_set_access_token(self): + """Test setting access token.""" + AsanaSource.set_access_token("test_token") + assert AsanaSource._asana_access_token == "test_token" + + def test_set_fields(self): + """Test setting fields.""" + test_fields = ["name", "notes", "completed"] + AsanaSource.set_fields(test_fields) + assert AsanaSource._asana_fields == test_fields + + @pytest.mark.asyncio + async def test_real_fetch_debug(self): + """ + Real test that fetches actual Asana tasks and shows debug output. + This test will be skipped if ASANA_TOKEN or ASANA_PROJECT_ID is not set. + """ + # Get ASANA_TOKEN and ASANA_PROJECT_ID from environment variables + asana_token = os.environ.get('ASANA_TOKEN') + asana_project_id = os.environ.get('ASANA_PROJECT_ID') + + if not asana_token: + pytest.skip( + "ASANA_TOKEN environment variable not set. Please set it:\n" + "export ASANA_TOKEN='your_token_here'" + ) + + if not asana_project_id: + pytest.skip( + "ASANA_PROJECT_ID environment variable not set. Please set it:\n" + "export ASANA_PROJECT_ID='your_project_id_here'" + ) + + print(f"\nDebugging AsanaSource fetch with real API...") + print(f"Found ASANA_TOKEN: {asana_token[:10]}...") + print(f"Using project ID: {asana_project_id}") + + # Set up the Asana source exactly like in our implementation + AsanaSource.set_access_token(asana_token) + + # Set fields to include assignee and other important fields + assignee_fields = [ + "name", "notes", "assignee", "assignee.name", "completed", + "created_at", "modified_at", "due_on", "start_on", "projects", + "projects.name", "tags", "tags.name", "custom_fields", + "custom_fields.name", "custom_fields.display_value", + "dependencies", "dependencies.name", "num_subtasks", "permalink_url" + ] + AsanaSource.set_fields(assignee_fields) + + source = AsanaSource(project_id=asana_project_id) + + print(f"Requested fields: {AsanaSource._asana_fields}") + + try: + print("\nTesting _get_client()...") + client = AsanaSource._get_client() + print(f"Client created: {type(client)}") + + print("\nTesting _get_tasks_api()...") + tasks_api = AsanaSource._get_tasks_api() + print(f"Tasks API created: {type(tasks_api)}") + + print("\nTesting get_tasks() with exact same parameters...") + # Use the exact same parameters as in the fetch method + tasks = list(tasks_api.get_tasks( + opts={ + "project": source.project_id, + "opt_fields": ",".join(AsanaSource._asana_fields) + } + )) + + print(f"Found {len(tasks)} tasks using AsanaSource methods!") + + # Show first few tasks + for i, task in enumerate(tasks[:3]): + task_name = task.get('name', 'Unnamed Task') + print(f" {i+1}. {task_name}") + + # Debug the first task structure + if tasks: + print(f"\nDebugging first task structure:") + task = tasks[0] + print(f"Type: {type(task)}") + print(f"Task data: {task}") + print(f"Keys: {list(task.keys()) if isinstance(task, dict) else 'Not a dict'}") + print(f"Name value: {task.get('name', 'NOT_FOUND')}") + print(f"GID value: {task.get('gid', 'NOT_FOUND')}") + print(f"Completed value: {task.get('completed', 'NOT_FOUND')}") + print(f"Notes value: {task.get('notes', 'NOT_FOUND')}") + print(f"Created at value: {task.get('created_at', 'NOT_FOUND')}") + print(f"Modified at value: {task.get('modified_at', 'NOT_FOUND')}") + print(f"Assignee value: {task.get('assignee', 'NOT_FOUND')}") + if task.get('assignee'): + print(f"Assignee name: {task.get('assignee', {}).get('name', 'NOT_FOUND')}") + print(f"Projects value: {task.get('projects', 'NOT_FOUND')}") + print(f"Tags value: {task.get('tags', 'NOT_FOUND')}") + + print("\nTesting actual fetch() method...") + result = await source.fetch() + + print(f"Fetch completed successfully!") + print(f"Result path: {result}") + print(f"Path exists: {result.exists()}") + print(f"Path name: {result.name}") + + # Check that task files were created in the asana_tasks subdirectory + asana_tasks_dir = result / "asana_tasks" + task_files = list(asana_tasks_dir.glob("*.txt")) if asana_tasks_dir.exists() else [] + print(f"Task files created: {len(task_files)}") + + # Show content of first task file + if task_files: + print(f"\nContent of task file ({task_files[0].name}):") + content = task_files[0].read_text() + print("=" * 50) + print(content) + print("=" * 50) + + # Verify the content has expected structure + assert "ASANA TASK INFORMATION" in content + assert "Task ID:" in content + assert "Name:" in content + print("Task file content structure looks correct!") + else: + print(f"No task files found in {asana_tasks_dir}") + if asana_tasks_dir.exists(): + print(f"Directory contents: {list(asana_tasks_dir.iterdir())}") + else: + print(f"asana_tasks directory does not exist in {result}") + print(f"Result directory contents: {list(result.iterdir())}") + + # Basic assertions + assert isinstance(result, Path) + assert result.exists() + # The result should contain an asana_tasks subdirectory + asana_tasks_dir = result / "asana_tasks" + assert asana_tasks_dir.exists(), f"asana_tasks directory not found in {result}" + assert len(task_files) > 0, f"No task files found in {result}" + + except Exception as e: + print(f"Error during fetch: {e}") + import traceback + traceback.print_exc() + raise +