-
Notifications
You must be signed in to change notification settings - Fork 23
Custom Metadata tool #163
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
abhinavmathur-atlan
wants to merge
36
commits into
main
Choose a base branch
from
MCP-8-Abhi
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Custom Metadata tool #163
Changes from all commits
Commits
Show all changes
36 commits
Select commit
Hold shift + click to select a range
22c024b
add: standard library imports
SatabrataPaul-GitAc 18c4b04
add: static methods for atlan api requests
SatabrataPaul-GitAc ff5f808
export: get_custom_metadata_context tool
SatabrataPaul-GitAc 56d4e6f
add: tool to fetch custom metadata context
SatabrataPaul-GitAc 99b8d55
add: custom_metadata_context tool registration
SatabrataPaul-GitAc 8652d58
remove: get_custom_metadata_context tool
SatabrataPaul-GitAc bf9ccb9
add: cache manager for perisisting custom metadata context between mu…
SatabrataPaul-GitAc 8f0eaf4
add: utility function to fetch all custom metadata context from a tenant
SatabrataPaul-GitAc 7eb389f
add: detect_custom_metadata_trigger tool
SatabrataPaul-GitAc d73074a
add: registration of detect_custom_metadata_from_query mcp tool
SatabrataPaul-GitAc 8bfe222
add: pre-commit fixes and update pre-commit versions
SatabrataPaul-GitAc fcd70a5
add: support for custom_metadata filterds in search_assets tool
SatabrataPaul-GitAc 150d08b
remove: custom metadata detector from query tool
SatabrataPaul-GitAc 042babe
remove: detect_custom_metadata_from_query tool registration
SatabrataPaul-GitAc 0a52351
add: custom_metadata_context tool
SatabrataPaul-GitAc 7512a6b
update: custom_metadata_context tool import
SatabrataPaul-GitAc 93cc2f6
add: get_custom_metadata_context_tool registration in server.py
SatabrataPaul-GitAc 017d738
update: procesisng logic for custom metadata filters
SatabrataPaul-GitAc dafac19
Merge branch 'main' into MCP-8
SatabrataPaul-GitAc e2f2654
fix: return type of get_custom_metadata_context_tool
SatabrataPaul-GitAc a08271c
fix: use active client loaded with env for custom_metadata_field search
SatabrataPaul-GitAc 069a2cc
fix: custom_metadata_conditions in search_assets_tool
SatabrataPaul-GitAc abe1430
fix: back earlier versions of pre-commit hooks
SatabrataPaul-GitAc 3fa116d
remove: main guard clause from custom_metadata_context.py file
SatabrataPaul-GitAc 9960d50
fix: return type of get_custom_metadata_context_tool
SatabrataPaul-GitAc b049283
update: base prompt for the entire result set, not every bm
SatabrataPaul-GitAc 2860c2f
update: remove extra examples from get_custom_metadata_context_tool a…
SatabrataPaul-GitAc 37142c5
fix: repitive description
SatabrataPaul-GitAc 582a125
fix: variable name
SatabrataPaul-GitAc 9f58b99
changes
abhinavmathur-atlan 890694f
chore: fix cm tool
abhinavmathur-atlan 89211d6
remove extra docs
abhinavmathur-atlan 300df26
precommit checks
abhinavmathur-atlan ade0e97
search changes
abhinavmathur-atlan 8d808ab
docstring fix
abhinavmathur-atlan d83fed6
fix: consistent naming
abhinavmathur-atlan File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,162 @@ | ||
| import logging | ||
| from typing import Any, Dict, List | ||
| from client import get_atlan_client | ||
| from pyatlan.cache.custom_metadata_cache import CustomMetadataCache | ||
| from pyatlan.cache.enum_cache import EnumCache | ||
|
|
||
| logger = logging.getLogger(__name__) | ||
|
|
||
|
|
||
| def process_custom_metadata( | ||
| cm_def: Any, | ||
| enum_cache: EnumCache, | ||
| ) -> Dict[str, Any]: | ||
| """ | ||
| Generates context prompt for a given Atlan business metadata definition. | ||
|
|
||
| Args: | ||
| cm_def: CustomMetadataDef object from PyAtlan | ||
| enum_cache: EnumCache instance for enriching enum attributes | ||
|
|
||
| Returns: | ||
| Dictionary containing prompt, metadata details, and id | ||
| """ | ||
| cm_name = cm_def.name or "" | ||
| cm_display_name = cm_def.display_name or "" | ||
| description = cm_def.description or "No description available." | ||
| guid = cm_def.guid | ||
|
|
||
| # For prompt: comma separated attribute names and descriptions | ||
| attributes_list_for_prompt: List[str] = [] | ||
| parsed_attributes_for_metadata: List[Dict[str, Any]] = [] | ||
|
|
||
| if not cm_def.attribute_defs: | ||
| attributes_str_for_prompt = "None" | ||
| metadata: Dict[str, Any] = { | ||
| "name": cm_name, | ||
| "display_name": cm_display_name, | ||
| "description": description, | ||
| "attributes": [], | ||
| } | ||
| prompt = f"""{cm_display_name}|{description}|{attributes_str_for_prompt}""" | ||
| return {"prompt": prompt, "metadata": metadata, "id": guid} | ||
|
|
||
| for attr_def in cm_def.attribute_defs: | ||
| attr_name = attr_def.display_name or attr_def.name or "Unnamed attribute" | ||
| attr_desc = attr_def.description or "No description" | ||
| attributes_list_for_prompt.append(f"{attr_name}:{attr_desc}") | ||
|
|
||
| base_description = attr_def.description or "" | ||
| enhanced_description = base_description | ||
| enum_enrichment = None | ||
|
|
||
| # Check if attribute is an enum type and enrich with enum values | ||
| if attr_def.options and attr_def.options.is_enum: | ||
| enum_type = attr_def.options.enum_type | ||
| if enum_type: | ||
| try: | ||
| enum_def = enum_cache.get_by_name(enum_type) | ||
| if enum_def and enum_def.element_defs: | ||
| enum_values = [ | ||
| elem.value for elem in enum_def.element_defs if elem.value | ||
| ] | ||
| if enum_values: | ||
| quoted_values = ", ".join( | ||
| [f"'{value}'" for value in enum_values] | ||
| ) | ||
| description_with_allowed_values = f"{base_description} This attribute can have enum values: {quoted_values}.".strip() | ||
| enhanced_description = description_with_allowed_values | ||
|
|
||
| # Create enum enrichment data - only include values field | ||
| enum_enrichment = { | ||
| "values": enum_values, | ||
| } | ||
| except Exception as e: | ||
| logger.debug(f"Could not enrich enum type {enum_type}: {e}") | ||
| enum_enrichment = None | ||
|
|
||
| attribute_metadata = { | ||
| "name": attr_def.name, | ||
| "display_name": attr_def.display_name, | ||
| "data_type": attr_def.type_name, | ||
| "description": enhanced_description, | ||
| } | ||
|
|
||
| if enum_enrichment: | ||
| attribute_metadata["enumEnrichment"] = enum_enrichment | ||
|
|
||
| parsed_attributes_for_metadata.append(attribute_metadata) | ||
|
|
||
| attributes_str_for_prompt = ( | ||
| ", ".join(attributes_list_for_prompt) if attributes_list_for_prompt else "None" | ||
| ) | ||
|
|
||
| metadata: Dict[str, Any] = { | ||
| "name": cm_name, | ||
| "display_name": cm_display_name, | ||
| "description": description, | ||
| "attributes": parsed_attributes_for_metadata, | ||
| } | ||
|
|
||
| prompt = f"""{cm_display_name}|{description}|{attributes_str_for_prompt}""" | ||
|
|
||
| return {"prompt": prompt, "metadata": metadata, "id": guid} | ||
|
|
||
|
|
||
| def get_custom_metadata_context() -> Dict[str, Any]: | ||
| """ | ||
| Fetch custom metadata context using PyAtlan's native cache classes. | ||
|
|
||
| Returns: | ||
| Dictionary containing context and custom metadata results | ||
| """ | ||
| custom_metadata_results: List[Dict[str, Any]] = [] | ||
|
|
||
| try: | ||
| # Get Atlan client | ||
| client = get_atlan_client() | ||
|
|
||
| # Initialize caches using PyAtlan's native classes | ||
| cm_cache = CustomMetadataCache(client) | ||
| enum_cache = EnumCache(client) | ||
|
|
||
| # Get all custom metadata attributes (includes full definitions) | ||
| all_custom_attributes = cm_cache.get_all_custom_attributes( | ||
| include_deleted=False, force_refresh=True | ||
| ) | ||
|
|
||
| # Process each custom metadata set | ||
| for set_name in all_custom_attributes.keys(): | ||
| try: | ||
| # Get the full custom metadata definition | ||
| cm_def = cm_cache.get_custom_metadata_def(set_name) | ||
|
|
||
| # Process and enrich with enum data | ||
| result = process_custom_metadata(cm_def, enum_cache) | ||
| custom_metadata_results.append(result) | ||
|
|
||
| except Exception as e: | ||
| logger.warning( | ||
| f"Error processing custom metadata set '{set_name}': {e}" | ||
| ) | ||
| continue | ||
|
|
||
| logger.info( | ||
| f"Fetched {len(custom_metadata_results)} custom metadata definitions with enum enrichment." | ||
| ) | ||
|
|
||
| except Exception as e: | ||
| logger.error( | ||
| f"Error fetching custom metadata context: {e}", | ||
| exc_info=True, | ||
| ) | ||
| return { | ||
| "context": "Error fetching custom metadata definitions", | ||
| "custom_metadata_results": [], | ||
| "error": str(e), | ||
| } | ||
|
|
||
| return { | ||
| "context": "This is the list of custom metadata definitions used in the data catalog to add more information to an asset", | ||
| "custom_metadata_results": custom_metadata_results, | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we need to send this?