[Integration][ADO] Added Support for boards and columns (#1034)

# Description

What - Added functionality to fetch boards and their respective columns
from Azure DevOps.

Why - This feature is necessary to support synchronization of Azure
DevOps boards and their columns, allowing for more detailed management
and tracking of board data within the system.

How - Implemented a method that fetches boards from Azure DevOps and
retrieves their columns. The method also handles cases where the
`columns` field may be missing from the board, ensuring that the system
does not break when such cases occur.

## Type of change

Please leave one option from the following and delete the rest:

- [x] New feature (non-breaking change which adds functionality)


<h4> All tests should be run against the port production
environment(using a testing org). </h4>

### Core testing checklist

- [ ] Integration able to create all default resources from scratch
- [ ] Resync finishes successfully
- [ ] Resync able to create entities
- [ ] Resync able to update entities
- [ ] Resync able to detect and delete entities
- [ ] Scheduled resync able to abort existing resync and start a new one
- [ ] Tested with at least 2 integrations from scratch
- [ ] Tested with Kafka and Polling event listeners
- [ ] Tested deletion of entities that don't pass the selector


### Integration testing checklist

- [x] Integration able to create all default resources from scratch
- [x] Resync able to create entities
- [x] Resync able to update entities
- [x] Resync able to detect and delete entities
- [x] Resync finishes successfully
- [x] If new resource kind is added or updated in the integration, add
example raw data, mapping and expected result to the `examples` folder
in the integration directory.
- [x] If resource kind is updated, run the integration with the example
data and check if the expected result is achieved
- [ ] If new resource kind is added or updated, validate that
live-events for that resource are working as expected
- [ ] Docs PR link [here](#)

### Preflight checklist

- [ ] Handled rate limiting
- [ ] Handled pagination
- [ ] Implemented the code in async
- [ ] Support Multi account

## Screenshots

Include screenshots from your environment showing how the resources of
the integration will look.

![image](https://github.com/user-attachments/assets/aa40ddde-294d-46dd-9ae9-c3238460c9d5)

![image](https://github.com/user-attachments/assets/1e5eb973-188c-48bd-bc0c-a0001bdde3ed)


## API Documentation

Provide links to the API documentation used for this integration.
[Columns
Documentation](https://learn.microsoft.com/en-us/rest/api/azure/devops/work/columns?view=azure-devops-rest-7.1)
[Board
Documentation](https://learn.microsoft.com/en-us/rest/api/azure/devops/work/boards?view=azure-devops-rest-7.1)

---------

Co-authored-by: Matan <51418643+matan84@users.noreply.github.com>
Co-authored-by: PagesCoffy <isaac.p.coffie@gmail.com>

Author: 3 people

integrations/azure-devops/.port/resources/blueprints.json

@@ -127,7 +127,11 @@
           "type": "string",
           "icon": "AzureDevops",
           "description": "The type of work item (e.g., Bug, Task, User Story)",
-          "enum": ["Issue", "Epic", "Task"],
+          "enum": [
+            "Issue",
+            "Epic",
+            "Task"
+          ],
           "enumColors": {
             "Issue": "green",
             "Epic": "orange",
@@ -190,6 +194,66 @@
       },
       "required": []
     },
+    "mirrorProperties": {
+      "board": {
+        "title": "Board",
+        "path": "column.board.$title"
+      }
+    },
+    "calculationProperties": {},
+    "aggregationProperties": {},
+    "relations": {
+      "project": {
+        "title": "Project",
+        "target": "project",
+        "required": true,
+        "many": false
+      },
+      "column": {
+        "title": "Column",
+        "description": "The column the entity belongs",
+        "target": "column",
+        "required": true,
+        "many": false
+      }
+    }
+  },
+  {
+    "identifier": "column",
+    "title": "Column",
+    "icon": "AzureDevops",
+    "schema": {
+      "properties": {},
+      "required": []
+    },
+    "mirrorProperties": {},
+    "calculationProperties": {},
+    "aggregationProperties": {},
+    "relations": {
+      "board": {
+        "title": "board",
+        "target": "board",
+        "required": true,
+        "many": false
+      }
+    }
+  },
+  {
+    "identifier": "board",
+    "title": "Board",
+    "icon": "AzureDevops",
+    "schema": {
+      "properties": {
+        "link": {
+          "title": "Link",
+          "type": "string",
+          "format": "url",
+          "icon": "AzureDevops",
+          "description": "Link to the board in Azure DevOps"
+        }
+      },
+      "required": []
+    },
     "mirrorProperties": {},
     "calculationProperties": {},
     "aggregationProperties": {},

integrations/azure-devops/.port/resources/port-app-config.yaml

@@ -54,6 +54,19 @@ resources:
           blueprint: '"service"'
           properties:
             workItemLinking: .isEnabled and .isBlocking
+  - kind: board
+    selector:
+      query: 'true'
+    port:
+      entity:
+        mappings:
+          identifier: .id | gsub(" "; "")
+          title: .name
+          blueprint: '"board"'
+          properties:
+            link: .url
+          relations:
+            project: .__project.id | gsub(" "; "")
   - kind: work-item
     selector:
       query: 'true'
@@ -76,3 +89,17 @@ resources:
             changedDate: .fields."System.ChangedDate"
           relations:
             project: .__projectId | gsub(" "; "")
+            column: >-
+              .fields."System.WorkItemType"+"-"+.fields."System.BoardColumn"+"-"+.__project.id
+              | gsub(" "; "")
+  - kind: column
+    selector:
+      query: 'true'
+    port:
+      entity:
+        mappings:
+          identifier: .__stateType+"-"+.name+"-"+.__board.__project.id | gsub(" "; "")
+          title: .name
+          blueprint: '"column"'
+          relations:
+            board: .__board.id | gsub(" "; "")

integrations/azure-devops/CHANGELOG.md

@@ -7,11 +7,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 <!-- towncrier release notes start -->
 
+## 0.1.74 (2024-10-10)
+
+
+### Improvements
+
+
+- Added support for ingesting boards and columns
+
+
 ## 0.1.73 (2024-10-09)
 
 
 ### Improvements
 
+
 - Bumped ocean version to ^0.12.3
 
 
@@ -20,6 +30,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Improvements
 
+
 - Bumped ocean version to ^0.12.2
 
 
@@ -28,6 +39,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Improvements
 
+
 - Bumped ocean version to ^0.12.1
 
 

integrations/azure-devops/azure_devops/client/azure_devops_client.py

@@ -258,6 +258,52 @@ async def get_repository(self, repository_id: str) -> dict[Any, Any]:
         repository_data = response.json()
         return repository_data
 
+    async def get_columns(self) -> AsyncGenerator[list[dict[str, Any]], None]:
+        async for boards in self.get_boards_in_organization():
+            for board in boards:
+                yield [
+                    {
+                        **column,
+                        "__board": board,
+                        "__stateType": stateType,
+                        "__stateName": stateName,
+                    }
+                    for column in board.get("columns", [])
+                    if column.get("stateMappings")
+                    for stateType, stateName in column.get("stateMappings").items()
+                ]
+
+    async def _enrich_boards(
+        self, boards: list[dict[str, Any]], project_id: str
+    ) -> list[dict[str, Any]]:
+        for board in boards:
+            response = await self.send_request(
+                "GET",
+                f"{self._organization_base_url}/{project_id}/{API_URL_PREFIX}/work/boards/{board['id']}",
+            )
+            board.update(response.json())
+        return boards
+
+    async def _get_boards(self, project_id: str) -> list[dict[str, Any]]:
+        get_boards_url = (
+            f"{self._organization_base_url}/{project_id}/{API_URL_PREFIX}/work/boards"
+        )
+        response = await self.send_request("GET", get_boards_url)
+        board_data = response.json().get("value", [])
+        logger.info(f"Found {len(board_data)} boards for project {project_id}")
+        return await self._enrich_boards(board_data, project_id)
+
+    @cache_iterator_result()
+    async def get_boards_in_organization(
+        self,
+    ) -> AsyncGenerator[list[dict[str, Any]], None]:
+        async for projects in self.generate_projects():
+            yield [
+                {**board, "__project": project}
+                for project in projects
+                for board in await self._get_boards(project["id"])
+            ]
+
     async def generate_subscriptions_webhook_events(self) -> list[WebhookEvent]:
         headers = {"Content-Type": "application/json"}
         try:

integrations/azure-devops/azure_devops/misc.py

@@ -19,6 +19,8 @@ class Kind(StrEnum):
     TEAM = "team"
     PROJECT = "project"
     WORK_ITEM = "work-item"
+    BOARD = "board"
+    COLUMN = "column"
 
 
 PULL_REQUEST_SEARCH_CRITERIA: list[dict[str, Any]] = [