Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incident reports docx #325

Merged
merged 44 commits into from
Oct 16, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
6f27dc9
`refactor: Update endpoint in provision.py`
taylorwalton Oct 2, 2024
56946b3
`refactor: Update endpoint in provision.py`
taylorwalton Oct 2, 2024
a3f51a7
chore: update dependencies in frontend
Linko91 Oct 3, 2024
d47d561
refactor: update case export service
Linko91 Oct 3, 2024
09817a4
add case report templates db table
taylorwalton Oct 3, 2024
866fe88
`chore: Add copilot-case-report-templates bucket creation during init…
taylorwalton Oct 3, 2024
4302a75
`refactor: Update route for generating incident report CSV`
taylorwalton Oct 3, 2024
bf3b732
refactor: Update route for generating incident case report CSV
Linko91 Oct 3, 2024
24a3e8c
upload case report template to datastore and db
taylorwalton Oct 3, 2024
4bb9195
Merge branch 'incident-reports-docx' of https://github.com/socfortres…
taylorwalton Oct 3, 2024
86ed39c
delete case report template functions and route
taylorwalton Oct 3, 2024
de8002f
download report template function and route
taylorwalton Oct 3, 2024
ad7e820
testing downloading case and applying jinja template
taylorwalton Oct 4, 2024
eac575b
loading alert with context
taylorwalton Oct 4, 2024
373d99a
`refactor: Remove commented out code for creating case context`
taylorwalton Oct 4, 2024
afbb02c
build for wazuh group template creation fix
taylorwalton Oct 4, 2024
741a172
`chore: Update branch name in Docker workflow`
taylorwalton Oct 4, 2024
0a4f7ae
add docxtpl to requirements
taylorwalton Oct 4, 2024
1f34c99
`chore: Update branch name in Docker workflow`
taylorwalton Oct 4, 2024
fcb9e28
bump lxml down
taylorwalton Oct 4, 2024
e05f0af
`chore: Update branch name in Docker workflow`
taylorwalton Oct 4, 2024
539d2c3
comment docx out for now
taylorwalton Oct 4, 2024
01674b2
`chore: Update branch name in Docker workflow`
taylorwalton Oct 4, 2024
90b242d
`chore: Update branch name in Docker workflow`
taylorwalton Oct 4, 2024
d680491
`chore: Update branch name in Docker workflow`
taylorwalton Oct 4, 2024
d7440d2
reimplment docx
taylorwalton Oct 14, 2024
865de63
`chore: Bump typing_extensions to version 4.9.0`
taylorwalton Oct 14, 2024
3520e3c
CaseDownloadDocxRequest model
taylorwalton Oct 14, 2024
b69cf63
chore: Update branch name in Docker workflow
taylorwalton Oct 14, 2024
a7420d7
`chore: Add endpoint to list case report template files`
taylorwalton Oct 14, 2024
ac2b7ac
`feat: Add endpoint to create default case report template`
taylorwalton Oct 14, 2024
9eb709b
`chore: Add file type validation for uploaded case report template`
taylorwalton Oct 14, 2024
63a6bbc
chore: update dependencies in frontend
Linko91 Oct 15, 2024
a9a93bb
feat: add case report api/types
Linko91 Oct 15, 2024
84d2c49
refactor: case list page
Linko91 Oct 16, 2024
fd00b7d
feat: add new utility type
Linko91 Oct 16, 2024
438ef7d
feat: add case report template store
Linko91 Oct 16, 2024
be54bd8
feat: add case report feature
Linko91 Oct 16, 2024
c947a09
`feat: Add endpoint to check if any of the default case report templa…
taylorwalton Oct 16, 2024
17e5df9
fix: case report component link
Linko91 Oct 16, 2024
186ad31
Merge remote-tracking branch 'origin/incident-reports-docx' into inci…
Linko91 Oct 16, 2024
e376c37
feat: add default template check
Linko91 Oct 16, 2024
912543f
some route docs
taylorwalton Oct 16, 2024
d288d3c
precommit fixes
taylorwalton Oct 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
with:
context: ./backend
push: true
tags: ghcr.io/socfortress/copilot-backend:latest
tags: ghcr.io/socfortress/copilot-backend:lab
build-args: |
COPILOT_API_KEY=${{ secrets.COPILOT_API_KEY }}

Expand Down Expand Up @@ -60,7 +60,7 @@ jobs:
with:
context: ./frontend
push: true
tags: ghcr.io/socfortress/copilot-frontend:latest
tags: ghcr.io/socfortress/copilot-frontend:lab

- name: Notify Discord
uses: appleboy/discord-action@v1.0.0
Expand Down
2 changes: 1 addition & 1 deletion backend/alembic/alembic.ini
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ version_path_separator = os # Use os.pathsep. Default configuration used for ne
# are written from script.py.mako
# output_encoding = utf-8

sqlalchemy.url = mysql+pymysql://copilot:H7U3AHsXWSGvE5L123B7$GQdLQz@10.255.254.2/copilot
sqlalchemy.url = mysql+pymysql://copilot:REPLACE_ME@copilot-mysql/copilot


[post_write_hooks]
Expand Down
1 change: 1 addition & 0 deletions backend/alembic/env.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
from app.incidents.models import Case
from app.incidents.models import CaseAlertLink
from app.incidents.models import CaseDataStore
from app.incidents.models import CaseReportTemplateDataStore
from app.incidents.models import Comment
from app.incidents.models import CustomerCodeFieldName
from app.incidents.models import FieldName
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
"""Add table for case report templates

Revision ID: a1e4beb87498
Revises: ba64d98dbdc5
Create Date: 2024-10-03 10:14:31.289126

"""
from typing import Sequence
from typing import Union

import sqlalchemy as sa

from alembic import op

# revision identifiers, used by Alembic.
revision: str = "a1e4beb87498"
down_revision: Union[str, None] = "ba64d98dbdc5"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None


def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table(
"incident_management_case_report_template_datastore",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("report_template_name", sa.String(length=255), nullable=False),
sa.Column("bucket_name", sa.String(length=255), nullable=False),
sa.Column("object_key", sa.String(length=1024), nullable=False),
sa.Column("file_name", sa.String(length=255), nullable=False),
sa.Column("content_type", sa.String(length=100), nullable=True),
sa.Column("file_size", sa.Integer(), nullable=True),
sa.Column("upload_time", sa.DateTime(), nullable=False),
sa.Column("file_hash", sa.String(length=128), nullable=False),
sa.PrimaryKeyConstraint("id"),
)
# ### end Alembic commands ###


def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.drop_table("incident_management_case_report_template_datastore")
# ### end Alembic commands ###
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ async def configure_wazuh_group(group_code, template_path, request: ProvisionNew

# Make the API request to update the group configuration
return await send_wazuh_put_request(
endpoint=f"groups/{group_code}/configuration",
endpoint=f"/groups/{group_code}/configuration",
data=group_config,
xml_data=True,
)
Expand Down
39 changes: 38 additions & 1 deletion backend/app/data_store/data_store_operations.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import os
from typing import Optional

import aiofiles
import aiohttp
Expand All @@ -7,6 +8,7 @@
from loguru import logger

from app.data_store.data_store_schema import CaseDataStoreCreation
from app.data_store.data_store_schema import CaseReportTemplateDataStoreCreation
from app.data_store.data_store_session import create_session


Expand Down Expand Up @@ -43,7 +45,31 @@ async def upload_case_data_store(data: CaseDataStoreCreation, file: UploadFile)
os.remove(temp_file_path)


async def download_case_data_store(bucket_name: str, object_name: str) -> bytes:
async def upload_case_report_template_data_store(data: CaseReportTemplateDataStoreCreation, file: UploadFile) -> None:
client = await create_session()
logger.info(f"Uploading file {file.filename} to bucket {data.bucket_name}")

# Define the temporary file path
temp_file_path = os.path.join(os.getcwd(), file.filename)

# Save the file to the temporary location
async with aiofiles.open(temp_file_path, "wb") as out_file:
content = await file.read()
await out_file.write(content)

# Upload the file to Minio
await client.fput_object(
bucket_name=data.bucket_name,
object_name=f"{file.filename}",
file_path=temp_file_path,
content_type=data.content_type,
)

# Optionally, remove the temporary file after upload
os.remove(temp_file_path)


async def download_data_store(bucket_name: str, object_name: str) -> bytes:
client = await create_session()
logger.info(f"Downloading file {object_name} from bucket {bucket_name}")
try:
Expand Down Expand Up @@ -77,6 +103,17 @@ async def list_case_data_store_files(bucket_name: str, case_id: int) -> list:
return objects


async def list_case_report_template_data_store_files(bucket_name: Optional[str] = "copilot-case-report-templates") -> list:
client = await create_session()
objects_list = []
logger.info(f"Listing objects in bucket {bucket_name}")
objects = await client.list_objects(bucket_name)
for obj in objects:
logger.info(f"Object: {obj.object_name}")
objects_list.append(obj.object_name)
return objects_list


async def create_buckets() -> None:
await create_bucket_if_not_exists("copilot-cases")

Expand Down
10 changes: 10 additions & 0 deletions backend/app/data_store/data_store_schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,13 @@ class CaseDataStoreCreation(BaseModel):
file_size: Optional[int] = None
upload_time: datetime = Field(default_factory=datetime.utcnow)
file_hash: str = Field(max_length=128)


class CaseReportTemplateDataStoreCreation(BaseModel):
report_template_name: str = Field(max_length=255)
bucket_name: str = Field(max_length=255)
object_key: str = Field(max_length=1024)
file_name: str = Field(max_length=255)
content_type: Optional[str] = Field(max_length=100, default=None)
upload_time: datetime = Field(default_factory=datetime.utcnow)
file_hash: str = Field(max_length=128)
1 change: 1 addition & 0 deletions backend/app/data_store/data_store_setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,4 @@ async def create_bucket_if_not_exists(bucket_name: str) -> None:

async def create_buckets() -> None:
await create_bucket_if_not_exists("copilot-cases")
await create_bucket_if_not_exists("copilot-case-report-templates")
14 changes: 14 additions & 0 deletions backend/app/incidents/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -165,3 +165,17 @@ class CaseDataStore(SQLModel, table=True):
file_hash: str = Field(max_length=128, nullable=False) # Hash of the file (e.g., SHA-256)

case: "Case" = Relationship(back_populates="data_store")


class CaseReportTemplateDataStore(SQLModel, table=True):
__tablename__ = "incident_management_case_report_template_datastore"
id: Optional[int] = Field(default=None, primary_key=True)

report_template_name: str = Field(max_length=255, nullable=False)
bucket_name: str = Field(max_length=255, nullable=False) # Name of the MinIO bucket
object_key: str = Field(max_length=1024, nullable=False) # Path/key of the file in MinIO
file_name: str = Field(max_length=255, nullable=False) # Original file name uploaded by the user
content_type: Optional[str] = Field(max_length=100, nullable=True) # MIME type of the file
file_size: Optional[int] = Field(nullable=True) # File size in bytes
upload_time: datetime = Field(default_factory=datetime.utcnow) # Time of upload
file_hash: str = Field(max_length=128, nullable=False) # Hash of the file (e.g., SHA-256)
95 changes: 95 additions & 0 deletions backend/app/incidents/routes/db_operations.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import io
import mimetypes
from typing import List
from typing import Optional

Expand All @@ -20,6 +21,9 @@
from app.connectors.wazuh_indexer.utils.universal import get_index_mappings_key_names
from app.connectors.wazuh_indexer.utils.universal import get_index_source
from app.customer_provisioning.routes.provision import check_customer_exists
from app.data_store.data_store_operations import (
list_case_report_template_data_store_files,
)
from app.db.db_session import get_db
from app.db.universal_models import Customers
from app.incidents.models import Alert
Expand All @@ -46,10 +50,13 @@
from app.incidents.schema.db_operations import CaseCreateFromAlert
from app.incidents.schema.db_operations import CaseDataStoreResponse
from app.incidents.schema.db_operations import CaseOutResponse
from app.incidents.schema.db_operations import CaseReportTemplateDataStoreListResponse
from app.incidents.schema.db_operations import CaseReportTemplateDataStoreResponse
from app.incidents.schema.db_operations import CaseResponse
from app.incidents.schema.db_operations import CommentCreate
from app.incidents.schema.db_operations import CommentResponse
from app.incidents.schema.db_operations import ConfiguredSourcesResponse
from app.incidents.schema.db_operations import DefaultReportTemplateFileNames
from app.incidents.schema.db_operations import FieldAndAssetNames
from app.incidents.schema.db_operations import FieldAndAssetNamesResponse
from app.incidents.schema.db_operations import ListCaseDataStoreResponse
Expand Down Expand Up @@ -114,8 +121,10 @@
from app.incidents.services.db_operations import delete_case
from app.incidents.services.db_operations import delete_field_name
from app.incidents.services.db_operations import delete_file_from_case
from app.incidents.services.db_operations import delete_report_template
from app.incidents.services.db_operations import delete_timefield_name
from app.incidents.services.db_operations import download_file_from_case
from app.incidents.services.db_operations import download_report_template
from app.incidents.services.db_operations import file_exists
from app.incidents.services.db_operations import get_alert_by_id
from app.incidents.services.db_operations import get_alert_context_by_id
Expand Down Expand Up @@ -147,12 +156,15 @@
from app.incidents.services.db_operations import replace_asset_name
from app.incidents.services.db_operations import replace_field_name
from app.incidents.services.db_operations import replace_timefield_name
from app.incidents.services.db_operations import report_template_exists
from app.incidents.services.db_operations import update_alert_assigned_to
from app.incidents.services.db_operations import update_alert_status
from app.incidents.services.db_operations import update_case_assigned_to
from app.incidents.services.db_operations import update_case_customer_code
from app.incidents.services.db_operations import update_case_status
from app.incidents.services.db_operations import upload_file_to_case
from app.incidents.services.db_operations import upload_report_template
from app.incidents.services.db_operations import upload_report_template_to_data_store
from app.incidents.services.db_operations import validate_source_exists

incidents_db_operations_router = APIRouter()
Expand Down Expand Up @@ -807,3 +819,86 @@ async def delete_case_data_store_file_endpoint(case_id: int, file_name: str, db:
@incidents_db_operations_router.get("/case/{case_id}", response_model=CaseOutResponse)
async def get_case_by_id_endpoint(case_id: int, db: AsyncSession = Depends(get_db)):
return CaseOutResponse(cases=[await get_case_by_id(case_id, db)], success=True, message="Case retrieved successfully")


@incidents_db_operations_router.get("/case-report-template", response_model=CaseReportTemplateDataStoreListResponse)
async def list_case_report_template_data_store_files_endpoint(db: AsyncSession = Depends(get_db)):
logger.info("Listing all files in the data store")
return CaseReportTemplateDataStoreListResponse(
case_report_template_data_store=await list_case_report_template_data_store_files(),
success=True,
message="Files retrieved successfully",
)


@incidents_db_operations_router.get("/case-report-template/do-default-template-exists")
async def check_default_case_report_template_exists_endpoint(db: AsyncSession = Depends(get_db)):
"""
Endpoint to check if any of the default case report template files exist in the data store.

If any of them do, return True, else return False.

Returns:
- success (bool): Indicates if the operation was successful.
- message (str): Success message.
- default_template_exists (bool): Indicates if any of the default case report template files exist in the data store.
"""
for template in DefaultReportTemplateFileNames:
if await report_template_exists(template.value, db):
return {"success": True, "message": "Default case report template exists", "default_template_exists": True}

return {"success": True, "message": "No default case report templates exist", "default_template_exists": False}


@incidents_db_operations_router.post("/case-report-template/default-template", response_model=CaseReportTemplateDataStoreListResponse)
async def create_default_case_report_template_endpoint(db: AsyncSession = Depends(get_db)):
"""
Create a default case report template in the data store.

Returns:
CaseReportTemplateDataStoreListResponse: The response containing the created case report template data store.

Raises:
None
"""
logger.info("Creating default file in the data store")
return CaseReportTemplateDataStoreListResponse(
case_report_template_data_store=await upload_report_template_to_data_store(db),
success=True,
message="Default file created successfully",
)


@incidents_db_operations_router.get("/case-report-template/download/{file_name}")
async def download_case_report_template_endpoint(file_name: str, db: AsyncSession = Depends(get_db)) -> StreamingResponse:
file_bytes, file_content_type = await download_report_template(file_name, db)
logger.info(f"Streaming file {file_name}")
output = io.BytesIO(file_bytes)
output.seek(0)

return StreamingResponse(output, media_type=file_content_type, headers={"Content-Disposition": f"attachment; filename={file_name}"})


@incidents_db_operations_router.post("/case-report-template/upload", response_model=CaseReportTemplateDataStoreResponse)
async def upload_case_report_template_endpoint(
file: UploadFile = File(...),
db: AsyncSession = Depends(get_db),
):
# Check if the file type is a .docx
mime_type, _ = mimetypes.guess_type(file.filename)
if mime_type != "application/vnd.openxmlformats-officedocument.wordprocessingml.document":
raise HTTPException(status_code=400, detail="Invalid file type. Only .docx files are allowed.")

if await report_template_exists(file.filename, db):
raise HTTPException(status_code=400, detail="File name already exists for this template")
return CaseReportTemplateDataStoreResponse(
case_report_template_data_store=await upload_report_template(file, db),
success=True,
message="File uploaded successfully",
)


@incidents_db_operations_router.delete("/case-report-template/{file_name}")
async def delete_case_report_template_endpoint(file_name: str, db: AsyncSession = Depends(get_db)):
await delete_report_template(file_name, db)
return {"message": "File deleted successfully", "success": True}
Loading
Loading