Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Threat detection #1783

Merged
merged 88 commits into from
Jan 27, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
88 commits
Select commit Hold shift + click to select a range
28fc386
pushing malicious requests to kafka
ag060 Nov 6, 2024
42a670f
added flush kafka messages task
ag060 Nov 9, 2024
6aa0a73
added window based aggregation support for malicious requests
ag060 Nov 12, 2024
4edb096
using separate mongo for storing suspect data
ag060 Nov 12, 2024
8273012
fixed kafka host
ag060 Nov 12, 2024
cce8673
fixed source ip generation logic
ag060 Nov 12, 2024
46a8a93
not syncing writes and read for RedisWriteBackCache at the same time
ag060 Nov 13, 2024
73b6de0
using redis backed increment counter cache
ag060 Nov 13, 2024
9de818f
added group id to aggregate notifier
ag060 Nov 14, 2024
7d5c5ba
added notification cooldown support
ag060 Nov 14, 2024
a372fcc
added start and end bucket ids with each detected alerts
ag060 Nov 14, 2024
4a51afc
added ttl for redis entries
ag060 Nov 14, 2024
f524ce4
added new collections for sample malicious requests and detected alerts
ag060 Nov 15, 2024
efc6da0
addded cleanup sample malicious records task
ag060 Nov 16, 2024
0676751
refactor code
ag060 Nov 18, 2024
c6c4e06
refactor code
ag060 Nov 18, 2024
fc5c2b0
refactor code
ag060 Nov 18, 2024
b2d8c96
add aggregation parse layer
ayushaga14 Nov 21, 2024
3589d09
grpc service for consuming malicious and smart events (#1748)
ag060 Nov 22, 2024
690f433
apply aggregations
ayushaga14 Nov 22, 2024
f46dc16
removed db service and using mongo client everywhere
ag060 Nov 22, 2024
d1d63a8
moved clean up task to threat protection service
ag060 Nov 22, 2024
1059e89
added authorization interceptor for grpc server
ag060 Nov 22, 2024
3202f7e
removed account_id field from consume event service requests (threat-…
ag060 Nov 22, 2024
1dbbbf4
added client side authentication support while saving malicious and s…
ag060 Nov 23, 2024
ebe7e57
added new api threat detection
ag060 Nov 25, 2024
595590c
saving malicious sample data to postgres db
ag060 Nov 25, 2024
2b7e280
abstracted out common kafka polling consumer task
ag060 Nov 26, 2024
f2d91b3
added tasks for sending alerts to protection backend
ag060 Nov 28, 2024
64ae52f
exposing single rpc method for recording alerts
ag060 Nov 28, 2024
5cd4383
updated proto definition
ag060 Nov 29, 2024
6c3d473
removed cleanup task from threat protection backend module
ag060 Nov 29, 2024
0eae5b6
updated kafka topic names
ag060 Nov 29, 2024
cc011a3
running flyway migrations programatically
ag060 Nov 29, 2024
d5d2232
refactored code
ag060 Nov 29, 2024
37e2553
added clean up for deleting all the malicious entries older than 7 days
ag060 Nov 29, 2024
cd525e1
added hibernate orm (#1786)
ag060 Dec 10, 2024
940bebf
threat protection producer consumer db writes
ayushaga14 Dec 10, 2024
9e4d118
fix create mongo client params
ayushaga14 Dec 10, 2024
474473b
deleted generated proto files
ag060 Dec 10, 2024
6c8734a
added dashboard rpc service (#1808)
ag060 Dec 10, 2024
6074ee4
updated proto definition to include malicious event type (SINGLE or A…
ag060 Dec 10, 2024
aaac144
storing api_collection_id in postgres
ag060 Dec 10, 2024
5304c11
fixed paths in ci for threat-detection module
ag060 Dec 11, 2024
e6a96ab
moved kafka config from threat detection to utils
ag060 Dec 11, 2024
0610252
moved consumer and producer on same machine for threat protection bac…
ag060 Dec 11, 2024
c2655b8
refactored dashboard proto messages
ag060 Dec 11, 2024
1950d9a
fixed MaliciousEventModel not populating data from mongo
ag060 Dec 16, 2024
8a70306
refactored code
ag060 Dec 17, 2024
b7a5715
disabled debug logging for hibernate
ag060 Dec 17, 2024
af3d124
removed unused imports
ag060 Dec 17, 2024
63aa1e4
disabled debug logging for hibernate
ag060 Dec 17, 2024
93cf428
committing kafka offsets manually now
ag060 Dec 17, 2024
2f9ae16
renamed threat detection backend service
ag060 Dec 17, 2024
c963494
added health check server
ag060 Dec 17, 2024
a339997
refactored generated proto files
ag060 Dec 18, 2024
1bfb70b
added health check for rpc and enabled reflection
ag060 Dec 18, 2024
60c115d
added health service for grpc
ag060 Dec 18, 2024
3de86e9
using vertx http server as threat detection backend (#1842)
ag060 Dec 19, 2024
84468d2
reading akto threat detection backend url from env
ag060 Dec 20, 2024
e059c2c
added try catch
ag060 Dec 21, 2024
6128b4b
fixed sample not being to backend
ag060 Dec 21, 2024
4709c35
clearing cache as soon the alert is raised
ag060 Dec 23, 2024
b53af2f
sending sample malicious events to backend only once
ag060 Dec 23, 2024
1457b51
use fastjson lib and increase template fetch interval
ayushaga14 Dec 29, 2024
207d99d
add log
ayushaga14 Dec 29, 2024
d22aab8
avoid reevaluating params for each filter check
ayushaga14 Dec 30, 2024
b013f83
using proto message envelope for kafka transport in threat detection …
ag060 Jan 1, 2025
1ced701
setting redis ttl while sync to redis instead of increment op
ag060 Jan 2, 2025
48056b4
fixed merge conflicts in staging and prod workflows
ag060 Jan 2, 2025
395c550
proto changes
ayushaga14 Jan 1, 2025
54c6921
remove logs and headers loop
ayushaga14 Jan 3, 2025
7577487
optimizations
notshivansh Jan 3, 2025
6b678ae
move httpresponse param proto generated files
ayushaga14 Jan 6, 2025
5091e59
fixed bad merges
ag060 Jan 6, 2025
b48ad81
refactored threat detection backend
ag060 Jan 6, 2025
282eec2
maxmind integration (#1924)
ag060 Jan 7, 2025
a804dd2
removed System.out.println
ag060 Jan 8, 2025
90c33f4
removed unusued proto messages
ag060 Jan 8, 2025
0999f81
reading kafka key and value serializer from config
ag060 Jan 8, 2025
83f32a5
updated custom generated httpresponse params files
ag060 Jan 8, 2025
41f4e21
save and parse agg rules
ayushaga14 Jan 14, 2025
6dbc96f
using local redis instead of centralized redis (#1979)
ag060 Jan 16, 2025
ab38baa
updated dockerfile for threat detection client
ag060 Jan 16, 2025
eb41974
fixed redis longvalue codec value decoder
ag060 Jan 16, 2025
0fed5d5
reading category and subCategory from info
ag060 Jan 20, 2025
f193fd3
using proper display names for threat detection filter category
ag060 Jan 21, 2025
49ebc58
reverting back to redis backed cache
ag060 Jan 22, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .github/workflows/prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,11 @@ on:
type: boolean
default: true
description: Internal
threat_detection:
type: boolean
default: true
description: Threat Detection Client

threat_detection_backend:
type: boolean
default: true
Expand Down Expand Up @@ -132,6 +137,11 @@ jobs:
echo "::set-output name=image::$ECR_REGISTRY/akto-internal:$IMAGE_TAG"
fi

if [[ "${{ github.event.inputs.threat_detection}}" == "true" ]]; then
cd ../threat-detection
docker buildx build --platform linux/arm64/v8,linux/amd64 -t $ECR_REGISTRY/akto-threat-detection:$IMAGE_TAG_1 -t $ECR_REGISTRY/akto-threat-detection:$IMAGE_TAG_2 . --push
fi

if [[ "${{ github.event.inputs.threat_detection_backend}}" == "true" ]]; then
cd ../threat-detection-backend
docker buildx build --platform linux/arm64/v8,linux/amd64 -t $ECR_REGISTRY/akto-threat-detection-backend:$IMAGE_TAG_1 -t $ECR_REGISTRY/akto-threat-detection-backend:$IMAGE_TAG_2 . --push
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/staging.yml
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,8 @@ jobs:
docker buildx build --platform linux/arm64/v8,linux/amd64 -t $ECR_REGISTRY/akto-internal:$IMAGE_TAG . --push
cd ../source-code-analyser
docker buildx build --platform linux/arm64/v8,linux/amd64 -t $ECR_REGISTRY/source-code-analyser:$IMAGE_TAG . --push
cd ../threat-detection
docker buildx build --platform linux/arm64/v8,linux/amd64 -t $ECR_REGISTRY/akto-threat-detection:$IMAGE_TAG . --push
cd ../threat-detection-backend
docker buildx build --platform linux/arm64/v8,linux/amd64 -t $ECR_REGISTRY/akto-threat-detection-backend:$IMAGE_TAG . --push

Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,4 @@ build: proto-gen
mvn install -DskipTests

build-clean: proto-gen
mvn clean install -DskipTests
mvn clean install -DskipTests
9 changes: 0 additions & 9 deletions apps/api-threat-detection/Dockerfile

This file was deleted.

190 changes: 0 additions & 190 deletions apps/api-threat-detection/pom.xml

This file was deleted.

This file was deleted.

Loading
Loading