Code

Threat detection#1783

Code

Merged

ayushaga14 merged 88 commits intomasterakto-api-security/akto:masterfrom threat-detection-v1akto-api-security/akto:threat-detection-v1Copy head branch name to clipboardJan 27, 2025

+6,567-468

Commits

Commits on Jan 22, 2025

• pushing malicious requests to kafka

  ag060

  committed
  28fc386

  Copy full SHA for 28fc386

• added flush kafka messages task

  ag060

  committed
  42a670f

  Copy full SHA for 42a670f

• added window based aggregation support for malicious requests

  ag060

  committed
  6aa0a73

  Copy full SHA for 6aa0a73

• using separate mongo for storing suspect data

  ag060

  committed
  4edb096

  Copy full SHA for 4edb096

• fixed kafka host

  ag060

  committed
  8273012

  Copy full SHA for 8273012

• fixed source ip generation logic

  ag060

  committed
  cce8673

  Copy full SHA for cce8673

• not syncing writes and read for RedisWriteBackCache at the same time

  ag060

  committed
  46a8a93

  Copy full SHA for 46a8a93

• using redis backed increment counter cache

  ag060

  committed
  73b6de0

  Copy full SHA for 73b6de0

• added group id to aggregate notifier

  ag060

  committed
  9de818f

  Copy full SHA for 9de818f

• added notification cooldown support

  ag060

  committed
  7d5c5ba

  Copy full SHA for 7d5c5ba

• added start and end bucket ids with each detected alerts

  ag060

  committed
  a372fcc

  Copy full SHA for a372fcc

• added ttl for redis entries

  ag060

  committed
  4a51afc

  Copy full SHA for 4a51afc

• added new collections for sample malicious requests and detected alerts

  ag060

  committed
  f524ce4

  Copy full SHA for f524ce4

• addded cleanup sample malicious records task

  ag060

  committed
  efc6da0

  Copy full SHA for efc6da0

• refactor code

  ag060

  committed
  0676751

  Copy full SHA for 0676751

• refactor code

  ag060

  committed
  c6c4e06

  Copy full SHA for c6c4e06

• refactor code

  ag060

  committed
  fc5c2b0

  Copy full SHA for fc5c2b0

• add aggregation parse layer

  

  ayushaga14

  authored and

  ag060

  committed
  b2d8c96

  Copy full SHA for b2d8c96

• grpc service for consuming malicious and smart events (#1748)

  Show description for 3589d09
  ag060

  committed
  3589d09

  Copy full SHA for 3589d09

• apply aggregations

  

  ayushaga14

  authored and

  ag060

  committed
  690f433

  Copy full SHA for 690f433

• removed db service and using mongo client everywhere

  ag060

  committed
  f46dc16

  Copy full SHA for f46dc16

• moved clean up task to threat protection service

  ag060

  committed
  d1d63a8

  Copy full SHA for d1d63a8

• added authorization interceptor for grpc server

  ag060

  committed
  1059e89

  Copy full SHA for 1059e89

• removed account_id field from consume event service requests (threat-detection-backend)

  Show description for 3202f7e
  ag060

  committed
  3202f7e

  Copy full SHA for 3202f7e

• added client side authentication support while saving malicious and smart events

  ag060

  committed
  1dbbbf4

  Copy full SHA for 1dbbbf4

• added new api threat detection

  Show description for ebe7e57
  ag060

  committed
  ebe7e57

  Copy full SHA for ebe7e57

• saving malicious sample data to postgres db

  ag060

  committed
  595590c

  Copy full SHA for 595590c

• abstracted out common kafka polling consumer task

  ag060

  committed
  2b7e280

  Copy full SHA for 2b7e280

• added tasks for sending alerts to protection backend

  ag060

  committed
  f2d91b3

  Copy full SHA for f2d91b3

• exposing single rpc method for recording alerts

  ag060

  committed
  64ae52f

  Copy full SHA for 64ae52f

• updated proto definition

  ag060

  committed
  5cd4383

  Copy full SHA for 5cd4383

• removed cleanup task from threat protection backend module

  ag060

  committed
  6c3d473

  Copy full SHA for 6c3d473

• updated kafka topic names

  ag060

  committed
  0eae5b6

  Copy full SHA for 0eae5b6

• running flyway migrations programatically

  ag060

  committed
  cc011a3

  Copy full SHA for cc011a3

• refactored code

  ag060

  committed
  d5d2232

  Copy full SHA for d5d2232

• added clean up for deleting all the malicious entries older than 7 days

  ag060

  committed
  37e2553

  Copy full SHA for 37e2553

• added hibernate orm (#1786)

  ag060

  committed
  cd525e1

  Copy full SHA for cd525e1

• threat protection producer consumer db writes

  

  ayushaga14

  authored and

  ag060

  committed
  940bebf

  Copy full SHA for 940bebf

• fix create mongo client params

  

  ayushaga14

  authored and

  ag060

  committed
  9e4d118

  Copy full SHA for 9e4d118

• deleted generated proto files

  ag060

  committed
  474473b

  Copy full SHA for 474473b

• added dashboard rpc service (#1808)

  ag060

  committed
  6c8734a

  Copy full SHA for 6c8734a

• updated proto definition to include malicious event type (SINGLE or AGGREGATED)

  ag060

  committed
  6074ee4

  Copy full SHA for 6074ee4

• storing api_collection_id in postgres

  ag060

  committed
  aaac144

  Copy full SHA for aaac144

• fixed paths in ci for threat-detection module

  ag060

  committed
  5304c11

  Copy full SHA for 5304c11

• moved kafka config from threat detection to utils

  ag060

  committed
  e6a96ab

  Copy full SHA for e6a96ab

• moved consumer and producer on same machine for threat protection backend

  ag060

  committed
  0610252

  Copy full SHA for 0610252

• refactored dashboard proto messages

  ag060

  committed
  c2655b8

  Copy full SHA for c2655b8

• fixed MaliciousEventModel not populating data from mongo

  ag060

  committed
  1950d9a

  Copy full SHA for 1950d9a

• refactored code

  ag060

  committed
  8a70306

  Copy full SHA for 8a70306

• disabled debug logging for hibernate

  ag060

  committed
  b7a5715

  Copy full SHA for b7a5715

• removed unused imports

  ag060

  committed
  af3d124

  Copy full SHA for af3d124

• disabled debug logging for hibernate

  ag060

  committed
  63aa1e4

  Copy full SHA for 63aa1e4

• committing kafka offsets manually now

  ag060

  committed
  93cf428

  Copy full SHA for 93cf428

• renamed threat detection backend service

  ag060

  committed
  2f9ae16

  Copy full SHA for 2f9ae16

• added health check server

  ag060

  committed
  c963494

  Copy full SHA for c963494

• refactored generated proto files

  ag060

  committed
  a339997

  Copy full SHA for a339997

• added health check for rpc and enabled reflection

  ag060

  committed
  1bfb70b

  Copy full SHA for 1bfb70b

• added health service for grpc

  ag060

  committed
  60c115d

  Copy full SHA for 60c115d

• using vertx http server as threat detection backend (#1842)

  ag060

  committed
  3de86e9

  Copy full SHA for 3de86e9

• reading akto threat detection backend url from env

  ag060

  committed
  84468d2

  Copy full SHA for 84468d2

• added try catch

  ag060

  committed
  e059c2c

  Copy full SHA for e059c2c

• fixed sample not being to backend

  ag060

  committed
  6128b4b

  Copy full SHA for 6128b4b

• clearing cache as soon the alert is raised

  ag060

  committed
  4709c35

  Copy full SHA for 4709c35

• sending sample malicious events to backend only once

  ag060

  committed
  b53af2f

  Copy full SHA for b53af2f

• use fastjson lib and increase template fetch interval

  

  ayushaga14

  authored and

  ag060

  committed
  1457b51

  Copy full SHA for 1457b51

• add log

  

  ayushaga14

  authored and

  ag060

  committed
  207d99d

  Copy full SHA for 207d99d

• avoid reevaluating params for each filter check

  

  ayushaga14

  authored and

  ag060

  committed
  d22aab8

  Copy full SHA for d22aab8

• using proto message envelope for kafka transport in threat detection client (#1874)

  ag060

  committed
  b013f83

  Copy full SHA for b013f83

• setting redis ttl while sync to redis instead of increment op

  ag060

  committed
  1ced701

  Copy full SHA for 1ced701

• fixed merge conflicts in staging and prod workflows

  ag060

  committed
  48056b4

  Copy full SHA for 48056b4

• proto changes

  

  ayushaga14

  authored and

  ag060

  committed
  395c550

  Copy full SHA for 395c550

• remove logs and headers loop

  

  ayushaga14

  authored and

  ag060

  committed
  54c6921

  Copy full SHA for 54c6921

• optimizations

  

  notshivansh

  authored and

  ag060

  committed
  7577487

  Copy full SHA for 7577487

• move httpresponse param proto generated files

  

  ayushaga14

  authored and

  ag060

  committed
  6b678ae

  Copy full SHA for 6b678ae

• fixed bad merges

  ag060

  committed
  5091e59

  Copy full SHA for 5091e59

• refactored threat detection backend

  ag060

  committed
  b48ad81

  Copy full SHA for b48ad81

• maxmind integration (#1924)

  ag060

  committed
  282eec2

  Copy full SHA for 282eec2

• removed System.out.println

  ag060

  committed
  a804dd2

  Copy full SHA for a804dd2

• removed unusued proto messages

  ag060

  committed
  90c33f4

  Copy full SHA for 90c33f4

• reading kafka key and value serializer from config

  ag060

  committed
  0999f81

  Copy full SHA for 0999f81

• updated custom generated httpresponse params files

  ag060

  committed
  83f32a5

  Copy full SHA for 83f32a5

• save and parse agg rules

  

  ayushaga14

  authored and

  ag060

  committed
  41f4e21

  Copy full SHA for 41f4e21

• using local redis instead of centralized redis (#1979)

  Show description for 6dbc96f
  ag060

  committed
  6dbc96f

  Copy full SHA for 6dbc96f

• updated dockerfile for threat detection client

  ag060

  committed
  ab38baa

  Copy full SHA for ab38baa

• fixed redis longvalue codec value decoder

  ag060

  committed
  eb41974

  Copy full SHA for eb41974

• reading category and subCategory from info

  ag060

  committed
  0fed5d5

  Copy full SHA for 0fed5d5

• using proper display names for threat detection filter category

  ag060

  committed
  f193fd3

  Copy full SHA for f193fd3

• reverting back to redis backed cache

  ag060

  committed
  49ebc58

  Copy full SHA for 49ebc58