Feature/SubmissionProfiles (master)

README.md

@@ -1,31 +1,124 @@
+[![Discord](https://img.shields.io/badge/chat-on%20discord-7289da.svg?sanitize=true)](https://discord.gg/GUAy9wErNu)
+[![](https://img.shields.io/discord/908084610158714900)](https://discord.gg/GUAy9wErNu)
+[![Static Badge](https://img.shields.io/badge/github-assemblyline-blue?logo=github)](https://github.com/CybercentreCanada/assemblyline)
+[![Static Badge](https://img.shields.io/badge/github-assemblyline--ui-blue?logo=github)](https://github.com/CybercentreCanada/assemblyline-ui)
+[![GitHub Issues or Pull Requests by label](https://img.shields.io/github/issues/CybercentreCanada/assemblyline/ui)](https://github.com/CybercentreCanada/assemblyline/issues?q=is:issue+is:open+label:ui)
+[![License](https://img.shields.io/github/license/CybercentreCanada/assemblyline-ui)](./LICENCE.md)
+
 # Assemblyline 4 - API and Socket IO server
 
 This component provides the User Interface as well as the different APIs and socketio endpoints for the Assemblyline 4 framework.
 
-### Components
+## Image variants and tags
+
+| **Tag Type** | **Description**                                                                                  |      **Example Tag**       |
+| :----------: | :----------------------------------------------------------------------------------------------- | :------------------------: |
+|    latest    | The most recent build (can be unstable).                                                         |          `latest`          |
+|  build_type  | The type of build used. `dev` is the latest unstable build. `stable` is the latest stable build. |     `stable` or `dev`      |
+|    series    | Complete build details, including version and build type: `version.buildType`.                   | `4.5.stable`, `4.5.1.dev3` |
 
-#### APIs
+## Components
 
-Assemblyline 4 provides a large set of API that can provide you with all the same information you will find in it's UI and even more. The list of APIs and their functionality is described in the help section of the UI.  
+### APIs
+
+Assemblyline 4 provides a large set of API that can provide you with all the same information you will find in it's UI and even more. The list of APIs and their functionality is described in the help section of the UI.
 
 All APIs in Assemblyline output their result in the same manner for consistency:
 
-    {
-       "api_response": {},            //Actual response from the API
-       "api_error_message": "",       //Error message if it is an error response
-       "api_server_version": "4.0.0"  //Assemblyline version and version of the different component
-       "api_status_code": 200         //Status code of the response
-    }
+```json
+{
+  "api_response": {},             //Actual response from the API
+  "api_error_message": "",        //Error message if it is an error response
+  "api_server_version": "4.0.0",  //Assemblyline version and version of the different component
+  "api_status_code": 200          //Status code of the response
+}
+```
+
+**NOTE**: All response codes return this output layout
 
- **NOTE**: All response codes return this output layout
+#### Running this component
 
-#### SocketIO endpoints
+```bash
+docker run --name ui cccs/assemblyline-service-ui
+```
+
+### SocketIO endpoints
 
 Assemblyline 4 also provide a list of SocketIO endpoints to get information about the system live. The endpoints will provide authenticated access to many Redis broadcast queues. It is a way for the system to notify user of changes and health of the system without having them to query for that information.
 
 The following queues can be listen on:
 
-* Alerts created
-* Submissions ingested
-* Health of the system
-* State of a given running submission
+- Alerts created
+- Submissions ingested
+- Health of the system
+- State of a given running submission
+
+#### Running this component
+
+```bash
+docker run --name socketio cccs/assemblyline-service-socketio
+```
+
+## Documentation
+
+For more information about this Assemblyline component, follow this [overview](https://cybercentrecanada.github.io/assemblyline4_docs/overview/architecture/) of the system's architecture.
+
+---
+
+# Assemblyline 4 - API et serveur Socket IO
+
+Ce composant fournit l'interface utilisateur ainsi que les différentes API et les points de terminaison Socket IO pour le framework Assemblyline 4.
+
+## Variantes et étiquettes d'image
+
+| **Type d'étiquette** | **Description**                                                                                                                    |  **Exemple d'étiquette**   |
+| :------------------: | :--------------------------------------------------------------------------------------------------------------------------------- | :------------------------: |
+|       dernière       | La version la plus récente (peut être instable).                                                                                   |          `latest`          |
+|      build_type      | Le type de compilation utilisé. `dev` est la dernière version instable. `stable` est la dernière version stable. `stable` ou `dev` |     `stable` ou `dev`      |
+|        séries        | Le détail de compilation utilisé, incluant la version et le type de compilation : `version.buildType`.                                               | `4.5.stable`, `4.5.1.dev3` |
+
+## Composants
+
+### APIs
+
+Assemblyline 4 fournit un grand nombre d'API qui peuvent vous fournir toutes les informations que vous trouverez dans l'interface utilisateur et même plus. La liste des API et de leurs fonctionnalités est décrite dans la section d'aide de l'interface utilisateur.
+
+Pour des raisons de cohérence, toutes les API d'Assemblyline produisent leurs résultats de la même manière :
+
+```json
+{
+  "api_response": {},             //Réponse réelle de l'API
+  "api_error_message": "",        //Message d'erreur s'il s'agit d'une réponse d'erreur
+  "api_server_version": "4.0.0",  //Assemblyline version et version des différents composants
+  "api_status_code": 200          //Code d'état de la réponse
+}
+```
+
+**NOTE** : Tous les codes de réponse renvoient cette présentation de sortie
+
+#### Exécuter ce composant
+
+```bash
+docker run --name ui cccs/assemblyline-service-ui
+```
+
+### Points d'extrémité SocketIO
+
+Assemblyline 4 fournit également une liste de points de contact SocketIO pour obtenir des informations sur le système en direct. Ces points de contact fournissent un accès authentifié à de nombreuses files d'attente de diffusion Redis. C'est un moyen utilisé par le système pour informer les utilisateurs des changements et de l'état du système sans qu'ils aient à faire des requêtes d'informations.
+
+Les files d'attente suivantes peuvent être écoutées :
+
+- Alertes créées
+- Soumissions reçues
+- Santé du système
+- État d'une soumission en cours
+
+#### Exécuter ce composant
+
+```bash
+docker run --name socketio cccs/assemblyline-service-socketio
+```
+
+## Documentation
+
+Pour plus d'informations sur ce composant Assemblyline, suivez ce [overview](https://cybercentrecanada.github.io/assemblyline4_docs/overview/architecture/) de l'architecture du système.

assemblyline_ui/api/base.py

@@ -13,7 +13,7 @@
 from assemblyline_ui.security.authenticator import BaseSecurityRenderer
 from assemblyline_ui.security.oauth_auth import validate_oauth_token
 from assemblyline_ui.config import LOGGER, QUOTA_TRACKER, STORAGE, SECRET_KEY, VERSION, CLASSIFICATION, \
-    DAILY_QUOTA_TRACKER
+    DAILY_QUOTA_TRACKER, AUDIT_LOG, AUDIT_LOGIN
 from assemblyline_ui.helper.user import login
 from assemblyline_ui.http_exceptions import AuthenticationException
 from assemblyline_ui.config import config
@@ -58,12 +58,14 @@ def auto_auth_check(self):
                     #       sub-sequent calls...
                     validated_user, roles_limit = validate_apikey(uname, apikey, STORAGE)
                 except AuthenticationException as ae:
-                    LOGGER.warning(f"Authentication failure. (U:{uname} - IP:{ip}) [{str(ae)}]")
+                    login_logger = AUDIT_LOG if AUDIT_LOGIN else LOGGER
+                    login_logger.warning(f"Authentication failure. (U:{uname} - IP:{ip}) [{str(ae)}]")
                     abort(401, str(ae))
                     return
 
                 if validated_user:
-                    LOGGER.info(f"Login successful. (U:{uname} - IP:{ip})")
+                    login_logger = AUDIT_LOG if AUDIT_LOGIN else LOGGER
+                    login_logger.info(f"Login successful. (U:{uname} - IP:{ip})")
 
                     return validated_user, roles_limit
 

assemblyline_ui/api/v4/authentication.py

@@ -32,6 +32,8 @@
     get_reset_queue,
     get_signup_queue,
     get_token_store,
+    AUDIT_LOG,
+    AUDIT_LOGIN
 )
 from assemblyline_ui.helper.oauth import fetch_avatar, parse_profile
 from assemblyline_ui.helper.user import API_PRIV_MAP, get_default_user_quotas, get_dynamic_classification
@@ -443,11 +445,13 @@ def login(**_):
             }, cookies={'XSRF-TOKEN': xsrf_token})
         except AuthenticationException as wpe:
             uname = auth.get('username', '(None)')
-            LOGGER.warning(f"Authentication failure. (U:{uname} - IP:{ip}) [{wpe}]")
+            login_logger = AUDIT_LOG if AUDIT_LOGIN else LOGGER
+            login_logger.warning(f"Authentication failure. (U:{uname} - IP:{ip}) [{wpe}]")
             return make_api_response("", err=str(wpe), status_code=401)
         finally:
             if logged_in_uname:
-                LOGGER.info(f"Login successful. (U:{logged_in_uname} - IP:{ip})")
+                login_logger = AUDIT_LOG if AUDIT_LOGIN else LOGGER
+                login_logger.info(f"Login successful. (U:{logged_in_uname} - IP:{ip})")
 
     return make_api_response("", "Not enough information to proceed with authentication", 401)
 

assemblyline_ui/api/v4/help.py

@@ -192,7 +192,7 @@ def get_systems_constants(**_):
 
     out = {
         "max_priority": constants.MAX_PRIORITY,
-        "priorities": constants.PRIORITIES,
+        "priorities": constants.PRIORITY_RANGES,
         "file_types": [[t,
                         sorted([x for x in accepts_map.keys()
                                 if re.match(accepts_map[x], t) and not re.match(rejects_map[x], t)])]

assemblyline_ui/api/v4/ingest.py

@@ -16,12 +16,14 @@
 from assemblyline.odm.messages.submission import Submission
 from assemblyline.odm.models.user import ROLES
 from assemblyline.remote.datatypes.queues.named import NamedQueue
+from assemblyline_core.ingester.constants import INGEST_QUEUE_NAME
 from assemblyline_ui.api.base import api_login, make_api_response, make_subapi_blueprint
 from assemblyline_ui.config import ARCHIVESTORE, CLASSIFICATION as Classification, IDENTIFY, TEMP_SUBMIT_DIR, \
-    STORAGE, config, FILESTORE, metadata_validator
+    STORAGE, config, FILESTORE, metadata_validator, LOGGER
 from assemblyline_ui.helper.service import ui_to_submission_params
 from assemblyline_ui.helper.submission import FileTooBigException, submission_received, refang_url, fetch_file, \
-    FETCH_METHODS
+    FETCH_METHODS, URL_GENERATORS, update_submission_parameters
+
 from assemblyline_ui.helper.user import check_async_submission_quota, decrement_submission_ingest_quota, \
     load_user_settings
 
@@ -31,7 +33,7 @@
 ingest_api._doc = "Ingest files for large volume processing"
 
 ingest = NamedQueue(
-    "m-ingest",
+    INGEST_QUEUE_NAME,
     host=config.core.redis.persistent.host,
     port=config.core.redis.persistent.port)
 MAX_SIZE = config.submission.max_file_size
@@ -150,15 +152,17 @@ def ingest_single_file(**kwargs):
       "base64": "<BINARY DATA OF THE FILE TO SCAN... ENCODED AS BASE64 STRING>",
 
       // OPTIONAL VALUES
-      "name": "file.exe",         # Name of the file to scan otherwise the sha256 or base file of the url
+      "name": "file.exe",                   # Name of the file to scan otherwise the sha256 or base file of the url
+
+      "submission_profile": "Static Analysis",    # Name of submission profile to use
 
-      "metadata": {               # Submission metadata
-        "key": val,                 # Key/Value pair for metadata parameters
+      "metadata": {                         # Submission metadata
+        "key": val,                             # Key/Value pair for metadata parameters
       },
 
-      "params": {                 # Submission parameters
-        "key": val,                 # Key/Value pair for params that differ from the user's defaults
-      },                            # Default params can be fetch at /api/v3/user/submission_params/<user>/
+      "params": {                           # Submission parameters
+        "key": val,                             # Key/Value pair for params that differ from the user's defaults
+      },                                        # Default params can be fetch at /api/v3/user/submission_params/<user>/
 
       "generate_alert": False,        # Generate an alert in our alerting system or not
       "notification_queue": None,     # Name of the notification queue
@@ -214,7 +218,7 @@ def ingest_single_file(**kwargs):
                     break
 
             hash = string_value
-            if string_type == "url":
+            if string_type in URL_GENERATORS:
                 string_value = refang_url(string_value)
                 name = string_value
             else:
@@ -227,7 +231,7 @@ def ingest_single_file(**kwargs):
             return make_api_response({}, "Invalid content type", 400)
 
         # Get default description
-        default_description = f"Inspection of {'URL' if string_type == 'url' else 'file'}: {name}"
+        default_description = f"Inspection of {'URL' if string_type in URL_GENERATORS else 'file'}: {name}"
 
         # Get file name
         if not name:
@@ -269,8 +273,11 @@ def ingest_single_file(**kwargs):
             "type": "INGEST"
         })
 
-        # Apply provided params
-        s_params.update(data.get("params", {}))
+        # Update submission parameters as specified by the user
+        try:
+            update_submission_parameters(s_params, data, user)
+        except Exception as e:
+            return make_api_response({}, str(e), 400)
 
         # Use the `default_external_sources` if specified as a param in request otherwise default to user's settings
         default_external_sources = s_params.pop('default_external_sources', []) or default_external_sources
@@ -287,8 +294,9 @@ def ingest_single_file(**kwargs):
                     if not found:
                         raise FileNotFoundError(
                             f"{string_type.upper()} does not exist in Assemblyline or any of the selected sources")
-                except FileTooBigException:
-                    return make_api_response({}, "File too big to be scanned.", 400)
+                except FileTooBigException as e:
+                    LOGGER.warning(f"[{user['uname']}] {e}")
+                    return make_api_response({}, str(e), 400)
                 except FileNotFoundError as e:
                     return make_api_response({}, str(e), 404)
                 except PermissionError as e:

assemblyline_ui/api/v4/search.py

@@ -281,6 +281,7 @@ def facet(index, field, **kwargs):
     timeout        =>   Maximum execution time (ms)
     use_archive    =>   Allow access to the malware archive (Default: False)
     archive_only   =>   Only access the Malware archive (Default: False)
+    size           =>   How many facets are returned? (Default: 10)
 
     Data Block (POST ONLY):
     {"query": "id:*",
@@ -307,6 +308,7 @@ def facet(index, field, **kwargs):
     fields = ["query", "mincount", "timeout"]
     multi_fields = ['filters']
     boolean_fields = ['use_archive', 'archive_only']
+    integer_fields = ["size"]
 
     if request.method == "POST":
         req_data = request.json
@@ -322,6 +324,10 @@ def facet(index, field, **kwargs):
                    for k in boolean_fields
                    if req_data.get(k, None) is not None})
 
+    params.update({k: int(req_data.get(k, '0'))
+                for k in integer_fields
+                if req_data.get(k, None) is not None})
+
     use_archive = params.pop('use_archive', False)
     archive_only = params.pop('archive_only', False)
     if archive_only:

assemblyline_ui/api/v4/service.py

@@ -614,6 +614,12 @@ def get_service(servicename, **_):
     version = request.args.get('version', None)
 
     service = STORAGE.get_service_with_delta(servicename, version=version, as_obj=False)
+    if 'update_config' in service:
+        # Fill in defaults for update sources based on update_config
+        for s in service['update_config']['sources']:
+            # Update update_interval to default to globally configured value by updater
+            if 'update_interval' not in s:
+                s['update_interval'] = service['update_config']['update_interval_seconds']
     if service:
         # Ensure service classification is set in response
         service['classification'] = service.get('classification', Classification.UNRESTRICTED)