WIP: New frontend

.github/workflows/ci.yml

@@ -17,6 +17,42 @@ env:
   CACHE_NUMBER: 1
 
 jobs:
+  unit-test-frontend2:
+    name: Frontend2 linting / unit tests
+    # Defines the type of runner the job runs on
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout to the repository
+        uses: actions/checkout@v3
+      - name: Set up NodeJS environment
+        uses: actions/setup-node@v3
+        with:
+          node-version: '18.11.0'
+      # Consider this as an add on to optimize the execution of actions
+      - name: Cache node modules
+        id: cache-npm
+        uses: actions/cache@v3
+        env:
+          cache-name: cache-node-modules
+        with:
+          # npm cache files are stored in `~/.npm` on Linux/macOS
+          path: ~/.npm
+          key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
+          restore-keys: |
+            ${{ runner.os }}-build-${{ env.cache-name }}-
+            ${{ runner.os }}-build-
+            ${{ runner.os }}-
+      - name: Install package dependencies
+        run: npm install
+        working-directory: ./frontend2
+      - name: Check linting and formatting
+        # Custom script for checking the linting and formatting being in place
+        run: npm run lint
+        working-directory: ./frontend2
+      # Run test cases and this could ensure minimum coverage as well if set
+      - name: Execute test cases
+        run: npm run test
+        working-directory: ./frontend2
   lint:
     name: Linter (pre-commit)
     runs-on: ubuntu-latest
@@ -40,8 +76,8 @@ jobs:
         run: |
           pre-commit install
           pre-commit run -a
-  unit-test:
-    name: Unit tests (django)
+  unit-test-backend:
+    name: Backend unit tests
     runs-on: ubuntu-latest
     permissions:
       contents: read

.pre-commit-config.yaml

@@ -4,16 +4,20 @@ repos:
     hooks:
       - id: check-merge-conflict
       - id: end-of-file-fixer
+        exclude: ^frontend2/src/api/_autogen
       - id: mixed-line-ending
         args: ["--fix=lf"]
+        exclude: ^frontend2/src/api/_autogen
       - id: trailing-whitespace
+        exclude: ^frontend2/src/api/_autogen
 
+  # This is only used for the old frontend. The prettier check for frontend2 exists in ci.yml
   - repo: https://github.com/pre-commit/mirrors-prettier
     rev: v2.7.1
     hooks:
       - id: prettier
         types_or: [css, html, javascript, json, jsx, markdown]
-        exclude: ^frontend/public/assets
+        exclude: ^(frontend/public/assets|frontend2)
 
   - repo: https://github.com/dnephin/pre-commit-golang
     rev: v0.5.1

README.md

@@ -5,7 +5,7 @@
 Galaxy is the framework that powers the infrastructure for MIT Battlecode.
 The galaxy consists of three main parts:
 
-- Siarnaq, the competitor dashboard that interfaces with our data storage.
+- Siarnaq, the competitor dashboard that interfaces with our data storage. (Siarnaq, in turn, has two parts: a frontend and a backend.)
 - Saturn, the compute cluster that compiles competitor bots and runs matches.
 - Titan, the malware scanner that scans all file uploads.
 
@@ -23,19 +23,10 @@ Please follow these steps carefully to ensure your development environment is in
 
 If the Conda specifications are updated upstream, you can refresh your local environment to match it by running `conda env update -n galaxy -f environment-dev.yml`.
 
-## Development workflow
-
-For specific development workflows in each module, see the README files in the respective folder.
-
-Develop new features on branches and use pull-requests to request code-review.
-The `main` branch is protected and pushes to `main` will be rejected.
+Afterwards, to run systems locally, see each of their respective directory's readme files.
 
-We will be using the [Projects](https://github.com/battlecode/galaxy/projects?type=classic) feature to track our todo list.
-Entries in the "To do" column are allowed to simply be items created in the project board, but once the entry moves to one of the other columns (e.g. "In progress"), please convert the entry into an issue for easier discussion and reference in PRs.
-
-To start a feature, _claim_ it on the Projects page by moving it to the "In Progress" column and adding your name to it. Then you can work on it on a git branch.
+## Development workflow
 
-In places where it makes sense, it could be good to write test cases, although a lot of the functionality might not be very testable.
+See [docs-general/workflow.md](docs-general/workflow.md) for workflow information. **Reading this page, especially the "Coding" and "Review" sections, before starting work is highly recommended.**
 
-GitHub has been configured to automatically verify all Pre-commit checks and in the future will also run all test cases.
-It's not required to make sure every commit on GitHub passes, but anything merged to main should pass. GitHub will send you an email if the checks fail.
+For specific development workflows in each module, see the README files in the respective folder.

backend/siarnaq/api/compete/serializers.py

@@ -236,9 +236,9 @@ def to_representation(self, instance):
 
 
 class MatchSerializer(serializers.ModelSerializer):
-    participants = MatchParticipantSerializer(many=True)
-    maps = serializers.SerializerMethodField()
-    replay_url = serializers.SerializerMethodField()
+    participants = MatchParticipantSerializer(many=True, allow_null=True)
+    maps = serializers.SerializerMethodField(allow_null=True)
+    replay_url = serializers.SerializerMethodField(allow_null=True)
     tournament_round = TournamentRoundSerializer(required=False, allow_null=True)
 
     class Meta:
@@ -469,3 +469,7 @@ def save(self, *args, **kwargs):
 class HistoricalRatingSerializer(serializers.Serializer):
     rating = RatingField()
     timestamp = serializers.DateTimeField()
+
+
+class EmptySerializer(serializers.Serializer):
+    pass

backend/siarnaq/api/compete/views.py

@@ -24,6 +24,7 @@
 )
 from siarnaq.api.compete.permissions import HasTeamSubmission
 from siarnaq.api.compete.serializers import (
+    EmptySerializer,
     HistoricalRatingSerializer,
     MatchReportSerializer,
     MatchSerializer,
@@ -108,6 +109,18 @@ def get_queryset(self):
             .order_by("-pk")
         )
 
+    @extend_schema(
+        request={
+            "multipart/form-data": {
+                "type": "object",
+                "properties": {
+                    "package": {"type": "string"},
+                    "description": {"type": "string"},
+                    "source_code": {"type": "string", "format": "binary"},
+                },
+            }
+        },
+    )
     def create(self, request, *, episode_id):
         """
         Create a new submission. This operation creates a submission record in the
@@ -149,8 +162,16 @@ def create(self, request, *, episode_id):
             headers=headers,
         )
 
+    @extend_schema(
+        # https://drf-spectacular.readthedocs.io/en/latest/faq.html#i-m-using-action-detail-false-but-the-response-schema-is-not-a-list
+        responses=TournamentSubmissionSerializer(many=True)
+    )
     @action(
-        detail=False, methods=["get"], serializer_class=TournamentSubmissionSerializer
+        detail=False,
+        methods=["get"],
+        serializer_class=TournamentSubmissionSerializer,
+        # needed so that the generated schema is not paginated
+        pagination_class=None,
     )
     def tournament(self, request, *, episode_id):
         """Retrieve the submissions used in tournaments by the current team.."""
@@ -447,7 +468,7 @@ def report(self, request, pk=None, *, episode_id):
         detail=True,
         methods=["post"],
         permission_classes=(IsAdminUser,),
-        serializer_class=None,
+        serializer_class=EmptySerializer,
         throttle_classes=(),
     )
     def rating_update(self, request, pk=None, *, episode_id):
@@ -462,7 +483,7 @@ def rating_update(self, request, pk=None, *, episode_id):
         detail=True,
         methods=["post"],
         permission_classes=(IsAdminUser,),
-        serializer_class=None,
+        serializer_class=EmptySerializer,
         throttle_classes=(),
     )
     def publish_public_bracket(self, request, pk=None, *, episode_id):

backend/siarnaq/api/episodes/serializers.py

@@ -1,5 +1,5 @@
 from drf_spectacular.types import OpenApiTypes
-from drf_spectacular.utils import extend_schema_field
+from drf_spectacular.utils import extend_schema_field, extend_schema_serializer
 from rest_framework import serializers
 
 from siarnaq.api.episodes.models import (
@@ -48,6 +48,10 @@ def get_frozen(self, obj):
         return obj.frozen()
 
 
+@extend_schema_serializer(
+    # workaround for https://github.com/OpenAPITools/openapi-generator/issues/9289
+    component_name="GameMap"
+)
 class MapSerializer(serializers.ModelSerializer):
     class Meta:
         model = Map

backend/siarnaq/api/teams/serializers.py

@@ -186,3 +186,7 @@ class UserPassedSerializer(serializers.Serializer):
 
 class TeamReportSerializer(serializers.Serializer):
     report = serializers.FileField(write_only=True)
+
+
+class TeamLeaveSerializer(serializers.Serializer):
+    pass

backend/siarnaq/api/teams/views.py

@@ -22,6 +22,7 @@
     TeamAvatarSerializer,
     TeamCreateSerializer,
     TeamJoinSerializer,
+    TeamLeaveSerializer,
     TeamPrivateSerializer,
     TeamPublicSerializer,
     TeamReportSerializer,
@@ -114,7 +115,7 @@ def me(self, request, *, episode_id):
     @action(
         detail=False,
         methods=["post"],
-        serializer_class=None,
+        serializer_class=TeamLeaveSerializer,
         permission_classes=(IsAuthenticated, IsEpisodeAvailable),
     )
     def leave(self, request, *, episode_id):

backend/siarnaq/api/user/views.py

@@ -75,6 +75,14 @@ def me(self, request):
                 serializer.save()
                 return Response(serializer.data)
 
+    @extend_schema(
+        responses={
+            status.HTTP_200_OK: {
+                "type": "object",
+                "additionalProperties": {"$ref": "#/components/schemas/TeamPublic"},
+            }
+        }
+    )
     @action(
         detail=True,
         permission_classes=(AllowAny,),

backend/siarnaq/settings.py

@@ -510,3 +510,18 @@ def dropper(logger, method_name, event_dict):
         raise structlog.DropEvent
 
     structlog.configure(processors=[dropper])
+
+SPECTACULAR_SETTINGS = {
+    # Split components into request and response parts where appropriate
+    # This setting is highly recommended to achieve the most accurate API
+    # description, however it comes at the cost of having more components.
+    "COMPONENT_SPLIT_REQUEST": True,
+    # Controls which authentication methods are exposed in the schema. If not None,
+    # will hide authentication classes that are not contained in the whitelist.
+    # Use full import paths like
+    # ['rest_framework.authentication.TokenAuthentication', ...].
+    # Empty list ([]) will hide all authentication methods. The default None shows all.
+    "AUTHENTICATION_WHITELIST": [
+        "rest_framework_simplejwt.authentication.JWTAuthentication"
+    ],
+}

docs-general/emails.md

@@ -0,0 +1,21 @@
+# Emails
+
+## Mailing Lists
+
+Battlecode maintains various email lists of competitors each year, as well as a general interest list. These lists are generally administered by presidents or directors of operation (although webinfra devs may certainly help with these tasks). More information and instructions about these lists can be found in our Google Drive.
+
+However, sets of competitor emails are derived from our backend, and so it is webinfra's job to produce these. To do this:
+
+- Go to the admin panel, go to the `users` page, then click `export as csv`
+- Then, format this as needed with your favorite csv editor (even Google Sheets is fine).
+- If you are formatting for the `mmblanche` tool, you can also the automated script that is coming soon as tracked in [https://github.com/battlecode/galaxy/issues/500](#500).
+
+## Backend Automated Emails
+
+Sometimes, our backend itself needs to send out emails, for example for the automated password reset process.
+
+We use [https://www.mailjet.com/](Mailjet), an emailing service especially catered for developers. Our account can be managed on their [https://app.mailjet.com/dashboard](dashboard).
+
+Sometimes, Mailjet emails will stop sending. The most likely cause has been that we have reached our account's email limit. Mailjet's free plan (which we usually stay on, especially not during competition season) only allows us 200 emails per day, which we can blow by during IAP.
+
+If emails stop sending, consider subscribing to the lowest-usage paid plan that Mailjet offers. **Don't forget to unsubscribe at the end of the month / during the beginning of the next!!** (It's possible that when we subscribe, Mailjet bills us for a full month -- meaning you can instantly click the unsubscribe button, have the premium subscrition for the rest of the month, and have the free plan after without having to remember to change it down the road. If this is possible, do so!) If necessary, writing a reminder is helpful.

docs-general/onboard.md

@@ -0,0 +1,57 @@
+# General onboarding
+
+(See frontend- and backend- specific folders for less specific materials.)
+
+## Get some experience!
+
+First and foremost, the easiest way to get a handle on our codebase is to use everything. Go to the deployed production website, make an account, join a team, upload a bot, and start playing.
+
+It similarly helps to run as much of the Battlecode stack as you can, especially the backend and frontend. Follow the steps in the top-level readme to set up conda, and then see each of the backend and frontend's readmes as well.
+
+## Goals of the product
+
+Allow a user to:
+
+- scrimmage
+- submit code
+- view Battlecode information
+- manage their team and user
+
+The latter two are fairly straightforward; the first two are more complex, and involve our entire system.
+
+## Webinfra system overview
+
+This is a rough depiction of how webinfra code runs in practice / in deployment. All of the systems in use are run through the Google Cloud Program (GCP).
+
+![System Diagram](system-diagram.png)
+
+## The big stuff
+
+Here, we introduce the systems through tracking the flow of information.
+
+First, a competitor uses their own computer to load up `play.battlecode.org`, thus visiting the _frontend_. Through the competitor's interactions, the frontend makes Internet requests to the _backend_. The backend allows these requests via the _API_, an interface that the backend defines to communicate with the outside world. Finally, the backend stores and retrieves bulk amounts of data in the _database_.
+
+For viewing general information or managing team/user data, the flow of data generally ends here. However, when a user wants to submit new source code for their bot, or request a scrimmage against another team... these are more involved.
+
+Most importantly, in these cases, the backend puts info about this request onto a _Pub/Sub_, which works very similarly to a queue. (We use two of these, one for submitting code and the other for requesting scrimmages.)
+Furthermore, the backend also stores other information into the database for records. Also, for a submission, the backend saves the competitor's source code in _Google Cloud Storage_. (This is Google's simple file storage, which works similarly to Google Drive or other file-and-folder based stores.)
+
+This request is then taken off the respective queue and processed by either a _compile runner_ (also called a compile server or submission server) or a _execute runner_ (also called an execute server or scrimmage server). A compile runner takes in the request, gets the competitor's source code from Cloud Storage, compiles this code through running Java commands, uploads the compiled code back to Cloud Storage, and tells the backend to update the database accordingly (to reflect the completion of compilation). An execute runner follows a similar process, though with matches rather than submissions.
+
+### Other parts
+
+While these parts need less active work and maintenance from devs, they are still important to know.
+
+_Titan_ is our integrated malware scanner. At the end of competition, competitors upload files ostensibly containing their resumes to our site, which are compiled into a resume book given to sponsors. To prevent funny business, Titan intercepts and scans these files, and marks safe ones.
+
+The _Secret Manager_ is a Google product, that holds our internal secrets, such as our password to manage the database. This prevents us from having to put passwords and authentication tokens in our code (ask older devs for horror stories). Instead, deployed components access Secret Manager to get the passwords, and can inject these into the flow of code.
+
+The _Artifact Registry_ is a Google product that holds and organizes Docker containers. We use containers to standardize and deploy our backend servers and our compile/execute runners. The number of total servers running at any time scales up and down depending on how much we need, through processes we have defined ourselves.
+
+(Docker is an amazing service that enables standardized servers to run across different operating systems. You can learn more at [their page](https://www.docker.com).)
+
+Google's _Cloud Build_ turns our source code into real machines running that code, and allows us to manage and customize this process as well.
+
+An extra layer of abstraction is created with the _Load Balancer_. This takes all HTTP requests to our domain, and points each request to the right place in our system.
+
+Furthermore, many of these systems can have multiple instances running. For example, GCP can run multiple backend servers, in order to do more operations at the same rate, thus handling more network traffic and competitor usage. This scalability is enabled by the Load Balancer, which can distribute requests across these multiple instances.

docs-general/operations.md

@@ -0,0 +1,21 @@
+# Operations
+
+## Get dev permissions
+
+TODO, tracked in #681
+
+## Add a dev to any team
+
+TODO, tracked in #681
+
+This enables lots of cool technical support things. Most notably, **this is the (currently) "unofficial-official" way to view and download a team's submissions and replays**, which is helpful in debugging.
+
+## Resetting people's passwords
+
+Sometimes, password resets don't go through, since users never get the emails, due to issues with our email backend. No email backend will ever be failproof, so here are some other methods:
+
+- Have users switch to a different email address, with which that they can receive emails from our backend. You can make this switch in the admin panel. Then, have them request a new password through the normal means.
+
+## Administering Backend Emails through Mailjet
+
+If automated emails from the backend server stop sending, or if you have other issues with these emails, see `emails.md` in this folder. Feel free to add any new notes there too!