Merge remote-tracking branch 'origin/master' into updating-tron

* origin/master: (44 commits)
  Released 3.2.8 via make release
  Released 3.2.7 via make release
  Handling timezone aware jobs
  Released 3.2.6 via make release
  Add default behaviour for jobs that dont have some keys and handle the non-existent of some json_vals (#1017)
  Released 3.2.5 via make release
  Rename run_time_str to be more good and less bad
  Use localize instead of replace when writing tz info
  Fix loop boundary on getting partitions
  Released 3.2.4 via make release
  Add check for empty val. Merge json correctly
  Released 3.2.3 via make release
  Replce --dry-run with --execute so that we default to the safer option
  Add sub_parsers and restructure args a bit
  Released 3.2.2 via make release
  Released 3.2.1 via make release
  Pass json_val back to the save queue on failure
  Cleanup tron scribereader (#1012)
  Released 3.2.0 via make release
  Add a separation for failed-keys so that we can investigate failures outside of deprecated keys
  ...

.github/workflows/ci.yml

@@ -51,7 +51,7 @@ jobs:
       # to bring those in
       - run: sudo apt-get install --quiet --assume-yes devscripts
       - run: make itest_${{ matrix.dist }}
-      - uses: actions/upload-artifact@v3
+      - uses: actions/upload-artifact@v4
         with:
           name: deb-${{ matrix.dist }}
           path: dist/tron_*.deb
@@ -61,11 +61,11 @@ jobs:
     steps:
       - uses: actions/checkout@v2
       - run: mkdir -p dist/
-      - uses: actions/download-artifact@v3
+      - uses: actions/download-artifact@v4
         with:
           name: deb-bionic
           path: dist/
-      - uses: actions/download-artifact@v3
+      - uses: actions/download-artifact@v4
         with:
           name: deb-jammy
           path: dist/

CODEOWNERS

@@ -0,0 +1,5 @@
+# NOTE: "we" in this file will refer to the Compute Infrastructure team at Yelp
+* @Yelp/paasta
+#
+# prevent cheeky modifications :)
+CODEOWNERS @Yelp/paasta

Makefile

@@ -1,5 +1,5 @@
 # Edit this release and run "make release"
-RELEASE=2.8.1
+RELEASE=3.2.8
 
 SHELL=/bin/bash
 

debian/changelog

@@ -1,3 +1,126 @@
+tron (3.2.8) jammy; urgency=medium
+
+  * 3.2.8 tagged with 'make release'
+    Commit: Merge pull request #1025 from Yelp/u/kkasp/DAR-2558-fix-
+    overstep  Fix loop boundary on getting partitions
+
+ -- Kevin Kaspari <kkasp@yelp.com>  Thu, 30 Jan 2025 11:59:51 -0800
+
+tron (3.2.7) jammy; urgency=medium
+
+  * 3.2.7 tagged with 'make release'
+    Commit: Merge pull request #1027 from Yelp/u/emanelsabban/TRON-2354
+    Handling timezone aware jobs
+
+ -- Eman Elsabban <emanelsabban@yelp.com>  Fri, 24 Jan 2025 12:38:21 -0800
+
+tron (3.2.6) jammy; urgency=medium
+
+  * 3.2.6 tagged with 'make release'
+    Commit: Add default behaviour for jobs that dont have some keys and
+    handle the non-existent of some json_vals (#1017)  * Add default
+    behaviour for jobs that dont have some keys and handle the non-
+    existent of some json_vals  * Remove get from command  * cap add and
+    drop should have values  * removing the try/except since it was
+    added in a separate pr  * maybe not all gets are necessary  * Adding
+    comments
+
+ -- Eman Elsabban <emanelsabban@yelp.com>  Fri, 17 Jan 2025 11:47:48 -0800
+
+tron (3.2.5) jammy; urgency=medium
+
+  * 3.2.5 tagged with 'make release'
+    Commit: Merge pull request #1026 from Yelp/u/kkasp/fix-timezones
+    Use localize instead of replace when writing tz info
+
+ -- Kevin Kaspari <kkasp@yelp.com>  Fri, 17 Jan 2025 11:00:55 -0800
+
+tron (3.2.4) jammy; urgency=medium
+
+  * 3.2.4 tagged with 'make release'
+    Commit: Merge pull request #1024 from Yelp/u/kkasp/empty-val  Add
+    check for empty val. Merge json correctly
+
+ -- Kevin Kaspari <kkasp@yelp.com>  Thu, 16 Jan 2025 08:49:56 -0800
+
+tron (3.2.3) jammy; urgency=medium
+
+  * 3.2.3 tagged with 'make release'
+    Commit: Merge pull request #1016 from Yelp/u/kkasp/TRON-2239-migrate-
+    pickle-only-state-to-JSON  U/kkasp/tron 2239 migrate pickle only
+    state to json
+
+ -- Kevin Kaspari <kkasp@yelp.com>  Tue, 14 Jan 2025 12:05:21 -0800
+
+tron (3.2.2) jammy; urgency=medium
+
+  * 3.2.2 tagged with 'make release'
+    Commit: Merge pull request #1014 from Yelp/u/kkasp/fix-tooltip-lol-
+    its-bootstrap-2.3.1  Disable the tooltip animation that is causing a
+    strange problem where you can"t get a tooltip when entering from the
+    top of an object
+
+ -- Kevin Kaspari <kkasp@yelp.com>  Mon, 13 Jan 2025 06:59:56 -0800
+
+tron (3.2.1) jammy; urgency=medium
+
+  * 3.2.1 tagged with 'make release'
+    Commit: Merge pull request #1022 from Yelp/u/kkasp/pass-json-back-to-
+    save-queue  Pass json_val back to the save queue on failure
+
+ -- Kevin Kaspari <kkasp@yelp.com>  Fri, 10 Jan 2025 07:27:17 -0800
+
+tron (3.2.0) jammy; urgency=medium
+
+  * 3.2.0 tagged with 'make release'
+    Commit: Merge pull request #1018 from Yelp/u/kkasp/update-delete-
+    item  Update delete_item logic to handle json partitions
+
+ -- Kevin Kaspari <kkasp@10-40-19-131-uswest1cdevc.dev.yelpcorp.com>  Thu, 09 Jan 2025 12:29:29 -0800
+
+tron (3.1.2) jammy; urgency=medium
+
+  * 3.1.2 tagged with 'make release'
+    Commit: Bump task_processing to v1.3.1 for capability fix (#1020)
+    Just like Yelp/paasta#3972 and Yelp/paasta#3973, we need to ensure
+    that there are no duplicates between cap_add and cap_drop -
+    otherwise, the cap_drop entry will "win" and the duplicate
+    capability will not be added.
+
+ -- Luis Perez <luisp@yelp.com>  Thu, 09 Jan 2025 09:21:33 -0800
+
+tron (3.1.1) jammy; urgency=medium
+
+  * 3.1.1 tagged with 'make release'
+    Commit: Merge pull request #1019 from Yelp/u/emanelsabban/capture-
+    json_val  Capturing exception when json_val doesnt exist
+
+ -- Eman Elsabban <emanelsabban@yelp.com>  Wed, 08 Jan 2025 13:04:07 -0800
+
+tron (3.1.0) jammy; urgency=medium
+
+  * 3.1.0 tagged with 'make release'
+    Commit: Add read_json to be a part of master.yaml config (#1015)  *
+    Add read_json to be a part of master.yaml config  * Fix the mypy
+    error & other failed tests  * Addressing reviews  * Add reas_json to
+    restore in shelve function
+
+ -- Eman Elsabban <emanelsabban@yelp.com>  Thu, 02 Jan 2025 12:51:55 -0800
+
+tron (3.0.0) jammy; urgency=medium
+
+  * 3.0.0 tagged with 'make release'
+    Commit: [TRON-2238] Reading from JSON to restore jobs" state instead
+    of pickles (#1010)  * Reading from Json to restore jobs" state
+    instead of pickles  * Fixing some bugs through testing  * Deleting
+    some comments  * mocking get_config_watcher  * Try mocking one more
+    time  * Toggling off read_json since we want to merge that way  *
+    Removing some comments  * Addressed reviews  * Toggling read_json
+    back off  * Addressing more reviews  * one more review plz  *
+    includes reviews except tz
+
+ -- Eman Elsabban <emanelsabban@yelp.com>  Mon, 23 Dec 2024 07:56:01 -0800
+
 tron (2.8.1) jammy; urgency=medium
 
   * 2.8.1 tagged with 'make release'

dev/logging.conf

@@ -1,5 +1,5 @@
 [loggers]
-keys=root, twisted, tron, tron.serialize.runstate.statemanager, tron.api.www.access, task_processing, tron.mesos.task_output, pymesos, yelp_lib.clog.StreamTailer
+keys=root, twisted, tron, tron.serialize.runstate.statemanager, tron.api.www.access, task_processing, tron.mesos.task_output, pymesos
 
 [handlers]
 keys=stdoutHandler, accessHandler, nullHandler
@@ -64,12 +64,6 @@ class=logging.NullHandler
 level=DEBUG
 args=()
 
-[logger_yelp_lib.clog.StreamTailer]
-level=CRITICAL
-handlers=nullHandler
-qualname=yelp_lib.clog.StreamTailer
-propagate=0
-
 [handler_accessHandler]
 class=logging.StreamHandler
 level=DEBUG

docs/source/generated/tron.utils.logreader.rst

@@ -0,0 +1,7 @@
+tron.utils.logreader module
+===========================
+
+.. automodule:: tron.utils.logreader
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/generated/tron.utils.rst

@@ -10,11 +10,11 @@ Submodules
    tron.utils.collections
    tron.utils.crontab
    tron.utils.exitcode
+   tron.utils.logreader
    tron.utils.observer
    tron.utils.persistable
    tron.utils.proxy
    tron.utils.queue
-   tron.utils.scribereader
    tron.utils.state
    tron.utils.timeutils
    tron.utils.trontimespec

mypy.ini

@@ -13,8 +13,5 @@ warn_unused_ignores = True
 [mypy-clusterman_metrics.*]
 ignore_missing_imports = True
 
-[mypy-clog.*]
-ignore_missing_imports = True
-
-[mypy-scribereader.*]
+[mypy-logreader.*]
 ignore_missing_imports = True

requirements.txt

@@ -80,7 +80,7 @@ setuptools==65.5.1
 six==1.15.0
 sshpubkeys==3.1.0
 stack-data==0.6.2
-task-processing==1.3.0
+task-processing==1.3.1
 traitlets==5.0.0
 Twisted==22.10.0
 typing-extensions==4.5.0

tests/config/config_parse_test.py

@@ -337,6 +337,7 @@ def make_tron_config(
     jobs=None,
     mesos_options=None,
     k8s_options=None,
+    read_json=False,
 ):
     return schema.TronConfig(
         action_runner=action_runner or {},
@@ -350,6 +351,7 @@ def make_tron_config(
         jobs=jobs or make_master_jobs(),
         mesos_options=mesos_options or make_mesos_options(),
         k8s_options=k8s_options or make_k8s_options(),
+        read_json=read_json,
     )
 
 
@@ -498,6 +500,7 @@ def test_attributes(self):
         assert test_config.nodes == expected.nodes
         assert test_config.node_pools == expected.node_pools
         assert test_config.k8s_options == expected.k8s_options
+        assert test_config.read_json == expected.read_json
         for key in ["0", "1", "2", "_actions_dict", "4", "_mesos"]:
             job_name = f"MASTER.test_job{key}"
             assert job_name in test_config.jobs, f"{job_name} in test_config.jobs"