PLGN-398: add logic to prevent searching beyond 24 hours

[joneill-r7]

Proposed Changes

Collector code when querying Okta will never query beyond 24 hours. Update plugin logic to match this.

Description

Describe the proposed changes:

• If a customer has paused the task from running and resumes days/weeks/months later we don't want to resume the polling logic from when it was paused. In this case we want to drop that saved timestamp and only query from the last 24 hours - in essence following the workflow of a first iteration.
• For pagination customers can't pause when there is next pages as in the flowchart when the task returns has_more_pages, it queues the next iteration/page.
  • in the collector code this differs slightly in that we run one large execution of the task, polling each next page until we get all pages <- up to a limit of 20 iterations per run and at the end will save the last time of the latest event in this iteration.

PR Requirements

Developers, verify you have completed the following items by checking them off:

Testing

1. Passing saved time stamp older than 24 hours -> logs this and then reverts to now.

rapid7/Okta:4.2.1. Step name: monitor_logs
Saved state 2022-09-28T11:16:05.334Z exceeds the cut off (24 hours). Reverting to use time: 2023-10-09T08:56:51.530Z
Subsequent run...
Calling Okta with parameters={'since': '2023-10-09T08:56:51.530Z', 'until': '2023-10-10T08:56:51.530Z', 'limit': 1000} and next_page=None
Returning 0 log event(s) from this iteration.
No record to use as last timestamp, will not move checkpoint forward. Keeping value of 2023-10-09T08:56:51.530Z

2. First run (no timestamp in the state) -> uses default 24 hour time period.

rapid7/Okta:4.2.1. Step name: monitor_logs
First run
Calling Okta with parameters={'since': '2023-10-09T09:00:00.975Z', 'until': '2023-10-10T09:00:00.975Z', 'limit': 1000} and next_page=None
Returning 0 log event(s) from this iteration.
No record to use as last timestamp, will not move checkpoint forward. Keeping value of 2023-10-09T09:00:00.975Z

4. Normal logic without the task being paused.

rapid7/Okta:4.2.1. Step name: monitor_logs
Subsequent run...
Calling Okta with parameters={'since': '2023-10-09T14:35:28.390Z', 'until': '2023-10-10T09:00:33.537Z', 'limit': 1000} and next_page=None
Returning 0 log event(s) from this iteration.
No record to use as last timestamp, will not move checkpoint forward. Keeping value of 2023-10-09T14:35:28.390Z

Unit Tests

Review our documentation on generating and writing plugin unit tests

• Unit tests written for any new or updated code

In-Product Tests

If you are an InsightConnect customer or have access to an InsightConnect instance, the following in-product tests should be done:

• Screenshot of job output with the plugin changes
• Screenshot of the changed connection, actions, or triggers input within the InsightConnect workflow builder

Style

Review the style guide

• For dependencies, pin OS package and Python package versions
• For security, set least privileged account with USER nobody in the Dockerfile when possible
• For size, use the slim SDK images when possible: rapid7/insightconnect-python-3-38-slim-plugin:{sdk-version-num} and rapid7/insightconnect-python-3-38-plugin:{sdk-version-num}
• For error handling, use of PluginException and ConnectionTestException
• For logging, use self.logger
• For docs, use changelog style
• For docs, validate markdown with insight-plugin validate which calls icon_validate to lint help.md

Functional Checklist

• Work fully completed
• Functional
  • Any new actions/triggers include JSON test files in the tests/ directory created with insight-plugin samples
  • Tests should all pass unless it's a negative test. Negative tests have a naming convention of tests/$action_bad.json
  • Unsuccessful tests should fail by raising an exception causing the plugin to die and an object should be returned on successful test
  • Add functioning test results to PR, sanitize any output if necessary
    • Single action/trigger insight-plugin run -T tests/example.json --debug --jq
    • All actions/triggers shortcut insight-plugin run -T all --debug --jq (use PR format at end)
  • Add functioning run results to PR, sanitize any output if necessary
    • Single action/trigger insight-plugin run -R tests/example.json --debug --jq
    • All actions/triggers shortcut insight-plugin run --debug --jq (use PR format at end)

Assessment

You must validate your work to reviewers:

1. Run insight-plugin validate and make sure everything passes
2. Run the assessment tool: insight-plugin run -A. For single action validation: insight-plugin run tests/{file}.json -A
3. Copy (insight-plugin ... | pbcopy) and paste the output in a new post on this PR
4. Add required screenshots from the In-Product Tests section