Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARXIVCE-2614 Create remote converter driver #66

Merged
merged 22 commits into from
Oct 25, 2024

Conversation

norbusan
Copy link
Collaborator

  • factor out remote submission functionality from compile_submissions into
    a new module remote_call
  • use the remote_call in compile_submissions script
  • generate a RemoteConverterDriver based on ConverterDriver with overriden generate_pdf
    resuing the remote_call facilities

@norbusan norbusan force-pushed the ARXIVCE-2614-RemoteConverterDriver branch 2 times, most recently from 59d7dbb to 793c516 Compare September 26, 2024 04:40
Base automatically changed from ARXIVCE-2542-use-new-preflight to master September 26, 2024 19:13
@norbusan norbusan force-pushed the ARXIVCE-2614-RemoteConverterDriver branch 2 times, most recently from 5d131e6 to b1dd0ef Compare October 3, 2024 17:05
@norbusan norbusan force-pushed the ARXIVCE-2614-RemoteConverterDriver branch from 5a5d46a to 6f8bf0c Compare October 4, 2024 18:57
@norbusan norbusan changed the title [WIP] ARXIVCE-2614 Create remote converter driver ARXIVCE-2614 Create remote converter driver Oct 4, 2024
@norbusan norbusan marked this pull request as ready for review October 4, 2024 18:58
@norbusan norbusan changed the title ARXIVCE-2614 Create remote converter driver [WIP] ARXIVCE-2614 Create remote converter driver Oct 6, 2024
@norbusan norbusan marked this pull request as draft October 6, 2024 01:18
@norbusan norbusan force-pushed the ARXIVCE-2614-RemoteConverterDriver branch from 23c62d4 to a8dfb43 Compare October 6, 2024 22:48
@norbusan norbusan force-pushed the ARXIVCE-2614-RemoteConverterDriver branch from f479ab7 to 3353f4a Compare October 13, 2024 20:18
@norbusan norbusan marked this pull request as ready for review October 24, 2024 04:02
@norbusan norbusan changed the title [WIP] ARXIVCE-2614 Create remote converter driver ARXIVCE-2614 Create remote converter driver Oct 24, 2024
dginev
dginev previously approved these changes Oct 24, 2024


def submit_tarball(service: str, tarball: str, outcome_file: str, tex2pdf_timeout: int, post_timeout: int, auto_detect: bool = False) -> bool:
"""Submit tarball to compilation service."""
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Naming discussion]

The submit verb is a bit loaded due to the arXiv "submission" process, which gets me a little confused. If you're only thinking of message passing here, how about send_tarball?

But I see that the function does the web request and handles the response, so this is the "full work". Maybe then process_tarball is even better?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed it to service_process_tarball, since process_tarball itself again has the same problem ;-)


with open(tarball, "rb") as data_fd:
uploading = {"incoming": (os.path.basename(tarball), data_fd, "application/gzip")}
while True:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

while True may be inviting an infinite loop during the worst of times (some DDoS 504 situation). How about the retry pattern of:

retries = 2
for attempt in range(1+retries):
  # do work

or some variation on that.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I don't know why there is this special case for 504 error anyway ... @ntai ?

(I stumbled over the retries = 2 .... range(1+retries) until I realized it is retries and not tries = 2 ;-)

Copy link

@dginev dginev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@norbusan norbusan merged commit 206b88a into master Oct 25, 2024
2 checks passed
@norbusan norbusan deleted the ARXIVCE-2614-RemoteConverterDriver branch October 25, 2024 00:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants