Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement offline DB update #25

Merged
merged 24 commits into from
Jul 21, 2022
Merged

Implement offline DB update #25

merged 24 commits into from
Jul 21, 2022

Commits on Jun 15, 2022

  1. Configuration menu
    Copy the full SHA
    f8cac74 View commit details
    Browse the repository at this point in the history
  2. Update Jenkins pipeline to submit job updating the database. The subm…

    …ission script now uses the environment variables set by pipeline to set the database name and path.
    micaeljtoliveira committed Jun 15, 2022
    Configuration menu
    Copy the full SHA
    82172c0 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    2dc9790 View commit details
    Browse the repository at this point in the history
  4. Remove the check that the file we are trying to create during the upd…

    …ate does not exist yet. If non concurrent Jenkins runs are allowed, then it should be safe to restart an indexing job from an unfinished one.
    micaeljtoliveira committed Jun 15, 2022
    Configuration menu
    Copy the full SHA
    e898053 View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2022

  1. Reorganize the different stages of the Jenkins pipeline and move seve…

    …ral steps to run at the end of the pipeline, depending on the result of the update.
    
    Change the name of the PBS logs, so that we can easily archive then with Jenkins.
    micaeljtoliveira committed Jun 16, 2022
    Configuration menu
    Copy the full SHA
    eca3c41 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2022

  1. Refactor to separate bash and python scripts.

    Use parallel to allow simultaneous indexing and shuffling inputs.
    
    Set file chunks to 1000.
    aidanheerdegen authored and micaeljtoliveira committed Jun 20, 2022
    Configuration menu
    Copy the full SHA
    bafb5c7 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    47f4ebf View commit details
    Browse the repository at this point in the history
  3. Update update script to use an older version of the cookbook. Allow t…

    …o pass the database to be updated through the command line.
    micaeljtoliveira committed Jun 20, 2022
    Configuration menu
    Copy the full SHA
    838ebbe View commit details
    Browse the repository at this point in the history
  4. Update Jenkinsfile to run then update script interactively, i.e., wit…

    …hout using PBS. We are still setting a time limit for the update, so that updates that run for longer than a reasonable amount of time don't go unnoticed. An update that fails to complete because of the time limit will mark the build as unstable.
    micaeljtoliveira committed Jun 20, 2022
    Configuration menu
    Copy the full SHA
    16d0697 View commit details
    Browse the repository at this point in the history
  5. Better handling of some situations (plus some comments):

    - Do not allow to update the database twice during the same day.
    - Remove the new database file if an update fails.
    - Remove the new database file if the run is aborted by a user.
    micaeljtoliveira committed Jun 20, 2022
    Configuration menu
    Copy the full SHA
    34a7895 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    bec388f View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2022

  1. Configuration menu
    Copy the full SHA
    3722473 View commit details
    Browse the repository at this point in the history

Commits on Jun 30, 2022

  1. Only prune old files if current update was successful. Otherwise we m…

    …ight end up deleting all databases if the update fails over a long period of time.
    micaeljtoliveira committed Jun 30, 2022
    Configuration menu
    Copy the full SHA
    ed744fc View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a3ca9a8 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2022

  1. Go back to using a PBS job to update the database, as the interactive…

    … job was being killed after some time.
    micaeljtoliveira committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    61a02bc View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2022

  1. Configuration menu
    Copy the full SHA
    ae401c2 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2022

  1. Configuration menu
    Copy the full SHA
    abe7177 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0fe7db5 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    699c060 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    5d011d6 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    f015ab9 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2022

  1. Configuration menu
    Copy the full SHA
    092b606 View commit details
    Browse the repository at this point in the history
  2. Fix some typos.

    micaeljtoliveira committed Jul 19, 2022
    Configuration menu
    Copy the full SHA
    1557726 View commit details
    Browse the repository at this point in the history

Commits on Jul 20, 2022

  1. Configuration menu
    Copy the full SHA
    ba4c72e View commit details
    Browse the repository at this point in the history