Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/add core functions #1

Merged
merged 35 commits into from
Dec 19, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
0b24fba
add inital Dockerfile
AlexAxthelm Dec 12, 2023
ad93160
Add GitHub actions
AlexAxthelm Dec 12, 2023
cbe4a73
Add first data pulling function
AlexAxthelm Dec 12, 2023
2134fb6
Add R package linter
AlexAxthelm Dec 12, 2023
deca16e
Formatting changes for `lintr`
AlexAxthelm Dec 12, 2023
c4f788a
Add overall exporting function
AlexAxthelm Dec 12, 2023
0bc81ea
install package deps separate from package
AlexAxthelm Dec 12, 2023
fce4809
Pull arguments for exporting function from envvars
AlexAxthelm Dec 12, 2023
6dc8862
improve error logging
AlexAxthelm Dec 13, 2023
8891ad4
Add Azure deploy Template
AlexAxthelm Dec 13, 2023
841cf65
linting
AlexAxthelm Dec 13, 2023
b52af91
Copy get_factset_financial_data
AlexAxthelm Dec 14, 2023
bf22e52
Add financial data to export
AlexAxthelm Dec 14, 2023
5842ca7
Increase memory request
AlexAxthelm Dec 14, 2023
574bc0e
Externalize DB connection from extraction function
AlexAxthelm Dec 14, 2023
3623846
Update namespace to include .data and .env
AlexAxthelm Dec 14, 2023
4ded8ce
Simplify variable names
AlexAxthelm Dec 14, 2023
ee5a6cd
simplify variable names
AlexAxthelm Dec 14, 2023
9b436d6
Terminate connection if created in function
AlexAxthelm Dec 14, 2023
0a68abd
improve lgging in autofinalizing DB connection
AlexAxthelm Dec 14, 2023
705c303
Improve log messages
AlexAxthelm Dec 14, 2023
f684b94
Improve logging
AlexAxthelm Dec 14, 2023
3f05126
Simplify variable names
AlexAxthelm Dec 14, 2023
cdbdf92
Copy factset functions from pacta.data.preparation
AlexAxthelm Dec 14, 2023
202dd36
Update Entity Financing Data function
AlexAxthelm Dec 14, 2023
dc570c3
Convert get_factset_isin_to_fund_table to package
AlexAxthelm Dec 15, 2023
d7d5ef2
Update fund data function
AlexAxthelm Dec 18, 2023
e8d85cb
update ISS emissions function
AlexAxthelm Dec 18, 2023
760e2e0
Add new exports to export function
AlexAxthelm Dec 18, 2023
e4c84a6
update documentation
AlexAxthelm Dec 18, 2023
7837823
Remove `browser()`
AlexAxthelm Dec 18, 2023
d3e0e6c
rename function
AlexAxthelm Dec 18, 2023
a81781f
Rename functions
AlexAxthelm Dec 19, 2023
d9dac56
rename files to match function names
AlexAxthelm Dec 19, 2023
e088e5b
Document package
AlexAxthelm Dec 19, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .Rbuildignore
Original file line number Diff line number Diff line change
@@ -1 +1,4 @@
^LICENSE\.md$
.git/
.github/
^\.github$
3 changes: 3 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
.git/
.github/
Dockerfile
1 change: 1 addition & 0 deletions .github/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
*.html
12 changes: 12 additions & 0 deletions .github/workflows/build-Docker-image-nightly.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
on:
schedule:
- cron: '0 0 * * 1,2,3,4,5'

jobs:
build_docker_image:
name: "Call build and push action"
uses: ./.github/workflows/build-and-push-Docker-image.yml
secrets: inherit
with:
image-name: workflow.factset
image-tag: nightly
12 changes: 12 additions & 0 deletions .github/workflows/build-Docker-image-on-push-to-main.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
on:
push:
branches: [main]

jobs:
build_docker_image:
name: "Call build and push action"
uses: ./.github/workflows/build-and-push-Docker-image.yml
secrets: inherit
with:
image-name: workflow.factset
image-tag: main
37 changes: 37 additions & 0 deletions .github/workflows/build-Docker-image-on-push-to-pr.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
on:
pull_request:

jobs:
build_docker_image:
name: "Call build and push action"
uses: ./.github/workflows/build-and-push-Docker-image.yml
secrets: inherit
with:
image-name: workflow.factset
image-tag: pr${{ github.event.pull_request.number }}

add_comment:
needs: build_docker_image
runs-on: ubuntu-latest
steps:
- name: Find Comment
# https://github.com/peter-evans/find-comment
uses: peter-evans/find-comment@v2
id: fc
with:
issue-number: ${{ github.event.pull_request.number }}
comment-author: 'github-actions[bot]'
body-includes: Docker image from this PR

- name: Create or update comment
# https://github.com/peter-evans/create-or-update-comment
uses: peter-evans/create-or-update-comment@v3
with:
comment-id: ${{ steps.fc.outputs.comment-id }}
issue-number: ${{ github.event.pull_request.number }}
body: |
Docker image from this PR (${{ github.event.pull_request.head.sha }}) created
```
docker pull ${{ needs.build_docker_image.outputs.full-image-name }}
```
edit-mode: replace
67 changes: 67 additions & 0 deletions .github/workflows/build-and-push-Docker-image.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
---
name: Build and push docker image

on:
workflow_call:
inputs:
image-name:
required: true
type: string
image-tag:
required: true
type: string
outputs:
full-image-name:
description: "Full pushed image name including host/registry, name, and tag"
value: ${{ jobs.docker.outputs.full-image-name }}

jobs:
docker:
runs-on: ubuntu-latest
permissions:
packages: write
contents: read
timeout-minutes: 25
outputs:
full-image-name: ${{ steps.image-name.outputs.full-image-name }}

steps:

- name: Define image name
id: image-name
run: |
full_image_name="ghcr.io/${{ github.repository_owner }}/${{ inputs.image-name }}:${{ inputs.image-tag }}"
full_image_name=$(echo $full_image_name | tr '[A-Z]' '[a-z]')
echo "full-image-name=$full_image_name" >> "$GITHUB_OUTPUT"
echo "$full_image_name" > full-image-name

- uses: actions/upload-artifact@v3
with:
name: full-image-name
path: .

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Build and push
uses: docker/build-push-action@v5
with:
push: true
tags: ${{ steps.image-name.outputs.full-image-name }}
cache-from: type=gha
cache-to: type=gha,mode=min
no-cache-filters: install-pacta

check-system-dependencies:
name: "Check System Dependencies"
needs: docker
uses: ./.github/workflows/check-R-sysdeps.yml
with:
image: ${{ needs.docker.outputs.full-image-name }}
32 changes: 32 additions & 0 deletions .github/workflows/check-R-sysdeps.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
---
name: Check R system dependencies

on:
workflow_call:
inputs:
image:
required: true
type: string

jobs:

check-system-dependencies:
runs-on: ubuntu-latest
steps:
- name: 'Pull image'
run: |
echo ${{ inputs.image }}
docker pull ${{ inputs.image }}
- name: 'Run pak::sysreqs_check_installed()'
run: |

docker run \
--rm \
--entrypoint "/bin/sh" \
${{ inputs.image }} \
-c "Rscript -e '
x <- pak::sysreqs_check_installed()
print(x)
is_installed <- as.data.frame(x)[[\"installed\"]]
stopifnot(all(is_installed))
'"
32 changes: 32 additions & 0 deletions .github/workflows/lint-package.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Workflow derived from https://github.com/r-lib/actions/tree/v2/examples
# Need help debugging build failures? Start at https://github.com/r-lib/actions#where-to-find-help
on:
push:
branches: [main, master]
pull_request:
branches: [main, master]

name: lint

jobs:
lint:
runs-on: ubuntu-latest
env:
GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout@v3

- uses: r-lib/actions/setup-r@v2
with:
use-public-rspm: true

- uses: r-lib/actions/setup-r-dependencies@v2
with:
extra-packages: any::lintr, local::.
needs: lint

- name: Lint
run: lintr::lint_package()
shell: Rscript {0}
env:
LINTR_ERROR_ON_LINT: true
11 changes: 11 additions & 0 deletions .github/workflows/run-hadolint.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
on: [push, pull_request]

jobs:
hadolint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: hadolint/hadolint-action@v3.1.0
with:
dockerfile: Dockerfile
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
.env
azure-deploy.parameters.json
10 changes: 10 additions & 0 deletions DESCRIPTION
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,13 @@ License: MIT + file LICENSE
Encoding: UTF-8
Roxygen: list(markdown = TRUE)
RoxygenNote: 7.2.3
Imports:
DBI,
dbplyr,
dplyr,
logger,
rlang,
RPostgres,
withr
Suggests:
rstudioapi
69 changes: 69 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# using rocker r-vers as a base with R 4.3.1
# https://hub.docker.com/r/rocker/r-ver
# https://rocker-project.org/images/versioned/r-ver.html
#
# sets CRAN repo to use Posit Package Manager to freeze R package versions to
# those available on 2023-10-30
# https://packagemanager.posit.co/client/#/repos/2/overview
# https://packagemanager.posit.co/cran/__linux__/jammy/2023-10-30

# set proper base image
ARG R_VERS="4.3.1"
FROM rocker/r-ver:$R_VERS AS base

# set Docker image labels
LABEL org.opencontainers.image.source=https://github.com/RMI-PACTA/workflow.factset
LABEL org.opencontainers.image.description="Extract FactSet Data for use in PACTA"
LABEL org.opencontainers.image.licenses=MIT
LABEL org.opencontainers.image.title=""
LABEL org.opencontainers.image.revision=""
LABEL org.opencontainers.image.version=""
LABEL org.opencontainers.image.vendor=""
LABEL org.opencontainers.image.base.name=""
LABEL org.opencontainers.image.ref.name=""
LABEL org.opencontainers.image.authors=""

# set apt-get to noninteractive mode
ARG DEBIAN_FRONTEND="noninteractive"
ARG DEBCONF_NOWARNINGS="yes"

RUN groupadd -r runner-workflow-factset \
&& useradd -r -g runner-workflow-factset runner-workflow-factset \
&& mkdir -p /home/runner-workflow-factset \
&& chown -R runner-workflow-factset /home/runner-workflow-factset
WORKDIR /home/runner-workflow-factset

# install system dependencies
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
libicu-dev=70.* \
libpq-dev=14.* \
&& chmod -R a+rwX /root \
&& rm -rf /var/lib/apt/lists/*

# set frozen CRAN repo
ARG CRAN_REPO="https://packagemanager.posit.co/cran/__linux__/jammy/2023-10-30"
RUN echo "options(repos = c(CRAN = '$CRAN_REPO'), pkg.sysreqs = FALSE)" >> "${R_HOME}/etc/Rprofile.site" \
# install packages for dependency resolution and installation
&& Rscript -e "install.packages(c('pak', 'jsonlite'))"

# Install R deopendencies
COPY DESCRIPTION /workflow.factset/DESCRIPTION

# install R package dependencies
RUN Rscript -e "\
deps <- pak::local_install_deps(root = '/workflow.factset'); \
"

# copy in everything from this repo
COPY . /workflow.factset

# install R package dependencies
RUN Rscript -e "\
pak::pkg_install('local::/workflow.factset'); \
"

USER runner-workflow-factset

# set default run behavior
CMD ["Rscript", "-e", "logger::log_threshold(Sys.getenv('LOG_LEVEL', 'INFO'));workflow.factset::export_pacta_files()"]
11 changes: 11 additions & 0 deletions NAMESPACE
Original file line number Diff line number Diff line change
@@ -1,2 +1,13 @@
# Generated by roxygen2: do not edit by hand

export(connect_factset_db)
export(export_pacta_files)
export(get_entity_financing_data)
export(get_entity_info)
export(get_financial_data)
export(get_fund_data)
export(get_isin_to_fund_table)
export(get_iss_emissions_data)
importFrom(dplyr,"%>%")
importFrom(rlang,.data)
importFrom(rlang,.env)
Loading