feat(s3-endpoint): Added endpoint url paramter for using non aws buck…

…ets for usersync

helm/fence/Chart.yaml

@@ -15,7 +15,7 @@ type: application
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 0.1.28
+version: 0.1.29
 
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application. Versions are not expected to

helm/fence/README.md

@@ -1,6 +1,6 @@
 # fence
 
-![Version: 0.1.27](https://img.shields.io/badge/Version-0.1.27-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: master](https://img.shields.io/badge/AppVersion-master-informational?style=flat-square)
+![Version: 0.1.29](https://img.shields.io/badge/Version-0.1.29-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: master](https://img.shields.io/badge/AppVersion-master-informational?style=flat-square)
 
 A Helm chart for gen3 Fence
 
@@ -184,10 +184,11 @@ A Helm chart for gen3 Fence
 | serviceAccount.create | bool | `true` | Specifies whether a service account should be created. |
 | serviceAccount.name | string | `"fence-sa"` | The name of the service account |
 | tolerations | list | `[]` | Tolerations for the pods |
-| usersync | map | `{"addDbgap":false,"custom_image":null,"onlyDbgap":false,"schedule":"*/30 * * * *","slack_send_dbgap":false,"slack_webhook":"None","syncFromDbgap":false,"userYamlS3Path":"s3://cdis-gen3-users/helm-test/user.yaml","usersync":false}` | Configuration options for usersync cronjob. |
+| usersync | map | `{"addDbgap":false,"custom_image":null,"onlyDbgap":false,"s3Endpoint":"https://s3.amazonaws.com","schedule":"*/30 * * * *","slack_send_dbgap":false,"slack_webhook":"None","syncFromDbgap":false,"userYamlS3Path":"s3://cdis-gen3-users/helm-test/user.yaml","usersync":false}` | Configuration options for usersync cronjob. |
 | usersync.addDbgap | bool | `false` | Force attempting a dbgap sync if "true", falls back on user.yaml |
 | usersync.custom_image | string | `nil` | To set a custom image for pulling the user.yaml file from S3. Default is the Gen3 Awshelper image. |
 | usersync.onlyDbgap | bool | `false` | Forces ONLY a dbgap sync if "true", IGNORING user.yaml |
+| usersync.s3Endpoint | string | `"https://s3.amazonaws.com"` | The endpoint url of your s3 service |
 | usersync.schedule | string | `"*/30 * * * *"` | The cron schedule expression to use in the usersync cronjob. Runs every 30 minutes by default. |
 | usersync.slack_send_dbgap | bool | `false` | Will echo what files we are seeing on dbgap ftp to Slack. |
 | usersync.slack_webhook | string | `"None"` | Slack webhook endpoint used with certain jobs. |

helm/fence/templates/usersync-cron.yaml

@@ -78,6 +78,8 @@ spec:
                       key: hostname
               - name: userYamlS3Path
                 value: {{ .Values.usersync.userYamlS3Path | quote }}
+              - name: s3Endpoint
+                value: {{ .Values.usersync.s3Endpoint | quote }}
               - name: slackWebHook
                 value: {{ .Values.usersync.slack_webhook | quote }}
               volumeMounts:
@@ -105,7 +107,7 @@ spec:
                     n=0
                     until [ $n -ge 5 ]; do
                       echo "Download attempt $n"
-                      aws s3 cp "${userYamlS3Path}" /mnt/shared/user.yaml && break
+                      aws s3 cp "${userYamlS3Path}" /mnt/shared/user.yaml --endpoint-url $s3Endpoint && break
                       n=$[$n+1]
                       sleep 2
                     done
@@ -216,4 +218,4 @@ spec:
                 echo "Exit code: $exitcode"
                 exit "$exitcode"
           restartPolicy: "Never"
-{{- end }}
+{{- end }}

helm/fence/values.yaml

@@ -114,6 +114,8 @@ usersync:
   onlyDbgap: false
   # -- (string) Path to the user.yaml file in S3.
   userYamlS3Path: s3://cdis-gen3-users/helm-test/user.yaml
+  # -- (string) The endpoint url of your s3 service
+  s3Endpoint: https://s3.amazonaws.com
   # -- (string) Slack webhook endpoint used with certain jobs.
   slack_webhook: None
   # -- (bool) Will echo what files we are seeing on dbgap ftp to Slack.

helm/gen3/Chart.yaml

@@ -128,7 +128,7 @@ type: application
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 0.1.56
+version: 0.1.57
 
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application. Versions are not expected to

helm/gen3/README.md

@@ -1,6 +1,6 @@
 # gen3
 
-![Version: 0.1.55](https://img.shields.io/badge/Version-0.1.55-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: master](https://img.shields.io/badge/AppVersion-master-informational?style=flat-square)
+![Version: 0.1.57](https://img.shields.io/badge/Version-0.1.57-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: master](https://img.shields.io/badge/AppVersion-master-informational?style=flat-square)
 
 Helm chart to deploy Gen3 Data Commons
 
@@ -25,7 +25,7 @@ Helm chart to deploy Gen3 Data Commons
 | file://../aws-es-proxy | aws-es-proxy | 0.1.13 |
 | file://../common | common | 0.1.16 |
 | file://../etl | etl | 0.1.9 |
-| file://../fence | fence | 0.1.27 |
+| file://../fence | fence | 0.1.28 |
 | file://../frontend-framework | frontend-framework | 0.1.6 |
 | file://../gen3-network-policies | gen3-network-policies | 0.1.2 |
 | file://../guppy | guppy | 0.1.16 |
@@ -66,9 +66,10 @@ Helm chart to deploy Gen3 Data Commons
 | elasticsearch.singleNode | bool | `true` |  |
 | etl.enabled | bool | `true` | Whether to deploy the etl subchart. |
 | fence.enabled | bool | `true` | Whether to deploy the fence subchart. |
-| fence.usersync | map | `{"addDbgap":false,"onlyDbgap":false,"schedule":"*/30 * * * *","slack_send_dbgap":false,"slack_webhook":"None","syncFromDbgap":false,"userYamlS3Path":"s3://cdis-gen3-users/helm-test/user.yaml","usersync":false}` | Configuration options for usersync cronjob. |
+| fence.usersync | map | `{"addDbgap":false,"onlyDbgap":false,"s3Endpoint":"https://s3.amazonaws.com","schedule":"*/30 * * * *","slack_send_dbgap":false,"slack_webhook":"None","syncFromDbgap":false,"userYamlS3Path":"s3://cdis-gen3-users/helm-test/user.yaml","usersync":false}` | Configuration options for usersync cronjob. |
 | fence.usersync.addDbgap | bool | `false` | Force attempting a dbgap sync if "true", falls back on user.yaml |
 | fence.usersync.onlyDbgap | bool | `false` | Forces ONLY a dbgap sync if "true", IGNORING user.yaml |
+| fence.usersync.s3Endpoint | string | `"https://s3.amazonaws.com"` | The endpoint url of your s3 service |
 | fence.usersync.schedule | string | `"*/30 * * * *"` | The cron schedule expression to use in the usersync cronjob. Runs every 30 minutes by default. |
 | fence.usersync.slack_send_dbgap | bool | `false` | Will echo what files we are seeing on dbgap ftp to Slack. |
 | fence.usersync.slack_webhook | string | `"None"` | Slack webhook endpoint used with certain jobs. |

helm/gen3/values.yaml

@@ -132,6 +132,8 @@ fence:
     onlyDbgap: false
     # -- (string) Path to the user.yaml file in S3.
     userYamlS3Path: s3://cdis-gen3-users/helm-test/user.yaml
+    # -- (string) The endpoint url of your s3 service
+    s3Endpoint: https://s3.amazonaws.com
     # -- (string) Slack webhook endpoint used with certain jobs.
     slack_webhook: None
     # -- (bool) Will echo what files we are seeing on dbgap ftp to Slack.