Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SIMPLE_FORMS] feat: add logic to properly upload files to S3 #18572

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -609,6 +609,7 @@ app/uploaders/uploader_virus_scan.rb @department-of-veterans-affairs/va-api-engi
app/uploaders/validate_pdf.rb @department-of-veterans-affairs/Disability-Experience @department-of-veterans-affairs/va-api-engineers @department-of-veterans-affairs/backend-review-group
app/uploaders/vets_shrine.rb @department-of-veterans-affairs/va-api-engineers @department-of-veterans-affairs/backend-review-group
app/validators/token_util.rb @department-of-veterans-affairs/backend-review-group
app/uploaders/veteran_facing_forms_remediation_uploader.rb @department-of-veterans-affairs/platform-va-product-forms @department-of-veterans-affairs/va-api-engineers @department-of-veterans-affairs/backend-review-group
app/sidekiq/account_login_statistics_job.rb @department-of-veterans-affairs/octo-identity
app/sidekiq/benefits_intake_remediation_status_job.rb @department-of-veterans-affairs/platform-va-product-forms @department-of-veterans-affairs/va-api-engineers @department-of-veterans-affairs/backend-review-group
app/sidekiq/benefits_intake_status_job.rb @department-of-veterans-affairs/platform-va-product-forms @department-of-veterans-affairs/Disability-Experience @department-of-veterans-affairs/va-api-engineers @department-of-veterans-affairs/backend-review-group
Expand Down
46 changes: 46 additions & 0 deletions app/uploaders/veteran_facing_forms_remediation_uploader.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# frozen_string_literal: true

class VeteranFacingFormsRemediationUploader < CarrierWave::Uploader::Base
include SetAWSConfig
include UploaderVirusScan

def size_range
(1.byte)...(100_000_000.bytes)
end

# All the same files allowed by benefits intake, with the
# addition of json for metadata and csv for manifest
def extension_allowlist
%w[bmp csv gif jpeg jpg json pdf png tif tiff txt]
end

def initialize(benefits_intake_uuid, directory)
raise 'The benefits_intake_uuid is missing.' if @benefits_intake_uuid.blank?

super
@benefits_intake_uuid = benefits_intake_uuid
@directory = directory

set_storage_options!
end

def store_dir
raise 'The s3 directory is missing.' if @directory.blank?

@directory
end

def set_storage_options!
# TODO: update this to vff specific S3 bucket once it has been created
s3_settings = Settings.reports.aws
# defaults to CarrierWave::Storage::File if not AWS unless a real aws_access_key_id is set
if s3_settings.aws_access_key_id.present?
set_aws_config(
s3_settings.aws_access_key_id,
s3_settings.aws_secret_access_key,
s3_settings.region,
s3_settings.bucket
)
end
end
end
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ def initialize(parent_dir: 'vff-simple-forms', **options) # rubocop:disable Lint
def upload
log_info("Uploading archive: #{benefits_intake_uuid} to S3 bucket")

upload_temp_folder_to_s3
upload_directory_to_s3(temp_directory_path)
cleanup
generate_presigned_url
rescue => e
Expand Down Expand Up @@ -72,11 +72,16 @@ def build_submission_archive(**)
SubmissionArchiveBuilder.new(**).run
end

def upload_temp_folder_to_s3
Dir.glob("#{temp_directory_path}/**/*").each do |path|
def upload_directory_to_s3(directory_path)
raise "Directory #{directory_path} does not exist" unless Dir.exist?(directory_path)

Dir.glob(File.join(directory_path, '**', '*')).each do |path|
next if File.directory?(path)

File.open(path, 'rb') { |file| save_file_to_s3(file.read) }
File.open(path) do |file_obj|
sanitized_file = CarrierWave::SanitizedFile.new(file_obj)
s3_uploader.store!(sanitized_file)
end
end
end

Expand All @@ -102,10 +107,6 @@ def generate_presigned_url
submission_object.presigned_url(:get, expires_in: 30.minutes.to_i)
end

def save_file_to_s3(content)
submission_object.tap { |obj| obj.put(body: content) }
end

def submission_object
s3_resource.bucket(target_bucket).object(s3_submission_file_path)
end
Expand All @@ -131,6 +132,10 @@ def s3_directory_path
@s3_directory_path ||= "#{parent_dir}/#{unique_file_name}"
end

def s3_uploader
@s3_uploader ||= VeteranFacingFormsRemediationUploader.new(benefits_intake_uuid, s3_submission_file_path)
end

def local_submission_file_path
@local_submission_file_path ||= build_local_file_path(s3_submission_file_path)
end
Expand Down
Loading