Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Chunked File Uploader - front end changes #3501

Merged
merged 1 commit into from
Dec 7, 2023
Merged

Chunked File Uploader - front end changes #3501

merged 1 commit into from
Dec 7, 2023

Conversation

athithyaaselvam
Copy link
Collaborator

@athithyaaselvam athithyaaselvam commented Oct 12, 2023 •
edited
Loading

What changes were proposed in this pull request?

The chunked fileuploader breaks each file that is selected to be uploaded into chunks and stores them first to local storage. This backend PR : #3504 picks these locally stored chunked files and uploads them to desired filesystem (HDFS, S3, ABFS, Ozone). This chunked upload feature can now help the user upload large GB files without hogging the Hue Server.

How was this patch tested?

Manual testing on hdfs, S3, ABFS

Please review Hue Contributing Guide before opening a pull request.

@athithyaaselvam athithyaaselvam force-pushed the as_fineuploader branch 2 times, most recently from 98799f5 to 5d01f70 Compare October 12, 2023 22:38
@Harshg999
Copy link
Collaborator

Please improve the commit message and PR description.

Harshg999
Harshg999 reviewed Oct 17, 2023
View reviewed changes
apps/filebrowser/src/filebrowser/conf.py Show resolved Hide resolved
apps/filebrowser/src/filebrowser/conf.py Outdated Show resolved Hide resolved
apps/filebrowser/src/filebrowser/conf.py Outdated Show resolved Hide resolved
apps/filebrowser/src/filebrowser/conf.py Outdated Show resolved Hide resolved
apps/filebrowser/src/filebrowser/conf.py Outdated Show resolved Hide resolved
desktop/core/src/desktop/api2.py
@@ -46,8 +46,8 @@
from beeswax.models import Namespace
from desktop import appmanager
from desktop.auth.backend import is_admin
from desktop.conf import ENABLE_CONNECTORS, ENABLE_GIST_PREVIEW, CUSTOM, get_clusters, ENABLE_SHARING
from desktop.conf import ENABLE_NEW_STORAGE_BROWSER
from desktop.conf import ENABLE_CONNECTORS, ENABLE_GIST_PREVIEW, CUSTOM, get_clusters, IS_K8S_ONLY, ENABLE_SHARING
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not used in this file?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

didnt change anything in this line. Not sure why this is showing up as a change.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try rebasing your branch with latest master changes once then.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still present

desktop/core/src/desktop/conf.py Outdated Show resolved Hide resolved
desktop/core/src/desktop/templates/global_js_constants.mako Outdated Show resolved Hide resolved
@athithyaaselvam athithyaaselvam force-pushed the as_fineuploader branch 3 times, most recently from 17b3ca2 to 1219c80 Compare October 24, 2023 23:11
Harshg999
Harshg999 reviewed Oct 25, 2023
View reviewed changes
desktop/core/src/desktop/settings.py Outdated Show resolved Hide resolved
desktop/core/src/desktop/api2.py
@@ -46,8 +46,8 @@
from beeswax.models import Namespace
from desktop import appmanager
from desktop.auth.backend import is_admin
from desktop.conf import ENABLE_CONNECTORS, ENABLE_GIST_PREVIEW, CUSTOM, get_clusters, ENABLE_SHARING
from desktop.conf import ENABLE_NEW_STORAGE_BROWSER
from desktop.conf import ENABLE_CONNECTORS, ENABLE_GIST_PREVIEW, CUSTOM, get_clusters, IS_K8S_ONLY, ENABLE_SHARING
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try rebasing your branch with latest master changes once then.

desktop/core/src/desktop/conf.py Outdated Show resolved Hide resolved
desktop/core/src/desktop/templates/global_js_constants.mako Outdated Show resolved Hide resolved
bjornalm
bjornalm reviewed Oct 25, 2023
View reviewed changes
Copy link
Collaborator

@bjornalm bjornalm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, see feedback and questions

apps/filebrowser/src/filebrowser/templates/listdir_components.mako Show resolved Hide resolved
desktop/core/src/desktop/templates/global_js_constants.mako Outdated Show resolved Hide resolved
apps/filebrowser/src/filebrowser/conf.py Outdated Show resolved Hide resolved
desktop/core/src/desktop/js/ext/fileuploader.custom.js Show resolved Hide resolved
desktop/core/src/desktop/js/ext/fileuploader.custom.new.js
@@ -0,0 +1,7726 @@
// Fine Uploader 5.16.2 - MIT licensed. http://fineuploader.com
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the custom changes made to this file?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ozone UI changes, CSRF token handling, Encoding filepath changes are ported to this file from the old code.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add comments in the file for this to explain why this custom version exists?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a comment "Utilizing this new version of fileuploader js code as it supports chunked file upload. Large files are broken into chunks and stored in local storage and then from local storage these small chunked files are transferred to filesystem. This method will prevent Hue from crashing or getting stuck in the file upload page when a large file is uploaded "

amitsrivastava
amitsrivastava requested changes Nov 1, 2023
View reviewed changes
apps/filebrowser/src/filebrowser/conf.py Outdated Show resolved Hide resolved
apps/filebrowser/src/filebrowser/templates/listdir_components.mako Outdated Show resolved Hide resolved
apps/filebrowser/src/filebrowser/templates/listdir_components.mako Show resolved Hide resolved
desktop/core/src/desktop/conf.py Outdated Show resolved Hide resolved
desktop/core/src/desktop/conf.py Outdated Show resolved Hide resolved
desktop/core/src/desktop/settings.py Outdated Show resolved Hide resolved
desktop/core/src/desktop/settings.py Show resolved Hide resolved
@athithyaaselvam athithyaaselvam force-pushed the as_fineuploader branch 3 times, most recently from 92f5e86 to 9c61e7a Compare November 1, 2023 23:36
Harshg999
Harshg999 approved these changes Nov 22, 2023
View reviewed changes
Copy link
Collaborator

@Harshg999 Harshg999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Backend part looks good to me!
There is some conflict in mako file.

When resolved, squash your commits and improve the commit title/message.

desktop/core/src/desktop/api2.py
@@ -46,8 +46,8 @@
from beeswax.models import Namespace
from desktop import appmanager
from desktop.auth.backend import is_admin
from desktop.conf import ENABLE_CONNECTORS, ENABLE_GIST_PREVIEW, CUSTOM, get_clusters, ENABLE_SHARING
from desktop.conf import ENABLE_NEW_STORAGE_BROWSER
from desktop.conf import ENABLE_CONNECTORS, ENABLE_GIST_PREVIEW, CUSTOM, get_clusters, IS_K8S_ONLY, ENABLE_SHARING
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still present

bjornalm
bjornalm requested changes Nov 23, 2023
View reviewed changes
desktop/core/src/desktop/js/ext/fileuploader.custom.new.js
@@ -0,0 +1,7726 @@
// Fine Uploader 5.16.2 - MIT licensed. http://fineuploader.com
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add comments in the file for this to explain why this custom version exists?

@athithyaaselvam athithyaaselvam force-pushed the as_fineuploader branch 2 times, most recently from 55e0df0 to 5ed1345 Compare November 28, 2023 22:45
@athithyaaselvam athithyaaselvam changed the title Fileuploader-chunking support UI changes Chunked File Uploader - front end changes Nov 28, 2023
bjornalm
bjornalm approved these changes Nov 29, 2023
View reviewed changes
Copy link
Collaborator

@bjornalm bjornalm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ranade1
Adding new chunking-fileuploader template
a81c378 
(cherry picked from commit 517072f)
Change-Id: I9f80d24959a725c8a4765de7759b8114be808460

pop up modal close and upload job progress UI changes

(cherry picked from commit f2fb733)

Fix - drag and drop functionality - the drop zone will automatically hide when the file is moved out of the dropzone area ,
fix file upload popup css styling,
name of the file uploading will be displayed instead of static text failed.

(cherry picked from commit 7d9737b)

chunksize and max number of concurrent connections config support

(cherry picked from commit 2cf8b98)

Flag to enable new file uploader

(cherry picked from commit 4df96aa)
Change-Id: Id1eae6e920d42668858eb2b3586c5b492195e482
@athithyaaselvam athithyaaselvam enabled auto-merge (squash) December 7, 2023 17:56
@athithyaaselvam athithyaaselvam merged commit f931e4f into master Dec 7, 2023
4 of 5 checks passed
@athithyaaselvam athithyaaselvam deleted the as_fineuploader branch December 7, 2023 18:11
@athithyaaselvam athithyaaselvam mentioned this pull request Dec 11, 2023
Merged
athithyaaselvam added a commit that referenced this pull request Feb 5, 2024
@athithyaaselvam @ranade1 @wing2fly
Adding new chunking-fileuploader template (#3501)
0589e46 
(cherry picked from commit 517072f)
Change-Id: I9f80d24959a725c8a4765de7759b8114be808460

pop up modal close and upload job progress UI changes

(cherry picked from commit f2fb733)

Fix - drag and drop functionality - the drop zone will automatically hide when the file is moved out of the dropzone area ,
fix file upload popup css styling,
name of the file uploading will be displayed instead of static text failed.

(cherry picked from commit 7d9737b)

chunksize and max number of concurrent connections config support

(cherry picked from commit 2cf8b98)

Flag to enable new file uploader

(cherry picked from commit 4df96aa)
Change-Id: Id1eae6e920d42668858eb2b3586c5b492195e482

Co-authored-by: Prakash Ranade <ranade@cloudera.com>
(cherry picked from commit f931e4f)
Change-Id: I4ca598d255a60613f6272c49e809da163e8bb714
athithyaaselvam added a commit that referenced this pull request Mar 14, 2024
@athithyaaselvam
Adding new chunking-fileuploader template (#3501)
07be6a2 
(cherry picked from commit 517072f)
Change-Id: I9f80d24959a725c8a4765de7759b8114be808460

pop up modal close and upload job progress UI changes

(cherry picked from commit f2fb733)

Fix - drag and drop functionality - the drop zone will automatically hide when the file is moved out of the dropzone area ,
fix file upload popup css styling,
name of the file uploading will be displayed instead of static text failed.

(cherry picked from commit 7d9737b)

chunksize and max number of concurrent connections config support

(cherry picked from commit 2cf8b98)

Flag to enable new file uploader

(cherry picked from commit 4df96aa)
Change-Id: Id1eae6e920d42668858eb2b3586c5b492195e482

Co-authored-by: Prakash Ranade <ranade@cloudera.com>
(cherry picked from commit f931e4f)
Change-Id: I4ca598d255a60613f6272c49e809da163e8bb714
(cherry picked from commit 0589e46)
(cherry picked from commit 4e74c2ab5e3eb7e28570cbab8be5cacb6e5d17fc)
athithyaaselvam added a commit that referenced this pull request Jun 10, 2024
@athithyaaselvam
CDPD-60716: Adding new chunking-fileuploader template (#3501)
a5b3219 
(cherry picked from commit 517072f)
Change-Id: I9f80d24959a725c8a4765de7759b8114be808460

pop up modal close and upload job progress UI changes

(cherry picked from commit f2fb733)

Fix - drag and drop functionality - the drop zone will automatically hide when the file is moved out of the dropzone area ,
fix file upload popup css styling,
name of the file uploading will be displayed instead of static text failed.

(cherry picked from commit 7d9737b)

chunksize and max number of concurrent connections config support

(cherry picked from commit 2cf8b98)

Flag to enable new file uploader

(cherry picked from commit 4df96aa)
Change-Id: Id1eae6e920d42668858eb2b3586c5b492195e482

Co-authored-by: Prakash Ranade <ranade@cloudera.com>
(cherry picked from commit f931e4f)
Change-Id: I4ca598d255a60613f6272c49e809da163e8bb714
(cherry picked from commit 0589e46)
(cherry picked from commit 302a7ad534e130fcbad21d45fafb6095a5c24592)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amitsrivastava amitsrivastava amitsrivastava approved these changes

@bjornalm bjornalm bjornalm approved these changes

@Harshg999 Harshg999 Harshg999 approved these changes

@JohanAhlen JohanAhlen Awaiting requested review from JohanAhlen

@tabraiz12 tabraiz12 Awaiting requested review from tabraiz12

@sreenaths sreenaths Awaiting requested review from sreenaths

@agl29 agl29 Awaiting requested review from agl29

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@athithyaaselvam @Harshg999 @amitsrivastava @bjornalm @ranade1