[pull] dev from galaxyproject:dev #948

[24.1] Backport PR 18630 "Add BlobToolkit to the list of interactive tools" to release_24.1

Also: - Remove unnecessary `unicodify()` calls

[24.1] Fix copying workflow with subworkflow step for step that you own

…le_handling [24.1] Fix response if dataset requested by display application is deleted

Parse stored config details to script-based visualizations

Bumps [express](https://github.com/expressjs/express) from 4.19.2 to 4.21.0. - [Release notes](https://github.com/expressjs/express/releases) - [Changelog](https://github.com/expressjs/express/blob/4.21.0/History.md) - [Commits](expressjs/express@4.19.2...4.21.0) --- updated-dependencies: - dependency-name: express dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>

[24.1] Use smtplib send_message to support utf-8 chars in to and from

…se `HistorySortByLiteral` type

…ker`

…oryDatasetPicker` a generic component

…cker`

…nDialog` directory and updated import paths accordingly

Fixes #18791

…job_failed [24.1] Collect job metrics also when job failed

…n there are no running progress

…tasetAssociation` class See #18773 (review)

Fix the following mypy 1.11 error: ``` lib/galaxy/managers/history_contents.py:638: error: Signature of "_add_parsers" incompatible with supertype "GenomeFilterMixin" [override] def _add_parsers(self): ^ lib/galaxy/managers/history_contents.py:638: note: Superclass: lib/galaxy/managers/history_contents.py:638: note: def _add_parsers(self, database_connection: str) -> Any lib/galaxy/managers/history_contents.py:638: note: Subclass: lib/galaxy/managers/history_contents.py:638: note: def _add_parsers(self) -> Any ```

…method Fix the following mypy 1.11 errors: ``` lib/galaxy/managers/hdas.py:616: error: Signature of "serialize" incompatible with supertype "DatasetAssociationSerializer" [override] def serialize(self, hda, keys, user=None, **context): ^ lib/galaxy/managers/hdas.py:616: note: Superclass: lib/galaxy/managers/hdas.py:616: note: def serialize(self, item: Any, keys: Any, **context: Any) -> Any lib/galaxy/managers/hdas.py:616: note: Subclass: lib/galaxy/managers/hdas.py:616: note: def serialize(self, hda: Any, keys: Any, user: Any = ..., **context: Any) -> Any lib/galaxy/managers/hdas.py:616: error: Signature of "serialize" incompatible with supertype "ModelSerializer" [override] def serialize(self, hda, keys, user=None, **context): ^ lib/galaxy/managers/hdas.py:616: note: Superclass: lib/galaxy/managers/hdas.py:616: note: def serialize(self, item: Any, keys: Any, **context: Any) -> Any lib/galaxy/managers/hdas.py:616: note: Subclass: lib/galaxy/managers/hdas.py:616: note: def serialize(self, hda: Any, keys: Any, user: Any = ..., **context: Any) -> Any ```

…ethod Fix the following mypy 1.11 errors: ``` lib/galaxy/tools/__init__.py:3989: error: Signature of "produce_outputs" incompatible with supertype "DatabaseOperationTool" [override] def produce_outputs(self, trans, out_data, output_collections, inc... ^ lib/galaxy/tools/__init__.py:3989: note: Superclass: lib/galaxy/tools/__init__.py:3989: note: def produce_outputs(self, trans: Any, out_data: Any, output_collections: Any, incoming: Any, history: Any, **kwds: Any) -> Any lib/galaxy/tools/__init__.py:3989: note: Subclass: lib/galaxy/tools/__init__.py:3989: note: def produce_outputs(self, trans: Any, out_data: Any, output_collections: Any, incoming: Any, history: Any, tag_handler: Any, **kwds: Any) -> Any lib/galaxy/tools/__init__.py:4026: error: Signature of "produce_outputs" incompatible with supertype "DatabaseOperationTool" [override] def produce_outputs(self, trans, out_data, output_collections, inc... ^ lib/galaxy/tools/__init__.py:4026: note: Superclass: lib/galaxy/tools/__init__.py:4026: note: def produce_outputs(self, trans: Any, out_data: Any, output_collections: Any, incoming: Any, history: Any, **kwds: Any) -> Any lib/galaxy/tools/__init__.py:4026: note: Subclass: lib/galaxy/tools/__init__.py:4026: note: def produce_outputs(self, trans: Any, out_data: Any, output_collections: Any, incoming: Any, history: Any, tag_handler: Any, **kwds: Any) -> Any ```

Fix the following mypy 1.11 errors: ``` lib/galaxy/workflow/modules.py:650: error: Signature of "save_to_step" incompatible with supertype "WorkflowModule" [override] def save_to_step(self, step, **kwd): ^ lib/galaxy/workflow/modules.py:650: note: Superclass: lib/galaxy/workflow/modules.py:650: note: def save_to_step(self, step: Any, detached: Any = ...) -> Any lib/galaxy/workflow/modules.py:650: note: Subclass: lib/galaxy/workflow/modules.py:650: note: def save_to_step(self, step: Any, **kwd: Any) -> Any lib/galaxy/workflow/modules.py:1030: error: Signature of "save_to_step" incompatible with supertype "WorkflowModule" [override] def save_to_step(self, step, **kwd): ^ lib/galaxy/workflow/modules.py:1030: note: Superclass: lib/galaxy/workflow/modules.py:1030: note: def save_to_step(self, step: Any, detached: Any = ...) -> Any lib/galaxy/workflow/modules.py:1030: note: Subclass: lib/galaxy/workflow/modules.py:1030: note: def save_to_step(self, step: Any, **kwd: Any) -> Any lib/galaxy/workflow/modules.py:1680: error: Signature of "save_to_step" incompatible with supertype "WorkflowModule" [override] def save_to_step(self, step, **kwd): ^ lib/galaxy/workflow/modules.py:1680: note: Superclass: lib/galaxy/workflow/modules.py:1680: note: def save_to_step(self, step: Any, detached: Any = ...) -> Any lib/galaxy/workflow/modules.py:1680: note: Subclass: lib/galaxy/workflow/modules.py:1680: note: def save_to_step(self, step: Any, **kwd: Any) -> Any ```

Fix the following mypy 1.11 error: ``` lib/galaxy/tools/__init__.py:3212: error: Signature of "exec_after_process" incompatible with supertype "Tool" [override] def exec_after_process(self, app, inp_data, out_data, param_dict, ... ^ lib/galaxy/tools/__init__.py:3212: note: Superclass: lib/galaxy/tools/__init__.py:3212: note: def exec_after_process(self, app: Any, inp_data: Any, out_data: Any, param_dict: Any, job: Any = ..., **kwds: Any) -> Any lib/galaxy/tools/__init__.py:3212: note: Subclass: lib/galaxy/tools/__init__.py:3212: note: def exec_after_process(self, app: Any, inp_data: Any, out_data: Any, param_dict: Any, job: Any, final_job_state: Any = ..., **kwds: Any) -> Any ``` Also: - Add some type annotions - Small refactorings

Remove now unneeded `type: ignore` .

I was trying to use the galaxy-data container for data source tools, but it fails with: ``` Traceback (most recent call last): File "/Users/mvandenb/src/galaxy/test/functional/tools/data_source.py", line 13, in <module> from galaxy.datatypes import sniff File "/usr/local/lib/python3.12/site-packages/galaxy/datatypes/sniff.py", line 40, in <module> import pylibmagic # noqa: F401 # isort:skip ^^^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named 'pylibmagic' ``` pylibmagic is just shipping the magic library and adds it to path, but that's not needed for the data-data container which can use magic just fine.

Update Mypy to 1.11.2 and fix new signature override errors

It is one of the more common client-side errors in Sentry (https://sentry.galaxyproject.org/share/issue/7cfa153372ab426485b0ea2d7cde8b49/), and this brings it in line with the API definition where both text and and synposis are nullable.

by running `make update-dependencies`.

Update Python dependencies

…story-dataset-picker Replace History Dataset Picker in Library Folder

…ndling [24.1] Fix optional text handling in LibraryEditField

[24.1] Make pylibmagic import optional

…ToHistoryLink [24.1] Add error handling for `SwitchToHistoryLink`

[24.1] Allow drag and drop for collection elements

…tput_to_required_input [24.1] Prevent optional output to required input connection in editor

[24.1] Fix unspecified ``oidc_endpoint`` variable overwriting specified ``redirect_url``

Fixes #18727 / https://sentry.galaxyproject.org/share/issue/8091a452f32645b89b37976362779532/: ``` Message (59965185/49195886) Job wrapper finish method failed Stack Trace Newest OperationalError: unable to open database file File "sqlalchemy/engine/base.py", line 146, in __init__ self._dbapi_connection = engine.raw_connection() File "sqlalchemy/engine/base.py", line 3300, in raw_connection return self.pool.connect() File "sqlalchemy/pool/base.py", line 449, in connect return _ConnectionFairy._checkout(self) File "sqlalchemy/pool/base.py", line 1263, in _checkout fairy = _ConnectionRecord.checkout(pool) File "sqlalchemy/pool/base.py", line 712, in checkout rec = pool._do_get() File "sqlalchemy/pool/impl.py", line 179, in _do_get with util.safe_reraise(): File "sqlalchemy/util/langhelpers.py", line 146, in __exit__ raise exc_value.with_traceback(exc_tb) File "sqlalchemy/pool/impl.py", line 177, in _do_get return self._create_connection() File "sqlalchemy/pool/base.py", line 390, in _create_connection return _ConnectionRecord(self) File "sqlalchemy/pool/base.py", line 674, in __init__ self.__connect() File "sqlalchemy/pool/base.py", line 900, in __connect with util.safe_reraise(): File "sqlalchemy/util/langhelpers.py", line 146, in __exit__ raise exc_value.with_traceback(exc_tb) File "sqlalchemy/pool/base.py", line 896, in __connect self.dbapi_connection = connection = pool._invoke_creator(self) File "sqlalchemy/engine/create.py", line 643, in connect return dialect.connect(*cargs, **cparams) File "sqlalchemy/engine/default.py", line 620, in connect return self.loaded_dbapi.connect(*cargs, **cparams) OperationalError: (sqlite3.OperationalError) unable to open database file (Background on this error at: https://sqlalche.me/e/20/e3q8) File "kombu/connection.py", line 472, in _reraise_as_library_errors yield File "kombu/connection.py", line 556, in _ensured return fun(*args, **kwargs) File "kombu/messaging.py", line 202, in _publish [maybe_declare(entity) for entity in declare] File "kombu/messaging.py", line 202, in <listcomp> [maybe_declare(entity) for entity in declare] File "kombu/messaging.py", line 107, in maybe_declare return maybe_declare(entity, self.channel, retry, **retry_policy) File "kombu/common.py", line 113, in maybe_declare return _maybe_declare(entity, channel) File "kombu/common.py", line 153, in _maybe_declare entity.declare(channel=channel) File "kombu/entity.py", line 617, in declare self._create_queue(nowait=nowait, channel=channel) File "kombu/entity.py", line 626, in _create_queue self.queue_declare(nowait=nowait, passive=False, channel=channel) File "kombu/entity.py", line 655, in queue_declare ret = channel.queue_declare( File "kombu/transport/virtual/base.py", line 537, in queue_declare self._new_queue(queue, **kwargs) File "kombu/transport/sqlalchemy/__init__.py", line 159, in _new_queue self._get_or_create(queue) File "kombu/transport/sqlalchemy/__init__.py", line 138, in _get_or_create obj = self.session.query(self.queue_cls) \ File "kombu/transport/sqlalchemy/__init__.py", line 133, in session _, Session = self._open() File "kombu/transport/sqlalchemy/__init__.py", line 125, in _open metadata.create_all(engine) File "sqlalchemy/sql/schema.py", line 5857, in create_all bind._run_ddl_visitor( File "sqlalchemy/engine/base.py", line 3250, in _run_ddl_visitor with self.begin() as conn: File "contextlib.py", line 137, in __enter__ return next(self.gen) File "sqlalchemy/engine/base.py", line 3240, in begin with self.connect() as conn: File "sqlalchemy/engine/base.py", line 3276, in connect return self._connection_cls(self) File "sqlalchemy/engine/base.py", line 148, in __init__ Connection._handle_dbapi_exception_noconnection( File "sqlalchemy/engine/base.py", line 2440, in _handle_dbapi_exception_noconnection raise sqlalchemy_exception.with_traceback(exc_info[2]) from e File "sqlalchemy/engine/base.py", line 146, in __init__ self._dbapi_connection = engine.raw_connection() File "sqlalchemy/engine/base.py", line 3300, in raw_connection return self.pool.connect() File "sqlalchemy/pool/base.py", line 449, in connect return _ConnectionFairy._checkout(self) File "sqlalchemy/pool/base.py", line 1263, in _checkout fairy = _ConnectionRecord.checkout(pool) File "sqlalchemy/pool/base.py", line 712, in checkout rec = pool._do_get() File "sqlalchemy/pool/impl.py", line 179, in _do_get with util.safe_reraise(): File "sqlalchemy/util/langhelpers.py", line 146, in __exit__ raise exc_value.with_traceback(exc_tb) File "sqlalchemy/pool/impl.py", line 177, in _do_get return self._create_connection() File "sqlalchemy/pool/base.py", line 390, in _create_connection return _ConnectionRecord(self) File "sqlalchemy/pool/base.py", line 674, in __init__ self.__connect() File "sqlalchemy/pool/base.py", line 900, in __connect with util.safe_reraise(): File "sqlalchemy/util/langhelpers.py", line 146, in __exit__ raise exc_value.with_traceback(exc_tb) File "sqlalchemy/pool/base.py", line 896, in __connect self.dbapi_connection = connection = pool._invoke_creator(self) File "sqlalchemy/engine/create.py", line 643, in connect return dialect.connect(*cargs, **cparams) File "sqlalchemy/engine/default.py", line 620, in connect return self.loaded_dbapi.connect(*cargs, **cparams) OperationalError: (sqlite3.OperationalError) unable to open database file (Background on this error at: https://sqlalche.me/e/20/e3q8) File "galaxy/jobs/runners/__init__.py", line 677, in _finish_or_resubmit_job job_wrapper.finish( File "galaxy/jobs/__init__.py", line 2070, in finish task_wrapper.delay() File "celery/canvas.py", line 353, in delay return self.apply_async(partial_args, partial_kwargs) File "celery/canvas.py", line 400, in apply_async return _apply(args, kwargs, **options) File "celery/app/task.py", line 594, in apply_async return app.send_task( File "celery/app/base.py", line 801, in send_task amqp.send_task_message(P, name, message, **options) File "celery/app/amqp.py", line 518, in send_task_message ret = producer.publish( File "kombu/messaging.py", line 186, in publish return _publish( File "kombu/connection.py", line 553, in _ensured with self._reraise_as_library_errors(): File "contextlib.py", line 155, in __exit__ self.gen.throw(typ, value, traceback) File "kombu/connection.py", line 476, in _reraise_as_library_errors raise ConnectionError(str(exc)) from exc ``` The issue was that we aren't setting `GALAXY_CONFIG_FILE` if the job and workflow handlers are started with `-c path/to/galaxy.yml`, and the celery config heuristic isn't looking at sys.argv.

Bumps [dompurify](https://github.com/cure53/DOMPurify) from 3.0.6 to 3.1.3. - [Release notes](https://github.com/cure53/DOMPurify/releases) - [Commits](cure53/DOMPurify@3.0.6...3.1.3) --- updated-dependencies: - dependency-name: dompurify dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>

This should fix #18822

Fixes #18782 / https://sentry.galaxyproject.org/share/issue/dca23c8d10e743509648cf6092a1dc15/: ``` Message Error closing cursor Stack Trace Newest ProgrammingError: named cursor isn't valid anymore File "sqlalchemy/engine/cursor.py", line 1255, in fetchmany new = dbapi_cursor.fetchmany(size - lb) ProgrammingError: named cursor isn't valid anymore File "sqlalchemy/engine/base.py", line 2213, in _safe_close_cursor cursor.close() ```

[24.1] Start job handler only after building InteractiveToolManager

[24.1] Fix ``named cursor is not valid anymore``

…started_via_main_py [24.1] Fix wrong celery_app config on job and workflow handlers

The stack trace is this: ``` __ TestSimplePriorInstallation.test_0005_create_convert_repository[chromium] ___ self = <tool_shed.test.functional.test_1170_prior_installation_required.TestSimplePriorInstallation object at 0x7f87c33834c0> def test_0005_create_convert_repository(self): """Create and populate convert_chars_0150.""" global running_standalone category = self.create_category(name=category_name, description=category_description) > self.login(email=common.test_user_1_email, username=common.test_user_1_name) lib/tool_shed/test/functional/test_1170_prior_installation_required.py:50: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ lib/tool_shed/test/base/twilltestcase.py:776: in login previously_created, username_taken, invalid_username = self.create( lib/tool_shed/test/base/twilltestcase.py:714: in create self.visit_url("/user/create", params) lib/tool_shed/test/base/twilltestcase.py:850: in visit_url return self._browser.visit_url(url, allowed_codes=allowed_codes) lib/tool_shed/test/base/playwrightbrowser.py:31: in visit_url response = self._page.goto(url) .venv/lib/python3.8/site-packages/playwright/sync_api/_generated.py:8851: in goto self._sync( .venv/lib/python3.8/site-packages/playwright/_impl/_page.py:524: in goto return await self._main_frame.goto(**locals_to_params(locals())) .venv/lib/python3.8/site-packages/playwright/_impl/_frame.py:145: in goto await self._channel.send("goto", locals_to_params(locals())) .venv/lib/python3.8/site-packages/playwright/_impl/_connection.py:59: in send return await self._connection.wrap_api_call( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <playwright._impl._connection.Connection object at 0x7f873c8bebe0> cb = <function Channel.send.<locals>.<lambda> at 0x7f87af067550> is_internal = False async def wrap_api_call( self, cb: Callable[[], Any], is_internal: bool = False ) -> Any: if self._api_zone.get(): return await cb() task = asyncio.current_task(self._loop) st: List[inspect.FrameInfo] = getattr(task, "__pw_stack__", inspect.stack()) parsed_st = _extract_stack_trace_information_from_stack(st, is_internal) self._api_zone.set(parsed_st) try: return await cb() except Exception as error: > raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None E playwright._impl._errors.Error: Page.goto: Navigation to "http://127.0.0.1:8723/user/create?cntrller=user&use_panels=False" is interrupted by another navigation to "http://127.0.0.1:8723/logout_success" E Call log: E navigating to "http://127.0.0.1:8723/user/create?cntrller=user&use_panels=False", waiting until "load" ```

Modelling for Test Assertions

Unfortunately, the OpenAPI schema of Discourse (https://docs.discourse.org/openapi.json) does not specify the proper type for posts in the "/search.json" response. There is at least one case where the post name is null (https://help.galaxyproject.org//search.json?q=tags:circos+tool-help%20status:solved) so let's play safe.

…ts_tests [24.1] Maybe fix flaky navigation v2 ts tests

[24.1] Make all fields optional for HelpForumPost

[24.1] prevent "missing refresh_token" errors by supporting <extra_scopes> also with Keycloak backend

And remove unused prop

So it can be used in other contexts as well.

Displays the name of a remote source by providing the URI

Make sure it also matches file sources without ID (stock file sources like FTP, etc.)

… function

To conditionally show file name input and include data checkbox

It will be handled now by the wizard logic

Extract logic into a service module and Integrate it into the wizard process.

The body default value will remind you to define an appropriate body for the step.

Co-authored-by: Alireza Heidari <itisalirh@gmail.com>

If we go back and change the export format after selecting some of the other options with a the previous format, we should ensure the options are compatible with that new format.

Fixes https://sentry.galaxyproject.org/share/issue/5b8d495edd6545278d4d93d0ed69ac3a/: ``` Message Uncaught exception in exposed API method: Stack Trace Newest gaierror: [Errno -2] Name or service not known File "urllib3/connection.py", line 174, in _new_conn conn = connection.create_connection( File "urllib3/util/connection.py", line 72, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "socket.py", line 962, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f8e35406450>: Failed to establish a new connection: [Errno -2] Name or service not known File "urllib3/connectionpool.py", line 715, in urlopen httplib_response = self._make_request( File "urllib3/connectionpool.py", line 416, in _make_request conn.request(method, url, **httplib_request_kw) File "urllib3/connection.py", line 244, in request super(HTTPConnection, self).request(method, url, body=body, headers=headers) File "http/client.py", line 1286, in request self._send_request(method, url, body, headers, encode_chunked) File "http/client.py", line 1332, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "http/client.py", line 1281, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "http/client.py", line 1041, in _send_output self.send(msg) File "http/client.py", line 979, in send self.connect() File "urllib3/connection.py", line 205, in connect conn = self._new_conn() File "urllib3/connection.py", line 186, in _new_conn raise NewConnectionError( MaxRetryError: HTTPConnectionPool(host='https', port=80): Max retries exceeded with url: //training.galaxyproject.org/training-material//api/ga4gh/trs/v2/tools/variant-analysis-microbial-variants/versions/microbial_variant_calling/GALAXY/descriptor (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8e35406450>: Failed to establish a new connection: [Errno -2] Name or service not known')) File "requests/adapters.py", line 486, in send resp = conn.urlopen( File "urllib3/connectionpool.py", line 799, in urlopen retries = retries.increment( File "urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) ConnectionError: HTTPConnectionPool(host='https', port=80): Max retries exceeded with url: //training.galaxyproject.org/training-material//api/ga4gh/trs/v2/tools/variant-analysis-microbial-variants/versions/microbial_variant_calling/GALAXY/descriptor (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8e35406450>: Failed to establish a new connection: [Errno -2] Name or service not known')) File "galaxy/web/framework/decorators.py", line 346, in decorator rval = func(self, trans, *args, **kwargs) File "galaxy/webapps/galaxy/api/workflows.py", line 261, in create archive_data = server.get_version_descriptor(trs_tool_id, trs_version_id) File "galaxy/workflow/trs_proxy.py", line 115, in get_version_descriptor return self._get(trs_api_url)["content"] File "galaxy/workflow/trs_proxy.py", line 126, in _get response = requests.get(url, params=params, timeout=DEFAULT_SOCKET_TIMEOUT) File "galaxy/util/requests.py", line 29, in wrapper rval = f(*args, **kwargs) File "requests/api.py", line 73, in get return request("get", url, params=params, **kwargs) File "requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) ``` which happened for https://usegalaxy.org/workflows/trs_import?run_form=true&trs_url=http%3A%2F%2Flocalhost%3A4000%2F%2Ftraining-material%2F%2Fapi%2Fga4gh%2Ftrs%2Fv2%2Ftools%2Fvariant-analysis-microbial-variants%2Fversions%2Fmicrobial_variant_calling"

…ws"`

…earch query

[24.1] Fix alert handling in workflow label handler

…client/express-4.21.0 Bump express from 4.19.2 to 4.21.0 in /client

Co-authored-by: Marius van den Beek <m.vandenbeek@gmail.com>

That happens if the user stops a job before it's queued (or if we have a job rule preventing queuing). Fixes https://sentry.galaxyproject.org/share/issue/8b462ee0d50a4de8bb9ca2c50f049b42/: ``` Message Could not recover job metrics Stack Trace Newest JobMappingException: This tool is currently being beta-tested and your account has not been given access. File "galaxy/jobs/mapper.py", line 214, in __handle_rule job_destination = self.__invoke_expand_function( File "galaxy/jobs/mapper.py", line 125, in __invoke_expand_function return expand_function(**actual_args) File "tpv/rules/gateway.py", line 51, in map_tool_to_destination return ACTIVE_DESTINATION_MAPPER.map_to_destination(app, tool, user, job, job_wrapper, resource_params, File "tpv/core/mapper.py", line 168, in map_to_destination evaluated_entity = self.match_combine_evaluate_entities(context, tool, user) File "tpv/core/mapper.py", line 138, in match_combine_evaluate_entities evaluated_entity = combined_entity.evaluate_rules(context) File "tpv/core/entities.py", line 489, in evaluate_rules rule = rule.evaluate(context) File "tpv/core/entities.py", line 750, in evaluate raise JobMappingException( JobMappingException: This tool is currently being beta-tested and your account has not been given access. File "galaxy/jobs/handler.py", line 719, in __verify_job_ready job_destination = job_wrapper.job_destination File "galaxy/jobs/__init__.py", line 2576, in job_destination return self.job_runner_mapper.get_job_destination(self.params) File "galaxy/jobs/mapper.py", line 276, in get_job_destination return self.__cache_job_destination(params) File "galaxy/jobs/mapper.py", line 257, in __cache_job_destination self.cached_job_destination = self.__determine_job_destination( File "galaxy/jobs/mapper.py", line 241, in __determine_job_destination job_destination = self.__handle_dynamic_job_destination(raw_job_destination) File "galaxy/jobs/mapper.py", line 204, in __handle_dynamic_job_destination return self.__handle_rule(expand_function, destination) File "galaxy/jobs/mapper.py", line 219, in __handle_rule raise e File "galaxy/jobs/mapper.py", line 208, in __handle_rule job_destination = self.__invoke_expand_function(rule_function, destination) File "galaxy/jobs/mapper.py", line 125, in __invoke_expand_function return expand_function(**actual_args) File "tpv/rules/gateway.py", line 51, in map_tool_to_destination return ACTIVE_DESTINATION_MAPPER.map_to_destination(app, tool, user, job, job_wrapper, resource_params, File "tpv/core/mapper.py", line 168, in map_to_destination evaluated_entity = self.match_combine_evaluate_entities(context, tool, user) File "tpv/core/mapper.py", line 138, in match_combine_evaluate_entities evaluated_entity = combined_entity.evaluate_rules(context) File "tpv/core/entities.py", line 489, in evaluate_rules rule = rule.evaluate(context) File "tpv/core/entities.py", line 750, in evaluate raise JobMappingException( ObjectNotFound: objectstore, _call_method failed: _get_filename on <galaxy.model.Job(60599388) at 0x7fcaec5b6450>, kwargs: {'base_dir': 'job_work', 'dir_only': True, 'obj_dir': True} File "galaxy/jobs/__init__.py", line 2163, in _collect_metrics job_metrics_directory = self.working_directory File "galaxy/jobs/__init__.py", line 1317, in working_directory self.__working_directory = self.app.object_store.get_filename( File "galaxy/objectstore/__init__.py", line 475, in get_filename return self._invoke("get_filename", obj, **kwargs) File "galaxy/objectstore/__init__.py", line 451, in _invoke return self.__getattribute__(f"_{delegate}")(obj=obj, **kwargs) File "galaxy/objectstore/__init__.py", line 1009, in _get_filename return self._call_method("_get_filename", obj, ObjectNotFound, True, **kwargs) File "galaxy/objectstore/__init__.py", line 1281, in _call_method raise default( ``` Minor follow up to #18809

Fixes https://sentry.galaxyproject.org/share/issue/c1c4c9b0ab324b4ea4f98c3b8c402d21/: ``` Message Uncaught Exception Stack Trace Newest FileNotFoundError: [Errno 2] No such file or directory: '' File "galaxy/web/framework/middleware/error.py", line 167, in __call__ app_iter = self.application(environ, sr_checker) File "galaxy/web/framework/middleware/statsd.py", line 29, in __call__ req = self.application(environ, start_response) File "/cvmfs/main.galaxyproject.org/venv/lib/python3.11/site-packages/paste/httpexceptions.py", line 635, in __call__ return self.application(environ, start_response) File "galaxy/web/framework/base.py", line 176, in __call__ return self.handle_request(request_id, path_info, environ, start_response) File "galaxy/web/framework/base.py", line 271, in handle_request body = method(trans, **kwargs) File "galaxy/webapps/galaxy/controllers/dataset.py", line 155, in display display_data, headers = data.datatype.display_data( File "galaxy/datatypes/tabular.py", line 226, in display_data chunk=self.get_chunk(trans, dataset, 0), File "galaxy/datatypes/tabular.py", line 149, in get_chunk ck_data, last_read = self._read_chunk(trans, dataset, offset, ck_size) File "galaxy/datatypes/tabular.py", line 159, in _read_chunk with compression_utils.get_fileobj(dataset.get_file_name()) as f: File "galaxy/util/compression_utils.py", line 79, in get_fileobj return get_fileobj_raw(filename, mode, compressed_formats)[1] File "galaxy/util/compression_utils.py", line 139, in get_fileobj_raw return compressed_format, open(filename, mode, encoding="utf-8") ```

[24.1] Extend on disk checks to running, queued and error states

…s_if_working_directory_hasnt_been_created_yet [24.1] Skip metric collection if job working directory doesn't exist

Fix transiently failing selenium tooltip issues

A [quota update via API](https://github.com/galaxyproject/galaxy/blob/67b4a69c1b36eec4dc5871c5eeb97b6bdea8a288/lib/galaxy/webapps/galaxy/services/quotas.py#L85) requires name or quota to be set, but the [manager](https://github.com/galaxyproject/galaxy/blob/67b4a69c1b36eec4dc5871c5eeb97b6bdea8a288/lib/galaxy/managers/quotas.py#L119) requires a name to be set. If one wants to change just the description of a quota the used select statement gets the quota itself and the elif branch hits resulting in an exception. With the change one can change just the description, but one still needs to specify the name of the quota. We could change this such that also in the manager the name of the quota needs to be specified.

Co-authored-by: Marius van den Beek <m.vandenbeek@gmail.com>

[24.1] Tighten TRS url check

Co-authored-by: Ahmed Hamid Awan <qe66653@umbc.edu>

…zard Add wizard-like export view for workflow invocations

@jennaj

Fixes a bug reported by @jennaj where using an input dataset from a history import (which contains implicitly converted datasets in discarded state) in an input that would require that exact same implicit conversion would fail with `Job output deleted by user before job completed`. This will make the whole implicit coversion system much more robust, since we didn't even exclude datasets in error state, so as soon as one conversion failed ... you'd be stuck.

[24.0] Ignore converted datasets in invalid input states

…-update [24.1] Allow to change only the description of a quota

Add docs on indexing for filter column

…_case_xml Improvements to parameter models for test case inputs

[24.1] Fix Workflow index bookmark filter

stores The problem is that `object_store.is_private` would check in which object store the dataset is stored, and that fails if we haven't written to the object store yet (which is why extended metadata is a workaround). Fixes #17208

…_step_workflow Fix merge conflict in advanced step workflow

Followup to #18846

[24.0] Fix discovered outputs with directory metadata and distributed object

[24.1] Fix check dataset check

avoiding injecting it as innerHTML and executing contents.

and enforces a unique constraint. Also incldues a script to update purged status for affected datasets that are not in any object store.

Format dev to fix linting.

…lboxFiltersSeleniumIntegration::test_toolbox_filters` Fixes #18832

Co-authored-by: Marius van den Beek <m.vandenbeek@gmail.com>

by running `make update-dependencies`.

fix link to public-server mailing list

[24.1] Retry container monitor POST if it fails (don't assume it succeeded)

Update Python dependencies

Fixes #18855 (comment). I don't think this had any effect in the previous version since we'd always set the state to final_job_state anyway.

and make sure we include the parent_hda when exporting histories. Fixes https://sentry.galaxyproject.org/share/issue/20ee27f675ef476588bbe8ff273eaee7/: ``` KeyError: '31e7840b5aedca43523bfcc8e2e9e56f' File "galaxy/jobs/__init__.py", line 2039, in finish task_wrapper = self.tool.exec_after_process( File "galaxy/tools/__init__.py", line 3100, in exec_after_process JobImportHistoryArchiveWrapper(self.app, job.id).cleanup_after_job() File "galaxy/tools/imp_exp/__init__.py", line 81, in cleanup_after_job model_store.perform_import(new_history, job=job, new_history=True) File "galaxy/model/store/__init__.py", line 416, in perform_import self._import_implicit_dataset_conversions(object_import_tracker) File "galaxy/model/store/__init__.py", line 1280, in _import_implicit_dataset_conversions idc.parent_hda = object_import_tracker.hdas_by_key[idc_attrs["parent_hda"]] ```

…taset_state [24.0] Fix wrong final state when init_from is used

Fix flaky `test/integration_selenium/test_toolbox_filters.py::TestToolboxFiltersSeleniumIntegration::test_toolbox_filters`

[24.1] Raise MessageException instead of assertions on rerun problems

…it_hda_not_serialized [24.1] Fix history import when parent_hda not serialized

Fixes https://sentry.galaxyproject.org/share/issue/d1cad9f916e8466f844ea3b49f52afcb/: ``` AttributeError: 'NoneType' object has no attribute 'get' File "galaxy/web/framework/middleware/error.py", line 167, in __call__ app_iter = self.application(environ, sr_checker) File "galaxy/web/framework/middleware/statsd.py", line 29, in __call__ req = self.application(environ, start_response) File "/cvmfs/main.galaxyproject.org/venv/lib/python3.11/site-packages/paste/httpexceptions.py", line 635, in __call__ return self.application(environ, start_response) File "galaxy/web/framework/base.py", line 176, in __call__ return self.handle_request(request_id, path_info, environ, start_response) File "galaxy/web/framework/base.py", line 271, in handle_request body = method(trans, **kwargs) File "galaxy/web/framework/decorators.py", line 74, in call_and_format rval = func(self, trans, *args, **kwargs) File "galaxy/webapps/galaxy/controllers/workflow.py", line 263, in load_workflow return workflow_contents_manager.workflow_to_dict(trans, stored, style="editor", version=version) File "galaxy/managers/workflows.py", line 926, in workflow_to_dict wf_dict = self._workflow_to_dict_editor(trans, stored, workflow) File "galaxy/managers/workflows.py", line 1239, in _workflow_to_dict_editor self.__set_default_label(step, module, step.tool_inputs) File "galaxy/managers/workflows.py", line 1958, in __set_default_label default_label = new_state.get("name") ``` and https://sentry.galaxyproject.org/share/issue/92e8f30a288341e7b219e2c7c91390da/: ``` AttributeError: 'NoneType' object has no attribute 'get' File "starlette/applications.py", line 123, in __call__ await self.middleware_stack(scope, receive, send) File "starlette/middleware/errors.py", line 186, in __call__ raise exc File "starlette/middleware/errors.py", line 164, in __call__ await self.app(scope, receive, _send) File "starlette_context/middleware/raw_middleware.py", line 92, in __call__ await self.app(scope, receive, send_wrapper) File "starlette/middleware/base.py", line 189, in __call__ with collapse_excgroups(): File "contextlib.py", line 155, in __exit__ self.gen.throw(typ, value, traceback) File "starlette/_utils.py", line 93, in collapse_excgroups raise exc File "starlette/middleware/base.py", line 191, in __call__ response = await self.dispatch_func(request, call_next) File "galaxy/webapps/galaxy/fast_app.py", line 109, in add_x_frame_options response = await call_next(request) File "starlette/middleware/base.py", line 165, in call_next raise app_exc File "starlette/middleware/base.py", line 151, in coro await self.app(scope, receive_or_disconnect, send_no_error) File "starlette/middleware/exceptions.py", line 65, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "starlette/routing.py", line 756, in __call__ await self.middleware_stack(scope, receive, send) File "starlette/routing.py", line 776, in app await route.handle(scope, receive, send) File "starlette/routing.py", line 297, in handle await self.app(scope, receive, send) File "starlette/routing.py", line 77, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "starlette/routing.py", line 72, in app response = await func(request) File "fastapi/routing.py", line 278, in app raw_response = await run_endpoint_function( File "fastapi/routing.py", line 193, in run_endpoint_function return await run_in_threadpool(dependant.call, **values) File "starlette/concurrency.py", line 42, in run_in_threadpool return await anyio.to_thread.run_sync(func, *args) File "anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread return await future File "anyio/_backends/_asyncio.py", line 851, in run result = context.run(func, *args) File "galaxy/webapps/galaxy/api/workflows.py", line 1159, in show_workflow return self.service.show_workflow(trans, workflow_id, instance, legacy, version) File "galaxy/webapps/galaxy/services/workflows.py", line 243, in show_workflow **self._workflow_contents_manager.workflow_to_dict(trans, stored_workflow, style=style, version=version) File "galaxy/managers/workflows.py", line 930, in workflow_to_dict wf_dict = self._workflow_to_dict_instance(trans, stored, workflow=workflow, legacy=False) File "galaxy/managers/workflows.py", line 1663, in _workflow_to_dict_instance step_label = step.label or step.tool_inputs.get("name") ``` This is for https://usegalaxy.org/workflows/export?id=ea97d3a23e6d38e6

We don't need a ModelMapping instance here; without it we can easily write unit tests.

Improves readability of test code

Fix typo Co-authored-by: Marius van den Beek <m.vandenbeek@gmail.com>

Add unique constraint, set not nullable for user-id, role-id

Add unique constraint, set not nullable for user-id, group-id

Add unique constraint, set not nullable for group-id, role-id

In the previous version, we only passed NEW associations, which were ADDED to the current associations. The new version implements a true UPDATE operation, which replaces the existing set of associations with the provided set of associations.

Co-authored-by: Marius van den Beek <m.vandenbeek@gmail.com>

[24.1] Fix loading very old workflows with data inputs

[24.1] Fix data_column ref to nested collection

…client/dompurify-3.1.3 Bump dompurify from 3.0.6 to 3.1.3 in /client

Add unique constraints to the email and username fields in the galaxy_user table

Backend handling of setting user-role, user-group, and group-role associations

When upgrading back to a valid revision, upgrade to head, not the revision under test: we need the latest version of the model definition in the database for the test to cleanup correctly.

Fix migration data fixes tests

- Do not explicitly import fixtures (a little duplication is safer/cleaner) - Move dangerous fixture into module to prevent accidental usage from other modules - Ensure only one database is used for all tests in module

Implement Pydantic model for workflow test format.

that support pagination to 10000. Clients can always request more data via offset requests, but since this is not a streaming API we need to build the response in memory before we can start sending.

data_column params: offer same columns with and without use_header_names

[24.1] Limit max number of items in dataproviders

# Conflicts: # lib/galaxy/version.py # packages/app/HISTORY.rst # packages/app/setup.cfg # packages/auth/HISTORY.rst # packages/auth/setup.cfg # packages/config/HISTORY.rst # packages/config/setup.cfg # packages/data/HISTORY.rst # packages/data/setup.cfg # packages/files/HISTORY.rst # packages/files/setup.cfg # packages/job_execution/HISTORY.rst # packages/job_execution/setup.cfg # packages/job_metrics/HISTORY.rst # packages/job_metrics/setup.cfg # packages/navigation/HISTORY.rst # packages/navigation/setup.cfg # packages/objectstore/HISTORY.rst # packages/objectstore/setup.cfg # packages/schema/HISTORY.rst # packages/schema/setup.cfg # packages/selenium/HISTORY.rst # packages/selenium/setup.cfg # packages/test_api/HISTORY.rst # packages/test_api/setup.cfg # packages/test_base/HISTORY.rst # packages/test_base/setup.cfg # packages/test_driver/HISTORY.rst # packages/test_driver/setup.cfg # packages/tool_shed/HISTORY.rst # packages/tool_shed/setup.cfg # packages/tool_util/HISTORY.rst # packages/tool_util/setup.cfg # packages/tours/HISTORY.rst # packages/tours/setup.cfg # packages/util/HISTORY.rst # packages/util/setup.cfg # packages/web_apps/HISTORY.rst # packages/web_apps/setup.cfg # packages/web_framework/HISTORY.rst # packages/web_framework/setup.cfg # packages/web_stack/HISTORY.rst # packages/web_stack/setup.cfg

Refactor migration testing setup code

Surprised (or not actually ...) that mypy didn't catch this.

- Allow class: File and class: Collection on file and collection test descriptions. - Allow collection_type: <type> in collection test descriptions. Implement this. - Update test workflows to reflect these two things as best practices.

Raise exception if CompressedFile used on incompatible file

Enable ``ignore-without-code`` mypy error code

Improvements to Planemo test format based on feedback/discussions.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[pull] dev from galaxyproject:dev #948

[pull] dev from galaxyproject:dev #948

Commits on Sep 12, 2024

Commits on Sep 13, 2024

Commits on Sep 14, 2024

Commits on Sep 16, 2024

Commits on Sep 17, 2024

Commits on Sep 18, 2024

Commits on Sep 19, 2024

Commits on Sep 20, 2024

Commits on Sep 21, 2024

Commits on Sep 22, 2024

Commits on Sep 23, 2024

Commits on Sep 24, 2024

Commits on Sep 25, 2024

Commits on Sep 26, 2024

Commits on Sep 27, 2024

[pull] dev from galaxyproject:dev #948

Are you sure you want to change the base?

[pull] dev from galaxyproject:dev #948

Commits on Sep 12, 2024

Commits on Sep 13, 2024

Commits on Sep 14, 2024

Commits on Sep 16, 2024

Commits on Sep 17, 2024

Commits on Sep 18, 2024

Commits on Sep 19, 2024

Commits on Sep 20, 2024

Commits on Sep 21, 2024

Commits on Sep 22, 2024

Commits on Sep 23, 2024

Commits on Sep 24, 2024

Commits on Sep 25, 2024

Commits on Sep 26, 2024

Commits on Sep 27, 2024