DM-47790: Switch to vo-models for nearly all UWS responses

[rra]

Finish the conversion to using vo-models to generate nearly all UWS responses. Templating remains only to generate the error VOTable document.

Due to limitations in pydantic-xml plus the complexity of the type system, clients will have to provide an additional argument to the UWS configuration specifying the qualified type of the vo_models.uws.JobSummary class used to represent jobs. Clients will also have to implement a new to_xml_model method on their ParametersModel class.

[rra]

This is still a bit awkward because internally list[UWSJobParameter] is still used because of the database schema. Once this is switched to use Wobbly, I'll be able to eliminate that in favor of storing just the API model in the database, and then this should be more straightforward (although alas still requiring three separate models).

[stvoutsin]

All looks great to me.

Initially at first glance it felt like there may be some increased complexity and burden of having to manage multiple models to represent the same params and keep them in sync, but ultimately they all seem necessary so I can't really think of an alternative pattern that would be better.

I like the separation of concerns between worker & API models, the API vs XML seems necessary I guess though that's the bit that in an ideal world we can merge if it weren't for the obscuring that you noted.

[stvoutsin]

Would it make sense to specify the schema for the response using the JobSummary model here (for the Openapi spec):

@uws_router.get(
    "/{job_id}",
    response_model=JobSummary,
    description="Get details of a specific job.",
    responses={
        200: {
            "description": "Successful Response",
            "content": {
                "application/xml": {
                    "schema": JobSummary.model_json_schema(),
                },
            },
        }
    },
    summary="Job details",
)

[stvoutsin]

After re-reading your documentation I guess this section implies that we shouldn't because the parameters would be obscured in the output?

Unfortunately, the same model cannot be used for 1 and 3 even for simple applications because the XML model requires additional structure that obscures the parameters and should not be included in the JSON API model

[rra]

Oh, huh, I was assuming that this would give you a schema that was only suitable for JSON and not for XML, but it looks like the XML isn't that horrible. I'll go ahead and do this; at the least, it doesn't seem like it would hurt. Thanks!

[stvoutsin]

Have you considered using something like tenacity (https://tenacity.readthedocs.io/en/latest/) to handle & hide the backoff and retry logic?

[rra]

I'm hoping to get rid of this retry loop entirely by doing something a bit more sophisticated, since we should at least block on completion of the arq job before we start waiting for the database update to happen. If I end up still needing a retry loop in Wobbly after that, I'll take a look. Thanks!

[rra]

Thanks for the review! It will be a bit simpler once we switch to Wobbly, although the separate XML model is still needed because it's the canonical representation of the flattened string view instead of the structured data view. But the list of UWS parameters goes away, and that makes everything a bit simpler. (Propagation of types through this code is really challenging, though.)