Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update binding modelling in client state #986

Merged
merged 69 commits into from
Mar 20, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
e2a6f0b
Create a binding store to replace the resource config store
kiahna-tucker Feb 16, 2024
805cf79
Use a UUID as a unique binding identifier
kiahna-tucker Feb 17, 2024
5a90c26
Update binding prefill comment in binding store hydrator
kiahna-tucker Feb 17, 2024
491fea5
Add resource schema-related state to binding store
kiahna-tucker Feb 17, 2024
b7ed959
Add resource schema input argument to updateResourceConfig action
kiahna-tucker Feb 20, 2024
de3ef57
Add resource config dictionary to binding store state
kiahna-tucker Feb 20, 2024
5f3b480
Hydrate resource config-related state in binding store
kiahna-tucker Feb 20, 2024
6574814
Add current collection-related state to binding store
kiahna-tucker Feb 20, 2024
25b8c65
Add resource config-related state to binding store
kiahna-tucker Feb 21, 2024
90567f7
Rename binding ID-related properties
kiahna-tucker Feb 21, 2024
300ebf2
Remove a single binding from the binding selector
kiahna-tucker Feb 21, 2024
a45661a
Index the resource config dictionary to set the current binding
kiahna-tucker Feb 21, 2024
7f466ac
Integrate the setResourceConfig action from the binding store
kiahna-tucker Feb 21, 2024
985e0d1
Merge branch 'main' into kiahna-tucker/refactor/bindings-data-model
kiahna-tucker Feb 22, 2024
7a32aeb
Evaluate discovered bindings in binding store
kiahna-tucker Feb 22, 2024
90edb7b
Add discovered collections array to binding store
kiahna-tucker Feb 22, 2024
0abae56
Add restricted discovered collections-related state to binding store
kiahna-tucker Feb 22, 2024
a71a892
Add service updated required-related state to binding store
kiahna-tucker Feb 22, 2024
c688df2
Remove all resource config store hydration-related logic
kiahna-tucker Feb 22, 2024
73aff9a
Remove multiple bindings from the binding selector
kiahna-tucker Feb 28, 2024
fe7e6f3
Toggle the binding disabled flag
kiahna-tucker Feb 28, 2024
421ed71
Extend error-related state of binding store
kiahna-tucker Feb 28, 2024
7fcd9e7
Add backfill-related state to the binding store
kiahna-tucker Feb 28, 2024
0b3b0cd
Update data extraction from resource config dictionary in generateTas…
kiahna-tucker Feb 29, 2024
f50d5b1
Prefill backfill-related binding store state
kiahna-tucker Feb 29, 2024
d812e46
Remove discovered bindings when catalog name changes
kiahna-tucker Feb 29, 2024
3a11588
Remove resource config store
kiahna-tucker Feb 29, 2024
ff505fa
Merge branch 'main' into kiahna-tucker/refactor/bindings-data-model
kiahna-tucker Feb 29, 2024
563dda8
Move recommend fields-related state to binding store
kiahna-tucker Mar 1, 2024
61898bc
Initialize field selection state in binding store
kiahna-tucker Mar 1, 2024
da67287
Move remaining field selection state into binding store
kiahna-tucker Mar 1, 2024
80ed495
Temper frequency of field selection state initialization
kiahna-tucker Mar 4, 2024
0424156
Add binding index to resource config metadata object
kiahna-tucker Mar 4, 2024
f8b4e6c
Update binding pruning logic in generateTaskSpec to be compatible wit…
kiahna-tucker Mar 5, 2024
f5887aa
Return a distinct numeric value if binding index cannot be determined
kiahna-tucker Mar 6, 2024
8ce16d7
Merge branch 'main' into kiahna-tucker/refactor/bindings-data-model
kiahna-tucker Mar 7, 2024
3941e26
Use binding UUID order to determine binding index
kiahna-tucker Mar 8, 2024
0a8205d
Set server update required flag when resource config order changes
kiahna-tucker Mar 8, 2024
a9a8dff
Remove legacy getBindingIndex util
kiahna-tucker Mar 8, 2024
6839645
Merge branch 'main' into kiahna-tucker/refactor/bindings-data-model
kiahna-tucker Mar 8, 2024
3f17ff0
Move full source state into binding store
kiahna-tucker Mar 8, 2024
bd51702
Merge branch 'main' into kiahna-tucker/refactor/bindings-data-model
kiahna-tucker Mar 8, 2024
f626c27
Correct binding hydrator merge error
kiahna-tucker Mar 8, 2024
7f79387
Initialize time travel state during hydration
kiahna-tucker Mar 8, 2024
5922d73
Initialize time travel state for materializations only
kiahna-tucker Mar 8, 2024
9913cfd
Update binding editor store initialization test
kiahna-tucker Mar 11, 2024
3bb1a45
Key time travel state off of binding UUID
kiahna-tucker Mar 11, 2024
12723d9
Reset time travel state for binding when form cleared
kiahna-tucker Mar 11, 2024
c1fddd0
Remove commented code fragment from workflow utils
kiahna-tucker Mar 11, 2024
8d727a9
Use binding UUID as reference for manual backfill
kiahna-tucker Mar 11, 2024
86b31e9
Split setResourceConfig into two separate actions
kiahna-tucker Mar 11, 2024
397bf46
Merge branch 'main' into kiahna-tucker/refactor/bindings-data-model
kiahna-tucker Mar 11, 2024
02617af
Rename binding store hooks which index the resource config object
kiahna-tucker Mar 11, 2024
62ee312
Merge branch 'main' into kiahna-tucker/refactor/bindings-data-model
kiahna-tucker Mar 13, 2024
7ff1a07
Expand the state that is reset in evaluateDiscoveredBindings
kiahna-tucker Mar 13, 2024
a9dfe2a
Preserve the active binding state property by default when state reset
kiahna-tucker Mar 13, 2024
6397aac
Comment the relationship between notable binding and resource config …
kiahna-tucker Mar 13, 2024
fbaaaa6
Set the config SectionError property in ResourceConfigErrors
kiahna-tucker Mar 13, 2024
a890cb0
Use binding index ref in generateTaskSpec if drafted bindings exist
kiahna-tucker Mar 14, 2024
1b989e3
Narrow type annotation for getBindingIndex array argument
kiahna-tucker Mar 19, 2024
1111c1b
Adjust draftedBindingIndex editor store hook arguments and selector
kiahna-tucker Mar 19, 2024
d6a7284
Remove instance of getBindingIndex from server update required hook
kiahna-tucker Mar 19, 2024
2d733e6
Remove ternary operator from initializeBinding function
kiahna-tucker Mar 19, 2024
aaafb77
Remove addressed todo comment
kiahna-tucker Mar 19, 2024
3107556
Merge branch 'main' into kiahna-tucker/refactor/bindings-data-model
kiahna-tucker Mar 19, 2024
83b398c
Clean up server update required hook
kiahna-tucker Mar 20, 2024
99fab64
Extract getBindingIndex filter into independent function
kiahna-tucker Mar 20, 2024
8a24cb7
Merge branch 'main' into kiahna-tucker/refactor/bindings-data-model
kiahna-tucker Mar 20, 2024
fde4613
Reset binding state when derivation create dialog closed
kiahna-tucker Mar 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/api/hydration.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@ import {
LiveSpecsExt_MaterializeCapture,
} from 'hooks/useLiveSpecsExt';
import {
TABLES,
handleFailure,
handleSuccess,
supabaseClient,
supabaseRetry,
TABLES,
} from 'services/supabase';
import { Entity } from 'types';

Expand Down Expand Up @@ -88,7 +88,7 @@ export const getLiveSpecsById_writesTo = async (
typeof liveSpecId === 'string' ? [liveSpecId] : liveSpecId
),
'getLiveSpecsById_writesTo'
).then(handleSuccess<LiveSpecsExt_MaterializeCapture>, handleFailure);
).then(handleSuccess<LiveSpecsExt_MaterializeCapture[]>, handleFailure);

return data;
};
10 changes: 6 additions & 4 deletions src/components/capture/Create/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,13 @@ import useDraftSpecs from 'hooks/useDraftSpecs';
import usePageTitle from 'hooks/usePageTitle';
import { useCallback, useEffect, useMemo, useState } from 'react';
import { CustomEvents } from 'services/types';
import BindingHydrator from 'stores/Binding/Hydrator';
import {
useDetailsForm_connectorImage,
useDetailsForm_entityNameChanged,
} from 'stores/DetailsForm/hooks';
import { DetailsFormHydrator } from 'stores/DetailsForm/Hydrator';
import { EndpointConfigHydrator } from 'stores/EndpointConfig/Hydrator';
import ResourceConfigHydrator from 'stores/ResourceConfig/Hydrator';
import { MAX_DISCOVER_TIME } from 'utils/misc-utils';

function CaptureCreate() {
Expand Down Expand Up @@ -85,14 +85,16 @@ function CaptureCreate() {
return (
<DetailsFormHydrator>
<EndpointConfigHydrator>
<ResourceConfigHydrator>
<BindingHydrator>
<MutateDraftSpecProvider value={updateDraftSpecs}>
<EntityCreate
entityType={entityType}
draftSpecMetadata={draftSpecsMetadata}
toolbar={
<EntityToolbar
waitTimes={{ generate: MAX_DISCOVER_TIME }}
waitTimes={{
generate: MAX_DISCOVER_TIME,
}}
GenerateButton={
<CaptureGenerateButton
entityType={entityType}
Expand Down Expand Up @@ -128,7 +130,7 @@ function CaptureCreate() {
}
/>
</MutateDraftSpecProvider>
</ResourceConfigHydrator>
</BindingHydrator>
</EndpointConfigHydrator>
</DetailsFormHydrator>
);
Expand Down
6 changes: 3 additions & 3 deletions src/components/capture/Edit.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ import useDraftSpecs from 'hooks/useDraftSpecs';
import usePageTitle from 'hooks/usePageTitle';
import { useCallback, useMemo } from 'react';
import { CustomEvents } from 'services/types';
import BindingHydrator from 'stores/Binding/Hydrator';
import { DetailsFormHydrator } from 'stores/DetailsForm/Hydrator';
import { EndpointConfigHydrator } from 'stores/EndpointConfig/Hydrator';
import ResourceConfigHydrator from 'stores/ResourceConfig/Hydrator';
import { MAX_DISCOVER_TIME } from 'utils/misc-utils';
import useValidConnectorsExist from 'hooks/connectors/useHasConnectors';

Expand Down Expand Up @@ -67,7 +67,7 @@ function CaptureEdit() {
<DraftInitializer>
<DetailsFormHydrator>
<EndpointConfigHydrator>
<ResourceConfigHydrator>
<BindingHydrator>
<MutateDraftSpecProvider value={updateDraftSpecs}>
<EntityEdit
title="routeTitle.captureEdit"
Expand Down Expand Up @@ -112,7 +112,7 @@ function CaptureEdit() {
}
/>
</MutateDraftSpecProvider>
</ResourceConfigHydrator>
</BindingHydrator>
</EndpointConfigHydrator>
</DetailsFormHydrator>
</DraftInitializer>
Expand Down
4 changes: 2 additions & 2 deletions src/components/capture/GenerateButton.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@ import { buttonSx } from 'components/shared/Entity/Header';
import { useEntityWorkflow_Editing } from 'context/Workflow';
import { Dispatch, SetStateAction, useEffect } from 'react';
import { FormattedMessage } from 'react-intl';
import { useBinding_rediscoveryRequired } from 'stores/Binding/hooks';
import {
useDetailsForm_connectorImage_connectorId,
useDetailsForm_entityNameChanged,
useDetailsForm_previousConnectorImage_connectorId,
} from 'stores/DetailsForm/hooks';
import { useFormStateStore_status } from 'stores/FormState/hooks';
import { FormStatus } from 'stores/FormState/types';
import { useResourceConfig_rediscoveryRequired } from 'stores/ResourceConfig/hooks';
import { Entity } from 'types';
import useDiscoverCapture from './useDiscoverCapture';

Expand All @@ -29,7 +29,7 @@ function CaptureGenerateButton({
createWorkflowMetadata,
}: Props) {
const isEdit = useEntityWorkflow_Editing();
const rediscoveryRequired = useResourceConfig_rediscoveryRequired();
const rediscoveryRequired = useBinding_rediscoveryRequired();

const { generateCatalog, isSaving, formActive } = useDiscoverCapture(
entityType,
Expand Down
17 changes: 14 additions & 3 deletions src/components/capture/useCaptureDraftUpdate.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,18 @@ import { useMutateDraftSpec } from 'components/shared/Entity/MutateDraftSpecCont
import { useEntityWorkflow_Editing } from 'context/Workflow';
import useEntityNameSuffix from 'hooks/useEntityNameSuffix';
import { useCallback } from 'react';
import {
useBinding_bindings,
useBinding_resourceConfigs,
useBinding_serverUpdateRequired,
} from 'stores/Binding/hooks';
import { useDetailsForm_connectorImage_imagePath } from 'stores/DetailsForm/hooks';
import {
useEndpointConfigStore_endpointConfig_data,
useEndpointConfigStore_setEncryptedEndpointConfig,
useEndpointConfigStore_setPreviousEndpointConfig,
} from 'stores/EndpointConfig/hooks';

import { useResourceConfig_resourceConfig } from 'stores/ResourceConfig/hooks';
import { modifyExistingCaptureDraftSpec } from 'utils/workflow-utils';

function useDiscoverDraftUpdate(options?: {
Expand All @@ -42,7 +46,10 @@ function useDiscoverDraftUpdate(options?: {
const setPreviousEndpointConfig =
useEndpointConfigStore_setPreviousEndpointConfig();

const resourceConfig = useResourceConfig_resourceConfig();
const bindings = useBinding_bindings();
const resourceConfig = useBinding_resourceConfigs();
const resourceConfigServerUpdateRequired =
useBinding_serverUpdateRequired();

const processedEntityName = useEntityNameSuffix(
!isEdit && options?.initiateDiscovery
Expand Down Expand Up @@ -92,7 +99,9 @@ function useDiscoverDraftUpdate(options?: {
imagePath,
encryptedEndpointConfigResponse,
resourceConfig,
existingTaskData
existingTaskData,
resourceConfigServerUpdateRequired,
bindings
);

if (draftSpecsResponse.error) {
Expand Down Expand Up @@ -123,6 +132,7 @@ function useDiscoverDraftUpdate(options?: {
return true;
},
[
bindings,
callFailed,
draftId,
endpointConfigData,
Expand All @@ -133,6 +143,7 @@ function useDiscoverDraftUpdate(options?: {
postGenerateMutate,
processedEntityName,
resourceConfig,
resourceConfigServerUpdateRequired,
setDraftId,
setEncryptedEndpointConfig,
setPreviousEndpointConfig,
Expand Down
15 changes: 7 additions & 8 deletions src/components/capture/useDiscoverCapture.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,20 +7,20 @@ import {
import { useEntityWorkflow_Editing } from 'context/Workflow';
import useEntityNameSuffix from 'hooks/useEntityNameSuffix';
import { useCallback, useMemo } from 'react';
import { useBinding_resourceConfigErrorsExist } from 'stores/Binding/hooks';
import { useDetailsForm_errorsExist } from 'stores/DetailsForm/hooks';
import {
useEndpointConfig_serverUpdateRequired,
useEndpointConfigStore_encryptedEndpointConfig_data,
useEndpointConfigStore_endpointConfig_data,
useEndpointConfigStore_errorsExist,
useEndpointConfig_serverUpdateRequired,
} from 'stores/EndpointConfig/hooks';
import {
useFormStateStore_isActive,
useFormStateStore_setFormState,
useFormStateStore_updateStatus,
} from 'stores/FormState/hooks';
import { FormStatus } from 'stores/FormState/types';
import { useResourceConfig_resourceConfigErrorsExist } from 'stores/ResourceConfig/hooks';
import { Entity } from 'types';
import useDiscoverConfigEncrypt from './useCaptureConfigEncrypt';
import useDiscoverDraftUpdate from './useCaptureDraftUpdate';
Expand All @@ -40,6 +40,9 @@ function useDiscoverCapture(

const isEdit = useEntityWorkflow_Editing();

// Binding Store
const resourceConfigErrorsExist = useBinding_resourceConfigErrorsExist();

// Draft Editor Store
const persistedDraftId = useEditorStore_persistedDraftId();
const isSaving = useEditorStore_isSaving();
Expand All @@ -61,10 +64,6 @@ function useDiscoverCapture(
const endpointConfigErrorsExist = useEndpointConfigStore_errorsExist();
const serverUpdateRequired = useEndpointConfig_serverUpdateRequired();

// Resource Config Store
const resourceConfigHasErrors =
useResourceConfig_resourceConfigErrorsExist();

// If we are doing an initial discovery add the name name to the name
// If not we are either refreshing collections during create OR during edit
// Refreshing during:
Expand All @@ -83,7 +82,7 @@ function useDiscoverCapture(
if (
detailsFormsHasErrors ||
endpointConfigErrorsExist ||
resourceConfigHasErrors
resourceConfigErrorsExist
) {
setFormState({
status: FormStatus.FAILED,
Expand Down Expand Up @@ -147,7 +146,7 @@ function useDiscoverCapture(
persistedDraftId,
processedEntityName,
resetEditorState,
resourceConfigHasErrors,
resourceConfigErrorsExist,
serverEndpointConfigData,
serverUpdateRequired,
setCatalogName,
Expand Down
37 changes: 16 additions & 21 deletions src/components/collection/Config.tsx
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
import { Typography, useTheme } from '@mui/material';
import MessageWithLink from 'components/content/MessageWithLink';
import BindingsMultiEditor from 'components/editor/Bindings';
import { useBindingsEditorStore_fullSourceErrorsExist } from 'components/editor/Bindings/Store/hooks';
import AlertBox from 'components/shared/AlertBox';
import WrapperWithHeader from 'components/shared/Entity/WrapperWithHeader';
import ErrorBoundryWrapper from 'components/shared/ErrorBoundryWrapper';
import { DraftSpecQuery } from 'hooks/useDraftSpecs';
import { WarningCircle } from 'iconoir-react';
import { ReactNode } from 'react';
import { FormattedMessage } from 'react-intl';
import { useFormStateStore_messagePrefix } from 'stores/FormState/hooks';
import {
useResourceConfig_collectionErrorsExist,
useResourceConfig_hydrationErrorsExist,
useResourceConfig_resourceConfigErrorsExist,
} from 'stores/ResourceConfig/hooks';
useBinding_bindingErrorsExist,
useBinding_fullSourceErrorsExist,
useBinding_hydrationErrorsExist,
useBinding_resourceConfigErrorsExist,
} from 'stores/Binding/hooks';
import { useFormStateStore_messagePrefix } from 'stores/FormState/hooks';

interface Props {
draftSpecs: DraftSpecQuery[];
Expand All @@ -31,26 +31,21 @@ function CollectionConfig({
}: Props) {
const theme = useTheme();

// Binding Store
const bindingHydrationErrorsExist = useBinding_hydrationErrorsExist();
const resourceConfigErrorsExist = useBinding_resourceConfigErrorsExist();
const bindingErrorsExist = useBinding_bindingErrorsExist();
const fullSourceErrorsExist = useBinding_fullSourceErrorsExist();

// Form State Store
const messagePrefix = useFormStateStore_messagePrefix();

// Resource Config Store
const resourceConfigHydrationErrorsExist =
useResourceConfig_hydrationErrorsExist();

const resourceConfigHasErrors =
useResourceConfig_resourceConfigErrorsExist();
const fullSourceErrorsExist =
useBindingsEditorStore_fullSourceErrorsExist();

const collectionsHasErrors = useResourceConfig_collectionErrorsExist();

const hasErrors =
resourceConfigHydrationErrorsExist ||
resourceConfigHasErrors ||
bindingHydrationErrorsExist ||
resourceConfigErrorsExist ||
fullSourceErrorsExist;

const hasWarnings = collectionsHasErrors;
const hasWarnings = bindingErrorsExist;

return (
<WrapperWithHeader
Expand Down Expand Up @@ -78,7 +73,7 @@ function CollectionConfig({
}
>
<ErrorBoundryWrapper>
{resourceConfigHydrationErrorsExist ? (
{bindingHydrationErrorsExist ? (
<AlertBox
severity="error"
title={
Expand Down
42 changes: 29 additions & 13 deletions src/components/collection/ResourceConfig.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -8,33 +8,42 @@ import { useEntityType } from 'context/EntityContext';
import { useEntityWorkflow_Editing } from 'context/Workflow';
import { FormattedMessage } from 'react-intl';
import {
useResourceConfig_hydrated,
useResourceConfig_resourceConfigOfCollectionProperty,
} from 'stores/ResourceConfig/hooks';
useBinding_currentBindingIndex,
useBinding_hydrated,
useBinding_resourceConfigOfMetaBindingProperty,
} from 'stores/Binding/hooks';
import { BindingsEditorConfigSkeleton } from './CollectionSkeletons';

interface Props {
bindingUUID: string;
collectionName: string;
readOnly?: boolean;
}

function ResourceConfig({ collectionName, readOnly = false }: Props) {
function ResourceConfig({
bindingUUID,
collectionName,
readOnly = false,
}: Props) {
const entityType = useEntityType();
const isEdit = useEntityWorkflow_Editing();

const hydrated = useResourceConfig_hydrated();
const hydrated = useBinding_hydrated();
const stagedBindingIndex = useBinding_currentBindingIndex();

const draftedBindingIndex =
useEditorStore_queryResponse_draftedBindingIndex(collectionName);
useEditorStore_queryResponse_draftedBindingIndex(
collectionName,
stagedBindingIndex
);

// If the collection is disabled then it will not come back in the built spec
// binding list. This means the user could end up clicking "See Fields" button
// forever and never get fields listed.
const collectionDisabled =
useResourceConfig_resourceConfigOfCollectionProperty(
collectionName,
'disable'
);
const collectionDisabled = useBinding_resourceConfigOfMetaBindingProperty(
bindingUUID,
'disable'
);

return (
<>
Expand All @@ -45,6 +54,7 @@ function ResourceConfig({ collectionName, readOnly = false }: Props) {
<Box sx={{ width: '100%' }}>
{hydrated ? (
<ResourceConfigForm
bindingUUID={bindingUUID}
collectionName={collectionName}
readOnly={readOnly}
/>
Expand All @@ -65,11 +75,17 @@ function ResourceConfig({ collectionName, readOnly = false }: Props) {
) : null}

{entityType === 'materialization' && !collectionDisabled ? (
<FieldSelectionViewer collectionName={collectionName} />
<FieldSelectionViewer
bindingUUID={bindingUUID}
collectionName={collectionName}
/>
) : null}

{entityType === 'materialization' ? (
<TimeTravel collectionName={collectionName} />
<TimeTravel
bindingUUID={bindingUUID}
collectionName={collectionName}
/>
) : null}
</>
);
Expand Down
Loading