diff --git a/examples/fields/GroupByField.README.md b/examples/fields/GroupByField.README.md new file mode 100644 index 0000000..110f460 --- /dev/null +++ b/examples/fields/GroupByField.README.md @@ -0,0 +1,145 @@ +# The expression language by example + +The expression language is the flatfile creation that allows concise code to be evaluated with the same syntax, whether that code is executing in a test in node, in a datahook in a validate function, or on our server for a sheetCompute. + +Let's dive in + +``` +When(GreaterThan(Val(), 5), Error(" val is greater than 5")) +``` + +What's going on there... a lot + +`Val()` is a special function, that references a variable for this interpret context.. more on that later. +We'll start from the inside `GreaterThan(Val(), 5)` runs `Val()` and returns true or false if it is greater or less than 5. + +Let me explain a simpler expression first +``` +When(true, Error(" val is greater than 5") +``` +`When` is a conditional, based on the value of the first argument, it will conditionally return the expression in the second argument. + +This all seems complex and a long way of saying +`if(val > 5) {return error("val greater than 5");}` + +The magic happens with this, because we aren't actually executing the code immediately. We can execute the code immmediately, or we can send that same parse tree to the server and execute it there. + +for a `validate` hook we execute it immediately, and `Val()` is the value being validated. + +For a sheetCompute, or groupConstraint, we send the parse tree to the server for execution there. + +# The expression language for validate Field hooks + +The easiest place to use the expression language is for `validate` field hooks. + +let's take the above example and put it in a test Sheet +``` +const SimpleSheet = new Sheet( + 'SimpleSheet', + {age: NumberField({validate: ErrorWhen(LessThan(Val(), 21), "too young to drink")}, +) +``` + + +note, when used in a `validate` function, there are special MessageConditionals `ErrorWhen`,`ErrorUnless`, `WarnWhen`, `WarnUnless`, `InfoWhen`, `InfoUnless` that should be used. `ErrorWhen` returns a function that accepts a value and calls the validate function on the expression, making `Val()` a valid way of accessing the argument for `validate`. + +# Expression language for sheet wide operations + +There are many operations and validations that need to be run across an entire sheet. These leverage expression language to allow declarative statements describing what you want done. GroupByField is the only way to accomplish this now. + +When thinking about using GroupByField, keep in mind the order of operations + +1. field cast +2. field compute +3. recordCompute +4. batchRecordsCompute +5. field validate +6. field egressFormat +----- Back to the server ---- +7. uniqueness checks +8. sheetCompute (groupByField) + + +Given that order of operations. Use the other hooks to prepare data for the groupByField. The other hooks are better suited to most tasks, but sheetwide operations require GroupByField. + +GroupByField is best suited for simple calculations (sum, count) and validating properties about a group. + +When developing with GroupByField, we recommend using local testing. Look at [GroupByField.spec.ts]( https://flatfile.com/docs/get-started/quickstart/) + +# GroupByField + + +``` +const ItemSummary = new Sheet( + 'ItemSummary', + { + orderID: NumberField({}), //parent item + itemId: NumberField({}), + price: NumberField({}), + orderTotal: GroupByField('orderId', compute: Sum(GetColumn('price', Group()))) + } +) +``` + +this sheet looks fairly straight forward until we get to GroupByField. + +the first argument is the column to group on +`compute` for GroupByField will be interpreted on the Flatfile server, once per group, with all relevant rows exposed as `Group()`. `GetColumn` returns a list of values for the column from the set sent in. `Sum` returns the sum of values passed in + + +# GroupConstraintSet +``` +const MixedRowSheet = new Sheet( + 'MixedRows', + { + parentId: NumberField({}), + rowType: TextField({}), //can be header or item + colForHeader: TextField({}), // required per header row + colForItem: NumberField({}), //unique per parentId for rowType = item + //the following field is just a placeholder to recieve errors + validityErrors: GroupByField('parentId', compute: + GroupConstraintSet( + [Group(), When(Equal(Count(Match({'rowType':'header'}), Group())), 0), + Error("At least 1 rowType='header' required")], + [Group(), When(Equal(Count(Match({'rowType':'item'}), MatchResult())), 0), + Error("At least 1 rowType='item' required")], + [Match({'rowType':'item'}), + Unless(Equal(Count(MatchResult()), Count(Uniq(GetColumn('colForItem', MatchResult())))), + Error("colForItem must be unique across parentId"))])) + } +) +``` + + +Wow, a lot going on there +`GroupConstraintSet` -- take arguments as (applicableSet, condition), add all Messages together and apply to field. + +note we use `When` here to differentiate from `ValidateWhen`. Still not sure how to handle this naming collision. + +for the first argument +``` + [Group(), When(Equal(Count(Match({'rowType':'header'}), Group())), 0), + Error("At least 1 rowType='header' required")], +``` +this will match each row in Group with `rowType`=`'header'` and count them. if the count is 0, add an Error message of "header required". + + +``` + [Group(), When(Equal(Count(Match({'rowType':'item'}), MatchResult())), 0), + Error("At least 1 rowType='item' required")], +``` +the second argument does much the same for rowType. note `MatchResult()` which is the same as the condition used as the first argument of the array + + +``` + [Match({'rowType':'item'}), + Unless(Equal(Count(MatchResult()), Count(Uniq(GetColumn('colForItem', MatchResult())))), + Error("colForItem must be unique across parentId"))])) +``` +the third argument only applies to rows where `rowType`=`'item'`. this enforces that `colForItem` is unique within the group. + + + + + + diff --git a/examples/fields/GroupByField.spec.ts b/examples/fields/GroupByField.spec.ts new file mode 100644 index 0000000..ece083c --- /dev/null +++ b/examples/fields/GroupByField.spec.ts @@ -0,0 +1,289 @@ +import { TextField, NumberField, GroupByField, Sheet, Workbook } from '@flatfile/configure' +import { SheetTester, matchMessages, matchSingleMessage } from '../../src/utils/testing/SheetTester' +import { + Group, + SumField, + Count, + Match, + Error, + GreaterThan, + Unless, + GroupConstraintItem, + Do, + NonUnique, + RecordPick, + Pick, + JSONLoad, + Debug, + JSONStringify, + SortedBy, + First, + Get, + Without, + StrConcat, + ArrayEquals +} from '../../src/expression-lang/EXPR' + +const tgrps = [ + { name: 'Odin_', age: 3., job: 'kid', weight: 30., eye_color: 'blue_', fav_group: 'Raffi', age_sum: '0', 'encoded_field': '' }, + { name: 'Sarah', age: 8., job: 'kid', weight: 60., eye_color: 'green', fav_group: 'Wiggles', age_sum: '0', 'encoded_field': '' }, + { name: 'Paddy', age: 40, job: 'eng', weight: 190, eye_color: 'green', fav_group: 'Wiggles', age_sum: '0', 'encoded_field': '' }, + { name: 'Kay__', age: 77, job: 'ret', weight: 160, eye_color: 'green', fav_group: 'Beach Boys', age_sum: '0', 'encoded_field': '' }, + { name: 'Cliff', age: 86, job: 'ret', weight: 160, eye_color: 'gray_', fav_group: 'The Stones', age_sum: '0', 'encoded_field': '' }, + { name: 'Franz', age: 72, job: 'ret', weight: 170, eye_color: 'blue_', fav_group: 'Beach Boys', age_sum: '0', 'encoded_field': '' } +] + + +const UniquePeopleSheet = new Sheet('People', + { + job: TextField(), + age: NumberField(), + age_sum: GroupByField( + ['job'], + GroupConstraintItem( + NonUnique(Group(), 'fav_group'), + Error('fav_group must be unique'), + 'fav_group', Group())) +}) +const UniquePeopleBook = new Workbook({name: 't', namespace: 't', sheets: {UniquePeopleSheet}}) + +describe('SampleGroupBy groupConstraint 2 ->', () => { + + + const testSheet = new SheetTester(UniquePeopleBook, 'UniquePeopleSheet') + test('GroupByField works properly with sum - multiple rows', async () => { + const res = await testSheet.testMessages(tgrps) + console.log(res) + expect(res[3][0]).toMatchObject({ + field: 'fav_group', + message: 'fav_group must be unique', + }) + expect(res[4]).toStrictEqual([]) + }) +}) + + +const CountSheet = new Sheet( + 'CountSheet', + { + category: TextField({}), + count_of_instances: GroupByField( + ['category'], + Count(Group()) + )}) +const CountBook = new Workbook({name: 't', namespace: 't', sheets: {CountSheet}}) + +describe('SampleGroupByField ->', () => { + const testSheet = new SheetTester(CountBook, 'CountSheet') + test('GroupByField works properly with count', async () => { + const res = await testSheet.testRecord({category: 'apple_', count_of_instances: '_' }) + expect(res).toStrictEqual({ category: 'apple_', count_of_instances: 1 }) + }) + + test('GroupByField works properly with count - multiple rows', async () => { + const res = await testSheet.testRecords( + [ + { category: 'apple_', count_of_instances: '_' }, + { category: 'orange', count_of_instances: '_' }, + { category: 'apple_', count_of_instances: '_' }, + ]) + + expect(res).toStrictEqual( + [ + { category: 'apple_', count_of_instances: 2 }, + { category: 'orange', count_of_instances: 1 }, + { category: 'apple_', count_of_instances: 2 }, + ] + ) + }) +}) + + + +const grps = [ + { name: 'Paddy', age: 40, job: 'eng', weight: 190, eye_color: 'green', age_sum: '0' }, + { name: 'Cliff', age: 86, job: 'ret', weight: 160, eye_color: 'gray_', age_sum: '0' }, + { name: 'Odin_', age: 3., job: 'kid', weight: 30., eye_color: 'blue_', age_sum: '0' }, + { name: 'Kay__', age: 77, job: 'ret', weight: 160, eye_color: 'green', age_sum: '0' }, + { name: 'Sarah', age: 8., job: 'kid', weight: 60., eye_color: 'green', age_sum: '0' }] + + +const JobAgeSheet = new Sheet( + 'JobAgeSheet', + { + job: TextField(), + age: NumberField(), + age_sum: GroupByField( + ['job'], + SumField(Group(), 'age') + ), + } +) + +const SumResults = [ + { name: 'Paddy', age: 40, job: 'eng', weight: 190, eye_color: 'green', age_sum: 40 }, + { name: 'Cliff', age: 86, job: 'ret', weight: 160, eye_color: 'gray_', age_sum: 163 }, + { name: 'Odin_', age: 3., job: 'kid', weight: 30., eye_color: 'blue_', age_sum: 11 }, + { name: 'Kay__', age: 77, job: 'ret', weight: 160, eye_color: 'green', age_sum: 163 }, + { name: 'Sarah', age: 8., job: 'kid', weight: 60., eye_color: 'green', age_sum: 11 }] + + +const JABook = new Workbook({name: 't', namespace: 't', sheets: {JobAgeSheet}}) + +describe('SampleGroupBy sum ->', () => { + const testSheet = new SheetTester(JABook, 'JobAgeSheet') + test('GroupByField works properly with sum - multiple rows', async () => { + const res = await testSheet.testRecords(grps) + expect(res).toStrictEqual(SumResults) + }) +}) + + + +const PeopleSheet = new Sheet('People', + { + job: TextField(), + age: NumberField(), + age_sum: GroupByField( + ['job'], + Do(GroupConstraintItem( + Group(), + Unless( + GreaterThan( + Count(Match(Group(), {eye_color: 'blue_'})), + 0), + Error('No Blue eyes')), + 'name', + Group()), "22222" )) + +}) +const PeopleBook = new Workbook({name: 't', namespace: 't', sheets: {PeopleSheet}}) + +describe('SampleGroupBy groupConstraint ->', () => { + + + const testSheet = new SheetTester(PeopleBook, 'PeopleSheet') + test('GroupByField works properly with sum - multiple rows', async () => { + const res = await testSheet.testMessages(grps) + expect(res[0][0]).toMatchObject({ + field: 'name', + message: 'No Blue eyes', + }) + expect(res[2]).toStrictEqual([]) + }) +}) + +const BothSheet = new Sheet( + 'BothSheet', + { + job: TextField(), + age: NumberField(), + age_sum: GroupByField( + ['job'], + // NOTE THE USE OF DO HERE... + // Do allows multiple expressions to be executed and the result + // of the last one is returned... in this case "SumField" + Do( + GroupConstraintItem( + Group(), + Unless( + GreaterThan( + Count(Match(Group(), {eye_color: 'blue_'})), + 0), + Error('No Blue eyes')), + 'name', + Group()), + SumField(Group(), 'age')) + ), + } +) + +const BothBook = new Workbook({name: 't', namespace: 't', sheets: {BothSheet}}) + +describe('SampleGroupBy groupConstraint and Comp ->', () => { + const testSheet = new SheetTester(BothBook, 'BothSheet') + test('GroupByField works properly with groupconstraint and sum - messages', async () => { + const res = await testSheet.testMessages(grps) + expect(res[0][0]).toMatchObject({ + field: 'name', + message: 'No Blue eyes', + }) + expect(res[2]).toStrictEqual([]) + }) + test('GroupByField works properly with groupconstraint and sum - rows', async () => { + const res = await testSheet.testRecords(grps) + expect(res).toStrictEqual(SumResults) + }) +}) + + + +const UnpackSheet = new Sheet( + 'UnpackSheet', + { + job: TextField(), + name: TextField(), + encoded_field: TextField({}), + age_sum: GroupByField( + ['job'], + GroupConstraintItem( + Group(), + Unless( + ArrayEquals( + SortedBy(Pick(First( + SortedBy(JSONLoad( + RecordPick(Group(), 'encoded_field', JSON.stringify({timestamp:'2000'}))), + 'timestamp', 'desc')), 'names')), + SortedBy(RecordPick(Group(), 'name')) + ), + Error( + //@ts-ignore + StrConcat( + Without( + SortedBy(Pick(First( + SortedBy(JSONLoad( + RecordPick(Group(), 'encoded_field', JSON.stringify({timestamp:'2000'}))), + 'timestamp', 'desc')), 'names')), + RecordPick(Group(), 'name')), + " is missing from the group") + )), + 'name', + Group()) +)}, + { + batchRecordsCompute: async (records) => { + // this batchRecordsCompute is supposed to mock making an HTTP request and setting it on fields + const rec1 = records.records[0]; + rec1.set('encoded_field', + JSON.stringify({timestamp: new Date(), names:["Odin_", "Sarah"]})) + const rec2 = records.records[2]; + //note Maire is an engineer that isn't present in the dataset + rec2.set('encoded_field', + JSON.stringify({timestamp: new Date(), names:["Paddy", "Maire"]})) + + } + } +) + +const UnpackBook = new Workbook({name: 't', namespace: 't', sheets: {UnpackSheet}}) + +describe('Unpack example ->', () => { +const tgrps = [ + { name: 'Odin_', job: 'kid', age_sum: '0', 'encoded_field': '' }, + { name: 'Sarah', job: 'kid', age_sum: '0', 'encoded_field': '' }, + { name: 'Paddy', job: 'eng', age_sum: '0', 'encoded_field': '' }, +] + + + const testSheet = new SheetTester(UnpackBook, 'UnpackSheet') + test('GroupByField works properly with sum - multiple rows', async () => { + const res = await testSheet.testMessages(tgrps) + + expect(res[0]).toStrictEqual([]) + expect(res[2][0]).toMatchObject({ + field: 'name', + message: "Maire is missing from the group", + }) + }) +}) + diff --git a/examples/sample-uploads/job-age-sample.csv b/examples/sample-uploads/job-age-sample.csv new file mode 100644 index 0000000..18d4a33 --- /dev/null +++ b/examples/sample-uploads/job-age-sample.csv @@ -0,0 +1,7 @@ +name,job,age,age_sum,eye_color +Paddy,eng,40,_,green +Cliff,ret,86,_,gray_ +Odin_,kid,3,_,blue_ +Kay__,ret,77,_,green +Sarah,kid,8,_,green +Stacy,kid,,_,gray_ diff --git a/examples/sheets/ValidateSheet.spec.ts b/examples/sheets/ValidateSheet.spec.ts new file mode 100644 index 0000000..c64a9ab --- /dev/null +++ b/examples/sheets/ValidateSheet.spec.ts @@ -0,0 +1,31 @@ +import { Sheet, NumberField, Workbook } from '@flatfile/configure' +import { SheetTester, matchMessages, matchSingleMessage } from '../../src/utils/testing/SheetTester' +import * as EXPR from '../../src/expression-lang/EXPR' +const [ErrorWhen, Val, GreaterThan] = [EXPR.ErrorWhen, EXPR.Val, EXPR.GreaterThan] + + +// note sheet must have same name as key in workbook it is shared as +const SubSheet = new Sheet( + 'SubSheet', + {numField: NumberField({validate: ErrorWhen(GreaterThan(Val(), 5), "more than 5")})}) + + +const TestWorkbook = new Workbook({ + name: `Test Workbook`, + namespace: 'test', + // saving SubSheet to workbook under key SubSheet + sheets: { SubSheet }, +}) + +describe('Workbook tests ->', () => { + // here we use Sheet tester + const testSheet = new SheetTester(TestWorkbook, 'SubSheet') + test('ErrorWhen ', async () => { + // for this inputRow + const inputRow = { numField: 6} + // we expect this output row + const res = await testSheet.testMessage(inputRow) + expect(matchSingleMessage(res, 'numField', 'more than 5', 'error')).toBeTruthy() + }) + +}) diff --git a/examples/sheets/complexUnique.spec.ts b/examples/sheets/complexUnique.spec.ts new file mode 100644 index 0000000..2329e53 --- /dev/null +++ b/examples/sheets/complexUnique.spec.ts @@ -0,0 +1,67 @@ +import { TextField, NumberField, GroupByField, Sheet, Workbook } from '@flatfile/configure' +import { SheetTester, matchMessages, matchSingleMessage } from '../../src/utils/testing/SheetTester' +import { + Group, + And, + Count, + Match, + Error, + GreaterThan, + When, + GroupConstraintItem, + +} from '../../src/expression-lang/EXPR' + +//any name that occurs with a job of 'kid', must be unique across the entire sheet +const tgrps = [ + { name: 'Odin_', job: 'kid', f:''}, //0 + { name: 'Odin_', job: 'ret', f:''}, //1 + { name: 'Sarah', job: 'kid', f:''}, //2 + { name: 'Paddy', job: 'eng', f:''}, + { name: 'Paddy', job: 'ret', f:''}, //4 + { name: 'Franz', job: 'ret', f:''}, + { name: 'Franz', job: 'ret', f:''}, //6 + { name: 'Kay__', job: 'ret', f:''}, + { name: 'Cliff', job: 'ret', f:''}, //8 +] + +const UniquePeopleSheet = new Sheet('People', + { + job: TextField(), + name: TextField(), + f: GroupByField( + ['name'], + GroupConstraintItem( + Group(), + When( + And( + GreaterThan(Count(Match(Group(), {job:'kid'})), 0), + GreaterThan(Count(Group()), 1)), + Error("name appears in row with 'kid', and other rows too")), + 'name')) +}) + +const UniquePeopleBook = new Workbook({name: 't', namespace: 't', sheets: {UniquePeopleSheet}}) + +describe('SampleGroupBy groupConstraint 2 ->', () => { + + + const testSheet = new SheetTester(UniquePeopleBook, 'UniquePeopleSheet') + test('GroupByField works properly with sum - multiple rows', async () => { + const res = await testSheet.testMessages(tgrps) + expect(res[0][0]).toMatchObject({ + field: 'name', + message: "name appears in row with 'kid', and other rows too", + }) + expect(res[1][0]).toMatchObject({ + field: 'name', + message: "name appears in row with 'kid', and other rows too", + }) + + expect(res[3]).toStrictEqual([]) + expect(res[4]).toStrictEqual([]) + expect(res[5]).toStrictEqual([]) + expect(res[6]).toStrictEqual([]) + }) +}) + diff --git a/examples/workbooks/GroupByBook.ts b/examples/workbooks/GroupByBook.ts new file mode 100644 index 0000000..4434a5e --- /dev/null +++ b/examples/workbooks/GroupByBook.ts @@ -0,0 +1,85 @@ +import { + Group, + SumField, + GroupConstraintItem, + Unless, + GreaterThan, + Count, + Match, + Error as ExprError, + Do +} from '../../src/expression-lang/EXPR' + +import { + NumberField, + Sheet, + TextField, + GroupByField, + Workbook, + Portal, +} from '@flatfile/configure' + + +const JobAgeSheet = new Sheet( + 'JobAgeSheet', + { + job: TextField(), + age: NumberField(), + age_sum: GroupByField( + ['job'], + SumField(Group(), 'age') + ), + } +) +const JobAgePortal = new Portal({ + name: 'JobAgePortal', + sheet: 'JobAgeSheet' +}) + + + +const PeopleSheet = new Sheet('People', + { + name: TextField(), + job: TextField(), + age: NumberField({ + validate: (val) => { + if (val > 10) { + throw new Error("too old") + } + } + } + ), + foo: TextField({stageVisibility: {review:true}}), + eye_color: TextField(), + age_sum: GroupByField( + ['job'], + Do( + GroupConstraintItem( + Group(), + Unless( + GreaterThan( + Count(Match(Group(), {eye_color: 'blue_'})), + 0), + ExprError('No Blue eyes')), + 'name', + Group()), + "33333")) +}) + +const PeoplePortal = new Portal({ + name: 'PeoplePortal', + sheet: 'PeopleSheet' +}) + + +export default new Workbook({ + name: 'GroupByWorkbook', + namespace: 'basic', + sheets: { + JobAgeSheet, + PeopleSheet + }, + portals: [JobAgePortal, PeoplePortal], +}) + diff --git a/package-lock.json b/package-lock.json index cae15dc..5cc5aa2 100644 --- a/package-lock.json +++ b/package-lock.json @@ -15,7 +15,8 @@ "date-fns-tz": "^1.3.7" }, "devDependencies": { - "@flatfile/configure": "^0.5.2", + "@flatfile/configure": "^0.5.4", + "@flatfile/expression-lang": "^0.0.2", "@flatfile/hooks": "^1.2.1", "@types/jest": "^28.1.4", "@types/lodash": "^4.14.186", @@ -621,38 +622,6 @@ "integrity": "sha512-0hYQ8SB4Db5zvZB4axdMHGwEaQjkZzFjQiN9LVYvIFB2nSUHW9tYpxWriPrWDASIxiaXax83REcLxuSdnGPZtw==", "dev": true }, - "node_modules/@esbuild/android-arm": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.15.18.tgz", - "integrity": "sha512-5GT+kcs2WVGjVs7+boataCkO5Fg0y4kCjzkB5bAip7H4jfnOS3dA6KPiww9W1OEKTKeAcUVhdZGvgI65OXmUnw==", - "cpu": [ - "arm" - ], - "dev": true, - "optional": true, - "os": [ - "android" - ], - "engines": { - "node": ">=12" - } - }, - "node_modules/@esbuild/linux-loong64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.15.18.tgz", - "integrity": "sha512-L4jVKS82XVhw2nvzLg/19ClLWg0y27ulRwuP7lcyL6AbUWB5aPglXY3M21mauDQMDfRLs8cQmeT03r/+X3cZYQ==", - "cpu": [ - "loong64" - ], - "dev": true, - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=12" - } - }, "node_modules/@eslint/eslintrc": { "version": "1.4.0", "resolved": "https://registry.npmjs.org/@eslint/eslintrc/-/eslintrc-1.4.0.tgz", @@ -682,9 +651,9 @@ "integrity": "sha512-bmPQLQSZC7761YHgREAYR1iocrYQAEdlnBjjyf8wUxLx5imwSj3C2vjHd0d30yE6K9CUDwcPch5UARQW98JdoQ==" }, "node_modules/@flatfile/configure": { - "version": "0.5.2", - "resolved": "https://registry.npmjs.org/@flatfile/configure/-/configure-0.5.2.tgz", - "integrity": "sha512-H3DA+N01/zRTustWZ7t5//gea9g5NVzy971Mf4JsIi+9b1sXqiLB/u3U4Qh1OYGuhJObL49JzZ3BUGaDBk452g==", + "version": "0.5.4", + "resolved": "https://registry.npmjs.org/@flatfile/configure/-/configure-0.5.4.tgz", + "integrity": "sha512-H2++RcWdNYb/T0uFlkOgbxhBeN9quOsrANC3q9UgP+eWIYcyXp7VsendxU0gDjFsgbgQckNBcvTo/bD6ZbTqvg==", "dev": true, "dependencies": { "@flatfile/hooks": "*", @@ -694,6 +663,12 @@ "date-fns": "^2.29.1" } }, + "node_modules/@flatfile/expression-lang": { + "version": "0.0.2", + "resolved": "https://registry.npmjs.org/@flatfile/expression-lang/-/expression-lang-0.0.2.tgz", + "integrity": "sha512-UIfkHh3rvCXlk5HYfIq7SGQHFi+/SOizbr4ipWjHItog/K0tMheoy5s7tVuV1jgqo5MS074aoJec9wBZsiLMmg==", + "dev": true + }, "node_modules/@flatfile/hooks": { "version": "1.2.1", "resolved": "https://registry.npmjs.org/@flatfile/hooks/-/hooks-1.2.1.tgz", @@ -2621,54 +2596,6 @@ "esbuild-windows-arm64": "0.15.18" } }, - "node_modules/esbuild-android-64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-android-64/-/esbuild-android-64-0.15.18.tgz", - "integrity": "sha512-wnpt3OXRhcjfIDSZu9bnzT4/TNTDsOUvip0foZOUBG7QbSt//w3QV4FInVJxNhKc/ErhUxc5z4QjHtMi7/TbgA==", - "cpu": [ - "x64" - ], - "dev": true, - "optional": true, - "os": [ - "android" - ], - "engines": { - "node": ">=12" - } - }, - "node_modules/esbuild-android-arm64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-android-arm64/-/esbuild-android-arm64-0.15.18.tgz", - "integrity": "sha512-G4xu89B8FCzav9XU8EjsXacCKSG2FT7wW9J6hOc18soEHJdtWu03L3TQDGf0geNxfLTtxENKBzMSq9LlbjS8OQ==", - "cpu": [ - "arm64" - ], - "dev": true, - "optional": true, - "os": [ - "android" - ], - "engines": { - "node": ">=12" - } - }, - "node_modules/esbuild-darwin-64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-darwin-64/-/esbuild-darwin-64-0.15.18.tgz", - "integrity": "sha512-2WAvs95uPnVJPuYKP0Eqx+Dl/jaYseZEUUT1sjg97TJa4oBtbAKnPnl3b5M9l51/nbx7+QAEtuummJZW0sBEmg==", - "cpu": [ - "x64" - ], - "dev": true, - "optional": true, - "os": [ - "darwin" - ], - "engines": { - "node": ">=12" - } - }, "node_modules/esbuild-darwin-arm64": { "version": "0.15.18", "resolved": "https://registry.npmjs.org/esbuild-darwin-arm64/-/esbuild-darwin-arm64-0.15.18.tgz", @@ -2685,262 +2612,6 @@ "node": ">=12" } }, - "node_modules/esbuild-freebsd-64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-freebsd-64/-/esbuild-freebsd-64-0.15.18.tgz", - "integrity": "sha512-TT3uBUxkteAjR1QbsmvSsjpKjOX6UkCstr8nMr+q7zi3NuZ1oIpa8U41Y8I8dJH2fJgdC3Dj3CXO5biLQpfdZA==", - "cpu": [ - "x64" - ], - "dev": true, - "optional": true, - "os": [ - "freebsd" - ], - "engines": { - "node": ">=12" - } - }, - "node_modules/esbuild-freebsd-arm64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-freebsd-arm64/-/esbuild-freebsd-arm64-0.15.18.tgz", - "integrity": "sha512-R/oVr+X3Tkh+S0+tL41wRMbdWtpWB8hEAMsOXDumSSa6qJR89U0S/PpLXrGF7Wk/JykfpWNokERUpCeHDl47wA==", - "cpu": [ - "arm64" - ], - "dev": true, - "optional": true, - "os": [ - "freebsd" - ], - "engines": { - "node": ">=12" - } - }, - "node_modules/esbuild-linux-32": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-linux-32/-/esbuild-linux-32-0.15.18.tgz", - "integrity": "sha512-lphF3HiCSYtaa9p1DtXndiQEeQDKPl9eN/XNoBf2amEghugNuqXNZA/ZovthNE2aa4EN43WroO0B85xVSjYkbg==", - "cpu": [ - "ia32" - ], - "dev": true, - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=12" - } - }, - "node_modules/esbuild-linux-64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-linux-64/-/esbuild-linux-64-0.15.18.tgz", - "integrity": "sha512-hNSeP97IviD7oxLKFuii5sDPJ+QHeiFTFLoLm7NZQligur8poNOWGIgpQ7Qf8Balb69hptMZzyOBIPtY09GZYw==", - "cpu": [ - "x64" - ], - "dev": true, - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=12" - } - }, - "node_modules/esbuild-linux-arm": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-linux-arm/-/esbuild-linux-arm-0.15.18.tgz", - "integrity": "sha512-UH779gstRblS4aoS2qpMl3wjg7U0j+ygu3GjIeTonCcN79ZvpPee12Qun3vcdxX+37O5LFxz39XeW2I9bybMVA==", - "cpu": [ - "arm" - ], - "dev": true, - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=12" - } - }, - "node_modules/esbuild-linux-arm64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-linux-arm64/-/esbuild-linux-arm64-0.15.18.tgz", - "integrity": "sha512-54qr8kg/6ilcxd+0V3h9rjT4qmjc0CccMVWrjOEM/pEcUzt8X62HfBSeZfT2ECpM7104mk4yfQXkosY8Quptug==", - "cpu": [ - "arm64" - ], - "dev": true, - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=12" - } - }, - "node_modules/esbuild-linux-mips64le": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-linux-mips64le/-/esbuild-linux-mips64le-0.15.18.tgz", - "integrity": "sha512-Mk6Ppwzzz3YbMl/ZZL2P0q1tnYqh/trYZ1VfNP47C31yT0K8t9s7Z077QrDA/guU60tGNp2GOwCQnp+DYv7bxQ==", - "cpu": [ - "mips64el" - ], - "dev": true, - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=12" - } - }, - "node_modules/esbuild-linux-ppc64le": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-linux-ppc64le/-/esbuild-linux-ppc64le-0.15.18.tgz", - "integrity": "sha512-b0XkN4pL9WUulPTa/VKHx2wLCgvIAbgwABGnKMY19WhKZPT+8BxhZdqz6EgkqCLld7X5qiCY2F/bfpUUlnFZ9w==", - "cpu": [ - "ppc64" - ], - "dev": true, - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=12" - } - }, - "node_modules/esbuild-linux-riscv64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-linux-riscv64/-/esbuild-linux-riscv64-0.15.18.tgz", - "integrity": "sha512-ba2COaoF5wL6VLZWn04k+ACZjZ6NYniMSQStodFKH/Pu6RxzQqzsmjR1t9QC89VYJxBeyVPTaHuBMCejl3O/xg==", - "cpu": [ - "riscv64" - ], - "dev": true, - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=12" - } - }, - "node_modules/esbuild-linux-s390x": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-linux-s390x/-/esbuild-linux-s390x-0.15.18.tgz", - "integrity": "sha512-VbpGuXEl5FCs1wDVp93O8UIzl3ZrglgnSQ+Hu79g7hZu6te6/YHgVJxCM2SqfIila0J3k0csfnf8VD2W7u2kzQ==", - "cpu": [ - "s390x" - ], - "dev": true, - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=12" - } - }, - "node_modules/esbuild-netbsd-64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-netbsd-64/-/esbuild-netbsd-64-0.15.18.tgz", - "integrity": "sha512-98ukeCdvdX7wr1vUYQzKo4kQ0N2p27H7I11maINv73fVEXt2kyh4K4m9f35U1K43Xc2QGXlzAw0K9yoU7JUjOg==", - "cpu": [ - "x64" - ], - "dev": true, - "optional": true, - "os": [ - "netbsd" - ], - "engines": { - "node": ">=12" - } - }, - "node_modules/esbuild-openbsd-64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-openbsd-64/-/esbuild-openbsd-64-0.15.18.tgz", - "integrity": "sha512-yK5NCcH31Uae076AyQAXeJzt/vxIo9+omZRKj1pauhk3ITuADzuOx5N2fdHrAKPxN+zH3w96uFKlY7yIn490xQ==", - "cpu": [ - "x64" - ], - "dev": true, - "optional": true, - "os": [ - "openbsd" - ], - "engines": { - "node": ">=12" - } - }, - "node_modules/esbuild-sunos-64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-sunos-64/-/esbuild-sunos-64-0.15.18.tgz", - "integrity": "sha512-On22LLFlBeLNj/YF3FT+cXcyKPEI263nflYlAhz5crxtp3yRG1Ugfr7ITyxmCmjm4vbN/dGrb/B7w7U8yJR9yw==", - "cpu": [ - "x64" - ], - "dev": true, - "optional": true, - "os": [ - "sunos" - ], - "engines": { - "node": ">=12" - } - }, - "node_modules/esbuild-windows-32": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-windows-32/-/esbuild-windows-32-0.15.18.tgz", - "integrity": "sha512-o+eyLu2MjVny/nt+E0uPnBxYuJHBvho8vWsC2lV61A7wwTWC3jkN2w36jtA+yv1UgYkHRihPuQsL23hsCYGcOQ==", - "cpu": [ - "ia32" - ], - "dev": true, - "optional": true, - "os": [ - "win32" - ], - "engines": { - "node": ">=12" - } - }, - "node_modules/esbuild-windows-64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-windows-64/-/esbuild-windows-64-0.15.18.tgz", - "integrity": "sha512-qinug1iTTaIIrCorAUjR0fcBk24fjzEedFYhhispP8Oc7SFvs+XeW3YpAKiKp8dRpizl4YYAhxMjlftAMJiaUw==", - "cpu": [ - "x64" - ], - "dev": true, - "optional": true, - "os": [ - "win32" - ], - "engines": { - "node": ">=12" - } - }, - "node_modules/esbuild-windows-arm64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-windows-arm64/-/esbuild-windows-arm64-0.15.18.tgz", - "integrity": "sha512-q9bsYzegpZcLziq0zgUi5KqGVtfhjxGbnksaBFYmWLxeV/S1fK4OLdq2DFYnXcLMjlZw2L0jLsk1eGoB522WXQ==", - "cpu": [ - "arm64" - ], - "dev": true, - "optional": true, - "os": [ - "win32" - ], - "engines": { - "node": ">=12" - } - }, "node_modules/escalade": { "version": "3.1.1", "resolved": "https://registry.npmjs.org/escalade/-/escalade-3.1.1.tgz", @@ -7207,20 +6878,6 @@ "integrity": "sha512-0hYQ8SB4Db5zvZB4axdMHGwEaQjkZzFjQiN9LVYvIFB2nSUHW9tYpxWriPrWDASIxiaXax83REcLxuSdnGPZtw==", "dev": true }, - "@esbuild/android-arm": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.15.18.tgz", - "integrity": "sha512-5GT+kcs2WVGjVs7+boataCkO5Fg0y4kCjzkB5bAip7H4jfnOS3dA6KPiww9W1OEKTKeAcUVhdZGvgI65OXmUnw==", - "dev": true, - "optional": true - }, - "@esbuild/linux-loong64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.15.18.tgz", - "integrity": "sha512-L4jVKS82XVhw2nvzLg/19ClLWg0y27ulRwuP7lcyL6AbUWB5aPglXY3M21mauDQMDfRLs8cQmeT03r/+X3cZYQ==", - "dev": true, - "optional": true - }, "@eslint/eslintrc": { "version": "1.4.0", "resolved": "https://registry.npmjs.org/@eslint/eslintrc/-/eslintrc-1.4.0.tgz", @@ -7244,9 +6901,9 @@ "integrity": "sha512-bmPQLQSZC7761YHgREAYR1iocrYQAEdlnBjjyf8wUxLx5imwSj3C2vjHd0d30yE6K9CUDwcPch5UARQW98JdoQ==" }, "@flatfile/configure": { - "version": "0.5.2", - "resolved": "https://registry.npmjs.org/@flatfile/configure/-/configure-0.5.2.tgz", - "integrity": "sha512-H3DA+N01/zRTustWZ7t5//gea9g5NVzy971Mf4JsIi+9b1sXqiLB/u3U4Qh1OYGuhJObL49JzZ3BUGaDBk452g==", + "version": "0.5.4", + "resolved": "https://registry.npmjs.org/@flatfile/configure/-/configure-0.5.4.tgz", + "integrity": "sha512-H2++RcWdNYb/T0uFlkOgbxhBeN9quOsrANC3q9UgP+eWIYcyXp7VsendxU0gDjFsgbgQckNBcvTo/bD6ZbTqvg==", "dev": true, "requires": { "@flatfile/hooks": "*", @@ -7256,6 +6913,12 @@ "date-fns": "^2.29.1" } }, + "@flatfile/expression-lang": { + "version": "0.0.2", + "resolved": "https://registry.npmjs.org/@flatfile/expression-lang/-/expression-lang-0.0.2.tgz", + "integrity": "sha512-UIfkHh3rvCXlk5HYfIq7SGQHFi+/SOizbr4ipWjHItog/K0tMheoy5s7tVuV1jgqo5MS074aoJec9wBZsiLMmg==", + "dev": true + }, "@flatfile/hooks": { "version": "1.2.1", "resolved": "https://registry.npmjs.org/@flatfile/hooks/-/hooks-1.2.1.tgz", @@ -8664,27 +8327,6 @@ "esbuild-windows-arm64": "0.15.18" } }, - "esbuild-android-64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-android-64/-/esbuild-android-64-0.15.18.tgz", - "integrity": "sha512-wnpt3OXRhcjfIDSZu9bnzT4/TNTDsOUvip0foZOUBG7QbSt//w3QV4FInVJxNhKc/ErhUxc5z4QjHtMi7/TbgA==", - "dev": true, - "optional": true - }, - "esbuild-android-arm64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-android-arm64/-/esbuild-android-arm64-0.15.18.tgz", - "integrity": "sha512-G4xu89B8FCzav9XU8EjsXacCKSG2FT7wW9J6hOc18soEHJdtWu03L3TQDGf0geNxfLTtxENKBzMSq9LlbjS8OQ==", - "dev": true, - "optional": true - }, - "esbuild-darwin-64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-darwin-64/-/esbuild-darwin-64-0.15.18.tgz", - "integrity": "sha512-2WAvs95uPnVJPuYKP0Eqx+Dl/jaYseZEUUT1sjg97TJa4oBtbAKnPnl3b5M9l51/nbx7+QAEtuummJZW0sBEmg==", - "dev": true, - "optional": true - }, "esbuild-darwin-arm64": { "version": "0.15.18", "resolved": "https://registry.npmjs.org/esbuild-darwin-arm64/-/esbuild-darwin-arm64-0.15.18.tgz", @@ -8692,118 +8334,6 @@ "dev": true, "optional": true }, - "esbuild-freebsd-64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-freebsd-64/-/esbuild-freebsd-64-0.15.18.tgz", - "integrity": "sha512-TT3uBUxkteAjR1QbsmvSsjpKjOX6UkCstr8nMr+q7zi3NuZ1oIpa8U41Y8I8dJH2fJgdC3Dj3CXO5biLQpfdZA==", - "dev": true, - "optional": true - }, - "esbuild-freebsd-arm64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-freebsd-arm64/-/esbuild-freebsd-arm64-0.15.18.tgz", - "integrity": "sha512-R/oVr+X3Tkh+S0+tL41wRMbdWtpWB8hEAMsOXDumSSa6qJR89U0S/PpLXrGF7Wk/JykfpWNokERUpCeHDl47wA==", - "dev": true, - "optional": true - }, - "esbuild-linux-32": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-linux-32/-/esbuild-linux-32-0.15.18.tgz", - "integrity": "sha512-lphF3HiCSYtaa9p1DtXndiQEeQDKPl9eN/XNoBf2amEghugNuqXNZA/ZovthNE2aa4EN43WroO0B85xVSjYkbg==", - "dev": true, - "optional": true - }, - "esbuild-linux-64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-linux-64/-/esbuild-linux-64-0.15.18.tgz", - "integrity": "sha512-hNSeP97IviD7oxLKFuii5sDPJ+QHeiFTFLoLm7NZQligur8poNOWGIgpQ7Qf8Balb69hptMZzyOBIPtY09GZYw==", - "dev": true, - "optional": true - }, - "esbuild-linux-arm": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-linux-arm/-/esbuild-linux-arm-0.15.18.tgz", - "integrity": "sha512-UH779gstRblS4aoS2qpMl3wjg7U0j+ygu3GjIeTonCcN79ZvpPee12Qun3vcdxX+37O5LFxz39XeW2I9bybMVA==", - "dev": true, - "optional": true - }, - "esbuild-linux-arm64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-linux-arm64/-/esbuild-linux-arm64-0.15.18.tgz", - "integrity": "sha512-54qr8kg/6ilcxd+0V3h9rjT4qmjc0CccMVWrjOEM/pEcUzt8X62HfBSeZfT2ECpM7104mk4yfQXkosY8Quptug==", - "dev": true, - "optional": true - }, - "esbuild-linux-mips64le": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-linux-mips64le/-/esbuild-linux-mips64le-0.15.18.tgz", - "integrity": "sha512-Mk6Ppwzzz3YbMl/ZZL2P0q1tnYqh/trYZ1VfNP47C31yT0K8t9s7Z077QrDA/guU60tGNp2GOwCQnp+DYv7bxQ==", - "dev": true, - "optional": true - }, - "esbuild-linux-ppc64le": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-linux-ppc64le/-/esbuild-linux-ppc64le-0.15.18.tgz", - "integrity": "sha512-b0XkN4pL9WUulPTa/VKHx2wLCgvIAbgwABGnKMY19WhKZPT+8BxhZdqz6EgkqCLld7X5qiCY2F/bfpUUlnFZ9w==", - "dev": true, - "optional": true - }, - "esbuild-linux-riscv64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-linux-riscv64/-/esbuild-linux-riscv64-0.15.18.tgz", - "integrity": "sha512-ba2COaoF5wL6VLZWn04k+ACZjZ6NYniMSQStodFKH/Pu6RxzQqzsmjR1t9QC89VYJxBeyVPTaHuBMCejl3O/xg==", - "dev": true, - "optional": true - }, - "esbuild-linux-s390x": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-linux-s390x/-/esbuild-linux-s390x-0.15.18.tgz", - "integrity": "sha512-VbpGuXEl5FCs1wDVp93O8UIzl3ZrglgnSQ+Hu79g7hZu6te6/YHgVJxCM2SqfIila0J3k0csfnf8VD2W7u2kzQ==", - "dev": true, - "optional": true - }, - "esbuild-netbsd-64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-netbsd-64/-/esbuild-netbsd-64-0.15.18.tgz", - "integrity": "sha512-98ukeCdvdX7wr1vUYQzKo4kQ0N2p27H7I11maINv73fVEXt2kyh4K4m9f35U1K43Xc2QGXlzAw0K9yoU7JUjOg==", - "dev": true, - "optional": true - }, - "esbuild-openbsd-64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-openbsd-64/-/esbuild-openbsd-64-0.15.18.tgz", - "integrity": "sha512-yK5NCcH31Uae076AyQAXeJzt/vxIo9+omZRKj1pauhk3ITuADzuOx5N2fdHrAKPxN+zH3w96uFKlY7yIn490xQ==", - "dev": true, - "optional": true - }, - "esbuild-sunos-64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-sunos-64/-/esbuild-sunos-64-0.15.18.tgz", - "integrity": "sha512-On22LLFlBeLNj/YF3FT+cXcyKPEI263nflYlAhz5crxtp3yRG1Ugfr7ITyxmCmjm4vbN/dGrb/B7w7U8yJR9yw==", - "dev": true, - "optional": true - }, - "esbuild-windows-32": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-windows-32/-/esbuild-windows-32-0.15.18.tgz", - "integrity": "sha512-o+eyLu2MjVny/nt+E0uPnBxYuJHBvho8vWsC2lV61A7wwTWC3jkN2w36jtA+yv1UgYkHRihPuQsL23hsCYGcOQ==", - "dev": true, - "optional": true - }, - "esbuild-windows-64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-windows-64/-/esbuild-windows-64-0.15.18.tgz", - "integrity": "sha512-qinug1iTTaIIrCorAUjR0fcBk24fjzEedFYhhispP8Oc7SFvs+XeW3YpAKiKp8dRpizl4YYAhxMjlftAMJiaUw==", - "dev": true, - "optional": true - }, - "esbuild-windows-arm64": { - "version": "0.15.18", - "resolved": "https://registry.npmjs.org/esbuild-windows-arm64/-/esbuild-windows-arm64-0.15.18.tgz", - "integrity": "sha512-q9bsYzegpZcLziq0zgUi5KqGVtfhjxGbnksaBFYmWLxeV/S1fK4OLdq2DFYnXcLMjlZw2L0jLsk1eGoB522WXQ==", - "dev": true, - "optional": true - }, "escalade": { "version": "3.1.1", "resolved": "https://registry.npmjs.org/escalade/-/escalade-3.1.1.tgz", diff --git a/package.json b/package.json index 6bcd99c..0640e17 100644 --- a/package.json +++ b/package.json @@ -27,8 +27,11 @@ "test": "jest --runInBand --detectOpenHandles --forceExit --passWithNoTests" }, "devDependencies": { - "@flatfile/configure": "^0.5.2", + "flatfile": "^3.1.5", + "@flatfile/configure": "^0.5.4", "@flatfile/hooks": "^1.2.1", + "@flatfile/expression-lang": "^0.0.2", + "node-fetch": "^3.2.10", "@types/jest": "^28.1.4", "@types/lodash": "^4.14.186", "@typescript-eslint/parser": "^5.31.0", diff --git a/src/expression-lang/EXPR.spec.ts b/src/expression-lang/EXPR.spec.ts new file mode 100644 index 0000000..ed6a0ac --- /dev/null +++ b/src/expression-lang/EXPR.spec.ts @@ -0,0 +1,165 @@ +import { FlatfileRecord } from '@flatfile/hooks' +import { sheetInterpret } from './sheetInterpret' +import { + When, + Error, + Val, + Group, + Match, + ImplicitLessThan, + GroupConstraintItem, + Count, + MatchResult, + GreaterThan, + Unless, + Debug, + ErrorWhen, + ErrorUnless +} from './EXPR' + +describe('VExpression Tests ->', () => { + test('Simple test', () => { + const valFunc = ErrorWhen(GreaterThan(Val(), 5), 'val greater than 5') + expect(valFunc(3)).toStrictEqual([]) + expect(valFunc(7)).toMatchObject([ + { level: 'error', message: 'val greater than 5' }, + ]) + }) + + test('Implicit Greater Than test', () => { + const valFunc = ErrorUnless(ImplicitLessThan(5), 'val less than 5') + expect(valFunc(7)).toStrictEqual([]) + expect(valFunc(3)).toMatchObject([ + { level: 'error', message: 'val less than 5' }, + ]) + }) +}) + +// [WholeGroup(), Unless(GreaterThan(Count(Match({'RowType':'Header'}, Group())), 0), ErrorC("Group must contain a Header row"))], +// [WholeGroup(), Unless(GreaterThan(Count(Match({'RowType':'Item'}, Group())), 0), ErrorC("Group must contain a Item row"))], +// [Match({'RowType':'Header'}, Group()), When(GreaterThan(Count(MatchResult()), 1), ErrorC("Group must contain only a single Header row"))], + +describe('GroupConstraint Tests ->', () => { + const getRecs = () => [ + new FlatfileRecord({ + rawData: { + name: 'Paddy', + age: 40, + job: 'eng', + weight: 190, + eyeColor: 'green', + }, + rowId: 1, + }), + new FlatfileRecord({ + rawData: { + name: 'Cliff', + age: 86, + job: 'ret', + weight: 160, + eyeColor: 'gray_', + }, + rowId: 2, + }), + new FlatfileRecord({ + rawData: { + name: 'Odin_', + age: 3, + job: 'kid', + weight: 30, + eyeColor: 'blue_', + }, + rowId: 3, + }), + new FlatfileRecord({ + rawData: { + name: 'Kay__', + age: 77, + job: 'ret', + weight: 160, + eyeColor: 'green', + }, + rowId: 4, + }), + new FlatfileRecord({ + rawData: { + name: 'Sarah', + age: 8, + job: 'kid', + weight: 60, + eyeColor: 'green', + }, + rowId: 5, + }), + ] + + test('Match', () => { + const recs = getRecs() + const matchResult = sheetInterpret( + Match(recs, { name: 'Paddy' }) + ) as any[] + expect(matchResult.length).toBe(1) + expect(matchResult[0].value).toMatchObject({ age: 40, name: 'Paddy' }) + + const matchResult2 = sheetInterpret( + Match({ name: 'no name' }, recs) + ) as any[] + expect(matchResult2).toStrictEqual([]) + }) + + + test('GroupConstraint', () => { + const recs = getRecs() + const greenEyes = sheetInterpret(Match(recs, { eyeColor: 'green' }), {}) + const retireds = sheetInterpret(Match({ job: 'ret' }, recs), {}) + + const gcResult = sheetInterpret( + GroupConstraintItem( + Group(), + Unless( + GreaterThan(Count(Match(MatchResult(), { job: 'kid' })), 0), + Error('No Kids') + ), + 'name', + greenEyes + ), + { group: greenEyes } + ) as FlatfileRecord[] + expect(gcResult[0].toJSON()['info']).toStrictEqual([]) + + const gcResult2 = sheetInterpret( + GroupConstraintItem( + Group(), + Unless( + GreaterThan(Count(Match(MatchResult(), { job: 'kid' })), 0), + Error('No Kids') + ), + 'name', + retireds + ), + {} + ) as FlatfileRecord[] + expect(gcResult2[0].toJSON()['info'][0]).toMatchObject({ + field: 'name', + message: 'No Kids', + }) + }) + test('GroupConstraintItem outputs properly', () => { + expect(GroupConstraintItem( + Group(), + Unless( + GreaterThan( + Count(Match(Group(), {job:'kid'})), + 0), + Error('No Kids')), + 'name', + Group())) + .toStrictEqual( + ['groupConstraintRow', + ['quote', ['variable', 'group']], + ['quote', ['when', ['not', ['>', ['count', ['match', ['variable', 'group'], {job:'kid'}]], 0 ]], + ['error', 'No Kids']]], + 'name', + ['variable', 'group']]) + }) +}) diff --git a/src/expression-lang/EXPR.ts b/src/expression-lang/EXPR.ts new file mode 100644 index 0000000..0751621 --- /dev/null +++ b/src/expression-lang/EXPR.ts @@ -0,0 +1,141 @@ +import { makeInterpreter, NestedIns } from '@flatfile/expression-lang' +import { error, warn, info, debug } from './sheetInterpret' +import { Message } from '@flatfile/configure' + +export const Add = (...args: any) => ['+', ...args] +export const Subtract = (...args: any) => ['-', ...args] +export const Mult = (...args: any) => ['*', ...args] +export const Div = (...args: any) => ['/', ...args] +export const Mod = (a: any, b: any) => ['mod', a, b] + +//Comparisons +export const GreaterThan = (a: any, b: any) => ['>', a, b] +export const GT = GreaterThan + +export const LessThan = (a: any, b: any) => ['<', a, b] +export const LT = LessThan + +export const GreaterThanEqual = (a: any, b: any) => ['>=', a, b] +export const GTE = GreaterThanEqual + +export const LessThanEqual = (a: any, b: any) => ['<', a, b] +export const LTE = LessThanEqual + +export const Equal = (a: any, b: any) => ['equal', a, b] + +export const Between = (a: any, test: any, b: any) => [ + 'and', + ['<', a, test], + ['<', test, b], +] + +//Math +export const Abs = (a: any) => ['abs', a] +export const Min = (...args: any) => ['min', ...args] +export const Max = (...args: any) => ['max', ...args] +export const Round = (...args: any) => ['round', ...args] + +//logic +export const Not = (a: any) => ['not', a] +export const And = (...args: any) => ['and', ...args] +export const Or = (...args: any) => ['or', ...args] // returns the first true element + + +export const Count = (expr: NestedIns) => ['count', expr] +export const NotEqual = (a: any, b: any) => ['neq', a, b] +export const When = (predicate: any, expr: any) => ['when', predicate, expr] +export const Unless = (predicate: any, expr: any) => [ + 'when', + ['not', predicate], + expr, +] + +export const NonUnique = (group:NestedIns, column:string) => ['nonUnique', group, column] + +export const Error = (message: string) => ['error', message] +export const Group = () => ['variable', 'group'] +export const Val = () => ['variable', 'val'] +export const MatchResult = () => ['variable', 'matchResult'] + +export const Match = (recordGrouping: any, matchSpec: any) => [ + 'match', + recordGrouping, + matchSpec +] +export const ImplicitGreaterThan = (comparand: any) => ['>', Val(), comparand] +export const ImplicitLessThan = (comparand: any) => ['>', Val(), comparand] + +export const GroupConstraintItem = ( + rowFilterExpr: NestedIns, + actionExpr: NestedIns, + origField: string, + group: NestedIns = ['variable', 'group'] +) => { + return [ + 'groupConstraintRow', + ['quote', rowFilterExpr], + ['quote', actionExpr], + origField, + group, + ] +} + +export const SumField = (grp:any, field:string) => ['sumField', grp, field] +export const Do = (...exprs:any[]) => ['do', ...exprs] + + +export const Debug = (expr: NestedIns) => ['debug', expr] + +export const RecordPick = (recordGroup:any, fieldName:any, defaultVal:any = undefined ) => + ['recordPick', recordGroup, fieldName, defaultVal] +export const Pick = (objs:any, fieldName:any) => ['pick', objs, fieldName] +export const JSONLoad = (str:any) => ['JSONLoad', str] +export const JSONStringify = (obj:any) => ['JSONStringify', obj] +export const SortedBy = (objs:any, field:any=undefined, direction:string='asc') => ['sortedBy', objs, field, direction] +export const First = (objs:any) => ['first', objs] +export const Get = (rec:any, field:any, defaultVal:any=undefined) => ['get', rec, field, defaultVal] +export const Without = (minuend:any, subtrahend:any) => ['without', minuend, subtrahend] +export const StrConcat = (a:any, b:any) => ['strConcat', a, b] +export const ArrayEquals = (a:any, b:any) => ['arrayEquals', a, b] +export const Uniq = (objs:any) => ['uniq', objs] + + +const simpleInterpret = makeInterpreter({error, warn, info, debug}) + +export const ErrorWhen = (predicate: any, errString: string) => { + return (val: any) => { + return simpleInterpret(['when', predicate, ['error', errString]], { val: val }) as Message[] + } +} + +export const ErrorUnless = (predicate: any, errString: string) => { + return (val: any) => { + return simpleInterpret(['when', ['not', predicate], ['error', errString]], { val: val }) as Message[] + } +} + + +export const WarnWhen = (predicate: any, errString: string) => { + return (val: any) => { + return simpleInterpret(['when', predicate, ['warn', errString]], { val: val }) as Message[] + } +} + +export const WarnUnless = (predicate: any, errString: string) => { + return (val: any) => { + return simpleInterpret(['unless', ['not', predicate], ['warn', errString]], { val: val }) as Message[] + } +} + +export const InfoWhen = (predicate: any, errString: string) => { + return (val: any) => { + return simpleInterpret(['when', predicate, ['info', errString]], { val: val }) as Message[] + } +} + +export const InfoUnless = (predicate: any, errString: string) => { + return (val: any) => { + return simpleInterpret(['when', ['not', predicate], ['info', errString]], { val: val }) as Message[] + } +} + diff --git a/src/expression-lang/sheetInterpret.ts b/src/expression-lang/sheetInterpret.ts new file mode 100644 index 0000000..b9dba40 --- /dev/null +++ b/src/expression-lang/sheetInterpret.ts @@ -0,0 +1,200 @@ +import { makeInterpreter, NestedIns } from '@flatfile/expression-lang' +import { Message } from '@flatfile/configure' +import { FlatfileRecord, FlatfileRecords } from '@flatfile/hooks' +import { Sheet } from '@flatfile/configure' +import * as _ from 'lodash' + +export type TRecordStageLevel = + | 'compute' + | 'validate' + | 'apply' + | 'other' + + +//I would love message to be able to accept some type of format string +export const error = (message: string, stage: TRecordStageLevel = 'validate') => { + // I don't like returning a list here, not sure where to deal with scalar/list + return [new Message(message, 'error', stage)] +} +export const warn = (message: string, stage: TRecordStageLevel = 'validate') => { + return [new Message(message, 'warn', stage)] +} +export const info = (message: string, stage: TRecordStageLevel = 'validate') => { + return [new Message(message, 'info', stage)] +} + + +const match = (records: FlatfileRecord[], matchSpec: object) => { + return _.filter(records, (rec: FlatfileRecord) => + _.isMatch(rec.originalValue, matchSpec) + ) +} + +const nonUnique = (records: FlatfileRecord[], column:string) => { + const groups = _.groupBy(records, (rec) => + rec.get(column)) + //_.forEach(groups, (group: FlatfileRecord[], gbKey) => { + const groupSets = _.map(groups, (group: FlatfileRecord[]) => group) + const nonUniqueSets = _.filter(groupSets, + (groupedBy:FlatfileRecord[]) => { + if (groupedBy.length > 1) { + return true} + return false}) + return _.flatten(nonUniqueSets) +} + + + + +const groupByCompute = ( + gbArgs: any, + sheet: Sheet, + records: FlatfileRecords +) => { + const { groupBy, expression, destination } = gbArgs + const recs = records.records + const groups: Record = _.groupBy(recs, (rec) => + rec.get(groupBy) + ) + + _.forEach(groups, (group: FlatfileRecord[], gbKey) => { + const res = simpleInterpret(expression, { sheet, group }) + for (const rec of group) { + //@ts-ignore + rec.set(destination, res) + } + }) + return records +} + + +const sumField = (records: FlatfileRecord[], field: string) => { + //@ts-ignore + const allVals = records.map((rec) => rec.get(field)) + const presentVals = _.remove(allVals) + //@ts-ignore + const numberVals = presentVals.map((val:string|number) => parseFloat(val)) + const sum_ = numberVals.reduce((a,b) => a+b) + return sum_ +} + + +const groupConstraintRow = ( + rowFilterExpr: NestedIns, + actionExpr: NestedIns, + origField: string, + group: any +) => { + // how to pass along variables from where this is evaluated??? + const appliccableRows = sheetInterpret(rowFilterExpr, { + group, + }) as FlatfileRecord[] + //some typeguard here + if (Array.isArray(appliccableRows) && appliccableRows.length > 0) { + const messagesToApply = sheetInterpret(actionExpr, { + group, + matchResult: appliccableRows, + }) as Message[] + if (Array.isArray(messagesToApply)) { + appliccableRows.map((record: FlatfileRecord) => { + messagesToApply.map((m) => { + //@ts-ignore + record.pushInfoMessage(origField, m.message, m.level, 'validate') + }) + }) + } + } + return appliccableRows +} + + +/* +RecordPick - takes a recordGroup and field name, returns a list of values for that field +Pick - takes a list of objects and a field name, returns a list of values for that field +JSONLoad - calls JSON.parse on a string returns the object +SortedBy - takes an array of objects, sorts them by field name +First - returns the first element of an array +Get - returns a field from an object +Without - performs set diff on two lists of primitives +strConcat - concatates objects/strings into a string +ArrayEquals - verifies that two arrays are equal (edited) +*/ + +const recordPick = (records:FlatfileRecord[], fieldName:string, defaultVal:any=undefined ) => { + return records.map((rec) => { + const v = rec.get(fieldName) + if( v === null) { + return defaultVal + } + return v + }) +} + +const pick = (objs:Record[], fieldName:string ) => { + if(!_.isArray(objs)) { + //once again a hack + return objs[fieldName] + } + return objs.map((o) => o[fieldName]) +} + +const JSONLoad = (str:string) => { + try { + if (_.isArray(str)) { + //this is a hack + //@ts-ignore + return str.map(JSON.parse) + } + return JSON.parse(str) + } + catch (e:any) {return {}} +} + +const JSONStringify = (obj:any) => JSON.stringify(obj) + +//@ts-ignore +const sortedBy = (objs:Record[], field:string, direction:string) => _.orderBy(objs, [field], [direction]) + +const first = (objs:any[]) => objs[0] + +const get = (rec:FlatfileRecord, field:string, defaultVal:any=undefined) => { + try { + rec.get(field) + } catch (e:any) { + return defaultVal + } +} +const without = (full:any[], subtrahend:any[]) => _.without(full, ...subtrahend) +const uniq = (full:any[]) => _.uniq(full) +const strConcat = (a:any, b:any) => a.toString() + b.toString() +const arrayEquals = _.isEqual + +export const debug = (expr: NestedIns) => { + console.log('debug', expr) + return expr +} + +const do_ = (...exprs: any) => exprs[exprs.length -1] + +//@ts-ignore +const simpleInterpret = makeInterpreter({ + sumField, groupConstraintRow, nonUnique, error, match, 'do': do_, + recordPick, pick, JSONLoad, JSONStringify, sortedBy, first, + //@ts-ignore + get, + debug, uniq, + without, strConcat, arrayEquals }) + +export const sheetInterpret = makeInterpreter({ + error, + match, + groupByCompute, + groupConstraintRow, + nonUnique, + debug, + recordPick, pick, JSONLoad, JSONStringify, sortedBy, first, + //@ts-ignore + get, uniq, + without, strConcat, arrayEquals +}) + diff --git a/src/utils/testing/SheetTester.ts b/src/utils/testing/SheetTester.ts index a7b3bf2..9799488 100644 --- a/src/utils/testing/SheetTester.ts +++ b/src/utils/testing/SheetTester.ts @@ -1,16 +1,37 @@ import _ from 'lodash' import { IRecordInfo, TRecordData, TPrimitive, FlatfileRecords, FlatfileSession, IPayload } from '@flatfile/hooks' -import { Workbook } from '@flatfile/configure' +import { NumberField, Sheet, TextField, Workbook } from '@flatfile/configure' +import { sheetInterpret } from '../../expression-lang/sheetInterpret' +const localSheetCompute = ( + sheet: Sheet, + records: FlatfileRecords +) => { + + const possibleSheetCompute = sheet.getSheetCompute() + if (possibleSheetCompute === undefined) { + return records + } else { + //@ts-ignore + const afterSheetCompute = sheetInterpret(possibleSheetCompute.sheetCompute, { + sheet, + modifiedRecords: records, + }) + return afterSheetCompute + } +} + export class SheetTester { public workbook public sheetName + private rawSheetName private testSession: IPayload constructor( public readonly passedWorkbook: Workbook, public readonly passedSheetName: string ) { this.sheetName = `${passedWorkbook.options.namespace}/${passedSheetName}` + this.rawSheetName = passedSheetName this.workbook = passedWorkbook this.testSession = { schemaSlug: '', @@ -50,7 +71,10 @@ export class SheetTester { const inputRecords = new FlatfileRecords(iRaw) await this.workbook.processRecords(inputRecords, session) - return inputRecords + + const sheet = this.workbook.options.sheets[this.rawSheetName] + return localSheetCompute(sheet, inputRecords) + //return inputRecords } public async transformField( @@ -98,20 +122,23 @@ export class SheetTester { public async testRecord(record: {}) { const transformedRecords = await this.transformRecords([record]) + //@ts-ignore return transformedRecords.records[0].value } public async testRecords(recordBatch: Record[]) { const transformedRecords = await this.transformRecords(recordBatch) - + //@ts-ignore return transformedRecords.records.map((r) => r.value) } public async testMessage(record: {}) { const transformedRecords = await this.transformRecords([record]) + //@ts-ignore return transformedRecords.records.map((r) => r.toJSON().info)[0] } public async testMessages(recordBatch: Record[]) { const transformedRecords = await this.transformRecords(recordBatch) + //@ts-ignore return transformedRecords.records.map((r) => r.toJSON().info) }