Modify clob handling (liquibase#6410)

tati-qalified · tatibeans · StevenMassaro · web-flow · commit 3db80b4197d7 · 2024-10-28T07:31:58.000-03:00
* changed code, started with test * Base tests * Modify tests * working on the groovy test * test: change order of stream closing and file deletion (liquibase#6380) * DAT-18563 :: Unhide tag parameter for UpdateTestingRollback command (liquibase#6330) - Unhide tag parameter for UpdateTestingRollback command. - Update command test. - Added integration test. * Removed usage of deprecated isEmpty() method (liquibase#6205) Removed usage of deprecated isEmpty method Co-authored-by: Daniel Mallorga <75833793+MalloD12@users.noreply.github.com> Co-authored-by: Daniel Mallorga <dmallorga@liquibase.com> Co-authored-by: filipe <flautert@liquibase.org> * liquibase#6281 fix NPE when default value is null (liquibase#6287) Sample: ```java java.lang.NullPointerException: Cannot invoke "String.replaceAll(String, String)" because "defaultValue" is null at liquibase.snapshot.jvm.ColumnSnapshotGenerator.readDefaultValue(ColumnSnapshotGenerator.java:571) at liquibase.snapshot.jvm.ColumnSnapshotGenerator.readColumn(ColumnSnapshotGenerator.java:291) at liquibase.snapshot.jvm.ColumnSnapshotGenerator.addTo(ColumnSnapshotGenerator.java:204) at liquibase.snapshot.jvm.JdbcSnapshotGenerator.snapshot(JdbcSnapshotGenerator.java:78) at liquibase.snapshot.SnapshotGeneratorChain.snapshot(SnapshotGeneratorChain.java:49) ``` * liquibase#6264 avoid ignoring the caught exception (liquibase#6288) * delete unused class UnknownConfigurationType * init field DatabaseIncapableOfOperation.operation it was not assigned in constructor; thus it was always NULL. P.S. This field is not used, so it should be removed. * liquibase#6264 avoid ignoring the caught exception I found all places where IDEA shows warning "catch block may ignore exception", and fixed many of them. * Adressing issue liquibase#6331, small changes to pom.xml files to reduce warnings shown in building (liquibase#6332) Use ${project.version} instead of ${version}; same goes for ${build.finalName}, which is recommended to switch to ${project.build.finalName}. Finally, use version for `maven-jar-plugin` consistently. Co-authored-by: Carlo Dapor <carlo.dapor@mimacom.com> * DAT-18743: dry-run: add current_branch_name (liquibase#6382) * missing ending fi statement * add simple logic, if the branch is neither "release" nor "master," it sets branch to the current branch name. * populate the value of the branch in dry-run-release.yml and pass it as outputs * populate the value of the branch in dry-run-release.yml and pass it as outputs --------- Co-authored-by: Sayali M <sayali@Sayalis-MacBook-Pro> * DAT-18329: comment out pieces of code that builds the Mac Installer (liquibase#6379) * comment out pieces of code that build the Mac Installer * delete macOS from install4j software and results produced are commented out * remove mac references --------- Co-authored-by: Sayali M <sayali@Sayalis-MacBook-Pro> * Handle Oracle blank schema names DAT-18199 (liquibase#6305) * Handle Oracle blank schema names DAT-18199 * Added comment DAT-18199 * Tweak last fix DAT-18199 * Another fix for the last fix DAT-18199 --------- Co-authored-by: suryaaki2 <80348493+suryaaki2@users.noreply.github.com> * DAT-18191 and DAT-18192 - Fix classpath loading and incorrect configuration for addForeignKeyConstraint (liquibase#6109) * fix: do not call DatabaseFactory before classpath parameter is evaluated by command line * chore: document changes/ refactor code * chore: remove offending method * fix: how sqlite supports initiallyDeferred and deferred if it doesn't work with addForeignKey ?! * fix: typo * Handle blank change set ID DAT-18687 (liquibase#6359) * WIP DAT-18687 * Handle blank change set IDs DAT-18687 * Add JavaDoc and another test --------- Co-authored-by: suryaaki2 <80348493+suryaaki2@users.noreply.github.com> * allow replaceIfExists in Postgres procedures (DAT-18495) (liquibase#6350) * allow replaceIfExists in Postgres procedures * reduce code duplication * CreateProcedure definition test updated to include postgresql as a supported database. --------- Co-authored-by: obovsunivskyii <baqaua@gmail.com> Co-authored-by: Daniel Mallorga <dmallorga@liquibase.com> Co-authored-by: suryaaki2 <80348493+suryaaki2@users.noreply.github.com> * fix CreateProcedureChange not working with mariadb (DAT-18495) (liquibase#6389) * Bump groovy.version from 4.0.22 to 4.0.23 (liquibase#6319) Bumps `groovy.version` from 4.0.22 to 4.0.23. Updates `org.apache.groovy:groovy-bom` from 4.0.22 to 4.0.23 - [Commits](https://github.com/apache/groovy/commits) Updates `org.apache.groovy:groovy-all` from 4.0.22 to 4.0.23 - [Commits](https://github.com/apache/groovy/commits) --- updated-dependencies: - dependency-name: org.apache.groovy:groovy-bom dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.apache.groovy:groovy-all dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add support for include columns in indexes (liquibase#6138) * Update ABOUT.txt * trying to get include columns to work with indexes * Update ABOUT.txt * Now with passing tests * got creating indexes to work properly. Still need to get generate changelog to work * got the generate-changelog command working with indexes that include columns * cleanup unnecesary changes in the diff * Added include field to the xsd * updated to use booleanutils * switched to non deprecated method * Added unit test coverage for index include feature * put generator into an if statement based on database * put more sections behind ifs on db type * do not strip "classpath:" when normalizing the path (liquibase#5894) This PR introduces an alternative behavior for sub-resource path resolution, namely if a root resource path starts with the classpath: prefix, it would preserve that prefix when resolving sub-resource locations. This behavior is activated with liquibase.preserveClasspathPrefixInNormalizedPaths property (it must be set to true), or via LIQUIBASE_PRESERVE_CLASSPATH_PREFIX_IN_NORMALIZED_PATHS variable or --preserve-classpath-prefix-in-normalized-paths CLI option. This feature allows to implement custom ResourceAccessors that work with abstract URLs without being directly tied to the filesystem, such as the one in Bootique.io. * Handle Snowflake add NOT NULL constraint DAT-18798 (liquibase#6391) * Fix generated SQL for Snowflake NOT NULL constraint DAT-18798 * Handle Snowflake NOT NULL constraint DAT-18798 --------- Co-authored-by: obovsunivskyii <baqaua@gmail.com> * chore: changes to make the test work - as it will be a PreparedStatement it needs to be supportsBatchUpdates - and as we are offline we need to override it. * Remove print statement * changed code, started with test * Base tests * Modify tests * working on the groovy test * chore: changes to make the test work - as it will be a PreparedStatement it needs to be supportsBatchUpdates - and as we are offline we need to override it. * Remove print statement * Changed string for the oracle clob test * Update names.csv * chore: take isRelativeToChangelogFile into account when evaluating the file location. * chore: fix variable name and tests --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: tatibeans <tatiana.f.ganz@gmail.com> Co-authored-by: Steven Massaro <steven.massaro.web@gmail.com> Co-authored-by: Daniel Mallorga <75833793+MalloD12@users.noreply.github.com> Co-authored-by: Anton Erofeev <antonerofeev11@gmail.com> Co-authored-by: Daniel Mallorga <dmallorga@liquibase.com> Co-authored-by: filipe <flautert@liquibase.org> Co-authored-by: Andrei Solntsev <andrei.solntsev@gmail.com> Co-authored-by: catull <catull@users.noreply.github.com> Co-authored-by: Carlo Dapor <carlo.dapor@mimacom.com> Co-authored-by: Sayali Mohadikar <76010603+sayaliM0412@users.noreply.github.com> Co-authored-by: Sayali M <sayali@Sayalis-MacBook-Pro> Co-authored-by: Wesley Willard <wwillard@liquibase.com> Co-authored-by: suryaaki2 <80348493+suryaaki2@users.noreply.github.com> Co-authored-by: obovsunivskyii <baqaua@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Stephen Atwell <78044713+stephenatwell@users.noreply.github.com> Co-authored-by: Andrus Adamchik <aadamchik@gmail.com> Co-authored-by: filipe <flautert@liquibase.com> Co-authored-by: rberezen <ruslan.berezenskyi@gmail.com>
diff --git a/liquibase-integration-tests/src/test/resources/changelogs/common/names.csv b/liquibase-integration-tests/src/test/resources/changelogs/common/names.csv
@@ -1,2 +1,2 @@
 id,name
-1,lorem.txt
+1,lorem.txt
diff --git a/liquibase-standard/src/main/java/liquibase/change/core/LoadDataChange.java b/liquibase-standard/src/main/java/liquibase/change/core/LoadDataChange.java
@@ -39,6 +39,7 @@
 import liquibase.util.StringUtil;
 import liquibase.util.csv.CSVReader;
 import lombok.Setter;
+import org.apache.commons.lang3.BooleanUtils;
 import org.apache.commons.lang3.StringUtils;
 
 import java.io.IOException;
@@ -387,9 +388,23 @@ public SqlStatement[] generateStatements(Database database) {
                                 needsPreparedStatement = true;
                             }
                         } else if (columnConfig.getTypeEnum() == LOAD_DATA_TYPE.CLOB) {
-                            // Similar to the blob case, we expect ALL clobs found using loadData to be a valid path to a file.
-                            // We then load the entire file into the value when executing the statement.
-                            valueConfig.setValueClobFile(value);
+                            // Previously, we expected all clobs found using loadData to be a valid path to a file.
+                            // To maintain backwards compatibility, we will first try to find the file.
+                            // If found, we then load the entire file into the value when executing the statement.
+                            // If not found, we load the value as a string.
+                            Resource r;
+                            if (getRelativeTo() != null) {
+                                r = Scope.getCurrentScope().getResourceAccessor().get(getRelativeTo()).resolveSibling(value);
+                            } else {
+                                r = Scope.getCurrentScope().getResourceAccessor().get(value);
+                            }
+
+                            if (r.exists()) {
+                                valueConfig.setValueClobFile(value);
+                            } else {
+                                LOG.fine(String.format("File %s not found. Inserting the value as a string. See https://docs.liquibase.com for more information.", value));
+                                valueConfig.setValue(value);
+                            }
                             needsPreparedStatement = true;
                         }  else if (columnConfig.getTypeEnum() == LOAD_DATA_TYPE.UUID) {
                             valueConfig.setType(columnConfig.getType());
diff --git a/liquibase-standard/src/main/java/liquibase/command/core/helpers/AbstractChangelogCommandStep.java b/liquibase-standard/src/main/java/liquibase/command/core/helpers/AbstractChangelogCommandStep.java
@@ -8,6 +8,7 @@
 import liquibase.command.CommandBuilder;
 import liquibase.command.CommandScope;
 import liquibase.exception.CommandValidationException;
+import org.apache.commons.lang3.StringUtils;
 
 import java.util.ArrayList;
 import java.util.Arrays;
@@ -32,7 +33,7 @@ public abstract class AbstractChangelogCommandStep extends AbstractCommandStep {
                 .defaultValue("none").description("Sets runOnChange=\"true\" for changesets containing solely changes of these types (e. g. createView, createProcedure, ...).").build();
         REPLACE_IF_EXISTS_TYPES_ARG = builder.argument("replaceIfExistsTypes", String.class)
                 .defaultValue("none")
-                .description("Sets replaceIfExists=\"true\" for changes of these types (supported types: createProcedure, createView)").build();
+                .description(String.format("Sets replaceIfExists=\"true\" for changes of these types (supported types: %s)", StringUtils.join(REPLACE_IF_EXISTS_TYPES_NAMES, ", "))).build();
         SKIP_OBJECT_SORTING = builder.argument("skipObjectSorting", Boolean.class)
                 .defaultValue(false)
                 .description("When true will skip object sorting. This can be useful on databases that have a lot of packages/procedures that are " +
@@ -55,7 +56,7 @@ protected static void validateReplaceIfExistsTypes(final CommandScope commandSco
         supportedReplaceIfExistsTypes.add("none");
         replaceIfExistsTypes.removeAll(supportedReplaceIfExistsTypes);
         if (!replaceIfExistsTypes.isEmpty())
-            throw new CommandValidationException("Invalid types for --replace-if-exists-types: " + replaceIfExistsTypes.stream().collect(Collectors.joining(", ")) + ". Valid types are: " + REPLACE_IF_EXISTS_TYPES_NAMES.stream().collect(Collectors.joining(", ")));
+            throw new CommandValidationException("Invalid types for --replace-if-exists-types: " + replaceIfExistsTypes.stream().collect(Collectors.joining(", ")));
     }
 
     protected static Stream<String> supportedRunOnChangeTypes() {
diff --git a/liquibase-standard/src/test/groovy/liquibase/change/core/LoadDataChangeTest.groovy b/liquibase-standard/src/test/groovy/liquibase/change/core/LoadDataChangeTest.groovy
@@ -21,6 +21,7 @@ import liquibase.resource.ResourceAccessor
 import liquibase.resource.SearchPathResourceAccessor
 import liquibase.snapshot.MockSnapshotGeneratorFactory
 import liquibase.snapshot.SnapshotGeneratorFactory
+import liquibase.statement.BatchDmlExecutablePreparedStatement
 import liquibase.statement.DatabaseFunction
 import liquibase.statement.ExecutablePreparedStatement
 import liquibase.statement.ExecutablePreparedStatementBase
@@ -61,6 +62,60 @@ class LoadDataChangeTest extends StandardChangeTest {
     }
 
 
+
+    def "column with clob datatype is path or string"(){
+        when:
+        def changelog = new DatabaseChangeLog("liquibase/changelog.xml")
+        ChangeSet changeSet = new ChangeSet(null, null, true, false,
+                "logical or physical file name",
+                null, null, false, null, changelog)
+
+        LoadDataChange change = new LoadDataChange()
+        LoadDataColumnConfig col1 = new LoadDataColumnConfig()
+        col1.setType(LoadDataChange.LOAD_DATA_TYPE.CLOB)
+        col1.setName("exists")
+        change.addColumn(col1)
+
+        LoadDataColumnConfig col2 = new LoadDataColumnConfig()
+        col2.setType(LoadDataChange.LOAD_DATA_TYPE.CLOB)
+        col2.setName("doesnt")
+        change.addColumn(col2)
+
+        LoadDataColumnConfig col3 = new LoadDataColumnConfig()
+        col3.setType(LoadDataChange.LOAD_DATA_TYPE.CLOB)
+        col3.setName("string")
+        change.addColumn(col3)
+
+        change.setSchemaName("SCHEMA_NAME")
+        change.setTableName("TABLE_NAME")
+        change.setRelativeToChangelogFile(Boolean.TRUE)
+        change.setChangeSet(changeSet)
+        change.setFile("change/core/sample.data.for.clob.types.csv")
+
+        def mockDBCsv = new MockDatabase() {
+            @Override
+            boolean supportsBatchUpdates() {
+                return true
+            }
+        }
+
+        SqlStatement[] stmt = change.generateStatements(mockDBCsv)
+
+        then:
+        stmt.length == 1
+        assert stmt[0] instanceof BatchDmlExecutablePreparedStatement
+
+        List<ExecutablePreparedStatementBase> insSt = ((BatchDmlExecutablePreparedStatement) stmt[0]).getIndividualStatements()
+        List<LoadDataColumnConfig> columns = insSt.get(0).getColumns()
+
+        "SCHEMA_NAME" == insSt.get(0).getSchemaName()
+        "TABLE_NAME" == insSt.get(0).getTableName()
+        "change/core/SQLFileTestData.sql" == columns.get(0).getValueClobFile()
+        "nothing.txt" == columns.get(1).getValue()
+        "sample text" == columns.get(2).getValue()
+
+    }
+
     def "loadDataEmpty database agnostic"() throws Exception {
         when:
         LoadDataChange refactoring = new LoadDataChange()
diff --git a/liquibase-standard/src/test/resources/liquibase/change/core/sample.data.for.clob.types.csv b/liquibase-standard/src/test/resources/liquibase/change/core/sample.data.for.clob.types.csv
@@ -0,0 +1,2 @@
+exists,doesnt,string
+change/core/SQLFileTestData.sql,nothing.txt,sample text
diff --git a/liquibase-standard/src/test/resources/liquibase/test-changelog-clob-types.xml b/liquibase-standard/src/test/resources/liquibase/test-changelog-clob-types.xml
@@ -0,0 +1,22 @@
+<?xml version="1.1" encoding="UTF-8" standalone="no"?>
+<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
+                   xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
+                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+                   xmlns:pro="http://www.liquibase.org/xml/ns/pro"
+                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd
+http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd
+  http://www.liquibase.org/xml/ns/pro http://www.liquibase.org/xml/ns/pro/liquibase-pro-latest.xsd">
+
+    <changeSet id="create-table-with-clob-types" author="tfernandez">
+        <createTable tableName="testClobs">
+            <column name="col1" type="text"/>
+            <column name="col2" type="longtext"/>
+            <column name="col3" type="longvarchar"/>
+        </createTable>
+    </changeSet>
+
+    <changeSet id="load-data-to-table" author="tfernandez">
+        <loadData tableName="testClobs" file="change/core/sample.data.for.clob.types.csv" relativeToChangelogFile="true"/>
+    </changeSet>
+
+</databaseChangeLog>

Original file line number	Diff line number	Diff line change
`@@ -1,2 +1,2 @@`
`1`	`1`	`id,name`
`2`		`-1,lorem.txt`
	`2`	`+1,lorem.txt`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,2 @@`
	`1`	`+exists,doesnt,string`
	`2`	`+change/core/SQLFileTestData.sql,nothing.txt,sample text`