v3.0.0

🎉 New features

• Adds off-heap caching to reduce memory footprint by @cmgrote in #984
• Parses precision, scale, max length and raw definition from SQL dataType by @cmgrote in #1006
• Adds new connector types by @Aryamanz29 in #1005

🧪 Experimental

• Adds a new Application type with creator and unit tests by @prateekrai-atlan in #1008

⛑️ Breaking changes

• Previously, there were at least 2-3 separate approaches used for caching across the SDK and custom package toolkits, despite all of them relying purely on on-heap memory. In this release caches have been revamped:
  • to be more consistent (same operations across caches)
  • to move them off-heap (with disk spilling)
• We have tried to minimize the impact of this change, but if you are using these caches directly (or related custom package runtime toolkit functionality) you may be impacted. See the end of the release notes for further details.
• The custom package testing toolkit's interface has also been changed slightly:
  • The PackageTest class now requires a "tag" parameter, that is used in all generated names for that test to make them more immediately identifiable.
  • Rather than calling the setup() method with your custom package's configuration and then directly invoking your custom package's code (typically via its main() method), you should now use the runCustomPackage() method — which takes both the configuration and your custom package's execution method as parameters. This ensures your custom package is executed with appropriate isolation from any others that may be tested in parallel.

🐞 Bug fixes

• Fixes an issue in asset batching related to case-insensitivity handling by @ErnestoLoma in #990

📦 Packages

• Fixes a dependency issue on Google Cloud by @ErnestoLoma in #974
• Fixes container name used on Azure by @ErnestoLoma in #1013
• Passes through failOnErrors in relational assets builder, to allow continuation beyond failures by @cmgrote in #994
• Asset import improvements by @cmgrote in #1014
• Extends off-heap caching to all custom package asset caches by @cmgrote in #988
• Fixes generalization-specialization relationship attribute names for data model assets by @Aryamanz29 in #975

🥗 QOL improvements

• Extends off-heap caching to asset batch processing tracking by @cmgrote in #988

Full Changelog: v2.3.2...v3.0.0

Caching changes

As mentioned in the breaking changes, this release introduces a complete revamp of the way caching is done across both the SDK and custom package toolkits.

Moved off-heap

Previously the caching was done entirely in-memory, on-heap. This naturally limited its scale — to the point where we had begun to observe OutOfMemory errors in some cases. With this release, all caches are moved off-heap and managed by an embedded, persistent key-value store that still offers high performance. (Note that this does mean some temporary files will be created locally to manage the caches — these are created using Files.createTempDirectory(), so the specific location will depend on how the JVM manages this on your system.)

Consistency changes

We've consolidated 2-3 different interfaces for caching that had evolved independently during the lifecycle of the SDK and package toolkit to date, resolving that tech debt so that all can directly benefit from the move to off-heap.

• Each cache now manages the following for each object in it:
  • an ID (always a UUID)
  • a human-readable name
  • an optional secondary ID (such as the hashed-string internal representation for tags and custom metadata)
  • the actual object that was cached
• Retrieval now uses these options almost exclusively, for consistency. For example, instead of .getCustomMetadataDef() on the custom metadata cache, you would now just use .getByName().
• Prior operations like lookupAssetByIdentity and lookupAssetByGuid in some cache variations have been made consistent, to lookupByIdentity and lookupById.
• Wherever possible the caches are now implemented using Java generics typing, so should return strong types without needing casting (rather than generic types like Asset).

Necessary interface changes

Various places that used to make use of collections (Collection<Asset>, List<Asset>, etc) now instead either directly make use of the new OffHeapAssetCache or of Stream<Asset> — to no longer require everything in-memory on the heap in order to be able to process it. All caches are now generally Closeable, too, to allow them to self-manage cleaning up any temporary files they use for persistence.

• Utils.updateConnectionCache() in the custom package runtime toolkit (now uses OffHeapAssetCache)
• PersistentConnectionCache.deleteAssets() and .addAssets() in the custom package runtime toolkit (now process Stream<Asset>)
• AssetGenerator.cacheCreated() interface (now processes Stream<Asset>)
• ImportResults.Details internal tracking of created, updated, restored and skipped assets (now use OffHeapAssetCache)
• AssetRemover.deleteAssets() in the custom package runtime toolkit (now uses OffHeapAssetCache)
• DeltaProcessor.run() in the custom package runtime toolkit (now uses OffHeapAssetCache)
• AtlanClient itself is now Closeable, so it can clean up its own managed caches
• AssetBatch and ParallelBatch no longer offer fluent-builder creation, but only overloaded constructors, and both now track their processing of created, updated, restored and skipped assets in OffHeapAssetCaches, and are now Closeable so they can clean up their own managed caches.

v2.3.2

🎉 New features

• Latest models by @cmgrote in #954

🧪 Experimental

• Adds a new attribute ModelAttributeAssociation to track entity relationship for which it provides details @Aryamanz29 in #960
• Adds generalization-specialization relationship to data model assets by @Aryamanz29 in #951

🐞 Bug fixes

• Fixes logic used to populate LinkCache by @ErnestoLoma in #959, #962 and #968
• Fixes an infinite loop in loading data products with the same name by @ErnestoLoma in #965
• Fixes construction of user-agent headers to avoid potential misinterpretation in back-end parsing by @cmgrote in #969

📦 Packages

• Makes LFTagsOnTable and LFTagsOnColumns optional in LF tag sync package by @ErnestoLoma in #952

🥗 QOL improvements

• Dependency bumps by @cmgrote in #963

Full Changelog: v2.3.1...v2.3.2

v2.3.1

🎉 New features

• Adds bulk streaming for non-indexsearch by @cmgrote in #926
• Enhances concurrency for parallel batching by @cmgrote in #927 and #940

🧪 Experimental

• Model asset attribute changes by @Aryamanz29 in #935 and #937

🐞 Bug fixes

• Adds missing deserialization for domain policy actions by @cmgrote in #933
• Fixes a bug where stale connection admins could prevent adding others by @cmgrote in #938

📦 Packages

• Changes default load option for relational assets builder back to incremental by @cmgrote in #944
• Corrects GCS -> ADLS for logging by @cmgrote in #936

🥗 QOL improvements

• Reuses version catalog and enforces versions during build by @cmgrote in #945

Full Changelog: v2.3.0...v2.3.1

v2.3.0

🎉 New features

• Adds INRIVER connector type by @Aryamanz29 in #923
• Adds stronger concurrency controls to parallel batching by @cmgrote in #927
• Adds bulk streaming (paging beyond 10,000 entries) for non-indexsearch searches by @cmgrote in #926

🧪 Experimental

• Adds a new relationship for the implementation of an entity by one or more assets by @Aryamanz29 and @cmgrote in #921 and #925
• Fixes model asset references to use a version-agnostic qualifiedName by @cmgrote in #916 and #918

🐞 Bug fixes

• Fixes attribute validations for assets to avoid potential NPE by @cmgrote in #917
• Fixes an issue where deleted Atlan tags in audit log entries could cause an NPE by @cmgrote in #926

📦 Packages

• Uses requested connection qualifiedName as fallback for the OpenAPI spec loader package by @cmgrote in #922
• Adds support for assuming roles in AWS to the package runtime toolkit by @ErnestoLoma in #924

Full Changelog: v2.2.6...v2.3.0

v2.2.6

🐞 Bug fixes

• Fixes bug with timestamp granularity by @cmgrote in #913

Full Changelog: v2.2.5...v2.2.6

v2.2.5

🧪 Experimental

• Further experimental work on data model assets by @cmgrote in #909

🐞 Bug fixes

• Fixes an issue where updatedBy missing from credentials would cause an error by @cmgrote in #910

📦 Packages

• Adds detailed search log options to adoption export by @cmgrote in #911

Full Changelog: v2.2.4...v2.2.5

v2.2.4

🧪 Experimental

• Updated metamodel for data model assets by @cmgrote in #884

🐞 Bug fixes

• Fixes missing updatedBy property on Credential object by @cmgrote in #905
• Ignores unknown properties during deserialization to avoid any similar bug in future by @cmgrote in #903

Full Changelog: v2.2.3...v2.2.4

v2.2.3

🐞 Bug fixes

• Restores main info logging by @cmgrote in #899

📦 Packages

• Adds an option to export all attributes by @cmgrote in #898
• Fixes a bug where timestamps of 0 would be included as a January 1, 1970 timestamp rather than being left out

Full Changelog: v2.2.2...v2.2.3

v2.2.2

🎉 New features

• Only debug-level log the Atlan SDK by @cmgrote in #896

🐞 Bug fixes

• Adds null-checking to order field of columns by @cmgrote in #895

🥗 QOL improvements

• Force updated transitive dependencies by @cmgrote in #892
• Dependency bumps by @cmgrote in #893

Full Changelog: v2.2.1...v2.2.2

v2.2.1

🧪 Experimental

• Adds version-agnostic qualifiedName attribute for data model assets by @cmgrote in #876

🐞 Bug fixes

• Adds typedef toolkit attribute naming validation to avoid conflicts by @cmgrote in #875
• Improve concurrency handling in caches by @cmgrote in #885

🥗 QOL improvements

• Adds CVE scanning and detection, fixes transitive versions by @cmgrote in #881

Full Changelog: v2.2.0...v2.2.1