[WIP] core: qdrant_migrator #2571

spolu · 2023-11-16T15:32:17Z

Tested locally (log below). A document was added and another one deleted during the migration.

spolu@spolu-box ~/code/dust/core (spolu-migrate_collection) $ cargo run --bin qdrant_migrator 
Tooling to migrate Qdrant collections

Usage: qdrant_migrator <COMMAND>

Commands:
  show                  Show qdrant state
  set-shadow-write      Set `shadow_write_cluster` (!!! creates collection on `shadow_write_cluster`)
  clear-shadow-write    Clear `shadow_write_cluster` (!!! deletes collection from `shadow_write_cluster`
  migrate-shadow-write  Migrate `cluster` collection to `shadow_write_cluster`
  commit-shadow-write   Switch `shadow_write_cluster` and `cluster` (!!! moves read traffic to `shadow_write_cluster`)
  help                  Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help

spolu@spolu-box ~/code/dust/core (spolu-migrate_collection) $ cargo run --bin qdrant_migrator set-shadow-write 4 test-migrate dedicated-0
  Finished dev [unoptimized + debuginfo] target(s) in 0.11s
   Running `target/debug/qdrant_migrator set-shadow-write 4 test-migrate dedicated-0`
[!] Error: Data source not found
spolu@spolu-box ~/code/dust/core (spolu-migrate_collection) $ cargo run --bin qdrant_migrator set-shadow-write 4 test-migrate dedicated-0
  Finished dev [unoptimized + debuginfo] target(s) in 0.10s
   Running `target/debug/qdrant_migrator set-shadow-write 4 test-migrate dedicated-0`
[!] Error: Data source not found
spolu@spolu-box ~/code/dust/core (spolu-migrate_collection) $ cargo run --bin qdrant_migrator set-shadow-write 4 test-migrate dedicated-0
"    Finished dev [unoptimized + debuginfo] target(s) in 0.10s
   Running `target/debug/qdrant_migrator set-shadow-write 4 test-migrate dedicated-0`
[!] Error: Data source not found
spolu@spolu-box ~/code/dust/core (spolu-migrate_collection) $ ^C
spolu@spolu-box ~/code/dust/core (spolu-migrate_collection) $ cargo run --bin qdrant_migrator set-shadow-write 4 "test-migrate" dedicated-0
  Finished dev [unoptimized + debuginfo] target(s) in 0.10s
   Running `target/debug/qdrant_migrator set-shadow-write 4 test-migrate dedicated-0`
[!] Error: Data source not found
spolu@spolu-box ~/code/dust/core (spolu-migrate_collection) $ cargo run --bin qdrant_migrator set-shadow-write 5 test-migrate dedicated-0
  Finished dev [unoptimized + debuginfo] target(s) in 0.11s
   Running `target/debug/qdrant_migrator set-shadow-write 5 test-migrate dedicated-0`
[✓] Created qdrant shadow_write_cluster collection: collection=ds_0fdeb4abe22bd528f9c7cb967b778c6a2fc45ba96340fd96e3c44654a9c77644 shadow_write_cluster=dedicated-0
[✓] Updated data source: collection=ds_0fdeb4abe22bd528f9c7cb967b778c6a2fc45ba96340fd96e3c44654a9c77644 cluster=main-0 shadow_write_cluster=dedicated-0
spolu@spolu-box ~/code/dust/core (spolu-migrate_collection) $ cargo run --bin qdrant_migrator migrate-shadow-write 5 test-migrate
  Finished dev [unoptimized + debuginfo] target(s) in 0.10s
   Running `target/debug/qdrant_migrator migrate-shadow-write 5 test-migrate`
[i] Migrated points: count=2 total=2
[i] Done migrating: total=2
spolu@spolu-box ~/code/dust/core (spolu-migrate_collection) $ cargo run --bin qdrant_migrator commit-shadow-write 5 test-migrate
  Finished dev [unoptimized + debuginfo] target(s) in 0.09s
   Running `target/debug/qdrant_migrator commit-shadow-write 5 test-migrate`
[i] Updated data source: collection=ds_0fdeb4abe22bd528f9c7cb967b778c6a2fc45ba96340fd96e3c44654a9c77644 cluster=dedicated-0 shadow_write_cluster=main-0
spolu@spolu-box ~/code/dust/core (spolu-migrate_collection) $ cargo run --bin qdrant_migrator clear-shadow-write 5 test-migrate
  Finished dev [unoptimized + debuginfo] target(s) in 0.11s
   Running `target/debug/qdrant_migrator clear-shadow-write 5 test-migrate`
[?] [DANGER] Are you sure you want to delete this qdrant shadow_write_cluster collection? (this is definitive) shadow_write_cluster=main-0 points_count=2 Confirm ([y]/n) ? 
[✓] Deleted qdrant shadow_write_cluster collection: collection=ds_0fdeb4abe22bd528f9c7cb967b778c6a2fc45ba96340fd96e3c44654a9c77644 shadow_write_cluster=main-0
[✓] Updated data source: collection=ds_0fdeb4abe22bd528f9c7cb967b778c6a2fc45ba96340fd96e3c44654a9c77644 cluster=dedicated-0 shadow_write_cluster=none

fontanierh

looks great

fontanierh · 2023-11-17T15:19:27Z

core/src/data_sources/data_source.rs

        format!("ds_{}", self.internal_id)
    }

-    pub async fn setup(
+    pub async fn update_config(


The only thing we ever want to update is qdrant_config no ? Would it be better to only expose that ?

The store function will remain the same so better align this one as it can be reused

I would have kept the sore fn flexible, but only expose what you can actually change in update_config as this is the interface for the rest of the app.
It's "clear" that you can do "unsupported" things by calling store, but to me data_source functions shouldn't leave things in an inconsistent state

fontanierh · 2023-11-17T15:23:27Z

core/src/data_sources/qdrant.rs

@@ -10,16 +12,37 @@ use serde::{Deserialize, Serialize};
 pub enum QdrantCluster {


nit: I think we can [serde(rename_all = "kebab-case")]

fontanierh · 2023-11-17T15:25:50Z

core/src/data_sources/qdrant.rs

+static QDRANT_CLUSTER_VARIANTS: &[QdrantCluster] =
+    &[QdrantCluster::Main0, QdrantCluster::Dedicated0];
+
+impl ToString for QdrantCluster {


out of scope but I am suprised that we have to implement these ourselves everytime, there must be a serde (or other) thing to do this for us

fontanierh · 2023-11-17T15:27:44Z

core/bin/qdrant_migrator.rs

+    },
+}
+
+/// A fictional versioning CLI


Not sure I understand what is fictional about it, wdym ?

cargo cult from a lib

fontanierh · 2023-11-17T15:29:39Z

core/bin/qdrant_migrator.rs

+    let args = Cli::parse();
+
+    let rt = tokio::runtime::Builder::new_multi_thread()
+        .worker_threads(32)


why not leave the default ? it will pick the # of cores available on the system.
I don't think we gain anything from having 32 since we'll be running this on machines that don't have many cores.

This is aligned with dust_api it's not only about the core and we know the default can be a foot gun if some things are blocking in unexpected ways

fontanierh · 2023-11-17T15:33:26Z

core/bin/qdrant_migrator.rs

+                // the embeedding size (which is what happens here). May need to be revisited in
+                // future.
+                let mut credentials = Credentials::new();
+                credentials.insert("OPENAI_API_KEY".to_string(), "foo".to_string());


this is a little bit of a code smell, but understandable if we don't want to fix

It is but it will error if it becomes irrelevant (eg we use another provider that require a credentials to perform this function)

fontanierh · 2023-11-17T15:35:55Z

core/bin/qdrant_migrator.rs

+                project_id,
+                data_source_id,
+            } => {
+                // This is the most dangerous command of all as it is the only one to actually


I think this comment is wrong

It's accurate

isn't the clear command the most dangerous one ?

fontanierh · 2023-11-17T15:36:21Z

core/bin/qdrant_migrator.rs

+
+                let qdrant_client = qdrant_clients.main_client(&ds.config().qdrant_config);
+
+                // Delete collection on shadow_write_cluster.


same here ?

moved 3 lines below

Is this function actually deleting ? I thought this one was copying ?

spolu added 2 commits November 16, 2023 16:31

WIP qdrant_migrator

5b007d5

WIP migrate

55495e7

spolu requested a review from fontanierh November 17, 2023 15:15

code

14f172c

fontanierh previously approved these changes Nov 17, 2023

View reviewed changes

nits

2629a5c

spolu dismissed fontanierh’s stale review via 2629a5c November 17, 2023 15:52

nit

69f77f4

spolu merged commit edceae6 into main Nov 17, 2023
1 check passed

spolu deleted the spolu-migrate_collection branch November 17, 2023 15:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] core: qdrant_migrator #2571

[WIP] core: qdrant_migrator #2571

spolu commented Nov 16, 2023 •

edited

Loading

fontanierh left a comment

fontanierh Nov 17, 2023

spolu Nov 17, 2023

fontanierh Nov 17, 2023

fontanierh Nov 17, 2023

fontanierh Nov 17, 2023

fontanierh Nov 17, 2023

spolu Nov 17, 2023

fontanierh Nov 17, 2023

fontanierh Nov 17, 2023

spolu Nov 17, 2023

fontanierh Nov 17, 2023

spolu Nov 17, 2023

fontanierh Nov 17, 2023

spolu Nov 17, 2023

fontanierh Nov 17, 2023

fontanierh Nov 17, 2023

spolu Nov 17, 2023

fontanierh Nov 17, 2023

		@@ -10,16 +12,37 @@ use serde::{Deserialize, Serialize};
		pub enum QdrantCluster {


		let qdrant_client = qdrant_clients.main_client(&ds.config().qdrant_config);

		// Delete collection on shadow_write_cluster.

[WIP] core: qdrant_migrator #2571

[WIP] core: qdrant_migrator #2571

Conversation

spolu commented Nov 16, 2023 • edited Loading

fontanierh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spolu commented Nov 16, 2023 •

edited

Loading