search image to CLI and copy matches (within a defined match value) to a folder

[rleaver152]

It would be great to be able to batch interrogate the database using a Bash script (within the virtual environment) and the CLI to:

eg input (eg -in flag) search image to CLI and copy matches (within a defined match value) to a (-o flag) named folder

[policratus]

Hey @rleaver152, how's it going? Thanks for the idea!

So, just let us check if we really understood this idea. Are you saying that would be great to do something like this using the CLI:

pupyl --data_dir /path/to/your/data/dir search /path/to/image/query.jpg --results /path/to/output/results

where pupyl will search inside database located in /path/to/your/data/dir and copy the image files associated with the search results to /path/to/output/results?

[rleaver152]

Hi - yes, exactly. Maybe also with a threshold setting for the similarity test. The benefit over imgdupes and other similarity programs is that with this, you can see on the serve interface the file images AND get copies to a directory of choice. Otherwise you just see on the web serve the matching pictures but no way of locating them easily (IMHO) Richard

…

On 26 Nov 2021, at 14:47, policratus ***@***.***> wrote: Hey @rleaver152, how's it going? Thanks for the idea! So, just let us check if we really understood this idea. Are you saying that would be great to do something like this using the CLI: pupyl --data_dir /path/to/your/data/dir search /path/to/output/results where pupyl will search inside database located in /path/to/your/data/dir and copy the image files associated with the search results to /path/to/output/results? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[policratus]

Awesome @rleaver152! Just opened issue #123 to address this. Please check if you agree with the specification and succeeding with that, we'll start to work on this.

[policratus]

Hey @rleaver152, just opened pull request #133 implementing this idea. After merging it, this will be possible:

pupyl --data_dir /tmp/pupyl search https://pbs.twimg.com/media/E-4t8mbVQAoK4Dd.jpg
pupyl --data_dir /tmp/pupyl export https://pbs.twimg.com/media/E-4t8mbVQAoK4Dd.jpg /tmp/output --top 100

Please check there, because the above is only the beginning of the new features.

[rleaver152]

Hi - I've pulled the latest version ( (0.13.0) but its not showing the export cli from a "pip install --upgrade pupyl". In anticipation, here is the revised search bash code for it (which assumes the pupyl build database bit has been done)

reverse_search_images_my_pupyl_search_and_export() {
thisDirectory="$PWD"
export_baseDirectory=/Volumes/ZX\ MEDIA\ ARCHIVE/MediaArchivePupylSearch
target_Directory="$(basename "$1")"
if [[ -z "$1" ]]; then
   echo "need directory  to search"
else
	
        if [[ -z "$2" ]]; then
           echo "need file to search"
        else
                if [[ -z "$3" ]]; then
                   number_to_export=2
                   else
                   number_to_export="$3"
                   fi
        export_Directory="$export_baseDirectory"/"$(basename "$1")"/"$(basename "$2" | cut -d. -f1 )"
        mkdir -p "$export_Directory"
        source $HOME/Scripts/PythonEnvironments/pupyl-venv/bin/activate
        pupyl_Directory="$HOME/.pupyl""_""$target_Directory"
        pupyl --data_dir "$pupyl_Directory" export "$2" "$export_Directory" --top "$number_to_export"
        deactivate
        cd_nolist "$thisDirectory"
        fi
fi
}
export -f reverse_search_images_my_pupyl_search_and_export

[rleaver152]

sorted - I should have done pip install -e pupyl from my local repo

[rleaver152]

all working great and made corresponding edits to bash script above, working like a dream!

[rleaver152]

It would be useful to preserve the original name of the exported near-matches, rather than renumber them.

[policratus]

Hey @rleaver152! Yep, it makes sense. The current main branch (which is considered unstable) has the last changes, but I couldn't publish a new version until now. So, what you testes will be part of version 0.13.1, and next version, 0.13.2, I'll add the feature request, regarding preserving the original ids on the export. Thanks a lot!

[policratus]

@rleaver152, version 0.13.1 just published, containing everything you already tested. Next thing is to work with keeping original ids on exported files, will be available on 0.13.2.

[rleaver152]

Hi - hope you are not having too many sleepless nights with the new arrival :-) Just wondering how 0.13.2 is coming along. The reason I mention is that I have a great usage case for "preserving the original ids on the export". Basically I do a lot of keyframe b/w-> colorisation work including the great EbSynth. This requires generation of keyframes using eg github keplerlab/katna or github video-keyframe-detector. The problem is that these latter programs output renumbered keyframes (0000.png etc) when EbSynth requires the actual numbered frame (so 0000.png might actually refer to 00201.png in the video frame sequence by way of example). This is a horrible task to do manually with up to 200 keyframes a movie, but its perfect for Pupyl if and only if the output is the original id on export. I tested it and it works perfectly. Massive time saver if I can get the original ids. A huge number of people use EbSynth so its going to be a lifesaver for them

This is my invocation (having first built an index using pupyl on directory "video" which contains all the video frames)

for file in keys/* ; do reverse_search_images_my_pupyl_search_and_export video "$file" 1 ; done

where reverse_search_images_my_pupyl_search_and_export is

reverse_search_images_my_pupyl_search_and_export () 
{ 
    thisDirectory="$PWD";
    export_baseDirectory=/Volumes/MEDIA/MediaArchivePupylSearch;
    target_Directory="$(basename "$1")";
    if [[ -z "$1" ]]; then
        echo "need directory  to search";
    else
        if [[ -z "$2" ]]; then
            echo "need file to search";
        else
            if [[ -z "$3" ]]; then
                number_to_export=2;
            else
                number_to_export="$3";
            fi;
            export_Directory="$export_baseDirectory"/"$(basename "$1")"/"$(basename "$2" | cut -d. -f1)";
            mkdir -pv -p "$export_Directory";
            source $HOME/Scripts/PythonEnvironments/pupyl-venv/bin/activate;
            pupyl_Directory="$HOME/.pupyl""_""$target_Directory";
            pupyl --data_dir "$pupyl_Directory" export "$2" "$export_Directory" --top "$number_to_export";
            deactivate;
            cd_nolist "$thisDirectory";
        fi;
    fi
}
___

[policratus]

Hi @rleaver152,

I'm so sorry, I couldn't even answer before this message: paternity is taking all my time right now. It's really simple to add this feature of keeping the original ids, but I can see how this can be a massive help on some pipelines like yours. So, this week I'll try to implement everything related with that and maybe next week, we'll have version 0.13.2 with this feature. Thanks again for all your help and support, users like you is everything that an open source project needs!

Therefore, stay tuned this week, maybe the keep_ids feature is coming!

[rleaver152]

Hi - brilliant news and many thanks. all best Richard

…

On 9 Feb 2022, at 00:17, policratus ***@***.***> wrote: Hi @rleaver152, I'm so sorry, I couldn't even answer before this message: paternity is taking all my time right now. It's really simple to add this feature of keeping the original ids, but I can see how this can be a massive help on some pipelines like yours. So, this week I'll try to implement everything related with that and maybe next week, we'll have version 0.13.2 with this feature. Thanks again for all your help and support, users like you is everything that an open source project needs! Therefore, stay tuned this week, maybe the keep_ids feature is coming! — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

[policratus]

Hey @rleaver152! Sorry, I'm late with this. But is on the go! Just wait #142 to be finished and version 0.13.2 will come!

[policratus]

Hi @rleaver152, version 0.13.2 was just released, with the feature you requested, implemented. Hope you enjoy!

[rleaver152]

brilliant - many thanks!! Richard

…

On 26 Feb 2022, at 06:48, policratus ***@***.***> wrote: Hi @rleaver152, version 0.13.2 was just released, with the feature you requested, implemented. Hope you enjoy! — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

[policratus]

If you have any doubts on how to use, please let me know.

[rleaver152]

A couple of examples would be welcome if you have time 😊 Thanks Richard

…

Sent from my iPhone

On 26 Feb 2022, at 16:43, policratus ***@***.***> wrote:  If you have any doubts on how to use, please let me know. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

[rleaver152]

I thought I had it working but still getting only numerical labels, not the original ids.

[policratus]

When you say "original ids", are you talking about "original names", like if the image file name before been inserted on the pupyl was called myimage.jpg, image name on export should also be called myimage.jpg, instead of be named as 75.jpg, where 75 is the id created for that image on indexation time?

[rleaver152]

exactly, on my invocation including --keep_ids, I'm still just getting a 75 rather than a myimage.jpg Pls could you give me an example of a complete command line that I can double check against mine. thanks :-)

[rleaver152]

Here's an example run.

https://we.tl/t-2IcZNpZFYP

(.pupyl_A index, A source and results ). Invoked as

reverse_search_images_my_pupyl_search_and_export A A/Adele_14.jpeg 5   where the code is:

reverse_search_images_my_pupyl_search_and_export () 
{ 
    thisDirectory="$PWD";
    export_baseDirectory=/Volumes/ZX\ MEDIA\ ARCHIVE/MediaArchivePupylSearch;
    target_Directory="$(basename "$1")";
    if [[ -z "$1" ]]; then
        echo "need directory  to search";
    else
        if [[ -z "$2" ]]; then
            echo "need file to search";
        else
            if [[ -z "$3" ]]; then
                number_to_export=2;
            else
                number_to_export="$3";
            fi;
            export_Directory="$export_baseDirectory"/"$(basename "$1")"/"$(basename "$2" | cut -d. -f1)";
            mkdir -pv -p "$export_Directory";
            source $HOME/Scripts/PythonEnvironments/pupyl-venv/bin/activate;
            pupyl_Directory="$HOME/.pupyl""_""$target_Directory";
            pupyl --data_dir "$pupyl_Directory" export "$2" "$export_Directory" --top "$number_to_export" --keep_ids;
            deactivate;
            cd_nolist "$thisDirectory";
        fi;
    fi
}

[rleaver152]

Python 3.9.10
my Pupyl directory https://we.tl/t-wdyucqmNw8
and pip list
https://we.tl/t-iqRV86vo3a

[policratus]

@rleaver152, now this is part of release 0.13.3. Thanks for all your help and ideas!