catch-all-advanced and flags

[felixlohmeier]

Hi @DannyBen, this project brings the fun back into bash scripting. Thanks a lot for this!

I have a question about the catch-all-advanced example. The download command has an option --force and catch-all for AWS params. What happens if the AWS params also contain --force?

Currently, the option defined in the CLI wins:

[felix@tux catch-all-advanced]$ ./cli download source target --force and --additional stuff
# this file is located in 'src/download_command.sh'
# code for 'cli download' goes here
# you can edit it freely and regenerate (it will not be overwritten)
args:
- ${args[--force]} = 1
- ${args[source]} = source
- ${args[target]} = target

other_args:
- ${other_args[*]} = and --additional stuff
- ${other_args[0]} = and
- ${other_args[1]} = --additional
- ${other_args[2]} = stuff

This is also the case when the --force option is placed at the very end of the command:

[felix@tux catch-all-advanced]$ ./cli download source target and --additional stuff --force
# this file is located in 'src/download_command.sh'
# code for 'cli download' goes here
# you can edit it freely and regenerate (it will not be overwritten)
args:
- ${args[--force]} = 1
- ${args[source]} = source
- ${args[target]} = target

other_args:
- ${other_args[*]} = and --additional stuff
- ${other_args[0]} = and
- ${other_args[1]} = --additional
- ${other_args[2]} = stuff

I think this is great because it also makes the following use case work:

cli import test.csv test2.csv --format csv
cli import --format csv test.csv test2.csv

Can I rely on this behavior? Maybe we could add it to the example then.

[DannyBen]

Hey Felix,

this project brings the fun back into bash scripting

This comment makes me very happy. It is great to learn that other people make good use of bashly.

As for the question at hand, it is a great observation on your part - nice catch.
I am wondering if this should be the behavior or not though. Right now this feature is there by chance, not by design.

Intuitively, I would imagine that once an "unknown argument" is introduced to the CLI, then this begins the "catch all" sequence and terminates any lookup for predefined arguments.

Need to think about this a little, and perhaps hear more opinions and see near-real-world use cases to decide - at this point, I am open to either way, but I agree that one way or the other, it needs to be by design, and documented.

[felixlohmeier]

Thanks for your quick feedback! I am looking for a way to support stdin, one or more files and flags (with args) in one command. Users would expect the following syntax for this, I think.

cli import "file name 1.csv" "file name 2.csv" --format csv

or

cat file* | cli import --format csv

This "feature by chance" would be the solution for this use case :-). But maybe I'm missing something.

[DannyBen]

I see. Can you paste the most minimal YAML file that uses this technique?

[felixlohmeier]

Most minimal, hmm... maybe something like this?

name: cli
help: Sample application
version: 0.1.0

catch_all:
  label: path
  help: Path to file(s) (reads from stdin if empty or "-")

flags:
- long: --format
  short: -f
  arg: format
  help: specify file format e.g. csv

examples:
- ./cli "file 1.csv" "file 2.csv" --format csv
- ./cli --format csv "file 1.csv" "file 2.csv"
- cat some-file | ./cli --format csv
- cat some-file | ./cli - --format csv

[DannyBen]

Excellent. Let me take a deeper look at it later today, so we can make a decision together, with these possibilities:

1. Endorse this use case, and ensure it is a documented and tested feature.
2. Change this behavior, if it is not the intended design.

I am strongly leaning towards option 1 at the moment.

[DannyBen]

Ok, I have looked a little deeper, so this will be a long post.

Bottom line

1. I believe it is safe to rely on this functionality.
2. I will add have added an example (which is also a test case for the automatic tests) to ensure it remains like this.
3. I might add support for repeatable args. The reasoning for not adding them can be alleviated by the same technique used in the repeatable flags-with-args (Can we add a repeatable for args? #200).
4. I want to make sure you are aware of the repeatable flags - which is another way to let users provide multiple items of the same.

---

Implementation using catch_all

At first, I tested with a variation of your YAML file. Note I made some changes you might want to consider:

name: cli
help: Sample application
version: 0.1.0

catch_all:
  label: path
  help: Path to file(s) (reads from stdin if empty or "-")

# Ideally, we would want to do this instead of catch_all
# args:
# - name: path
#   help: Path to file(s) (reads from stdin if empty or "-")
#   repeatable: true
#   default: '-'

flags:
- long: --format
  short: -f
  arg: format
  help: Specify file format
  default: json
  allowed: [csv, json]

examples:
- cli file1 file2 --format csv
- cli --format csv file1 file2
- cat file1 | cli --format csv
- cat file* | cli - --format csv

The code to process the input for this correctly is a little bloated, but I could not see a way around it:

# src/root_command.sh
inspect_args

# Collect the catch_all
files=( ${other_args[*]} )

# Read contents of the provided file(s)
content=""
for file in "${files[@]}"; do
  content+="$(cat "$file")"
  content+=$'\n'
done

# Read stdin if file(s) not provided as arguments
if [[ -z "$content" ]]; then
  content="$(cat -)"
fi

echo "$content"

Implementation using repeatable flags

This example uses the repeatable flags to achieve the same, although with a slightly less appealing user interface. I am pasting it here in case it helps. The upside is a slightly less complex code to handle the use case.

name: cli
help: Sample application
version: 0.1.0

flags:
- long: --input
  short: -i
  arg: path
  help: Path to input file(s) (reads from stdin if empty or "-")
  repeatable: true
  default: "-"
- long: --format
  short: -f
  arg: format
  help: Specify file format
  default: json
  allowed: [csv, json]

examples:
- cli -i file1 -i file2 --format csv
- cli --format csv -i file1 file2
- cat file1 | cli --format csv
- cat file* | cli --format csv -i - 

# src/root_command.sh
inspect_args

eval "files=(${args[--input]})"

# Read contents of the provided file(s)
content=""
for file in "${files[@]}"; do
  content+="$(cat "$file")"
  content+=$'\n'
done

echo "$content"

[felixlohmeier]

Wow, that's awesome. I prefer repeatable args over repeatable flags because it allows easy globbing (example: cli file*). So great that I can use this catch-all-stdin variant for the time being. I also like the proposed form for repeatable args even better because it's easier to understand.

Some minor points:

1. In other examples (e.g. repeatable), you also included the contents of src/root_command.sh in README.md. That would probably be good here too, so that it is understandable at a glance.

2. In your example in the above answer I would replace
   files=( ${other_args[*]} )
   to
   files=( "${other_args[@]}" )
   to also support file names with spaces.

Thank you very much for the super fast response and the great solutions!

[DannyBen]

I prefer repeatable args over repeatable flags because it allows easy globbing

You are absolutely right. Bashly - like Ruby - aims to make developers happy. There is no doubt repeatable args are needed. I opened an issue to track this, and I hope to implement it soon.

In other examples (e.g. repeatable), you also included the contents of src/root_command.sh

Correct and helpful again. Fixed.

In your example in the above answer I would replace files=( ${other_args[*]} )

I got rid of this assignment. It is now:

bashly/examples/catch-all-stdin/src/root_command.sh

Line 5 in 011c648

for file in "${other_args[@]}"; do

---

I also added the examples index to the documentation site so that searching for "stdin" would have something to show.

Thanks for the great suggestions, and please share anything else that doesn't work smoothly or as expected with Bashly.

[DannyBen]

I have now implemented repeatable args in #202.
It is available in the master branch, in case you want to test it.

[felixlohmeier]

Thanks again for the quick help and implementing new features! I will use repeatable arg then, along with a non-blocking support for stdin. These code snippets seem to work:

bashly.yml (without default:"-")

args:
- name: file
  help: Path to one or more files. Reads from stdin if empty or "-".
  repeatable: true

root_command.sh

eval "files=(${args[file]})"

if [[ -z ${files} ]]; then
  if [ -t 0 ]; then
    printf "missing required argument or standard input\nusage: cli FILE...\n"
    exit 1
  else
    files="-"
  fi
fi

For me, this would make the catch-all-stdin variant dispensable.

I am not sure whether the -t 0 test works in all scenarios. Source: https://stackoverflow.com/questions/2456750/how-to-check-if-stdin-is-from-the-terminal-or-a-pipe-in-a-shell-script

[DannyBen]

Whatever works for you. I agree the catch_all now has less of a reason to exist. The only one left (outside of backwards compatibility) is the fact that catch_all comes as an array, and does not need the eval trick.

That said, why aren't you using the default: "-"? or the required?

And lastly - if you find yourself putting code that needs to validate the argument, you can make use of the Custom Validations feature, in order to keep your code clean and neat.

If you feel any of the problems you face can be solved generically, for everybody, I am open to suggestions.

[felixlohmeier]

That said, why aren't you using the default: "-"? or the required?

I am trying to follow CLIG's recommendation:

If your command is expecting to have something piped to it and stdin is an interactive terminal, display help immediately and quit. This means it doesn’t just hang, like cat. Alternatively, you could print a log message to stderr.

A variant with default: "-" would be

args:
- name: file
  help: Path to one or more files. When FILE is -, read standard input.
  default: "-"
  repeatable: true

eval "files=(${args[file]})"

if [ "${files[*]}" = "-" ] && [ -t 0 ]; then
  printf "missing required argument or standard input\nusage: cli FILE...\n"
  exit 1
fi

True, that's a bit prettier.

... you can make use of the Custom Validations feature, in order to keep your code clean and neat.

Good point, I will give it a try!

[DannyBen]

I am trying to follow CLIG's recommendation:

Nice. I might read this more thoroughly, to ensure Bashly makes it easy to be compliant.