Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions tests/curl-mock-version
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/bin/sh

# wcurl - a simple wrapper around curl to easily download files.
#
# This is wcurl's mock curl binary.
#
# Copyright (C) Samuel Henrique <samueloph@debian.org>, Sergio Durigan
# Junior <sergiodj@debian.org> and many contributors, see the AUTHORS
# file.
#
# Permission to use, copy, modify, and distribute this software for any purpose
# with or without fee is hereby granted, provided that the above copyright
# notice and this permission notice appear in all copies.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. IN
# NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
# OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE
# OR OTHER DEALINGS IN THE SOFTWARE.
#
# Except as contained in this notice, the name of a copyright holder shall not be
# used in advertising or otherwise to promote the sale, use or other dealings in
# this Software without prior written authorization of the copyright holder.
#
# SPDX-License-Identifier: curl

# This is a simple binary to mock curl's version string. It will
# print a version specified in the CURL_MOCK_VERSION variable and just
# exit. If CURL_MOCK_VERSION is empty, print 0.0.0.

printf "%s\n" "curl ${CURL_MOCK_VERSION:-0.0.0}"
exit 0
42 changes: 42 additions & 0 deletions tests/tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,48 @@ testUrlDecodingNonLatinLanguages()
assertContains "Verify whether 'wcurl' successfully decodes percent-encoded Korean in URLs" "${ret}" '퍼센트_인코딩'
}

testCurlBinaryOption()
{
url='example.com'
ret=$(${WCURL_CMD} --dry-run "${url}")
bin=$(printf "%s" "${ret}" | head -n1)
assertEquals "Verify whether 'wcurl' invokes 'curl' when '--curl-binary' is not provided" "${bin}" "curl"

curlbin="${ROOTDIR}/tests/curl-mock-version"
ret=$(${WCURL_CMD} --dry-run --curl-binary "${curlbin}" "${url}")
bin=$(printf "%s" "${ret}" | head -n1)
assertContains "Verify whether 'wcurl' invokes the binary specified by '--curl-binary'" "${bin}" "${curlbin}"
}

testCurlVersionComparison()
{
# Verify that using a very old curl version correctly omits
# --parallel, --no-clobber and --parallel-max-host.
url='example.com'
curlbin="${ROOTDIR}/tests/curl-mock-version"
ret=$(CURL_MOCK_VERSION="1.0.0" ${WCURL_CMD} --dry-run --curl-binary "${curlbin}" "${url}" "${url}")
assertNotContains "Verify whether 'wcurl' correctly omits --parallel when using very old curl" "${ret}" '--parallel'
assertNotContains "Verify whether 'wcurl' correctly omits --no-clobber when using very old curl" "${ret}" '--no-clobber'
assertNotContains "Verify whether 'wcurl' correctly omits --parallel-max-host when using very old curl" "${ret}" '--parallel-max-host'

# Verify that using a curl version that's >= 7.66.0 and < 7.83.0
# correctly adds --parallel but omits --no-clobber and --parallel-max-host.
ret=$(CURL_MOCK_VERSION="7.66.0" ${WCURL_CMD} --dry-run --curl-binary "${curlbin}" "${url}" "${url}")
assertContains "Verify whether 'wcurl' correctly adds --parallel when using curl >= 7.66.0" "${ret}" '--parallel'
assertNotContains "Verify whether 'wcurl' correctly omits --no-clobber when using curl >= 7.66.0 && curl < 7.83.0" "${ret}" '--no-clobber'
assertNotContains "Verify whether 'wcurl' correctly omits --parallel-max-host when using curl >= 7.66.0 && curl < 8.16.0" "${ret}" '--parallel-max-host'

# Verify that using a curl version that's >= 7.83.0 and < 8.16.0 correctly adds --no-clobber but omits --parallel-max-host.
ret=$(CURL_MOCK_VERSION="7.83.0" ${WCURL_CMD} --dry-run --curl-binary "${curlbin}" "${url}" "${url}")
assertContains "Verify whether 'wcurl' correctly adds --no-clobber when using curl >= 7.83.0" "${ret}" '--no-clobber'
assertNotContains "Verify whether 'wcurl' correctly omits --parallel-max-host when using curl >= 7.83.0 && curl < 8.16.0" "${ret}" '--parallel-max-host'

# Verify that using a curl version that's >= 8.16.0 correctly adds
# --parallel-max-host.
ret=$(CURL_MOCK_VERSION="8.16.0" ${WCURL_CMD} --dry-run --curl-binary "${curlbin}" "${url}" "${url}")
assertContains "Verify whether 'wcurl' correctly adds --parallel-max-host. when using curl >= 8.16.0" "${ret}" '--parallel-max-host'
}

## Ideas for tests:
##
## - URL with whitespace
Expand Down
102 changes: 77 additions & 25 deletions wcurl
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,8 @@ usage()
${PROGRAM_NAME} -- a simple wrapper around curl to easily download files.

Usage: ${PROGRAM_NAME} <URL>...
${PROGRAM_NAME} [--curl-options <CURL_OPTIONS>]... [--no-decode-filename] [-o|-O|--output <PATH>] [--dry-run] [--] <URL>...
${PROGRAM_NAME} [--curl-options=<CURL_OPTIONS>]... [--no-decode-filename] [--output=<PATH>] [--dry-run] [--] <URL>...
${PROGRAM_NAME} [--curl-options <CURL_OPTIONS>]... [--curl-binary <CURL_BINARY>] [--no-decode-filename] [-o|-O|--output <PATH>] [--dry-run] [--] <URL>...
${PROGRAM_NAME} [--curl-options=<CURL_OPTIONS>]... [--curl-binary=<CURL_BINARY>] [--no-decode-filename] [--output=<PATH>] [--dry-run] [--] <URL>...
${PROGRAM_NAME} -h|--help
${PROGRAM_NAME} -V|--version

Expand All @@ -59,6 +59,8 @@ Options:
--curl-options <CURL_OPTIONS>: Specify extra options to be passed when invoking curl. May be
specified more than once.

--curl-binary <CURL_BINARY>: Specify the curl binary to be used. By default, "curl" is used.

-o, -O, --output <PATH>: Use the provided output path instead of getting it from the URL. If
multiple URLs are provided, resulting files share the same name with a
number appended to the end (curl >= 7.83.0). If this option is provided
Expand Down Expand Up @@ -89,6 +91,9 @@ error()
exit 1
}

# The curl binary to be used.
CURL_BINARY="curl"

# Extra curl options provided by the user.
# This is set per-URL for every URL provided.
# Some options are global, but we are erroring on the side of needlessly setting
Expand Down Expand Up @@ -123,14 +128,49 @@ readonly UNSAFE_PERCENT_ENCODE="%2F %5C"
# Whether to invoke curl or not.
DRY_RUN="false"

# The current version of curl.
CURL_VERSION=""
# The normalized curl version, in the format "XXYYZZ", where "XX" is
# the zero-padded major version, "YY" is the zero-padded minor
# version and "ZZ" is the zero-padded patch version.
CURL_NORMALIZED_VERSION=""

# Sanitize parameters.
sanitize()
{
if [ -z "${URLS}" ]; then
error "You must provide at least one URL to download."
fi

readonly CURL_OPTIONS URLS DRY_RUN HAS_USER_SET_OUTPUT
CURL_VERSION=$(${CURL_BINARY} --version | head -n1 | cut -f2 -d' ')
if [ -z "${CURL_VERSION}" ]; then
error "Unable to determine curl version. Is curl installed?"
fi

CURL_NORMALIZED_VERSION=$(normalize_version "${CURL_VERSION}")

readonly CURL_OPTIONS \
CURL_BINARY \
URLS \
DRY_RUN \
HAS_USER_SET_OUTPUT \
CURL_VERSION \
CURL_NORMALIZED_VERSION
}

# Print the normalized format of a version specified as the first argument.
#
# The normalized version has the format "XXYYZZ", where "XX" is the
# zero-padded major version, "YY" is the zero-padded minor version and
# "ZZ" is the zero-padded patch version.
normalize_version()
{
version="${1}"
vermaj=$(printf "%s" "${version}" | cut -f1 -d.)
vermin=$(printf "%s" "${version}" | cut -f2 -d.)
verpatch=$(printf "%s" "${version}" | cut -f3 -d.)

printf "%02d%02d%02d" "${vermaj}" "${vermin}" "${verpatch}"
}

# Indicate via exit code whether the string given in the first parameter
Expand Down Expand Up @@ -198,36 +238,38 @@ get_url_filename()
# No slash means there was just a hostname and no path; return empty string.
}

# Given a version (in the format MAJOR.MINOR) as the first argument
# and an operator as the second argument, perform a comparison against
# the current curl version.
compare_curl_version()
{
# Any of: -lt, -le, -eq, -gt, -ge
operator="${1}"
version="${2}"
version_to_compare=$(normalize_version "${version}")

test "${CURL_NORMALIZED_VERSION}" "${operator}" "${version_to_compare}"
}

# Execute curl with the list of URLs provided by the user.
exec_curl()
{
CMD="curl "

# Store version to check if it supports --no-clobber, --parallel and --parallel-max-host.
curl_version=$($CMD --version | cut -f2 -d' ' | head -n1)
curl_version_major=$(echo "$curl_version" | cut -f1 -d.)
curl_version_minor=$(echo "$curl_version" | cut -f2 -d.)
CMD="${CURL_BINARY} "

CURL_NO_CLOBBER=""
CURL_PARALLEL=""

if [ "${curl_version_major}" -ge 8 ]; then
# --parallel is only supported since 7.66.0.
if compare_curl_version -ge "7.66.0"; then
CURL_PARALLEL="--parallel"
fi
# --no-clobber is only supported since 7.83.0.
if compare_curl_version -ge "7.83.0"; then
CURL_NO_CLOBBER="--no-clobber"
CURL_PARALLEL="--parallel --parallel-max-host 5"

# --parallel-max-host is only supported since 8.16.0.
if [ "${curl_version_major}" -eq 8 ] && [ "${curl_version_minor}" -lt 16 ]; then
CURL_PARALLEL="--parallel"
fi
elif [ "${curl_version_major}" -eq 7 ]; then
# --no-clobber is only supported since 7.83.0.
if [ "${curl_version_minor}" -ge 83 ]; then
CURL_NO_CLOBBER="--no-clobber"
fi
# --parallel is only supported since 7.66.0.
if [ "${curl_version_minor}" -ge 66 ]; then
CURL_PARALLEL="--parallel"
fi
fi
# --parallel-max-host is only supported since 8.16.0.
if compare_curl_version -ge "8.16.0"; then
CURL_PARALLEL="${CURL_PARALLEL} --parallel-max-host 5"
fi

# Detecting whether we need --parallel. It is easier to rely on
Expand Down Expand Up @@ -285,6 +327,16 @@ while [ -n "${1-}" ]; do
CURL_OPTIONS="${CURL_OPTIONS} ${1}"
;;

--curl-binary=*)
opt=$(printf "%s\n" "${1}" | sed 's/^--curl-binary=//')
CURL_BINARY="${opt}"
;;

--curl-binary)
shift
CURL_BINARY="${1}"
;;

--dry-run)
DRY_RUN="true"
;;
Expand Down
6 changes: 4 additions & 2 deletions wcurl.1
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@
.SH SYNOPSIS
\fBwcurl <URL>...\fP

\fBwcurl [\--curl\-options <CURL_OPTIONS>]... [\--dry\-run] [\--no\-decode\-filename] [\-o|\-O|\--output <PATH>] [\--] <URL>...\fP
\fBwcurl [\--curl\-options <CURL_OPTIONS>]... [\--curl\-binary <CURL_BINARY>] [\--dry\-run] [\--no\-decode\-filename] [\-o|\-O|\--output <PATH>] [\--] <URL>...\fP

\fBwcurl [\--curl\-options=<CURL_OPTIONS>]... [\--dry\-run] [\--no\-decode\-filename] [\--output=<PATH>] [\--] <URL>...\fP
\fBwcurl [\--curl\-options=<CURL_OPTIONS>]... [\--curl\-binary=<CURL_BINARY>] [\--dry\-run] [\--no\-decode\-filename] [\--output=<PATH>] [\--] <URL>...\fP

\fBwcurl \-V|\--version\fP

Expand Down Expand Up @@ -61,6 +61,8 @@ if there is none in the URL.
.IP "--curl-options, --curl-options=\<CURL_OPTIONS\>..."
Specify extra options to be passed when invoking curl. May be specified more
than once.
.IP "--curl-binary, --curl-binary=\<CURL_BINARY\>..."
Specify the curl binary to be used. By default, "curl" is used.
.IP "-o, -O, --output, --output=\<PATH\>"
Use the provided output path instead of getting it from the URL. If multiple
URLs are provided, resulting files share the same name with a number appended to
Expand Down
8 changes: 6 additions & 2 deletions wcurl.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ Added-in: n/a

**wcurl \<URL\>...**

**wcurl [--curl-options \<CURL_OPTIONS\>]... [--dry-run] [--no-decode-filename] [-o|-O|--output \<PATH\>] [--] \<URL\>...**
**wcurl [--curl-options \<CURL_OPTIONS\>]... [--curl-binary \<CURL_BINARY\>] [--dry-run] [--no-decode-filename] [-o|-O|--output \<PATH\>] [--] \<URL\>...**

**wcurl [--curl-options=\<CURL_OPTIONS\>]... [--dry-run] [--no-decode-filename] [--output=\<PATH\>] [--] \<URL\>...**
**wcurl [--curl-options=\<CURL_OPTIONS\>]... [--curl-binary=\<CURL_BINARY\>] [--dry-run] [--no-decode-filename] [--output=\<PATH\>] [--] \<URL\>...**

**wcurl -V|--version**

Expand Down Expand Up @@ -78,6 +78,10 @@ By default, **wcurl** does:
Specify extra options to be passed when invoking curl. May be specified more
than once.

## --curl-binary, --curl-binary=\<CURL_BINARY\>

Specify the curl binary to be used. By default, "curl" is used.

## -o, -O, --output, --output=\<PATH\>

Use the provided output path instead of getting it from the URL. If multiple
Expand Down