Multiple widths

[norm]

We need webkit2png to grab at multiple widths for testing responsive designs.

Rather than loading the same URL over and over, webkit2png should resize its internal WebKit instance and save multiple images.

[norm]

Oh, this contains the controversial moves of removing:

• multiple URL captures
• clipped and thumbnail images

so I'll understand if you don't merge this. :)

[paulhammond]

I love the idea here, and I want to get this merged. But there are a few caveats. As I mentioned in #63 I'm in favor of removing the ability to capture multiple URLs. But I'm not in favor of removing the ability to resize and clip the image before saving it - I know you can use convert or sips to do this after the event but this is a common enough use case that I think it should be built in.

So the question is how do we allow one instance of webkit2png to create multiple screenshots of the same page, some with the browser at different sizes, some resized, and some cropped? This is definitely related to #12 and #32.

For each output we want to be able to specify starting browser width (-W), starting browser height (-H), browser zoom (-z) factor, resize ratio (-s), cropped width (--clipwidth), cropped height (--clipheight) and a filename for the output. We also need the ability to add more options in the future (--selector points at some of the possibilities here, as does the request in #49 for PDF generation).

I think the answer here is something like X11's geometry setting. But that only does position and size, so it's not quite what we want. srcset is also close, but doesn't have quite the right options and doesn't work as a command line interface.

So, as a strawman, how about we add a new option called -O replacing all the other options above. You can add this multiple times, each one specifies an output file. The format is something like -O WxHxZ-wxhxs:name with every element being optional, defaulting to the some sensible values that mean by default you get a full size un-cropped screenshot from a normal desktop browser (probably 1024x768x1-1024x768x1). The crop and resize options default to whatever size the output from the browser is. If you don't specify a filename we use the geometry.

Here's some examples:

# full size screenshot
./webkit2png -O :full
# crop screenshot to 800x600, do not resize:
./webkit2png -O -800x600:cropped
# resize to quater size, crop to 200x150:
./webkit2png -O -200x150x0.25:resized
# iphone screenshot, at full size, output as examplecom-640x1136.png
./webkit2png -O 640x1136

We could then add --full, --thumb, --iphone5, --iphone4 etc as aliases to useful pre-baked geometries so the most obvious uses of webkit2png don't require understanding all of this.

I've been pondering this idea for a long time, but never really tried to write it down before so I'm not sure I've got it exactly right. -O -200x150x0.25 looks a bit strange now I've written it down, and this isn't extensible. I'd love some feedback or other proposals...

[paulhammond]

One more idea (thanks to @shanewholloway in #52): if the scale isn't specified we could calculate it based on the ratio between the starting browser size and the requested output size. This seems so obvious in retrospect.

[norm]

I like this proposal. Admittedly I based my pull request entirely upon my own use case, not the more general one. I was hoping it might spur some action :)

I had two main problems — that I need to grab the same page at different widths (and reloading it each time seems wasteful) and that pages that aren't properly responsive/fluid/buzzword mean that specifying -W320 doesn't give me a 320px wide capture. I think the logic for creating clipped images makes this fuzzier than it needs to be.

So yes, if we can break that out by separating the idea of "get this url in this size browser" from "now save it at 25% size, at exactly X by Y pixels" it works better in my head. It also means you can get the same page at multiple widths (eg 320, 640, 768, 1024) and still only need to produce one clip/thumbnail not four, three of which are essentially useless.

I think you're right that output options are good as {size of browser} plus {output modifiers}, but I would drop the -O and just have arguments be webkit2png [-options] output-format […] URL

size of browser

• actual width (if I say 320, I mean 320)
• minimum width
• minimum height
• zoom

output modifiers

• size ratio
• maximum width/height (thumb?)
• fixed width/height (clip?)
• filename

(some confusion perhaps because I don't use clips/thumbs so I'm not entirely clear on what the difference is – clip is a portion, thumb is just smaller?)

Now, because my use case is grabbing full pages no matter how tall, I see max width/height as output modifiers, but it could be argued that they're browser size modifiers.

In terms of what that might look like, there's enough complexity that I don't think you'll find a method that doesn't descend into ugliness. But to address that I do really really like the idea of shorthands wrapping up common options, much like HandBrake's presets.

I favour more {number}{descriptor} (eg 1024w768h) because WxH means you have to give a height and it's hard to make it clear if it's minimum or absolute. 320w could be "at least 320 pixels wide" and 320W "exactly 320 pixels wide`. I suppose WxH could be shorthand for "exactly x by y". I would make the (optional) filename the first part. Also, if you want to allow the arguments to be more readable, ignore whitespace. Examples:

# 320, 640 and 1024 width, full page height
# output: 320W.png, 640W.png, 1024W.png
webkit2png 320W 640W 1024W https://www.gov.uk

# 1000 width and 200x150 thumbnail grabs
# output: full.png, thumb.png
webkit2png full:1000W thumb:1000W@200W150H https://www.gov.uk

# same as before but spaced for readability, using WxH shorthand
webkit2png "full: 1000W" "thumb: 1000W @ 200x150" https://www.gov.uk

[paulhammond]

I can't work out how to structure a reply. So instead here's some bullet points:

• The thumb is a resized version of the page. The clipped is a (badly named) cropped thumb.
• The problem with saying "I want the browser to be 320px wide" is that this is a concept that doesn't actually exist with browsers - if the browser is 320px wide and there's a 400px wide fixed width element then they scroll. If you mean "give me an image 320px wide ignoring the stuff that gets clipped out" then that's possible. Or did you mean something else?
• Yes, I think this will probably descend into madness if you want to specify everything. But that's ok, as long as the simple stuff is easy.
• I like the {number}{suffix} idea a lot.
• I'm on the fence about keeping -O. If we're only allowing one URL to be captured at a time then I guess it's superflous.
• I also like the idea of ignoring whitespace - although unix command line semantics make that harder (how do we tell when a second image starts?). I think people will have to quote their specifications if they want whitespace?

Here's another attempt at a spec:

• name: (has to be first) the filename for the image. We possibly allow %s (sanitized url) %x (md5 sum) etc. Or we expect people to provide their own filenames when they are capturing multiple pages. If there are no other options and the name does not start with a number then the : is optional. Likewise the .png on the end is optional if the name does not contain a ..
• W - initial browser width in pixels, defaults to 1024
• H - initial browser height in pixels, defaults to 768
• Z - initial browser zoom, defaults to 1
• w - actual output image width, cropping or empty pixels if needed, defaults to the final width of the content multiplied by the resize factor (that is, defaults to the whole image)
• h - actual output image height, cropping or empty pixels if needed, defaults to the final height of the content multiplied by the resize factor
• x - image resize factor (as in "0.5x size"), defaults to 1 unless an actual width and height are provided, in which case we use the largest scale that will fill the output image.

Some examples:

# full size screenshot, called full.png
webkit2png full https://www.gov.uk

# crop screenshot to 800x600, do not resize:
webkit2png cropped:800w600h https://www.gov.uk/

# resize to quarter size then crop to 200x150:
webkit2png resized:200w150h0.25x https://www.gov.uk/

# iphone screenshot, at full size, output as iphone.png
webkit2png iphone:640W1136H https://www.gov.uk
# as above, but output as wwwgovuk-640W1136H.png
webkit2png 640W1136H https://www.gov.uk

# 320, 640 and 1024 width, full page height
webkit2png 320W 640W 1024W https://www.gov.uk

# 1000 width and 200x150 thumbnail grabs, output: full.png, thumb.png
webkit2png full:1000W thumb:1000W200w150h https://www.gov.uk

Thoughts?

[norm]

Fair point about the wider content; so yes, I mean clipping when referring to narrow browsers; like "if I sized my user agent to 320px and I hadn't scrolled then capture what is visible". Which just gives initial size (minimum) and output clip (maximum) as you've shown.

Being able to use whitespace within options would definitely mean having to use a quoting mechanism rather than magic to divine when one set of options had finished and the next started. I did illustrate that in the last of my examples.

So in summary, like it. Also really like %s etc in the filename portion.

[paulhammond]

Sounds like we have a plan. Now one of us just needs to find the time to implement it...

[freshface]

Any idea when this will be implemented?

[dcsan]

@norm

I also need this width setting feature, I was wondering if you had added it to your fork? i can't seem to see any commits mentioning width

https://github.com/norm/webkit2png/commits/master
https://github.com/norm/webkit2png

[paulhammond]

As discussed in #108 any breaking changes will probably happen at the same time we rewrite the code, so this PR isn't going to land as is. I've opened #109 to track the ideas we discussed here.