Remove notifications from website-stalker

[EdJoPaTo]

It might also be neat to rethink some things of the current website-stalker. It bundles multiple things which are annoying to use separately then.

Currently it does the web requests, the editors to adapt the content, the git integration and the notification integration.

Something like this would be neat but is not possible currently:

#/usr/bin/env bash

summary=$(website-stalker run --config bla.yaml --all)

git add -A
git commit -m "$summary"

commit=$(git rev-parse HEAD)
commiturl="github.com/bla/$commit"

curl telegrambotapi --data "$summary bla bla $commiturl"

This would remove a bunch of dependencies and allow for easier combination with other use cases.

This mini example also already shows some downsides: there could only be one output so having a commit text and some human friendly notification text will probably differ. Maybe output stuff as JSON. Tools like jq can then be used to get whats needed.

Combining info into readable texts could be done with mustache like it is done currently already.

Another benefit of splitting things up would be a library usage of website-stalker to be used in other projects.

The biggest downside is simplicity of usage. Currently the executable can be run with a single command and then everything works.
Maybe keeping flags like --commit would be a good idea while simplifying its logic. Maybe let it just use the local git executable with git add -A && git commit --author foo --message bar and remove all the magic git assumptions.

[EdJoPaTo]

maybe a stdout-template config entry might be interesting replacing the current notification-template.

stdout_template: |
  Changes on:
  {{#sites}}
  - {{.}}
  {{/sites}}

  Diff:
  {{#commit}}
  https://github.com/bla/commit/{{.}}
  {{/commit}}
sites:
  - …

which could then be used in these cases:

summary=$(website-stalker run --all --commit`)

curl telegrambotapi --data "$summary"

The notification logic could be removed and integrated via other tools which are better suited for their jobs.

Outputting json might still be more helpful in some use cases.

Maybe output a default / custom stdout-template when no extra flag is given and output JSON when --stdout-json is given?

[Teufelchen1]

I would agree that notification should be split from the tool. I could easily see people using a tool that "stalks website" but not "stalks websites and notifies you but only on telegram". Before just throwing it out, we can module it as a feature see https://doc.rust-lang.org/cargo/reference/features.html

I like the idea of a stdout-template under the promise that it is completely optional / transparent to users who do not care about the output.

[EdJoPaTo]

The current notification stuff is also completely optional and supports more than just Telegram.

Having it under a feature seems not that different. I would prefer to just remove it completely. The default would be to have this feature enabled so it doesn't change much to the current situation.

[Teufelchen1]

I do not agree. Having it as a feature speaks a clear language: This is an optional, additional feature of the core utility 'website-stalker'. This is not the case at the moment. Further, cargo features are not enabled by default, for many users who do not use the notification, this will result in a shorter compilation time, smaller binary and potentially smaller attack vector (not that I would worry at the moment). This could also simplify the on-boarding of new users as the first impression of the tool might get more lightweight, only revealing the documentation about additional features when needed.

In case I could convince you to model it that way, I would be interested in doing the refactor work.

On a note: I just looked up the README again, indeed, it does support much more than telegram. This might be me looking at code instead of documentation - or the first impression of the tool might have mislead me.

[EdJoPaTo]

This would be the case for people compiling website stalker themselves. But people just using the precompiled binary (like in the GitHub Action or the OCI container) it should keep that feature enabled. Therefore it's still enable by default so it's not a huge difference to no feature for it.

Personally I think this feature is not really the core feature of the website-stalker. It's also not working perfectly. Having external tools doing that kind of things seems better to me than using a half baked internal part.

[EdJoPaTo]

Having a mustache stdout template with the --json-summary data as input might be a neat compromise…

This still wouldn't be great for people having multiple notification targets, but it's still easier to use compared to JSON. 🤔

[EdJoPaTo]

Currently, my leading thought for that is JSON on stdout and when the GitHub Actions environment is detected, JSON to GHA_OUTPUT.
This output can then be parsed and used with other tools especially made for their job: notifications to the service you want.

The notification feature would print a deprecation warning linking to this GitHub discussion or a dedicated issue / discussion. On a later release its fully removed.

What would be needed to simplify the migration from notifications to the simpler JSON output? What would you require on the JSON output to be easy to use?

[Teufelchen1]

Whats the reason for detecting github actions environment? What about passing a flag like --json or something?

[EdJoPaTo]

There is no point in not outputting to GHA_OUTPUT when available. It’s transparent to the user as it’s a write to file. If it’s not there its not done. If it’s there it’s very likely only on CI. Also this tool also does other logging stuff specifically when on GHA so it’s not the first GHA related thing.

the stdout could be hidden behind a --json-summary

[Teufelchen1]

Ah I see. Quickly scrolled through the PR. Makes much more sense now.

This leaves the questions: How should the JSON look like, whats required? Do you already have a plan/poc to use the output in a new notification application/service?

[EdJoPaTo]

Using the output should be relatively easy to achieve for programmers. For non programmers it might be neat to have some examples? Not sure if I want to keep examples as they are A kinda simple and B heavily dependent on the programming language and service. Implementing Slack would be different from Telegram and its part of the reason I don't want to have notifications as part of website-stalker.
My public examples will probably contain some Telegram example as I personally like to have that.

Also, GitHub has a lot of Actions which can be used and already do stuff needed by the user (with varying code quality). Maybe creating (yet another) Telegram sendMessage GitHub Action where I know there code quality as I built it myself.

Proof of concept (I assume that is what you mean by poc?) using Node.js follows.
On GitHub Actions it would use env for the input too.
You could also achieve something similar with curl. GitHub Actions has Curl and Node.js preinstalled and apt is available. There are many ways to reach what the user wants to have.

website-stalker run --all --commit --json-summary | node telegram.mjs

import { readFileSync } from "node:fs";
import { env } from "node:process";

const BOT_TOKEN = env["BOT_TOKEN"];
const TARGET_CHAT = env["TARGET_CHAT"];

const input = readFileSync("/dev/stdin", "utf-8");
const summary = JSON.parse(input);
console.log("summary", summary);

if (summary.change) {
  const url = `https://api.telegram.org/bot${BOT_TOKEN}/sendMessage`;
  const response = await fetch(url, {
    method: "POST",
    headers: {
      "Content-Type": "application/json; charset=utf8",
    },
    body: JSON.stringify({
      chat_id: TARGET_CHAT,
      text: toMessage(summary),
      link_preview_options: { is_disabled: true },
    }),
  });

  console.log("response", response.status, response.statusText);
  if (response.status !== 200) {
    console.log("response json", await response.json());
  }
}

function toMessage(summary) {
  const changes = summary.changed_sites.map((url) => `- ${url}`).join("\n");
  return `Änderungen auf:
${changes}

Diff: https://github.com/EdJoPaTo/website-stalker-example/commit/${summary.commit}`;
}

Use Telegrams sendMessage with curl. (find a way to generate $TEXT yourself :P )
(This GitHub Action could be interesting to fork and improve)

curl --silent -X POST --connect-timeout 10 \
  "https://api.telegram.org/bot$BOT_TOKEN/sendMessage" \
  -d "chat_id=$TARGET_CHAT" \
  -d "text=$TEXT"

[EdJoPaTo]

The JSON summary seems like another bunch of complexity which will be hard to deprecate and migrate properly. Which sparks the thought: what is actually needed. Currently I only know of Telegram being used for notifications. Notification Text could be hardcoded (human) language independent. No need to have that customisable.

The only thing that comes to my mind is the Commit URL that is added in my notification template. website-stalker could parse git remote -v and parse GitHub or use another CLI —-flag for the Commit URL prefix (https://gith…/commithash/)

My pling library currently used for notifications provides more than just Telegram and more than just reqwest. Stuff like Slack can be done with the same dependencies, throw out the rest like email which adds a huge bunch of dependencies which aren’t used by anyone.

Another downside of pling is the bad discoverability. Provide a clap derive feature and include this in the —-help. Also solvable.

I think this solves a lot of problems: less complexity and less dependencies while the stuff that is actually wanted (Telegram notifications) still works.

[EdJoPaTo]

#200 is the way going forward for this.