Wikipedia:Bots/Requests for approval/SDZeroBot 9

SDZeroBot 9Edit

Operator: SD0001 (talk · contribs · SUL · edit count · logs · page moves · block log · rights log · ANI search)

Time filed: 12:57, Thursday, November 12, 2020 (UTC)

Automatic, Supervised, or Manual: automatic

Programming language(s): TypeScript

Source code available: GitHub

Function overview: Monitor activity of other bots. Issue alerts to bot operators via talk page or email if they subscribe.

Links to relevant discussions (where appropriate): Wikipedia:Bot_requests/Archive_80#A_bot_to_monitor_the_activity_level_of_other_bots

Edit period(s): Continuous

Estimated number of pages affected: -

Exclusion compliant (Yes/No): No

Already has a bot flag (Yes/No): Yes

Function details: Based on pre-configured information about bot tasks (name of bot account, what edit summaries it uses, what pages/namespaces it edits, how many edits are expected in the last x days, etc), it identifies bots and bot tasks which have stopped working. Stalled bot tasks can be identified even if the bot account is still running other tasks. Bots which perform actions other than editing (deletions/blocks/patrols etc) can also be monitored. A bot status table would be generated and posted to WP:Bot activity monitor.

If configured, this bot can also issue alerts to the operator to let them know that their bot tasks are not running. Alerts can be sent via talk page or email or least intrusively, via a ping from a central page.

I expect anyone should be able to set up a bot for tracking (to be included in status table), but of course only the operator(s) should set up alerts for themselves.


Pinging some users from the old BOTREQ discussion: @Sdkb, GreenC, Redrose64, Headbomb, Primefac, Majavah, and Amorymeltzer:. – SD0001 (talk) 15:34, 12 November 2020 (UTC)

  • The configuration parameters for describing bot tasks are given at WP:Bot activity monitor. However, I'm a bit confused on how and where should people set up these configurations. My initial thought was to have a central JSON page: Wikipedia:Bot activity monitor/config.json but the problems with JSON are (i) it requires regexes to be weirdly escaped (eg. \d needs to be written as \\d, though it will show up as \d while viewing the page) and (ii) it looks so clumsy, especially when there would be 100s of tasks. It seems using template markup is better to describe the configurations – but should they go on a central page or be decentralized on bot user pages? The drawback of the latter is that it discourages users from setting up tracking for others' bots. – SD0001 (talk) 15:35, 12 November 2020 (UTC)
  • I'm super happy to see this; thanks for your work on it, SD0001! I don't have the expertise to comment on the technical questions, but as far as monitoring goes, my sense is that many bots that stop working have retired operators, so it would be good for there to be notifications not just to the talk page of the operator. Looking forward to seeing this in operation! {{u|Sdkb}}talk 15:58, 12 November 2020 (UTC)
  • Okay, so am I reading it correctly that this is an opt in situation, only "checking up" on whichever specific bots are listed? Also, why do we need a second process when Wikipedia:Bots/Requests for approval/MajavahBot 3 exists? Primefac (talk) 14:12, 13 November 2020 (UTC)
    @Primefac: This is a lot more advanced than MajavahBot 3. See User:SD0001/Bot_activity_monitor/Report for the kind of output it produces – that's based on the data for a cherry-picked set of bots at Wikipedia:Bot activity monitor/config.json. And as mentioned it also supports sending notifications to botops. Because all of this requires data about bot tasks in a machine-readable form, it necessarily has to be "opt-in" (through folks can opt-in others' bots). – SD0001 (talk) 19:07, 13 November 2020 (UTC)
    Fair enough. Per the general precedent, there's no issue with creating a database-style report for these bots (i.e. "only edits one page") but when it starts getting towards notifications there come more questions. Speaking as a bot operator, I don't really care if someone keeps tabs on my bot, but I don't want any sort of automated notice if I happen to decide not to run one of my "active" tasks for some period of time, and I'd rather not find out after receiving a notification that someone's added my name to the "notify" list. Primefac (talk) 20:15, 13 November 2020 (UTC)
    Agreed. I myself wouldn't want these notifications – I've implemented error handling in my own bot tasks so that whenever an error occurs, I get an email with the stack trace of the error – which would be more useful than a generic message from a third-party bot which says the task didn't run. This is why I say above notifications would (should) only be enabled by botops themselves. But I think we can just let this be a convention rather than try to restrict it at the technical level, and hope that people won't be jerks? Remember that it's technically also possible for a random guy to subscribe you to random wikiprojects' newsletters – but this doesn't seem to happen in practise. – SD0001 (talk) 11:33, 14 November 2020 (UTC)
  • I'll drop a note at WP:BON about this bot. One point is worth noting, just in case it isn't obvious, is that the "monitoring" is intended for bots that run fully automatically – with zero human intervention. It wouldn't make sense to track bots that are one-time or on-demand, or the even the ones which require any level of operator intervention to run. The intent is to "catch" bot stoppages which the operator may not be aware of, typically occurring due to unforeseen issues such as the ones Anomie mentioned at this discussion, quoting:
  • Something changes that causes the bot to fail unless its code or configuration is updated ...
  • A software update by the hosting provider breaks the bot's code, again requiring a code update.
  • The bot's process stops running or locks up, and the operator isn't paying attention to notice and restart it.
  • The hosting provider where it bot is being run closes the account (or closes entirely).

SD0001 (talk) 12:04, 14 November 2020 (UTC)

  • I think this is helpful and not particularly problematic. Of course, operators are not obligated to run tasks, but many times the downtime is accidental not intentional. For example, when my task 3 stops we lose a day of Main Page history. It did lock up once after some maintenance from my host, so Template:2020 Main Page history is missing November 9b and 10. Setting up good app/server monitoring is not what most bots do. Note I haven't looked too closely at the implementation yet to say if I have any concerns with that part. ProcrastinatingReader (talk) 15:21, 14 November 2020 (UTC)
  • Sounds like a great idea. I would prefer JSON where each bot monitor is its own object with fields for summary regexes, expected run times, number of runs per day, an array of pages that are expected to have been edited, etc. JSON has the added benefit of being highly extensible since it can contain other objects allowing for more complex configurations. That said it may not be widely accessible to less-technical botops, and the regex escape problem is always a nuisance. Either way, sounds great and I look forward to seeing it in operation! Wug·a·po·des 03:25, 15 November 2020 (UTC)
  • BAG question: is the notification system up and running? Primefac (talk) 10:52, 16 November 2020 (UTC)
    Not yet, the notifications code as presently written will keep spamming the botop every half an hour until their bot comes back up again! I'll probably have to further use SQLite to keep track of notifications sent to avoid repetitions. – SD0001 (talk) 15:12, 16 November 2020 (UTC)
    If you can have the dbase/check running without the notifications enabled, feel free to start running that part while the rest gets hammered out. Primefac (talk) 17:40, 16 November 2020 (UTC)
  • @SD0001: Happy new year! Gently checking in on the status of this. — The Earwig talk 05:10, 12 January 2021 (UTC)