Oh, oh, oh! I got one not mentioned yet:

ELK.

Well, not the whole of the ELK stack (Elastic, Logstash and Kibana, though the full stack size is much larger nowadays), but their watchers. A watcher is a piece of JSON with some search specifications on when to trigger and send an alert to email/slack/teams/whatever. We’re basically abusing it as an alerting system, and generally it works… Fine… Presuming Filebeat actually ingests our logs (which is partially our fault, as there’s a fix, but it takes too damn long to drag 3 teams along to implement what needs implementing to fix that problem).

Anyway, the problem is not the watcher itself, even though it is painful (heh) to learn the structure. It’s “Painless”, the JVM-based scripting language available in a watcher. It’s anything but. It is SO painful to write code, inside of a JSON object, making sure everything is exactly as it should be, having to use the DevTools in Kibana to try and trigger it, wait to see what enormous error comes out while praying it works. No IDE, no nothing. Ah, I lied. It does have Syntax Highlighting, for non-Painless code, IIRC…

Oh, having to dig information out of the data you get is super unintuitive too.

At least the UI/Kibana is good, and Elastic is pretty good too. Fuck Filebeat though. And Painless.