Building a CLI So I Didn't Have to Click Through Kibana

Open Distro for Elasticsearch launched on March 11th, 2019, and Alerting was one of the plugins that shipped on day one. Two weeks after that, on March 25th, I merged my first PR against it — support for Kibana 6.6.2. Over the next six months I kept shipping PRs against the alerting Kibana plugin, mostly chasing whatever Kibana version had just shipped: 6.7.1 in April, 7.0.1 in May, plus a steady stream of smaller fixes — where-clause support for visual monitors, a monitor update bug, moving the schedule field per UX feedback.

Somewhere in the middle of all that, I got annoyed at how I was actually managing monitors day to day. Alerting monitors — the rules that watch your data and fire notifications — lived entirely in the Kibana UI. If you wanted the same monitor on two clusters, you clicked through the UI twice, by hand, and hoped you didn’t fat-finger a threshold. If you wanted to know what changed in a monitor over time, you didn’t, because there was no history — just whatever state it was currently in.

So I built odfe-monitor-cli, a command-line tool that let you define monitors as YAML files instead. Write the monitor once, commit it to git, and push it to as many clusters as you wanted. Diff it before you applied a change. Actually know what changed and when, because now it was a file in version control instead of a state hidden inside Kibana.

On June 12th, 2019, I wrote it up for the AWS Open Source Blog — the first time I had a byline on an official AWS blog post. Then in July, at OSCON 2019 in Portland, I gave a talk on the same idea, “Gourmet Alerting with Open Distro for Elasticsearch”. Same tool, same argument: alerting rules are configuration, and configuration belongs in version control, not locked inside a UI you have to click through by hand.

I spent months in the actual UI, fixing the actual bugs real users hit, and the CLI came directly out of being annoyed at my own workflow while doing that. It wasn’t a roadmap item — I was the one shipping the plugin every week, had to live with how clunky it was to operate, and eventually just built the thing I wanted.

odfe-monitor-cli on GitHub · AWS blog post · OSCON 2019 talk