We Had 100 Dead Alerts Firing for Services That No Longer Existed. So I Built a Kubernetes Operator.

TL;DR: I built and open sourced a Kubernetes operator that manages Grafana Cloud dashboards, alert rules, and SLOs as code — with automatic cleanup when services are decommissioned. It solves the "...

By · · 1 min read
We Had 100 Dead Alerts Firing for Services That No Longer Existed. So I Built a Kubernetes Operator.

Source: DEV Community

TL;DR: I built and open sourced a Kubernetes operator that manages Grafana Cloud dashboards, alert rules, and SLOs as code — with automatic cleanup when services are decommissioned. It solves the "100 orphaned alerts" problem by coupling Grafana resource lifecycle to Kubernetes resource lifecycle. It was a Tuesday afternoon when someone on the team noticed that Grafana was still sending alerts for a service we'd decommissioned four months ago. Not one alert. Not five. We found over 100 alert rules in Grafana Cloud that had no corresponding live service. Some went back almost a year. No one cleaned them up — ownership was unclear after teams changed. The alerts just stayed there, quietly firing, quietly getting ignored, quietly eroding trust in the entire alerting system. That's when I started building the Grafana Cloud Operator. The Problem With Managing Grafana Manually If you've worked on a platform team, this scenario is probably familiar. Grafana is great for interactive observabil

Related Posts

Trending on ShareHub

  1. Understanding Modern JavaScript Frameworks in 2026
    by Alex Chen · Feb 12, 2026 · 0 likes
  2. The System Design Primer
    by Sarah Kim · Feb 12, 2026 · 0 likes
  3. Just shipped my first open-source project!
    by Alex Chen · Feb 12, 2026 · 0 likes
  4. OpenAI Blog
    by Sarah Kim · Feb 12, 2026 · 0 likes
  5. Building Accessible Web Applications: A Practical Guide
    by Alex Chen · Feb 12, 2026 · 0 likes
  6. Rapper Lil Poppa dead at 25, days after releasing new music
    Rapper Lil Poppa dead at 25, days after releasing new music
    by Anonymous User · Feb 19, 2026 · 0 likes
  7. write-for-us
    by Volt Raven · Mar 7, 2026 · 0 likes
  8. Before the Coffee Gets Cold: Heartfelt Story of Time Travel and Second Chances
    Before the Coffee Gets Cold: Heartfelt Story of Time Travel and Second Chances
    by Anonymous User · Feb 12, 2026 · 0 likes
    #coffee gets cold #the #time travel
  9. Best DoorDash Promo Code Reddit Finds for Top Discounts
    Best DoorDash Promo Code Reddit Finds for Top Discounts
    by Anonymous User · Feb 12, 2026 · 0 likes
    #doordash #promo #reddit
  10. Premium SEO Services That Boost Rankings & Revenue | VirtualSEO.Expert
    by Anonymous User · Feb 12, 2026 · 0 likes
  11. NBC under fire for commentary about Team USA women's hockey team
    NBC under fire for commentary about Team USA women's hockey team
    by Anonymous User · Feb 18, 2026 · 0 likes
  12. Where to Watch The Nanny: Streaming and Online Viewing Options
    Where to Watch The Nanny: Streaming and Online Viewing Options
    by Anonymous User · Feb 12, 2026 · 0 likes
    #streaming #the nanny #where
  13. How Much Is Kindle Unlimited? Subscription Cost and Plan Details
    How Much Is Kindle Unlimited? Subscription Cost and Plan Details
    by Anonymous User · Feb 12, 2026 · 0 likes
    #kindle unlimited #subscription #unlimited
  14. Russian skater facing backlash for comment about Amber Glenn
    Russian skater facing backlash for comment about Amber Glenn
    by Anonymous User · Feb 18, 2026 · 0 likes
  15. Google News
    Google News
    by Anonymous User · Feb 18, 2026 · 0 likes

Latest on ShareHub

Browse Topics

#ai (4023)#news (2440)#webdev (1684)#programming (1266)#business (1142)#opensource (962)#security (936)#productivity (878)#/business (823)#javascript (760)

Around the Network