If you don’t watch your infrastructure costs, you may end up with a huge bill at the end of the month. This is how an experienced engineer approached reducing observability costs.
The "OK" health check logged every few seconds burning hundreds of dollars a month is such a believable story.
Nobody looks at a library's default logging when they add it, and by the time it's costing real money it's buried under a year of other changes, that's exactly why regular pattern audits catch what a one-time review misses.
Yup, I was horrified when I realised the cost! I do wish libraries would stop randomly logging things that aren't configurable automatically without the user asking for them. I had to block this by hacking a lambda into the configuration of the logger, but there was no other way short of forking the library or submitting a PR that could take months to get through.
Absolutely. Auditing line items, in my experience, if possible needs to be done more often than annually so that one can respond to the increasing costs and get them under control before the contract renewal comes along.
Indeed, the cost accumulates over time, and when it's already too late, you would need to rebuilt the full architecture in order to cut the costs significantly.
Thanks! I was observing a Ruby on Rails app with a small team. One piece of context that I should emphasise is that I'd already added a ton of instrumentation to the app so this was kind of inevitable!
Cutting observability costs by 60% in two days is impressive, but it raises a question: why was the waste allowed to accumulate in the first place? Cost optimization should be a continuous discipline, not a reactive sprint. Let's prioritize proactive monitoring over panic-driven fixes.
Agreed - this was a learning experience for our team at the time. We were also under pressure to deliver. As I said above, this was also the first time I'd heavily instrumented an app and the big drivers of cost were counterintuitive to me.
This is why its important for every organisation to first measure important metrics and review them periodically. High performance organisations always focus on maintenance of systems.
The "OK" health check logged every few seconds burning hundreds of dollars a month is such a believable story.
Nobody looks at a library's default logging when they add it, and by the time it's costing real money it's buried under a year of other changes, that's exactly why regular pattern audits catch what a one-time review misses.
Yup, I was horrified when I realised the cost! I do wish libraries would stop randomly logging things that aren't configurable automatically without the user asking for them. I had to block this by hacking a lambda into the configuration of the logger, but there was no other way short of forking the library or submitting a PR that could take months to get through.
This is death by a thousand paper cuts — I see the same pattern with clients' SIPs and insurance premiums.
Small recurring costs compound quietly; by the time they're visible, the architecture (or portfolio) is already built.
Auditing line items annually, not just totals, is the real discipline.
Absolutely. Auditing line items, in my experience, if possible needs to be done more often than annually so that one can respond to the increasing costs and get them under control before the contract renewal comes along.
Indeed, the cost accumulates over time, and when it's already too late, you would need to rebuilt the full architecture in order to cut the costs significantly.
what exactly were you observing again?
I would like to learn more.
Thanks! I was observing a Ruby on Rails app with a small team. One piece of context that I should emphasise is that I'd already added a ton of instrumentation to the app so this was kind of inevitable!
Cutting observability costs by 60% in two days is impressive, but it raises a question: why was the waste allowed to accumulate in the first place? Cost optimization should be a continuous discipline, not a reactive sprint. Let's prioritize proactive monitoring over panic-driven fixes.
Agreed - this was a learning experience for our team at the time. We were also under pressure to deliver. As I said above, this was also the first time I'd heavily instrumented an app and the big drivers of cost were counterintuitive to me.
Great article!
“if you can’t measure it, you can’t improve it”
This is why its important for every organisation to first measure important metrics and review them periodically. High performance organisations always focus on maintenance of systems.
Glad the article resonated!