Log usage?
https://medium.com/@tjholowaychuk
Hello!
I'd love to know what kind of volume your product(s) produce monthly, and if you're unhappy with your current logging solution I would love to know why, or any deal-breakers would be useful as well.
I've worked with a few startups which produce well over $6000/mo worth of logs alone, but I'm not sure how common this is, I'm curious to see what other indie hackers use!
I've been working on a product to dramatically reduce the cost (by an order of magnitude), and of course retain features and performance for huge volumes.
If you're interested in the product itself it's still a few months out, but I'll be mentioning it on Twitter (https://twitter.com/tjholowaychuk) and Medium (https://medium.com/@tjholowaychuk).
I spend a couple hundred dollars a month on Papertrail for 2–4 GB a day, with a 30 day search duration. I serve ~10MM API requests a month currently, with lots of background jobs. I like the simplicity of their product and I can always find what I need with their search, but I do wish I didn't have to pay so much. It's one of my largest bills, next to databases.
Nice! I think that's around the range where mine might start to make sense, anything cheaper it's kind of just a "why bother" situation. I use Papertrail as well, generally happy with it other than the lack of structured logging. I haven't decided on a pricing model yet — I'd love to hear what everyone thinks there as well — but it should be at least 5-10x cheaper.
What I usually see is that big companies with big budgets log almost anything it happens in their apps. While small companies with small budgets are very careful with what they log.
So I think that if you can make logging cheap and properly searchable you will have a lot of small companies coming to you and increasing the amount of data they log.
Yeah we'll see haha, the stuff I'm working on is on-prem, so small people could still use it but it'll definitely be tailored towards larger companies. So far from what I've built, targeting the lower end of the markets doesn't seem wise.
I'm still working on getting canarykit.com off the ground, so I'm not presently using any sort of logging service. Most of the useful log entries are on a single host, which makes them easily greppable. I'm probably running ~2.5-5million lines of logs per month right now.
I've been eyeing DataDog's logging product for when I get to the point that I need something for several hosts. It costs $1.27/mo for every 1 million lines of logs, and the killer feature for me is that it automagically correlates to your APM and metrics data so you can easily trace/find what you're looking for -- which I think would help me save a bunch of time when I need to dive into the logs for debugging.
I've run a few pretty large ELK clusters in the past for different sized teams -- iirc they were in the ~$5000/mo to run, just because of the log throughput, and need to add more indexing nodes to keep up with the throughput. I'd be willing to bet this is fairly common for medium/large/growing vc-backed startups -- but I would be amazed if indiehacker type folks would spend anywhere near that level.
I have two standout pains that I've had when working with logging stuff. The first thing is that running ELK clusters is complicated and gets expensive very fast. Operating elasticsearch is a beast. I feel like having a logging-centric data store with less moving pieces for log storage and querying would be amazing (and probably a great way to cut costs and increase performance).
The other thing is that setting up log parsers to get all of the data extracted from the logs is kind of annoying. I've spent a ton of time trying to get things parsed and working in janky grok parser debugger web apps -- having an intuitive ui for auto-discovery and splitting log patterns would be pretty cool.
--
p.s. Your work on Apex ping and up were both a huge inspiration to me and how I think about serverless and monitoring -- so thank you! If you'd like to bounce ideas or ask any more questions, feel free to shoot me an email -- my email address is in my indiehackers profile.
Thanks man!Good luck with your project, and I appreciate the input!
I had a similar experience with ELK, the performance is great but it's crazy expensive. I haven't decided on how the parsers will work yet, if that'll be on ingestion or not, ideally I'd like to keep all the logs structured, but there's no reason there couldn't be another endpoint that discovers the type (syslog, nginx, whatever).
DD's pricing model is interesting, wonder why they did that vs the standard per gb, their pricing is pretty expensive, but they definitely have a crazy suite of features!
No immediate comments on the topic (other than the fact that a bug in the log shipper made me blow past logz.io 3GB daily allowance today, minutes before a launch), I just want to voice my surprise about bumping into a NodeJS legend here.
Everyone— pretty much any service that runs on NodeJS is likely to rely on TJ's open source code in one way or another, possibly a critical/foundational one (see ExpressJS). If a problem he's addressing is one you're dealing with, I'd be all ears.
Thanks man! :D