[HN Gopher] Show HN: Log collector that runs on a $4 VPS ___________________________________________________________________ Show HN: Log collector that runs on a $4 VPS Hey guys, I'm building erlog to try and solve problems with logging. While trying to add logs to my application, I couldn't find any lightweight log platform which was easy to set up without adding tons of dependencies to my code, or configuring 10,000 files. ErLog is just a simple go web server which batch inserts json logs into an sqlite3 server. Through tuning sqlite3 and batching inserts, I find I can get around 8k log insertions/sec which is fast enough for small projects. This is just an MVP, and I plan to add more features once I talk to users. If anyone has any problems with logging, feel free to leave a comment and I'd love to help you out. Author : Nevin1901 Score : 15 points Date : 2023-02-11 20:44 UTC (2 hours ago) (HTM) web link (github.com) (TXT) w3m dump (github.com) | withinboredom wrote: | Neat! Have you considered using query params instead of bodies, | then just piping the access logs to a spool (no program actually | on the server, just return an empty file). Then your program can | just read from the spool and dump them into sqlite. | | That should tremendously improve throughput, at the expense of | some latency. | unxdfa wrote: | I see your idea but you could drop the JSON and use rsyslogd + | logrotate + grep? You can grep 10 gig files on a $5 VPS easily | and quickly! I can't speak for a $4 one ;) | folmar wrote: | Sorry, but I don't see the selling point yet. Rsyslog has | omlibdbi module that send your data to sqlite. It can consume | pretty much any standard protocol on input, is already available | and battle proven. | keroro wrote: | If anyones looking for similar services Im using vector.dev to | move logs around & it works great & has a ton of | sources/destinations pre-configured. | Dachande663 wrote: | I've found the hard part is not so much the collection of logs | (especially at this scale), but the eventual querying. If you've | got an unknown set of fields been logged, queries very quickly | devolve into lots of slow table scans or needing materialised | views that start hampering your ingest rate. | | I settled on a happy/ok midpoint recently whereby I dump logs in | a redis queue using filebeat as it's very simple. Then have a | really simple queue consumer that dumps the logs into clickhouse | using a schema Uber detailed (split keys and values), so queries | can be pretty quick even over arbitrary fields. 30,00 logs an | hour and I can normally search for anything in under a second. | harisamin wrote: | Ah cool! Somewhat related I built a json log query tool recently | using rust and SQLite. Didn't build the server part of it | | https://github.com/hamin/jlq | Hamuko wrote: | I feel like if you're going to use "$4 VPS" as a quantifier, you | could at least specify which $4 VPS is being used. | teruakohatu wrote: | DO's 512mb basic VPS starts at $4, so I am guessing it is that. ___________________________________________________________________ (page generated 2023-02-11 23:00 UTC)