I used a TimescaleDB in OpsBuddy to support high-throughput log ingestion. TimescaleDB is just an extension on top of PostgreSQL similar to how the uuid-ossp extension adds UUID generation on demand in write queries.
Understanding TimescaleDB
When using TimescaleDB, you create a normal PostgreSQL table and mark it as a hypertable. Under the hood, TimescaleDB automatically splits this table into chunks by time. By default, the chunk interval is 7 days, but you can configure it.
For example, if you set the chunk interval to 1 day, then each day's data will be stored in a separate chunk where each chunk is a table.
Why Split Data into Chunks?
Now, you might ask: Why split data into chunks instead of storing everything in one big table?
Imagine you have a table for log ingestion. If you're ingesting 10k logs every minute, in just 100 mins you'll already have 1 million rows. Writing to a single large table with indexes becomes expensive because every insert requires updating the index and the full table structure. This is not ideal for high throughput scenarios.
With TimescaleDB, each chunk is a smaller table (in our example, one day's worth of logs). Maintaining and writing to these smaller tables is much faster, since inserts and index updates are limited to that chunk. That's how TimescaleDB supports high-throughput ingestion.
Querying Across Chunks
Timescale also provides optimized tools for querying across hypertables, so even though the data is chunked, queries remain efficient.