Conversion is async. The whole point is to never deal with CDC which is error prone and taxing Postgres with occupying a replication slot and burning memory and cpu in the OLTP system.
Taxing Postgres is one thing, which can be overcome with ways like using standbys. There could other more native ways (than unifying storage), which you’ll hear about in a few weeks. Also I don’t fully agree on logical replication taxing Postgres, if the client is built with care and precision.
In regards to error prone and speed (lag, latency at real-world scale), I wish the blog went into more detail and gave evidence than talk theory.
Moonlink was arguably one of the best ZeroETL(mirroring) solution, and there's a reason we built LTAP instead (not just because it's cool and we can do it)
To me,
two big issues with CDC/mirroring are:
1. For ad-hoc querying, CDC is usually not there when you need them, and it is too expensive to maintain for all tables if you don't query them often.
2. For true data pipeline, a simple mirror is not enough, you will end up building all kinds of transformation and it essentially becomes ETL like spark.
One small issue is CDC + merge into columnstore means huge write amplification if you want fresh read on Analytics side, it is absurd especially for lakehouse tables (I have seen 100X for many OLTP workloads).
The only solution I found that we can trust agents creating, managing, running OLTP & OLAP queries is LTAP.
If you product is CDC based (peerdb) you don’t want storage to support this :)
This architecture is better for OLTP because all maintenance operations are moved to storage AND it has all other benefits such as LTAP that emerge from having a scalable storage.
Oh no, I did read the blog. Not throwing shade at anyone here—the blog is great. It just doesn’t provide real-world evidence, and it opens up a bunch of technical questions that I’m trying to understand. that’s exactly what HN is for. :)
I assume the parquet files are way larger than the page format, doesn't this cause a lot of read amplification? OLTP side needs to read a lot more data to fetch a single old row that's stored in parquet format.
Lakebase is referring to the fact that in addition to disaggregated storage s3 is authoritative storage for older data.
Since data is on s3 (or lake) you can perform direct to s3 type operations like data loading, reading this data by engines that are not Postgres and more
> in addition to disaggregated storage s3 is authoritative storage for older data
Suppose a person retrives cold data from another Object Storage protocol rather than S3. This is no longer a "Lakebase", so we have to come up with a different name to avoid confusion.
But if you say "Disaggregated Storage on S3" then you have the flexibility to change that to "Disaggregated Storage on FOOBAR" to avoid confusion.
> Suppose a person retrives cold data from another Object Storage protocol rather than S3. This is no longer a "Lakebase", so we have to come up with a different name to avoid confusion.
I've never seen "lake" or adjacent terminology refer to S3 specifically like that vs other object storage. A data lake on Ceph would still be a data lake.
(My quibble would be that "lake" often refers to inconsistent or unstructured, and itself has always been a bit handwavy compared to "warehouse," whereas this is very structured data on object storage.)
Maybe I’m wrong, but AFAICT this is block (page) storage backed by S3, tuned for Postgres with some paxos-linked storage/caching servers sitting in front? Sounds good, but I’m not sure “lake” or “warehouse” is a word I’d choose… much closer to Litestream-with-reads, or the somewhat-famous “I ran out of RAM so I downloaded some more” blog article.
Again, when the users of this app are just 3 admins, over basic metrics from ArcGIS Monitor, I dont need to deviate from their recommended, tune, or much of anything.
This isnt some high throughput app where every IOPS matters.
And in my experiences, most apps that need a database arent serving 1000's of people per hour. Most are fine with a few users per hour.
The whole "scale to scale cause youre 1 minute away from HN hug of death" is just wishful thinking a budget bloat from the hyperscalers selling 'butwhatifs'. Ive called AWS to task on their 5 pillars crap, which basically says "pay double for redundancy". Gee, I wonder why AWS would recommend customers to pay double (eyeroll).
This applies to our storage implementation. In Lakebase architecture storage serves pages and it doesn't always have the most recent version of the page and therefore it reconstructs it on demand.
In the past we relied on Postgres compute to periodically send a full page so reconstructive a page was always a bounded process. Once we turned it off (and got all those perf gains) we got another problem: unbounded page reconstruction which we had to solve separately.
In the blog article[1] that linked to, it says "Unified transactional and analytical workloads: Lakebase integrates seamlessly with the Lakehouse, sharing the same storage layer across OLTP and OLAP. This makes it possible to run real-time analytics, machine learning, and AI-driven optimization directly on transactional data without moving or duplicating it."
Is the "without moving or duplicating" part actually a true statement? If the actual table state is only reconstructed by the pageserver, its not like Spark can just read it from S3.
Read replicas can be "shallow". You don't need to replicate all the data to create a replica. This allows to create them very very quickly (sub second).
All the extension still work. We don't support Citus today, but mostly because customers are not asking for it rather due to technical limitations. We support lots of extensions: https://docs.databricks.com/aws/en/oltp/projects/extensions
Thanks for offering. In the graph labeled "Prod customer throughput: (higher is better)" eyeballing it within a week you are seeing ~2k qps peak increase over the previous week.
Operationally, how do you handle landing that large of a perf improvement? If my data store changed that much in a week it could break something.
Generally the more throughput the system supports the better. In this case we were hitting limits (btw each operation is many queries of different sizes) and the customer observed higher latencies which is typical if the system can't sustain the throughput required.
After this change latencies are back to normal and throughput increased.
* Rate limiting on proxy in front of compute fleet
* Large tenants are broken up into shards, reducing hotspots
* Each shard is throttled to a fixed req/s rate
* We do not run pageservers at their redline in terms of CPU load, so there is some slack to take up bursts
* Capacity quotas which selectively throttle write traffic to the largest databases if they are competing with others for disk space, until the larger database is migrated away.
Not really. OrioleDB solve the vacuum problem with the introduction of the undo log. Neon gives you scale out storage which is in a way orthogonal to OrielDB. With some work you can run OrioleDB AND neon storage and get benefits of both.
