If I was a user of this product I'd be really worried about the "We rewrote our software 5 months ago, but that didn't solve the issue we wanted to solve so we're doing it again, maybe this time we'll fix it" message of this post.
Unironically, I think we will see a lot more of this given how engineering is being done currently. I have been involved in 2 green field projects in the last 2 years. Both have been rewritten or had massive "ship of theseus" type changes multiple times over that time window.
Engineers working on new products, thanks to LLMs, have very little understanding of the technical details of the product. They have the same level of understanding a traditional high level manager had of products they owned. If you ever worked in an engineering org with an M2 or M3 who always attended your outage review, and in every outage review something has to involve some issue with Postgres or your Kafka queue, that disconnected M3 would conclude:
"Almost every outage involves some problem with Postgres. My conclusion, observed over the last 6-12 months, Postgres is bad. I see this new FancyDB or DynamoDB thing is popular on Twitter. We should try that"
I have lost count of the number of idiotic takes like that from M2s or M3s who have no idea how things work but instead focus on keywords they can latch onto that seems to be in the intersection of all issues. Then the logical solution is just to remove that and see what happens. This used to be very expensive, but if you can throw an LLM at a code base and say "Replace Postgres with FancyDB, split it to a list of logical checkpoints adding regression testing when missing to each checkpoint and open stacked PRs for all checkpoints", let it churn for an after noon, and 46k-line change PR (and $800 token bill) later you are ready to try FancyDB, why would you not give it a try? Then if you have issues with FancyDB, are you gonna spend the time to understand what's going on or just say "lets try DynamoDB next"?
This is what this new feature called “loops” are for. Tell your agent to identify a list of databases and try them all, in prod, until it finds a good one. Be sure to tell it “no outages”.
To me to key “insight” is that if you are going to spin up single VMs (or even groups) per-customer and then route queries, you need to be able to estimate the query cost first.
I know snowflake does this, they do the query optimization before sending it for execution. So it knows about how many resources will be needed (it can obviously be wrong).
How does Snowflake use that information? You still need to pick the warehouse. It would be cool if there was an "elastic warehouse" where heavier queries are routed to bigger Warehouses, but that's not the case
The entire website is a bit of a red flag to me, starting with the cookie banner featuring an image of Ursula von der Leyen and some misinformation (cookie banners are not legally required).
> Five months ago, we rebuilt our warehouse on DuckDB instead of ClickHoue
They switched from ClickHouse to DuckDB five months ago, then switched from DuckDB to their own Trino derivative?
The article then ends before saying if the new setup solved their problems.
It’s great that we can now make these migrations with little downtime (I assume?) and try different things, but I wish we could have read more about what testing they did before jumping. The fact that they jumped to DuckDB without realizing it wasn’t going to work at their scale and spent only a few months with it before jumping again is not convincing.
I hoe there’s more to this story that just didn’t make it into the blog posts. As written it’s giving me flashbacks to some teams I worked with who changed tech stacks every six months because they liked migrating more than tuning and maintaining.
Founder here -- this is about our warehouse product which is separate from our main analytics DB. Our DuckDB warehouse was used by a handful of pilot customers. Our clickhouse database is serving all of our analytical queries.
Things like this are a part of why I push back on people telling me we can do important things later. I don’t want to swap the engine out of a machine that’s moving that fast. There’s no time to learn about how it behaves with real versus synthetic traffic and what the failure modes are when you’re doing many thousands of transactions per second.
Why wouldn’t you want to sort stuff like that out when the problem is still relatively small?
The only other reasonable option is to wait until the problem gets so big that it splits into several domains. And then you can move one domain over, see how that goes, and move more over if it goes well.
If you can’t know for certain how the system will behave under organic load that may mean migrating traffic onto the new system and then back to the old one several times, which means you need migrations in both directions and each transition is an opportunity for human error to destroy data or take the service offline.
There are many good reasons why the industry largely moved away from having DBAs as a specialist position. But one downside I've seen is that it's increasingly common to have teams decide to just switch platforms without fully exploring whether that's really necessary.
I'm not really surprised, data storage and data access technology is a pretty large subject all by itself, and a "jack of all, master of none" developer might never develop the expertise needed to properly optimize a database.
I think it was closer to what they need; it comes down to keeping TB of data on local SSD and having a query engine next to it. That takes persistent, shared CPU, caching, sharding, resource management, etc.
Duckdb is an interesting approach to resource management (all ephemeral), but TB's are TB's. I doubt it was finding data where it needed it.
Clickhouse seems slow at getting to the top end of scaling; it's stuck with the 99% of its userbase doing 1 GB queries. I'm hoping they put some of their money into it.
I don’t know? They have blog posts about it but I don’t know how much I’d trust them because their chosen replacement only lasted 5 months before the next replacement.
The intro to that post is so bad: "PostHog's original data warehouse had a single goal: delay the moment a company needs to hire their first data engineer."
It is not to serve high quality and timely data to make good decisions, is to avoid hiring a specific role, lol. So bad.
I mean, mostly it just comes off as egregious self-indulgence to try one thing and then jump to "we built the solution ourselves!" when the problem is not novel at all. These things are just tiresome.
Sometimes a company just isn't able to manage a data warehouse; I've worked for a few.
I even phone screened with this one but couldn't cover their crazy quilt of requirements; at that time they were also looking for someone to rewrite the clickhouse optimizer, and they were stuck trying to reshard their cluster, which should take an afternoon.
I still like the company because their business prop reminds me of what we were doing in 2009, a kind of federated analytics organization on a centralized data warehouse that enables a lot of tools to be developed quickly at large scale.
But goddammit they keep picking these half-working platforms and flip-flopping from one to another. We were crushing millions of queries against 2T-row tables on Vertica in 2010, and it's only gotten better for customer-facing data apps, which is what Posthog is. The company doesn't seem to have a baseline of what to expect; the market is too saturated with slow tools that lower expectations.
I wonder if they have considered moving back to ClickHouse. ClickHouse Iceberg support has improved a lot over the past few months. You can easily run stateless ClickHouse on top of Iceberg. This solves the tenancy issue, and it can easily handle petabytes scale [1].
I actually like the menus. They work great on mobile too.
We built up all these UI conventions from decades of desktop development, then abandoned them for the web (and worse, electron). It's annoying learning a new convention for every website. We could do a lot worse than bringing back menus.
I kind of liked the originality. It's what happens when you give software developers free reign on the marketing pages as well, I guess.
But it seems to have gotten a re-skin recently, and I preferred the previous more pixelated version. Also, there used to be a way to render a "normal" version of the website, but I can't find it anymore.
And what's with that bs cookie banner?
If the only cookies are functional/essential, they don't need to show a cookie banner.
If it's for analytics, then I should be able to reject it.
Otherwise, it's just a dick-over.
Now they have their "own flavor" of Trino and a brand new open table format called "HogLake". This gives the impression of making excuses with little introspection.
It would be great to see their ducklake fork, their patches and github issues so the community can learn.
The key insight I am getting from this is that as migrations get easier to do with agents, there will be a lot more of them.
But more importantly a lot of them will happen because we will all get more trigger happy to start one. On the other hand I also don't know if at scales like PostHog, a successful migration can be tested without doing all of it.
This reminds me so much of when we would bring a new marketing leader and they would plant their flag by nuking all of the existing marketing automation processes to bring in theirs, with no net impact on the business. Bike shedding.
I would have appreciated more technical details with methodologies and explicit failures observed. Oh and a human voice would be nice for a change.
> Customers ran more complex queries than we expected
Did the queries change after DuckDB was introduced? Did they do a study of query patterns that could have anticipated this ahead of time?
> Predicting CPU and memory for a specific query was even harder.
What attempts / methodologies were used to try this? What was hard? What failed?
> We'd rather have contributed to the existing community and helped it grow. But we couldn't land changes as fast as we needed, so we ended up carrying patches in a fork
This isn't a BAD approach per se. IME it's common (and necessary) for the large scale users of a platform to maintain a fork for their needs and work with the community and the maintainers to upstream changes when/if the community is ready to accept. That's called influencing. On the other hand, if you toss of a few slop grenades to the community and they reject you shouldn't be surprised. (Not suggesting that's what happening here but the defeatest conclusion may imply it?)
> Missing reconnects and managing TTLs and idle connections at scale is hard
One thing I really appreciate about CloudFlare blog posts is that often they show their work, by showing what tickets they opened, what improvements they land upstream. With this march across clickhouse, duckdb, now trino, I'd love to know what if any places the team worked with upstream, what improvements got tried.
I'm pretty ok with the march across products myself. It felt like the big shift was from shared giant clickhouse to lots of duckdbs. I wonder what juggling many more clickhouses would have been like. Excited and hopeful to here more back on Trini, with hope!
It's wild how many really really good data stacks are out there. Apache has been collecting some absolutely stellar projects. And it feels like the rest of world meaning everyone but China is kind of sleeping on a lot of these options. Maybe it's just that these projects often go out of their way to have a place for emblems, for who is using the project, but a lot of these have pretty wide ranges of companies using them.
I do wish we were still in an age of company blogging, where people did talk more to the tech getting played with, talk to what's cooking in the shop. It's great to have these outreaches. There's so much tech and we are just so quiet about it! There's so much less signal to go off of, so much less shared awareness than the aughts and 10's.
This idiotic churn used to be restricted to front end devs and backend folks were spared. Thanks for Agents now every part of the stack is going to be in constant churn thanks to resume builders. At no point in the article did you get a sense they have a handle on what their problem is and why a certain technology and architecture would be a better fit. Just try it out, churn out, try it out.
6 months from now: We are moving our warehouse off of Trino + HogLake to newer technology XXX.
This is ridiculous. Why can you not foresee such issues? Do you not do testing with mock data at the size you need? Migrating production between stacks so often means you are not getting new features in. This is pure churn.
It feels like a lot of churn and not a lot of measurements or engineering.
Engineers working on new products, thanks to LLMs, have very little understanding of the technical details of the product. They have the same level of understanding a traditional high level manager had of products they owned. If you ever worked in an engineering org with an M2 or M3 who always attended your outage review, and in every outage review something has to involve some issue with Postgres or your Kafka queue, that disconnected M3 would conclude:
I have lost count of the number of idiotic takes like that from M2s or M3s who have no idea how things work but instead focus on keywords they can latch onto that seems to be in the intersection of all issues. Then the logical solution is just to remove that and see what happens. This used to be very expensive, but if you can throw an LLM at a code base and say "Replace Postgres with FancyDB, split it to a list of logical checkpoints adding regression testing when missing to each checkpoint and open stacked PRs for all checkpoints", let it churn for an after noon, and 46k-line change PR (and $800 token bill) later you are ready to try FancyDB, why would you not give it a try? Then if you have issues with FancyDB, are you gonna spend the time to understand what's going on or just say "lets try DynamoDB next"?This is what this new feature called “loops” are for. Tell your agent to identify a list of databases and try them all, in prod, until it finds a good one. Be sure to tell it “no outages”.
I know snowflake does this, they do the query optimization before sending it for execution. So it knows about how many resources will be needed (it can obviously be wrong).
They are big on shipping slop at lightnigg speed. So this is not surprising. Checkout posts by their ceo on cringedin.
They switched from ClickHouse to DuckDB five months ago, then switched from DuckDB to their own Trino derivative?
The article then ends before saying if the new setup solved their problems.
It’s great that we can now make these migrations with little downtime (I assume?) and try different things, but I wish we could have read more about what testing they did before jumping. The fact that they jumped to DuckDB without realizing it wasn’t going to work at their scale and spent only a few months with it before jumping again is not convincing.
I hoe there’s more to this story that just didn’t make it into the blog posts. As written it’s giving me flashbacks to some teams I worked with who changed tech stacks every six months because they liked migrating more than tuning and maintaining.
Why wouldn’t you want to sort stuff like that out when the problem is still relatively small?
The only other reasonable option is to wait until the problem gets so big that it splits into several domains. And then you can move one domain over, see how that goes, and move more over if it goes well.
If you can’t know for certain how the system will behave under organic load that may mean migrating traffic onto the new system and then back to the old one several times, which means you need migrations in both directions and each transition is an opportunity for human error to destroy data or take the service offline.
I'm not really surprised, data storage and data access technology is a pretty large subject all by itself, and a "jack of all, master of none" developer might never develop the expertise needed to properly optimize a database.
Duckdb is an interesting approach to resource management (all ephemeral), but TB's are TB's. I doubt it was finding data where it needed it.
Clickhouse seems slow at getting to the top end of scaling; it's stuck with the 99% of its userbase doing 1 GB queries. I'm hoping they put some of their money into it.
https://news.ycombinator.com/item?id=33082224
https://benchmark.clickhouse.com/#system=+Hoaus|ucqs|Tuts&ty...
Why did they write this post if they had not even allowed things to settle. It sounds like someone wanted to claim victory (and promo?)
https://posthog.com/blog/why-we-rebuilt-our-data-warehouse
It is not to serve high quality and timely data to make good decisions, is to avoid hiring a specific role, lol. So bad.
I even phone screened with this one but couldn't cover their crazy quilt of requirements; at that time they were also looking for someone to rewrite the clickhouse optimizer, and they were stuck trying to reshard their cluster, which should take an afternoon.
I still like the company because their business prop reminds me of what we were doing in 2009, a kind of federated analytics organization on a centralized data warehouse that enables a lot of tools to be developed quickly at large scale.
But goddammit they keep picking these half-working platforms and flip-flopping from one to another. We were crushing millions of queries against 2T-row tables on Vertica in 2010, and it's only gotten better for customer-facing data apps, which is what Posthog is. The company doesn't seem to have a baseline of what to expect; the market is too saturated with slow tools that lower expectations.
[1] https://clickhouse.com/blog/jump-trading-uses-clickhouse-wit...
We built up all these UI conventions from decades of desktop development, then abandoned them for the web (and worse, electron). It's annoying learning a new convention for every website. We could do a lot worse than bringing back menus.
But it seems to have gotten a re-skin recently, and I preferred the previous more pixelated version. Also, there used to be a way to render a "normal" version of the website, but I can't find it anymore.
It would be great to see their ducklake fork, their patches and github issues so the community can learn.
But more importantly a lot of them will happen because we will all get more trigger happy to start one. On the other hand I also don't know if at scales like PostHog, a successful migration can be tested without doing all of it.
> Customers ran more complex queries than we expected
Did the queries change after DuckDB was introduced? Did they do a study of query patterns that could have anticipated this ahead of time?
> Predicting CPU and memory for a specific query was even harder.
What attempts / methodologies were used to try this? What was hard? What failed?
> We'd rather have contributed to the existing community and helped it grow. But we couldn't land changes as fast as we needed, so we ended up carrying patches in a fork
This isn't a BAD approach per se. IME it's common (and necessary) for the large scale users of a platform to maintain a fork for their needs and work with the community and the maintainers to upstream changes when/if the community is ready to accept. That's called influencing. On the other hand, if you toss of a few slop grenades to the community and they reject you shouldn't be surprised. (Not suggesting that's what happening here but the defeatest conclusion may imply it?)
> Missing reconnects and managing TTLs and idle connections at scale is hard
Wonder if they tried pgdog?
Who did the growth estimation of the data? This looks like an example of bad planning.
[0] https://github.com/PostHog/hoglake
I'm pretty ok with the march across products myself. It felt like the big shift was from shared giant clickhouse to lots of duckdbs. I wonder what juggling many more clickhouses would have been like. Excited and hopeful to here more back on Trini, with hope!
It's wild how many really really good data stacks are out there. Apache has been collecting some absolutely stellar projects. And it feels like the rest of world meaning everyone but China is kind of sleeping on a lot of these options. Maybe it's just that these projects often go out of their way to have a place for emblems, for who is using the project, but a lot of these have pretty wide ranges of companies using them.
I do wish we were still in an age of company blogging, where people did talk more to the tech getting played with, talk to what's cooking in the shop. It's great to have these outreaches. There's so much tech and we are just so quiet about it! There's so much less signal to go off of, so much less shared awareness than the aughts and 10's.
This is ridiculous. Why can you not foresee such issues? Do you not do testing with mock data at the size you need? Migrating production between stacks so often means you are not getting new features in. This is pure churn.