Been working on open-source projects involving storing and querying observability data (metrics, logs, traces) in parquet[0] and have been frustrated with the usability of Apache Iceberg … despite strongly agreeing and wanting to use an open storage format and catalog.

This makes Ducklake much more interesting for my use case, excited where this is going.

[0] https://github.com/smithclay/duckdb-otlp

Are you using it to replace Mimir?

Not yet.

That said… think duckdb/ducklake/quack could potentially be a future replacement for Mimir or Clickstack with way less operational complexity.

sprites.dev / fly.io has publicly said they are using a variant of JuiceFS for the object-storage-to-VM-filesystem stuff, it's cool tech.

* https://fly.io/blog/design-and-implementation/ * https://juicefs.com

The Claude Code analytics space is really interesting to me right now as well, this is cool.

I'm coming at it from more of the data infrastructure side (e.g. send all of your logs and metrics to a cheap Iceberg catalog in the cloud so you have a central place to query[1]) but also check out https://github.com/tobilg/ai-observer -- duckdb popping up everything to make this interesting and easy.

[1] https://github.com/smithclay/otlp2pipeline

That is great, thanks for sharing your work and that other link. Codex also supports OTel.

I love that you made the OTel DuckDB extension last year and then were able to flex it months later for these pursuits.

High-quality digital twins of complex software does not bode well at all for a lot of SaaS companies.

For customers, it makes migrations much easier and less-risky between vendors.

For the vendors themselves, it means you can cheaply and reliably port features your competitors have that you don’t have.

Personally: love the decorator pattern after I got used to it :)

Posted this yesterday as well, but seems like a really nice emerging pythonic way to call out to remote infrastructure (see: Modal[1]).

[1]: https://modal.com/docs/examples/hackernews_alerts#defining-t...

This looks cool, love the approach of a more "pythonic" infra as code -- have also noticed this pattern in Modal [1] and Ray [2] and all for it.

[1] https://modal.com/docs/examples/hackernews_alerts [2] https://docs.ray.io/en/latest/ray-overview/getting-started.h...

edit: forgot to mention StackQL [3], which is somewhat similar but takes a SQL-first vs python-first approach. The IaC space is about to get a lot more interesting.

[3] https://stackql.io/docs

Thank you, just had a look at modal, which I’ve not heard before. Looks useful

We need more rigorous benchmarks for SRE tasks, which is much easier said that done.

The only other benchmark I've come across is https://sreben.ch/ ... certainly there must be others by now?

We publish the benchmarks for HolmesGPT (CNCF sandbox project) at https://holmesgpt.dev/development/evaluations/

Anthropic ships an official plugin to create linters for you based on your Claude Code history or instructions, it’s great. You can vibe code your lint rules per repo.

https://github.com/anthropics/claude-code/tree/main/plugins/...

This is part of a bigger consolidation trend, AI hype or not: which general-purpose data vendor gets to store and query all of your observability and business data?

Snowflake acquired Observe last week, AWS made it easy in December to put logs from Cloudwatch in their managed iceberg catalog, and Azure is doing a bunch of interesting stuff with Fabric.

The line between your data lake/analytics vendor and observability vendor is getting blurry.

This is an emerging pattern that’s surprisingly powerful: thick clients that embed wasm query engines (pglite, duckdb) and do powerful analytics (with or without AI agents writing the queries).

Below are two examples using duckdb under the hood for similar purposes. Like the author, excited for this type of architecture making semi-advanced analytics more attainable if you’re not a data engineer.

- https://pandas-ai.com/ - https://marimo.io/