The data labeling objection baffles me. Even if you don’t need labels for training, how do you know your model is working if you’re not evaluating it?
My company specializes in statistical long document text classification, but nowadays we mainly work with audit trail requirements because we got tired of hearing complaints about our 5 example learning curve. Seems like the industry standard is telling an llm to label and telling an llm to eval, and crossing your fingers that it’s correct.
I’m a big fan of tmducken. We use it heavily in our prod systems. That said, we’ve recently started exploring ducktape [1] in our new projects and have been really impressed with the performance. It also support more complex types on insertions and queries which has been helpful for us. Not affiliated with the project, but just wanted to show it some love since it’s a bit newer. It was created by an active contributor to tmducken.
It really is - we use duckdb as our main workhorse in our entire stack.
As a side note, thank you for all the work you have done in the Clojure ecosystem! Techascent’s data science packages was a major tipping point in my company’s decision to build our data science ecosystem in Clojure and has been wonderful to work with.
I see a fair number of comments here advocating for either codex to hand-roll this themselves, or to simply punt to SQL. I do want to advocate for the difficulty of the problem, even if I can't speak to the company itself.
At the scale of a few hundred to a few thousand documents, especially short documents, there are a few out of the box methods that can yield reasonable results, whether it be embedding clustering or leveraging LLMs for tagging.
However as your (1) datasets gets larger (2) documents expand from tweets and text messages to 30+ minute conversations and (3) you build downstream analytics on top of the learned semantic units, you really start to feel the limitations of LLMs and embedding for reliable annotation. That doesn't even get into the nuances associated with taxonomy management, seasonality, and model drift.
TLDR; this problem solved effectively has a lot of value and is a lot harder than it seems.
Why is it hard? Ultimately you take whatever your signal is and send it to some relatively cheap LLM.
How is it easier to sign up and manage a different service, implement a different API, etc.
And from the company side the fatal flaw is that these types of tools rely upon 1% of their users having huge spend. Nobody is going to be a huge spender here because it's easier to hand roll than navigate procurement on this (not to mention impossible to justify the spend, additional security/privacy risk, etc.)
It feels approximately impossible for this company to have large accounts.
it gets hard when you need this continuously across lots of chats/calls, with metadata, changing clusters, going deeper into a user journey, etc. the LLM call is just one part of it lol
we're keeping it useful every week, finding out insights that the teams can extract value out of, work with them to understand users better.
the procurement what we've seen is v similar to how one would have for any analytics product? and we're selling this to companies when/once it becomes someone's job to do this
yea, at our volume which we still consider small as we've been able to figure out a way with llms & embeddings, its still fine. + we onboarded a voice ai company with more than 2 hour calls and thats when it was super hard to solve since there were so many elements to consider.
model drifting is something a lot of folks do face after 5th/6th turn as per my understanding and it usually the median, how did you tackle it if you have yet?
also yea, thats why we went for a per customer taxonomy than a general one, yeilded better results + easier to improve upon.
To clarify, I wasn't criticizing your approach or product, more responding to the people dismissing the problem you are solving.
Regarding my experience, I have done a fair amount of work in the contact center space with long calls. I used statistical Bayesian approaches which I found to be much more resilient especially on long documents than embeddings/transformers. It also provided a joint modeling foundation for classification with much lower label requirements than BERT or traditional ML.
IMHO this would make more sense if provided as part of a larger "platform" like Langfuse/Langsmith/etc. Otherwise you just end with a dozen SaaS products for highly specific use cases which might not scale that well.
Realistically do you also need to have this live with a fancy? i.e. a custom solution maybe even Jupiter notebooks initially might be sufficient. It's not like 100k messages is a very large dataset. It's not trivial to make a generic solution that fits every use cases (besides of basic customer chatbots) to get actual value for more agentic products.
i get the push, most teams come to us after they've done/tired of the claude running analysis thing manually and want a pro-active thing.
we're also targeting conversation first use cases and for them this serves as their everything custoemrs. we obv do not sell the fancy part, idts that sells anymore lol. lot of our queries come from our slack app/mcp.
The author really extracted the core tenants of exactly how my former research mentor and I ended up building our business.
We started with the second two points: our core technology was a sampler that enables arbitrary hierarchical Bayesian graph models for sparse data, our constraint was cpu bound tractable compute. The piece that took us the longest to discover was the fact that our end products need to be separate from our underlying technology.
We were given that advice in various words from many people even before we started but some lessons need to be lived to be learned.
One unique core concept per tenant, because the SaaS is nominally multi-tenant but it's really N unrelated websites for N enterprise customers in a trenchcoat
Roughtime is a really cool protocol we came across when we were hardening a license server. It provides a distributed mechanism for cryptographically verifiable time via chained requests. It’s not as precise as NTP (hence rough) but in practice it’s more than precise enough. It also has some nice additional properties: for example, NTP servers are often used as DDOS amplifiers, whereas roughtime servers return a smaller payload than the request.
The ecosystem is currently very young. Each additional deployment meaningfully strengthens the ecosystem (ours is only the fifth server) and each additional implementation
helps harden the spec (which is soon approaching 1.0).
Yeah Show HN has a pretty interesting distribution compared to standard posts due to the long-term visibility on the Show page. The odds of a Show HN post breaking 10 points is significantly higher than an average post, but of the posts that clear 10 points, I recall the likelihood of breaking 100 points to be similar to a regular post.
As a sidenote: That clock is so cool: I was just mesmerized for multiple minutes!
The code provided is to reproduce the analytical results from the annotated data; my impression is that you're more interested in the details of the annotation process than running into an issue with that code?
My company's core technology extends topic models to enable arbitrary hierarchical graphs, with additional branches beyond the topic and word branch. We expose those annotations in a SQL interface. It's an alternative/complementary approach to embeddings/LLMs for working with text data. In this case, the hierarchy broke submissions down into paragraphs added a layer to pool them into submissions, and added one more layer to pool them by year (on the topic branch).
Our word branch is a bit more complicated, but we have some extended documentation on our website if you are interested in digging a bit deeper. Always happy to chat more about the technical details of our topic models if you have any questions!
I totally agree that the metric is imperfect for a long term analysis. I was initially leaning toward a quantile based approach to really focus in on topic trends over time, but when I was initially exploring the data, the relative challenge of having a Show HN become popular in 2025 compared to previous years caught my curiosity, and for this decade I felt a static cutoff provided a simple and easy to understand threshold.
I do think as a metric for total reach, a static cutoff actually works reasonably well. I think some form of square root normalization over total users is probably the best balance.
Thank you! I currently don’t have much insight to this current trend. At the time of this analysis I hadn’t even heard of Clawd but that would definitely be worth my revisiting.
I was planning on doing this yearly but the Clawd excitement is definitely worth diving into.
My company specializes in statistical long document text classification, but nowadays we mainly work with audit trail requirements because we got tired of hearing complaints about our 5 example learning curve. Seems like the industry standard is telling an llm to label and telling an llm to eval, and crossing your fingers that it’s correct.
reply