> AI results in Google mark the first time I have ever encountered incorrect or patently untrue information at the top of a Google query.

Google search ads have been rife with scams for a while[1]. I think that counts as patently untrue information at the top of a google query.

[1] https://searchengineland.com/google-search-ads-brands-fraud-...