There is a difference between bugs and attacks. I think we are trying to solve attacks here. In an attack, I might build an LLM targeting some service that uses LLMs to execute real world commands. Adding providence to LLMs seems like a reasonable layer of security.
Now we shouldn’t be letting a random blob of binary run commands though right? Well that is exactly what you are doing when you install say Chrome.
Undoability is going to be a consideration. We let people use credit cards with practically no security for convenience, because the cost of reversing a few transactions of refunding people for fraud is low enough.
Now we shouldn’t be letting a random blob of binary run commands though right? Well that is exactly what you are doing when you install say Chrome.