In the past at least ChatGPT would reply "Building a perpetual motion machine sounds like a great idea, here are some plans on how to get started. Let me know if you need help with any of the details".

This has been a problem with using LLMs for design and brainstorming problems in general. It is virtually impossible to make them go "no, that's a stupid idea and will never work", or even to push back and give serious criticism. No matter what you ask they're just so eager to please.