I'm really trying to do the same, for both my work, and personal projects. But the type of answers I need for work (enterprise software, large codebase built over 20+ years) requires a ton of context that I simply cannot provide to ChatGPT, not only for legal reasons, but just due to the amount of code that would be required to provide enough context for the LLM to chew on.
Even personal projects, where I'm learning new languages and libraries, I've found that the code that gets generated in most cases is incorrect at best, and won't compile at worst. So I have to go through and double-check all of its "work" anyway- just like I'd have to do if I had a junior engineer sidekick who didn't know how to run the compiler.
I think for the work problems, if our company could train and self-host an LLM system on all of our internal code, it would be interesting to see if that could be used to assist building out new features and fixes.
This is the use-case that OpenAI on Azure is trying to solve. It supports finetuning, too.
Not cheap, though. I think most companies will end up hosting an LLM on local code rather than using OpenAI on Azure, but for now, there is no equivalently-capable replacement.
Even personal projects, where I'm learning new languages and libraries, I've found that the code that gets generated in most cases is incorrect at best, and won't compile at worst. So I have to go through and double-check all of its "work" anyway- just like I'd have to do if I had a junior engineer sidekick who didn't know how to run the compiler.
I think for the work problems, if our company could train and self-host an LLM system on all of our internal code, it would be interesting to see if that could be used to assist building out new features and fixes.