I think some of the contention comes in the statement "first make it work, then make it work well and then make it pretty or fast (depending on what the project is)" and the belief that without good design in the first place it doesn't really work. Leaving design till the end means it won't really have a positive impact on the use of the software.
While my statement probably doesn't covey this very well, I certainly am not an advocate of amalgamating dirty hacks or throwing any thoughts about good design out the window in the name of speed. You are quite right that leaving design until the very end isn't incredibly efficient or effective.
I'm also not a big believer in the "big design effort up front" methodologies. I tend to think of design (of code, at least) as more of an iterative process. I can't think of a single project I've worked on where a big design effort up front hasn't gone through a huge change before project completion.
Why spend all that time on a huge, detailed design that will end up being partially thrown out in the end when you can make iterative changes and design some things as you go?
Granted, this doesn't work in all situations and you do need good developers who are willing and able to write good, testable code and throw it away at a moment's notice.
If I code something in the "first make it work" phase that isn't throwaway code or can't be completely ripped out and rewritten without too much risk/effort, then I have failed as a developer.
Another mantra I try to code by is: never let perfect be the enemy of good. Perfection is never done and the best design in the world means nothing if it never gets finished.
As a designer, I feel that the design process is meant to highlight better routes to take with any artefact. It develops it further whilst keeping the goals more focused as they are nearer.
