If you want to build a real community around it there needs to be give and take, you can't just throw the stuff over the wall for the hoi polloi. That might not be suitable for a small project you are content with doing yourself or you might want to set yourself up as the unquestioned god-emperor, who allows others to work for your greater glory but who will not allow any competing vision. I don't like the latter kind of project at all.
Open source is about you, or could be about you, if you find the right kind of open source community.
What open source means and what different people choose to use it for are different things. Of course as a maintainer you may get more out of it if you put effort in for building a community, which then can help you improve the project, contribute, etc. But that's extra effort, and nobody should assume that everybody commits to that extra effort just because they release source code under a FOSS license.
Open source is about you, or could be about you, if you find the right kind of open source community.