The main concern is not that it's like Skynet and wishes us harm, but that it does things harmful to us because the means to accomplish the AI's goals are at odds with human values, which was unanticipated by the human creators, since the AGI is coming up with its own solutions. And the AGI doesn't have the same values as humans, so it doesn't care if its solutions are harmful.
I guess the risk seems more real if it's a question of "does it share human values". Which is possibly not a one in a million chance it doesn't - it could be a 50/50 chance or worse.
