This is it exactly. It opens up the door to even more complexity in the design...

Now that you can unselect floors, if the elevator is on its way to the 5th floor and you unselect it, should it skip that floor mid-movement?

If there's only one floor selected, we'll probably want to disallow unselecting it while the elevator is moving, but what if you just hopped on and it hasn't moved yet?

This would probably make for an interesting interview question.

Minor to you... Introducing a great experience is the job of good UX. Not simply limiting options and saying 'done'. I am sure there are better options to handle this. Seems like no-one has put the effort into it (yet).

Unnecessarily stopping at a floor for a few seconds is minor to literally everyone riding in a public elevator.

But yeah, I'm sure that at all of the various elevator companies, over all of the years that elevators have had electronic controls, nobody has bothered to put any thought into this.

Limiting options is exactly the point. Elevators are supposed to be easy to use for the entire population, including the very young, the very old, those with disabilities, those not able to read or write the dominant language, etc.

In these cases, it is simply the best UX to keep everything really simple.