Unit tests are of course important, but they don't test for higher level failures like network issues, high latency, increased load, etc. Your components must be designed to be isolated from incidents as much as possible, possibly using the techniques implemented in Hystrix [1], an open source library from Netflix.
In my experience if a client adds their own X-Forwarded-For header trying to spoof their IP, nginx simply prepends it to the X-Forwarded-For header like "1.2.3.4, 33.33.33.1", where 1.2.3.4 is the address the client supplied in their spoofing attempt, and 33.33.33.1 is the actual IP address forwarded by nginx.
So you can choose to trust only the rightmost one, if there are several entries in the list.
I remember being intrigued by Google's "Sesame" experiment (covered on HN at https://news.ycombinator.com/item?id=3469692) where they logged you in via a QR code processed via your mobile.
Relying on something you have (mobile phone with a trusted app on a trusted network) instead of something you know (passwords) can be an interesting choice. Ideally you'd require both (something you know and something you have), but we want to avoid passwords.
I find these visualizations pretty cool, and I forked the original WebGL globe demo to support real time activity. Check it out at https://github.com/zsolt/globestats . It uses node.js + socket.io for the sample server.
[1] https://github.com/Netflix/Hystrix