There is one more caveat that I wanted to briefly mention. As you can imagine, internet RTTs vary greatly. If someone's connected to you via a wireless connection, and turns on their microwave oven, that could lead to some jitter that throws the rtt values off.
To resolve this, we use certain low-pass filters and smoothing functions to make sure that outliers don't completely kill the values you're getting. However, in order for these to be effective, we need a number of round-trip samples to converge and make that determination. Thus, the longer a host has been talking to us, the better this estimation will be. It needn't be in the same connection (we remember the rtts we calculated for hosts we spoke to in the recent past even if they're not connected to us), but in case you see jitter right after the 3-way handshake completes for the first time, this is usually due to our not having a large enough sample set.