That's true, with the send-to-IP option, you are sending to whoever answers that IP.  Sending to a bitcoin address doesn't have that problem.

The plan is to implement an IP + bitcoin address option that would have the benefits of both.  It would still use a different address for each transaction, but the receiver would sign the one-time-use address with the given bitcoin address to prove it belongs to the intended receiver.