We're currently in the process of rewriting the wallet for much greater scalability in terms of historical transaction volume, but not yet in a way specifically designed to handle a large number of addresses. The first alpha with that should be out in a month or so.
At that point the main scalability problem will be the number of UTXOs in the wallet, and it will start slowing down when there are tens of thousands. This is a possible scenario if you have many thousands of watch-only addresses - it depends on the way in which you are building transactions.
So my recommendation would be to wait for this new version, test how it performs in your usage scenario, and then decide how to proceed. None of the actual APIs will be affected so you can start working with the current version.
