In general I would say that just because it needs to be migrated, is not a reason to downvote.
One of the ways to approach the question of Should I downvote, is to look at the text from the downvote button tooltip:
This question does not show any research effort; it is unclear or not useful.
![enter image description here](https://cdn.statically.io/img/i.sstatic.net/0LEux.png)
So, a question that is merely off-topic does not fit into this description. This also means that the other reason you stated: off topic that should be deleted is likely also not be a reason for downvoting. If a question is merely off-topic, the proper response, as Sean said in his comment, is to flag/vote to close.
If the question is good, but should be someplace else, then you can flag the moderator to migrate.
One important note: if you are going to flag for migration, try to be sure that the question will be on-topic at the suggested destination. A poorly written question will off-topic everywhere.
All sites in the network have a page at /help/on-topic to try and describe what is expected at that site. Something like:
https://{sitename}.stackexchange.com/help/on-topic
So to summarize:
- A well written, interesting, on-topic question should be upvoted.
- A question that is poorly written should be downvoted.
- A question that is off-topic should be flagged/voted for closure.
- If the off-topic question would be good on another site, then flag for migration.