I was surprised to learn that Puerto Ricans, despite living in a US territory, were not entitled to vote in the presidential elections.
I was even more surprised to learn that US citizens are allowed to vote for president from anywhere in the world - EXCEPT if they happen to live in Puerto Rico.
What is the legal/political rationale behind this? What is it about Puerto Rico that magically removes one's right to vote? Has anyone ever challenged this?
This is a peculiarity as a result of the federal nature of the USA and the exceptional position of Puerto Rico as a territory but not a state. Within the States and Territories of the USA, your voting rights depend on residence. If you leave the States and Territories your voting rights depend on former residence or inheritance.
In general most citizens of the USA are also citizens of a state of the USA. Since states don't issue their own passports, your citizenship of a state is determined by residence. If a New Yorker moves to Florida, they become Floridians, and so can vote in state elections in Florida, but can't now vote in elections in New York.
Now if our New Yorker moves to Timbuktu, this is treated slightly differently. They remain a US citizen and a citizen of New York, and so retain the right to vote in New York elections. Their children could also claim New York citizenship.
But Puerto Rico is both part of the USA but not a State or part of a State of the USA. When our New Yorker moves to Puerto Rico, they can vote in Puerto Rican elections but not elections in New York.
Now the nature of elections in the US is that there are no national elections There are statewide elections of Senators, Governors and Presidential electors. There are district elections of Representatives and there are local elections of many kinds. Our New Yorker has lost the right to vote in New York elections (just as they would have done if they had moved to Florida) but not gained the right to vote in Puerto Rican elections for Presidential electors (because there are none).
This is odd, but something similar is true if the New Yorker moves to DC. They cease to have the right to vote in elections to the Senate.
The US doesn't have a presidential election. Get that idea out of your head. Instead, all 51 states* have their own presidential elections on the same day.
This selects a set number of electors for each state. Those electors get together weeks later, and they decide who is President.
To be eligible to vote, you have to be a US citizen resident in one of those states. Most (all?) states have mechanisms for allowing residents who have to be out of town during the election to vote (eg: in Timbuktoo or even in Puerto Rico). However, you have to have a state to send that ballot to.
Puerto Rico, like other US territories such as Guam and American Samoa, is not a state. Therefore, they (and people resident there) are left out of that system. However, as full-blown US Citizens, they are allowed to move to any US state of their choosing and vote there.
This is fundamental to the design of the US Constitution, and can only be changed through the Amendment process. However, how a state picks its electors is up to each state. A state could, out of the goodness of its heart, "adopt" a territory and let its residents vote in that state's election. However, unless something like the National Popular Vote Interstate Compact is adopted, that seems quite unlikely, as it would dilute the power of the state's own voters for that state's set number of electors. However, if the compact goes into effect, there'd be no real downside to doing that, so we could then see US citizens resident in territories being allowed to vote for President.
* - Technically there are only 50 states, but residents of the stateless federal district of DC are allowed to vote there as well, via a special constitutional amendment that was passed in 1960.
"What is the legal reason why an American visiting Timbuktu can vote for president but an American resident of San Juan cannot?"
An American who is a resident of San Juan, and who is visiting Timbuktu, can not vote for president.
The US has a concept of "states" and a concept of "territories". There are various differences, particularly in relation to taxes and voting.
The answer to the question "what is the 'legal reason' that residents of territories do not vote for president?" is simply "because it is the law", a historical issue.
The issue of the exact details of what determines "residency", in the US, in various territories/states, is a detailed issue often seen discussed in (say) a tax discussion.
The issue that Americans can (or can't) vote while visiting overseas locations such as Timbuktu is completely fatuous and irrelevant.
Regarding the curiosity "which state (/territory, /city, /county etc) does a US citizen vote as if in, when, overseas for many years", this is the common issue and it's simply (what else could it be?) your previous address.
Residents of Puerto Rico are not entitled to vote because Puerto Rico is part of the United States commonwealth. A territory part of the US Common wealth is "An organized United States insular area, which has established with the Federal Government, a more highly developed relationship, usually embodied in a written mutual agreement", according to the Department of Interior. Moreover, an "insular area" is "A jurisdiction that is neither a part of one of the several States nor a Federal district" (from that same DOI link).
Hence, because it isn't a state nor a federal district, it does not have any voting rights in Congress, nor does it have electoral college votes. The exception to this rule is the District of Colombia which was awarded electoral college votes via the 23rd Amendment to the US Constitution, in 1961.
US citizens who live abroad are entitled an absentee ballot as stated on the website of the Department of State. More specifically, "U.S. citizens living outside of the U.S. register and vote in the state and county where they last established residence (domicile) in the U.S. before moving outside of the country", cf. here. As such, a US Citizen living outside of the US whose previous residence was in a US State or in a Federal District is entitled to vote when they live abroad.
Because Americans in Timbuktu pay federal income tax, unlike those in Puerto Rico, and there's "no taxation without representation." D.C. residents get to vote for president for the same reason.
