USSR defended them with nukes.

Korea and Vietnam are very close to China / USSR, and at the limit of US force projection range on the western pacific so if the US wanted to preserve its sphere of influence there they had to fight for it.

Cuba is well within the US sphere of influence even as a communist state because no other power can really challenge the US there - with the single exception of nuclear weapons hence why the US was willing to go to the wall on that issue since that’s the one thing the USSR could have done to make the Caribbean no longer a US province.

They will sanction and isolate Cuba because they’re ruled by psychopaths and the existence of Cuba challenges their ideological foundations, but they aren’t actually threatened by Cuba and their sphere of influence isn’t actually threatened by Cuba so it’s not worth the cost of invading.

Probably better phrased as “the same way they did Korea.”

1. Initially, the leftist credentials of the Cuban Revolution were in question. Castro visited Harlem right after winning, even. There was a real question of whether this was a government that would be friendly to U.S. business interests or not.
2. There were no airstrikes called to aid the Bay of Pigs invasion (which occurred before the USSR stationed nukes there) because the civilian leadership (Kennedy) didn’t want a full war, and didn’t want to be seen as overtly knocking off a national liberation movement. This was when the U.S. still had a pretty positive image throughout the developing world. They didn’t want another Korean War. The civilian leadership of the U.S. didn’t really want the actual Korean War; that was (like the Bay of Pigs) more driven and escalated by the budding military-industrial complex (a term coined by Eisenhower, who had to delicately manage public dissatisfaction with the war during his '52 campaign).
3. The U.S. had plenty of success with covert ops or small invasions in Latin America before, and would continue to do so throughout the Cold War, so an isolated Cuba was more of a hassle than a threat. Unlike Korea and later Vietnam, there was no strategic need to encircle China, or maintain an imperial foothold in mainland Asia.

They tried. The Soviets threatened to retaliate with nukes.

It could open a bigger can of worms for them then it is worth. Imagine if Cuban militias started raiding ships and launching missiles at oil wells in the Gulf of Mexico like the Houthis are doing now.

Could have created a huge refugee crisis off the coast of Florida

Because that would draw the ire of the public