Before considering how to disengage from a boarding action, I think you have to consider the difficulties and risks of bringing about a boarding action in the first place.
To evade a boarding attempt by manoeuvre was relatively easy. It was in practice very difficult to get near enough to a ship to attempt to board her unless it was also her intention to settle the matter hand to hand. A ship could be forced into a boarding action but this required some measure of heroics.
Fighting at Sea in the Eighteenth Century, S. Willis (Boydell, 2008) pg. 148
If you've disabled the enemy ship's ability to manoeuvre, so that it cannot escape a boarding action, it can actually be a better option not to board. You just position your own ship across the bows or stern of the target and use your guns to rake them, until they surrender.
Another possibility for creating a boarding opportunity (and the one in which a disengagement is more likely to be desired) is in the midst of battle when the ships have inadvertently come together and become entangled.
Just having two ships close together was a risk to both vessels (especially when one or both is likely to have battle damage to both hull and rigging). If the ships come together too quickly, the collision can cause severe damage to the hull and rigging, cause crew to fall (possibly from a height) and can dismount the guns. A dismounted gun is both a offensive loss and a risk the crew around it (great guns could weight over a ton).
A successful boarding action was only likely in a calm sea. If the ships are rolling in a strong swell, the risk of collision damage increases as does the difficulty of getting men from one deck to the next (which can be difficult enough in a flat calm).
The captain would also be careful to engage on the lee side of his enemy if at all possible, as the sea would be smoother where it was sheltered by the ship he was to board, and the lee position made it much more easy to haul off if the attempt failed.
Fighting at Sea in the Eighteenth Century, S. Willis (Boydell, 2008) pg. 149
And if the attacker was in the lee position, it made disengagement by the defending vessel that much more difficult because the wind would press it into the attacker.
The classic 'Hollywood' boarding action where the ships come alongside each other tends to miss some of the practical problems. For much of the Age of Sail, the ships had a 'tumblehome' hull design. This meant that while the ships were touching at the waterline, the upper decks, where the boarding parties were, could be several feet apart (too far to comfortably jump onto a hostile, moving deck). A fall between the ships would almost certainly be fatal.
An alternative option was to run your bowsprit to snare the other vessel.
Most commonly, however, the bowsprit of the aggressive ship would be run on to the enemy. This had the distinct advantage that it could then be lashed in some way to the enemy ship to prevent the two from drifting apart. Ideally the bowsprit might deliberately be run into the enemy's main rigging in the hope that it would become entangled and hold fast without the inherent danger of having to lash it in place. Once the two ships were secured by the bowsprit, it could itself be used as a makeshift bridge for men to cross from one ship to another.
Fighting at Sea in the Eighteenth Century, S. Willis (Boydell, 2008) pg. 150
So, if you were being boarded (and you want to get away) what options do you have? Certainly, getting your own crew organised and into a good defensive position gave you the best chance.
In any scenario, however, boarding was almost impossible in the face of a well prepared and steady enemy who could form defensive positions and sweep any boarding party with musketry and grape shot, and repel them with pikes. Moreover, once a boarding party had made it onto an enemy ship, it was also difficult for them to get back in retreat, help defend their own ship from boarding, or to receive reinforcements. Indeed, there appear to be as many examples of ships successfully defending themselves from being boarded as there are of being taken by boarding.
Fighting at Sea in the Eighteenth Century, S. Willis (Boydell, 2008) pg. 150-151
So a strong defence, followed by a swift counter attack could leave the original attackers in a disorganised and demoralised 'heap' on their ship, giving the defenders enough time to disengage their own ship.
Bluffing could also work. Fire was (and is) a great risk to all ships. So an option for forcing a disengagement was to make enough smoke to convince the other ship that a fire was raging out of control. For the aggressor, there was little to be gained by capturing a burning vessel and there was, potentially, the risk of losing their own ship in the process. Therefore, it was potentially possible to trick an enemy into disengaging by faking a fire but it would only work where the attacker couldn't just move off a little and watch.
[I've assumed for this answer that we're not considering 'cutting out' actions where ships boats were used to board moored vessels in river estuaries and harbours.]
