Ask your DM if it's possible to further modify your homebrew longbow, or learn to embrace consumables.
There aren't many permanent magic items published that add to weapon damage that aren't themselves weapons. Let's break it down:
Wondrous Items
There are a few wondrous items that grant a bonus to damage, usually paired with the same bonus to attack rolls. They are almost all restricted to melee (and sometimes also unarmed) attacks.
Bracers of Archery, which you already have, is effectively the only officially published item that applies a damage bonus to ranged weapon attacks.
Any item that increases your Dexterity score up to its maximum value, or increases the maximum value, will naturally increase attack and damage. Notable entries include the Manual of Quickness of Action (raises current and max DEX by 2) and the Ioun Stone of Agility (+2 DEX up to cap).
Armor
Surprisingly, there exists one published item, a shield, that can be used to grant a bonus to a nonmagical weapon. This does not work for your purposes, as the homebrewed bow is already magic, but it's here for completion.
- The Shield of the Uven Rune (Waterdeep: Dungeon of the Mad Mage). It can be used regularly as a shield and has additional benefits, or it can be consumed to enchant a weapon. This requires an 8-hour ritual and a nonmagical weapon, with the result being a +1 weapon that gets treated like a +3 weapon when used against a creature type specified at the time of the ritual.
Weapons
You have stated that you don't want to trade in your current, homebrew bow for a new one. That said, I think you could benefit from comparing the current features of the bow to published Very Rare and Legendary weapons, and talking to your DM about 'reforging' it to add things that might be missing. Note that weapons of this power level usually require attunement, and if your bow doesn't require this already, your DM may consider adding that property if you further increase its power level.
- Generally speaking, a basic +1/2/3 attack and damage bonus is a feature of the weapon. Is the longbow already benefitting from this? (According to your comment, it is, and will hit +3 once you hit Ranger 17. Good.)
- Does the bow have any other additional base features aside from the cast-from-hit points nova? Published Legendary weapons frequently seem to have a small (1-die) damage bonus on each hit, and then a triggerable additional ability that can be used selectively for great effect. While you have the damage nova already, do you add anything on a basic hit?
Consumables
Will this strategy be the most cost-effective? Probably not; once you use a consumable, it goes away forever even if you miss every shot. But it remains a reliable way to boost both maximum and minimum damage on a hit, and one of the few avenues you have not yet mentioned exploring.
+X ammunition. +X weapon bonuses do stack with +X ammunition bonuses, so a +3 arrow from a +3 bow ends up with a +6 attack and damage bonus overall. Are you already using magic ammunition? Pick some up, and get extras--the magic goes away once fired, even if you miss.
Arrows of Slaying. Locked to a specific type of creature, but when used against that type of creature, takes an additional 6d10 weapon damage on a failed DC 17 Constitution save, and half that on a success.
Oil of Sharpness. Requires additional prep, but one vial can coat 5 pieces of ammunition for up to an hour, treating the coated arrows as magical and giving a +3 bonus to attack and damage rolls. Also stacks with magic weapon and ammunition bonuses.
Poison. I say this with a caveat, since poison is one of the most resisted and most-likely immune damage types in 5th edition, and we don't know what sorts of things you're fighting. If you're in the lucky position to be fighting monsters that are susceptible, however, there exist some fairly oomphy poisons in the DMG--purple worm poison, in particular, deals 12d6 poison damage on a failed DC 19 Constitution save. Look for poison (injury) for poisons meant to coat weapons.
Spell scrolls, to allow for more castings of a beneficial spell, if you need that.
Not Items, But Options
Since you're interested. You already have solid choices locked in, but there are a few things you can consider.
Spells
A lot of Ranger class spells require concentration, including the ones you listed in the question as your go-to choices. There are still a few more that can help boost damage, especially when we consider the 5th-level spells you'll get at Ranger 17:
- Hail of Thorns (1st) (concentration) allows the normal weapon effects/damage to land on the target and then explodes for narrow AOE that includes the target. Though the normal weapon effects don't transfer to the explosion, upcasts nicely at a rate of +d10 per level.
- Lightning Arrow (3rd) (concentration) replaces the regular weapon damage with 4d8 lightning damage and a burst effect damaging additional targets. It's a decent choice for the spell slot level, since hunter's mark only increases in maximum duration when you upcast it.
- Swift Quiver (5th) (concentration) lets you make 2 weapon attacks with your bonus action, increasing your rate of fire, and therefore increasing benefits from effects that boost every normal weapon hit.
(I assume you're using Guardian of Nature (4th) for the Tree form's advantage on DEX-based attacks, since the Beast form's additional damage on hit is limited to melee attacks.)
There are a few spells off other class lists that would require shenanigans or multiclassing to get access to, but do boost damage off of a weapon hit:
- Banishing Smite and Branding Smite (5th, 2nd) are the only "smite" spells not specifically restricted to melee attacks only in their wording. Like lightning arrow, both are concentration spells that go off on the next successful hit. Both are on the Paladin, Hexblade Warlock, and Battlesmith Artificer spell lists, and branding smite is also available to Zariel Tieflings.
Multiclassing
I wouldn't necessarily recommend this per se, but there are a few options you can use to pile a bit more on a few key hits. Remember that by RAW there are stat prerequisites!
- Fighter 3, Battlemaster. Four maneuver dice (d8s) per short or long rest, and pick maneuvers that either increase accuracy or add to the damage roll. (Some maneuvers are limited to melee-only by RAW, so read the text carefully.) This one is the one you're mostly likely to qualify for out of the box, stat-wise.
- Rogue any, for sneak attack. Depending on your party setup and tactics, this might be worth doing if opportunities for triggering sneak attack occur regularly. Remember it only goes off once per turn, and you'll max out at +2d6 with your remaining levels.
- (Paladin 2, for Divine Smite, is counterindicated here. Aside from needing too many good stats, it's explicitly limited to melee attacks only.)
Feats
Again, like items, if you are not already at maximum Dexterity, technically speaking any feat that further increases that score will also increase attack and damage. Still, there are a few that will be a bit more effective than simply an ASI.
- Piercer (Tasha's Cauldron of Everything): +1 DEX and allows one die's reroll for piercing damage as well as an additional damage die on a critical hit with a piercing weapon. You must use the new result on the reroll.
- Lucky (PHB): grants a small pool of additional d20s for rerolls. Boosts accuracy, for resulting increased overall damage.
- Gift of the Chromatic Dragon (Fizban's Treasury of Dragons): Use a bonus action to add +d4 of a chromatic damage type to a simple or martial weapon. 1 minute duration, 1/long rest. Not the most efficient, but might be useful in a pinch.
Some damage-boosting feats are found in specific setting books. Check with your DM to see what they're willing to work with.
- Vital Sacrifice (Tal'Dorei Campaign Setting Reborn) is the Blood-Hunter/Hemocraft version of the 'class dip' feats like Martial Adept or Metamagic Adept. Take d6 necrotic damage, and for the next hour, receive a benefit off a list. 1/day. One possible benefit is +2d6 necrotic damage per weapon hit.
- Cruel (Tal'Dorei Campaign Setting Reborn) grants you a pool of d6s equal to your proficiency bonus per long rest, and once per turn you can add a die to a damage roll (among other benefits).
- Sigil and the Outlands sourcebook contains feats with "A Planescape Setting" as prerequisite. Taking Scion of the Outer Planes and then either Agent of Order or Baneful Scion grants a proficiency-bonus-uses-per-long-rest ability that adds damage on a hit and has additional benefits.
