I mentioned in my first look at Spelljammer that ship combat appears to largely be missing from the Astral Adventurer's Guide. I've fixed that with some custom rules, adapted from my well-worn Starfinder ship combat ruleset.
I didn't have high hopes for ship combat rules, honestly, because I've played Starfinder and I've read Of Ships and Seas from Ghosts of Saltmarsh. Ship combat in a tabletop RPG is hard. Actually, it's not just hard, it goes against everything a tabletop RPG is about. People play an RPG to roleplay an individual character. Putting lots of characters into a boat removes autonomy. Instead of roleplaying one character, the player now has to coordinate with other characters in an attempt to keep an imaginary machine running. It can work (as it does in Starfinder) but at best it's boring for some and at worst frustrating for all.
But I've played a lot of Starfinder, and over the years I've developed rules for simplified starship combat. I've used these rules in my games, and I've never gotten complaints about ship combat. The same can't be said of the rules as written in the Core Rulebook, which invariably cause players to consider leaving the game at the threat of every having to do ship combat every again.
Spelljammer's Astral Adevnturer's Guide strongly implies that ship combat is not something you should do:
Personally, I'd have written into Spelljammer that due to whatever fantasy space law, ranged attacks receive a boost within Wildspace. Maybe it's the air envelopes, or the magical influence of a Spelljammer helm, or just the unpredictability of Wildspace, but somehow ranged attacks use line-of-sight instead of feet. I'm sure that rule could be abused, so I guess you could constrain it with something: Ranged attacks are line-of-sight unless an object's air envelope is greater than 180 feet, or whatever.
In Starfinder, that's how I use magic in ship combat. If you see it on a ship screen, then you can cast magic on it as if it were line-of-sight. So far, it hasn't betrayed me and has kept magic users engaged during battle. I haven't had a chance to playtest this rule in Spelljammer, though, so it's just an idea for now.
The good news is that I don't think ship combat in Spelljammer has to be all that complex. Certainly I wouldn't go anywhere near the Of Ships and Sea rules, with its three-person command structure. I believe Astral Adventurer's Guide was pretty close, actually. Here's what it it lacks:
And here's how I'm working to hack around that.
At the start of combat, roll sides initiative, as described in Astral Adventurer's Guide. The winner may go first, or choose to delay in order to force the enemy to make the first move.
During a round, each player gets their usual allotment of actions. The enemy enemy ship has actions equal to 2 × the number of crew members listed in its stat block.
Only the Spelljammer (the creature attuned to the helm) can move a ship, and attacks may not be taken during movement. You can double-move, and attack during movement, if the Spelljammer can succeed at a Manoeuver check.
A manoeuver check is the Spelljammer's Wisdom with a +10 bonus (the helm adds to the Spelljammer's magical navigational skills). The DC is equal to the ship's beam. A ship's beam is listed in its stat block, and is essentially the width of the vessel. I'm imagining that the width of the vessel represents the mavoeuverability of it.
I know nothing about boats. I have no idea if this is physically accurate, but I needed a reliably attainable number as a DC. The beam is less variable than the keel, while retaining enough of a connection to the physical object to make it believable that it would influence navigation.
During combat, players can take their usual player actions (30 feet of movement around the ship, attacks, aid, dash, and so on.) Alternately, they can spend their actions to contribute to loading and aiming and firing a cannon. My assumption is that 1 player is at the helm, leaving 3 players available for gunnery. That's about 6 actions to spend on getting a shot off during a round. No more dead rounds spent loading and aiming and firing.
Ship distance matters. If you're 2 squares away from your target, you roll your attack with advantage. If you're 3 to 6 squares away, you roll normally. If you're 7 or more, you roll with disadvantage.
This encourages ships to actually move rather than just parking and firing cannons at one another.
The Spelljammer (the one at the helm) can move the ship, and also make a Manoeuver check to double move. Rules as written essentially make chases impossible because the fastest ship will always win. A chance to double move (or take damage on a fail) makes chases possible, because sometimes the slow ship can leap ahead.
The 5e system loves simplicity, and that's what we've gotten for ship combat in Spelljammer. Maybe it's too simple for some, but I think ship combat is inherently problematic due to the way it forces a loss of player individuality. With a slight rules adjustment, you can at least ensure that ranged attackers have something to do, but some characters are going to sit back and help load a cannon...again. That's not fun for anybody.
In my experience, the best ship combat is quick combat. Exchange a few volleys, and then have a ship either run away or bear down and board. In my mind, I use a soft rule that there are no more rounds in ship-to-ship combat than there are players. If you've got a table of 4 player characters, then ship combat stops, one way or another, after 4 rounds.
You can pick up my ship combat rules as a PDF from dmsguild.com. I hope it helps!