I have this idea for an alternate universe where guns were... flat out rejected. Basically, while the powers that were recognized that guns were more powerful and had greater range, they were considered to be too loud and inaccurate for use in war.
How would a society which rejected guns improve on the crossbow? Could they ever reach AR capabilities?
Pump guns or air rifles might become popular. There are guns in assault-rifle format which shoot bolts (or flechettes) with compressed air which are used for special operations missions. Additionally, air rifles, that is guns which fire balls or bullets using the pressure of an air reservoir have been around since the American Civil war. The advantage of these is that they are comparatively quiet, easy to operate and use, and almost as dangerous as gunpowder based weaponry.
An "Airbow", shoots bolts:
Self-loading crossbows go back to antiquity, predating the invention of gunpowder, in China. Modern developments along these lines already approach AR capacity with light crossbow power and accuracy. YouTuber Jorge Sprave has even built units powered by a battery drill motor that can continue to repeat as long as there are arrows in the magazine and power in the drill's battery.
The combination of total draw energy needed for long range and ability to source that energy from other than a deflagration mean you'll never have a crossbow with both the range and rate of fire of our AR class weapons -- but tactics will adapt if your powered crossbows only have a range of, say, three hundred meters, instead of a thousand, and a rate of fire of two or three bolts per second compared to fifteen bullets in the same time.
I pity the soldiers who carry these, though; the batteries will be killin' heavy.
Perhaps the main issue wouldn't be the rejection of firearms, but rather the adoption of crossbows.
Between the fall of the Roman Empire in the West and the adoption of firearms in the 1400's, military power depended on highly skilled men at arms (called Knights in the West, but also exemplified by Samurai in Japan and the Janissaries in the Ottoman Empire). These men required a lifetime of training, starting as young boys, and the time spent in training, and the expense of equipping these men tended to limit their numbers. The English/Welsh longbow was similar in may ways, only by starting training in boyhood could archers handle the 100-120 lb draw weight of full sized bows and draw and shoot fast enough to create "arrow storms".
Crossbows, and complimentary weapons like pikes and pole arms along with proper tactics allowed men with relatively little training to go into battle against Knights, Samurai and Janissaries with the expectation of winning. Polities which adopted "Infantry Revolution" weapons and tactics in the 1400's could field much larger armies than their aristocratic Knightly (etc.) counterparts, and more importantly, could make up losses far more quickly. A townsman killed in battle could be replaced in a matter of weeks by drilling a new man, a dead knight could only be replaced when a child had finished training and achieved manhood, and by furnishing them with expensive equipment.
Assuming the banning of firearms is a cultural issue, as the OP seems to suggest, then what will happen will likely replicate the situation in Japan once the Tokugawa Shogunate established itself. During the wars which raged up to that point, the Japanese were enthusiastic users of firearms and "Infantry Revolution" troops (Ashigaru) armed with pikes, arquebuses and other "modern" tools of warfare. Once the Japanese islands were consolidated, the Shogunate immediately banned the importation and manufacture of firearms, and collected firearms, swords, spears and other weapons from the population to prevent the prospect of revolts and preserve the social rank of the Samurai.
In such a situation, crossbows would actually be a terrible danger to the "Knightly" class and existing social structure. Anyone who made crossbows could issue them out to peasants, cooks and townsmen and have the ability to rapidly raise an army capable of putting lots of effective "fire" on the Knightly attackers, or use the firepower to sweep the walls of fortifications and overwhelm defenders without similar arms. (The term firepower is actually inaccurate, since it is a conceit of using firearms, but I'm not clear of the analogous term for arrows).
So if firearms are going to be banned, then crossbows will also be banned (and military geniuses who experimented with pike or pole-arm tactics will also be looked at with great suspicion). Long range attacks will be done with longbows, since the training for effective use of 120 lb draw longbows also excludes a large segment of the population from effectively participating in warfare (even in England, special laws were passed to ensure people were constantly practicing archery).
In social terms, the banning of firearms will also lead to the banning of "Infantry Revolution" weapons and a great stagnation in battlefield tactics in order to preserve the aristocratic privilege of the Knightly class. So long as the territory is insular enough to remain isolated from outside threats (like the Japanese Home Islands), the situation may endure, but once outside threats enter the picture, the polity will either need to adapt rapidly to the introduction of new technologies and tactics, or die.
If explosive technology exists, then this is a much more complicated question than just "what would crossbows look like". I'm going to preface by saying guns just never happening in a world with explosive technology is super unrealistic, but if somehow the entire human population just turns an eternal blind eye the to idea of guns, I would expect a timeline that looks something like this:
16-17th century: Instead of falling out of use, bows, crossbows, and platemail continue to take center stage in many conflicts. Without guns to content with we might see some more modern inventions come earlier like compound bows or alloyed steel, or a revival of repeating crossbows or gastraphetes so that crossbow could better contend with archers rate of fire. Medieval castles would still have fallen out of use at about the same time as they did in our history since trebuchets could just hurrell massive impact explosives to similar effect as artillery bombardment. Hand grenades would take on a more prominent role since it becomes easier to get into throwing range when guns are not a concern.
18th century: As metallurgy continues to develop and good spring steel becomes more common, crossbows will be able to have much longer draw lengths and therefore fire much heavier projectiles than bows. They can now fire impact grenades over significant distances making them the preferred battlefield weapon. If air-rifles are not considered "guns", they might make a bigger debut than they did in our timeline, but ultimately they are still too far ahead of their time to be mainstreamed at this point. Rapid fire crossbows are not a prerogative at this point. Since the army that can fire first pretty much wins the battle, it will be all about range. Tight battle formations become obsolete ahead of schedule since you are just inviting the enemy to kill more of your men per shot. Melee weapons are also beginning to fall out of use as ranged kill capacity goes up. Platemail would begin to fall out favor as stronger crossbows would have similar penetrating ability to firearms. Gambeson or other textile armors may still be pretty mainstream for their effectiveness against shrapnel.
19th century: Introduction of shape charged explosives means crossbows and air-rifles will now have the ability to penetrate any armor. If platemail was not already obsolete by this point, it certainly is now. Long-draw crossbows firing massive salvos of explosives remain the dominant battle-field technology, but air-guns may mainstream as personal defense/hunting/side-arm weapons.
20th century: The invention of the RPG would have a much more profound role on the course of history. Rockets can out range and fire more rapidly than crossbows making crossbows obsolete. Instead of just being anti-armor/anti-fortification weapons, you would see a lot of smaller RPGs as well, many only being in the 10-20mm range allowing them to take on combat roles similar to guns. WWI era airplanes would be nearly impossible to shoot down, but by WWII time-fused rockets similar to flak shells would be employed en masse. Air-to-Air weapons would all be mostly ineffective until the advent of guides missiles. Rapid fire grenade launchers like the MK19 would be pretty commonplace by the end of the century.
21th century: By the current era, the need for more but smaller faster firing RPGs will be much more easily met with computerized manufacturing and advanced alloys. Gun like rocket launchers will fire bursts of small projectiles almost indistinguishable from modern firearms. Cops will have 9mm, pistol shaped rocket launchers. Assault weapons will fire mini rockets from banana clips, so on and so forth. Warfare will look almost the same as it does in our world, it would just work off of different scientific principles.
Conclusion: The crossbow would continue to develop after it did in our timeline but not in the direction of the AR-15. Instead of becoming faster, it would become stronger. This is because the crossbow does not lend itself to rapid fire as well as the bow or air gun even when you include things like self-loading bolt clips. However, at the end of their historical effective lives when spring-steel arms were most popular, they typically only had an 8" draw length because impurities in the steel too often made for weak points that could cause them to critically fail and harm the user if you draw them farther than that. This meant that even though the windlass crossbow could have a draw-weight of over 1000 lb, they could not deliver much more force than a 150 lb warbow. However, as metallurgy approached industrial era standards, impurities became more minimal meaning you could have made crossbows with much longer draws. These crossbows could pack enough force to go through platemail whereas historical crossbows could typically only kill by hitting between the plates and penetrating the chainmail underneath or going through the eye slits. Rockets would still eventually supplant them though because rocket technology can be made synonymous with bullet technology once your manufacturing methods become good enough.
The issue with crossbows is the loading and powering mechanism. Assault rifles are self loading and the ammo is self powered. With crossbows this is not so easy. There are things like repeating crossbows, but they are still hand powered and not very powerful.
Perhaps electricity could be a factor to power the crossbow into resetting the string, or maybe compressed air. But it is unlikely to reach the power and speed of assault rifles without using anything explosive.
Accuracy isn't your problem, as crossbows can be very accurate and have a lot of stopping power. Perhaps with large (powered) compound contraptions you could even get close to sniper-like weapons.
I imagine a type of pedaled crossbow contraption could be possible at low-tech, but this would be a crew weapon at best.
Using a compound crossbow and a pump-action forward grip, you can produce a good fire rate with decent velocity on the bolts. Your warriors will have to work out, but you might get 160 foot-pounds (270 joules), fired once every few seconds, which is nothing to sneeze at.
I'm with Keith Morrison that crossbows are not that accurate. William Tell and Robin Hood (for bows) are Medieval tall tales.
Unless...
You laser-guide the bolts.
DARPA's already got prototype laser-guided .50-calbier bullets, and laser-guiding a crossbow bolt would be comparatively simple; it's larger, it deploys more gently, and it already has fins.
At that point, you could have an effective range of a mile (1.6 km) against unaware targets or targets with no cover or concealment (This requires a ~280mi/h [450 km/h] which is high, but achievable. The bolt would retain it's velocity well during the 13 seconds it travels through the air.); Against an aware target, you'd have to consider how long it takes the bolt to reach them (again, up to 13 seconds) versus how long before they can reach something that will stop the bolt.
My first thought is: why reject guns. My answer is: war should be about human energy, not chemical energy. This would then also reject any explosives, and anything powered by electricity, water, fire, etc...
Having said that, one can imagine a spring in a metal box, that is stretched in the box, and then latched stretched. This can become the stored energy for a repeating crossbow. At this point, the speed limit on your crossbow becomes how quickly you can load a spring and projectile (possibly as one unit).
I find it interesting that answers talk about the different ranges of rifles vs crossbows. I believe that the energy involved in each is actually comparable. The difference is that the crossbow shoots a much more massive projectile. So the secret to a long-range crossbow is imparting the same energy into a tiny projectile. This may means some form of mechanical advantage - a system of levers that makes your tiny pellet go very fast. An air compression system (akin to airguns others mentioned) might also work.
I think for a crossbow to lead to a weapon as capable an modern assault rifle that they would need to solve the problem of getting the bolt or quarrel to start its stabilizing spin before it leaves the bow string.
As others have very rightly observed, with electrical mechanisms they could create very fast firing repeating weapons. The next challenge would be increase the projectiles velocity
But, given the mass difference between a quarrel and bullet, they’ll very challenged by the weapon’s recoil if they increase the bolts velocity to approach that of an assault rifle.
Maybe, if, their society developed their technological and scientific expertise in such a fashion that they developed mastery of electrical and magnetic forces before mastering the chemistry necessary for explosives and rocket propellants then, maybe, they’d conceivably see gauss rifles as the natural follow on to cross bows.
The evolution of these weapons might involve replacing bow component with coiled spring, then powerful electric motor, then to coil guns — which we have yet to realize as anything between silly toys or fodder for sci-fi stories.
