From what I understand, the Istari were sent in to help the peoples of Middle-earth get through the third age. Why then, did Gandalf help the Dwarves along (and get Bilbo involved) in their quest to rob Smaug?
It doesn't seem to have any use, and I don't remember him having shown any sign of precognition that support a theory relating to knowledge of the finding of the ring or the death of Smaug.
First, keep in mind that when The Hobbit was published, it was essentially a standalone story. Tolkien had the idea of hobbits independently of his already well-developed but unpublished work on what is now known as the First Age. In modern terms, we'd say that The Lord of the Rings is a giant retcon. The now-common text of The Hobbit is itself a revised edition of the 1937 original, made to fit the sequel better.
Appendix A of The Lord of the Rings includes the story Durin's Folk. This story is the canonical answer to your question. It briefly tells the tale of the Dwarves' time in Erebor before and after the coming of Smaug, the longing for a reconquest of Khazad-dûm (Moria), and the fate of Thráin (Thorin's father). It also gives an account of Gandalf's meeting with Thorin. Gandalf wanted Smaug destroyed in preparation for a full-fledged war against Sauron, in order for the Dwarves in the Iron Hills not to have to fight on two fronts.
Among many cares [Gandalf] was troubled in mind by the perilous state of the North; because he knew then already that Sauron was plotting war, and intended, as soon as he felt strong enough, to attack Rivendell. But to resist any attempt from the East to regain the lands of Angmar and the northern passes in the mountains there were now only the Dwarves of the Iron Hills. And beyond them lay the desolation of the Dragon. The Dragon Sauron might use with terrible effect. How then could the end of Smaug be achieved?
Tolkien started to write a more detailed story of the background to The Hobbit, but he never finished it. It is published in the Unfinished Tales under the title The Quest of Erebor.
After Gandalf's meeting with Thorin, he was unconvinced by the latter's plan to fight Smaug directly. He thought the Dwarves didn't have a chance, and would sacrifice their lives in a foolish venture. As Gandalf happened to visit the Shire next, he thought of sending a hobbit along, to contribute stealth to the party.
Suddenly in my mind these three things came together: the great Dragon with his lust, and his keen hearing and scent; the sturdy heavy-booted Dwarves with their old burning grudge; and the quick, soft-footed Hobbit, sick at heart (I guessed) for a sight of the wide world.
Gandalf further realizes that he has obtained from the dying Thráin in Sauron's jail a way to enter Erebor unnoticed (the map and key). It is not clear from that version if Gandalf has an actual premonition that Bilbo's participation is necessary, or if this is merely a realistic evaluation of the party's chances and an argument to convince Thorin to take Bilbo on. In Gandalf's words:
I knew in my heart that Bilbo must go with him, or the whole quest would be a failure. (…) ‘Listen to me, Thorin Oakenshield !’ I said. ‘If this hobbit goes with you, you will succeed. If not, you will fail. A foresight is on me, and I am warning you.’
The history has to do with one of the seven Rings of Power given to Dwarves.
Quoting from Wiki:
As Thráin grew older, he was driven by the malice of his ring and the desire for gold gnawed at his heart. He left [his son] Thorin for the Wilderlands, but was captured and imprisoned in Dol Guldur in southern Mirkwood. There, the last of the Dwarven Rings of Power was taken from him and he was left in the dungeon to die.
Gandalf the Grey found Thráin while on a mission to Dol Guldur to discover the identity of "The Necromancer", a mysterious Dark Lord who was later revealed to be Sauron.
Moments away from death, Thráin gave him the old map. Gandalf promised to deliver it to Thráin's son, but Thráin, who was delirious with pain, had forgotten his name, so Gandalf did not expect to be able to fulfil this promise. Thráin died soon afterwards.
Later, Gandalf met with Thorin near Bree, and realized that he was the son of the captured dwarf in Dol Guldur. He promised to help Thorin with the Quest of Erebor, and invited a hobbit named Bilbo Baggins along to take the role of burglar.
Parts of this were covered in Appendix A and B of LOTR
Tolkien wrote a short story "The Quest of Erebor", which was later published in Unfinished Tales in 1980. It consists almost entirely of Gandalf telling Frodo (after King Elessar's coronation in Minas Tirith) how he got involved with the Thorin et al.:
Gandalf knew that Smaug the Dragon could pose a serious threat if used by Sauron, then dwelling in Dol Guldur in Mirkwood. He was thinking about the matter when he met Thorin Oakenshield at Bree. Thorin also was concerned about Smaug, but had the different motive of revenge and the reclaiming of the Dwarves' treasure in the Lonely Mountain. Gandalf agreed to help Thorin.
Gandalf thought Bilbo, an unlikely choice, to be a suitable companion to Thorin and his Dwarves for a number of reasons. First, he had observed that Bilbo took more of an interest in the world at large than was usual for Hobbits, and was thus more likely to be adventurous. Another reason was that Smaug would not recognize the scent of a Hobbit, advantageous to a stealthy operation and likely to distract the dragon's attention. Thorin also did not think highly of Hobbits, and putting Bilbo in the expedition might prevent the proud Thorin from rash actions—such as openly challenging Smaug.
Note that it's kind of quasi-canonical, since it was published after Tolkien's death. But it seems to fit very nicely, and I wouldn't be surprised if it gets worked into the upcoming movies.
One simple in-universe explanation is that Gandalf needs to maintain goodwill with all the races and has to strengthen them (in ways he deems good) whenever possible.
So when Thorin proposes his expedition to the Lonely Mountain, Gandalf achieves both goals simultaneously by helping them- earning goodwill and strengthening the dwarves.
I agree with the accepted answer that Gandalf was afraid of the damage that Smaug might do in the coming war, and helped the Dwaves in the hope of getting rid of Smaug. However, his concern was not just for the Dwarves of the Iron Mountains. He was afraid that the whole of the north would be at risk.
‘I grieved at the fall of Thorin,’ said Gandalf; ‘and now we hear that Dáin has fallen, fighting in Dale again, even while we fought here. I should call that a heavy loss, if it was not a wonder rather that in his great age he could still wield his axe as mightily as they say that he did, standing over the body of King Brand before the Gate of Erebor until the darkness fell.
‘Yet things might have gone far otherwise and far worse. When you think of the great Battle of the Pelennor, do not forget the battles in Dale and the valour of Durin’s Folk. Think of what might have been. Dragon-fire and savage swords in Eriador, night in Rivendell. There might be no Queen in Gondor. We might now hope to return from the victory here only to ruin and ash. But that has been averted - because I met Thorin Oakenshield one evening on the edge of spring in Bree. A chance-meeting, as we say in Middle-earth.’
The Lord of the Rings Appendix A, Section 3: Durin's Folk
Page 1077 (Single volume 50th Anniversary Edition)
I especially like the last sentence with its understated nod towards the existence of a higher power influencing events.
