It is said of Mandos:
he knows all things that shall be, save only those that lie still in the freedom of Ilúvatar
In the Unfinished Tales, Gandalf says:
If this hobbit goes with you, you will succeed. If not, you will fail. A foresight is on me, and I am warning you.
Does Gandalf have a foresight like Mandos does of certain events (surely he didn't foresee the Balrog for example) or is this just some intuition?
As a Maia, Gandalf was participant of the Music of the Ainur, and thus saw (most of) the history of the world unfold:
[W]hen they were come into the Void, Ilúvatar said to them: 'Behold your Music!' And he showed to them a vision, giving to them sight where before was only hearing; arid they saw a new World made visible before them, and it was globed amid the Void, and it was sustained therein, but was not of it. And as they looked and wondered this World began to unfold its history, and it seemed to them that it lived and grew.
The Silmarillion I Ainulindalë
So in his original form, he certainly would have had a measure of foreknowledge. However, it's never entirely clear how much of that he retained when he was incarnated as an Istar; it's said later in Unfinished Tales:
For it is said indeed that being embodied the Istari had needs to learn much anew by slow experience, and though they knew whence they came the memory of the Blessed Realm was to them a vision from afar off
Unfinished Tales Part 4 Chapter II: "The Istari"
Which isn't very helpful.
However, I said that I think he is predicting the future, though perhaps not with an actual vision. The reason I say this is because Gandalf qualifies his statement as a narrator earlier in the text, and remarks:
I knew in my heart that Bilbo must go with him, or the whole quest would be a failure - or, as I should say now, the far more important events by the way would not come to pass.
Unfinished Tales Part 3: "The Third Age" Chapter III: "The Quest of Erebor"
This isn't the first time Gandalf has known something in his heart; he says something very similar in Fellowship, with regards to Gollum (emphasis mine):
I have not much hope that Gollum can be cured before he dies, but there is a chance of it. And he is bound up with the fate of the Ring. My heart tells me that he has some part to play yet, for good or ill, before the end; and when that comes, the pity of Bilbo may rule the fate of many - yours not least.
Fellowship of the Ring Book 1 Chapter II: "The Shadow of the Past"
Of course we know how that turned out.
In Tolkien's writing, foresight doesn't mean always knowing what will happen in the future. It refers to occasional insight into what will happen.
The questions asks about Gandalf's statement to Thorin that he will only succeed if he takes Bilbo with him. Gandalf himself declares this to be foresight and the subsequent story shows how Bilbo was invaluable to the quest. That seems to be reasonable evidence of foresight.
The existing answers discuss how Gandalf's foresight may be enhanced by his nature as a Maia. Those answers also provide more examples of Gandalf's foresight. I don't disagree with those answers, but I want to point out that Gandalf is not the only one do display foresight in Tolkien's works.
The mothers of the Eldar
In an essay on the customs of name-giving among the Eldar in Valinor, Tolkien explicitly states that some of the mothers of the Eldar had prophetic foresight.
The second name was given later, sometimes much later but sometimes soon after the birth, by the mother; and these mother-names had great significance, for the mothers of the Eldar had insight into the characters and abilities of their children, and many also had the gift of prophetic foresight.
Unfinished Tales Part Two, Chapter IV: The History of Galadriel and Celeborn (Appendix E: The Names of Celeborn and Galadriel)
Page 255 (Houghton Mifflin Harcourt; 2012 Kindle Edition)
Fëanor
In Valinor, Fëanor foresees the loss of the light of the Trees and makes the Silmarils to preserve it.
In that time were made those things that afterwards were most renowned of all the works of the Elves. For Fëanor, being come to his full might, was filled with a new thought, or it may be that some shadow of foreknowledge came to him of the doom that drew near; and he pondered how the light of the Trees, the glory of the Blessed Realm, might be preserved imperishable. Then he began a long and secret labour, and he summoned all his lore, and his power, and his subtle skill; and at the end of all he made the Silmarils.
The Silmarillion: Quenta Silmarillion, Chapter 7: Of the Silmarils and the Unrest of the Noldor
Page 59 (Houghton Mifflin Harcourt; 2012 Kindle Edition)
Finrod
Finrod foresees his death and the destruction of his kingdom of Nargothrond.
It came to pass that Nargothrond was full-wrought (and yet Turgon still dwelt in the halls of Vinyamar), and the sons of Finarfin were gathered there to a feast; and Galadriel came from Doriath and dwelt a while in Nargothrond. Now King Finrod Felagund had no wife, and Galadriel asked him why this should be; but foresight came upon Felagund as she spoke, and he said: ‘An oath I too shall swear, and must be free to fulfil it, and go into darkness. Nor shall anything of my realm endure that a son should inherit.’
The Silmarillion: Quenta Silmarillion, Chapter 15: Of the Noldor in Beleriand
Page 124 (Houghton Mifflin Harcourt; 2012 Kindle Edition)
His foresight is correct, Barahir saves his life in battle and Finrod swears friendship and aid in every need to Barahir and all his kin. Later, Barahir's son Beren is sent by Thingol on a hopeless quest to retrieve a Silmaril. Finrod keeps his oath to Barahir and helps Beren, but dies in the process. Not long afterwards, his land of Norgothrond was destroyed.
Aragorn's maternal grandparents
When Aragorn's father, Arathorn, wants to marry Gilraen, her father and mother, Dírhael and Ivorwen, disagree on whether the marriage is advisable and both show foreknowledge of what will happen.
“Moreover,” he said, “Arathorn is a stern man of full age, and will be chieftain sooner than men looked for; yet my heart forebodes that he will be short-lived.”
But Ivorwen, his wife, who was also foresighted, answered: “The more need of haste! The days are darkening before the storm, and great things are to come. If these two wed now, hope may be born for our people; but if they delay, it will not come while this age lasts.”
The Lord of the Rings Appendix A, Section 1: The Númenórean Kings
Page 1057 (Houghton Mifflin Harcourt; Single Volume 50th Anniversary Edition)
Gilraen and all the Dúnedain
When Aragorn tells his mother of his love for Arwen, Gilraen is worried and tells him:
I do not think that you will have the good will of Elrond in this matter.”
“Then bitter will my days be, and I will walk in the wild alone,” said Aragorn.
“That will indeed be your fate,” said Gilraen; but though she had in a measure the foresight of her people, she said no more to him of her foreboding, nor did she speak to anyone of what her son had told her.
The Lord of the Rings Appendix A, Section 1: The Númenórean Kings
Page 1057 (Houghton Mifflin Harcourt; Single Volume 50th Anniversary Edition)
So we are told that not only Gilraen, but also her people, are foresighted.
Aragorn
There are several instances of Aragorn's foresight, but here is my favourite.
Before Aragorn leaves to take the Paths of the Dead, he has this conversation with Éomer.
‘Alas! Aragorn my friend!’ said Éomer. ‘I had hoped that we should ride to war together; but if you seek the Paths of the Dead, then our parting is come, and it is little likely that we shall ever meet again under the Sun.’
‘That road I will take, nonetheless,’ said Aragorn. ‘But I say to you, Éomer, that in battle we may yet meet again, though all the hosts of Mordor should stand between.’
The Lord of the Rings Book Five, Chapter 2: The Passing of the Grey Company
Page 779 (Houghton Mifflin Harcourt; Single Volume 50th Anniversary Edition)
At the Battle of the Pelennor Fields, Éomer arrives from the north to find the army of Mordor attacking Minas Tirith and Aragorn arrives by river from the south on the ships captured from the corsairs. "All the hosts of Mordor" did indeed stand between them, and when they met during the battle, Aragorn reminds Éomer of his words and Éomer realises that they were foresighted.
‘Thus we meet again, though all the hosts of Mordor lay between us,’ said Aragorn. ‘Did I not say so at the Hornburg?’
‘So you spoke,’ said Éomer, ‘but hope oft deceives, and I knew not then that you were a man foresighted.
The Lord of the Rings Book Five, Chapter 6: The Battle of the Pelennor Fields
Page 848 (Houghton Mifflin Harcourt; Single Volume 50th Anniversary Edition)
Frodo
When Gollum is leading him towards Cirith Ungol, Frodo warns Gollum not to try to take the Ring from him:
You will never get it back. In the last need, Sméagol, I should put on the Precious; and the Precious mastered you long ago. If I, wearing it, were to command you, you would obey, even if it were to leap from a precipice or to cast yourself into the fire. And such would be my command. So have a care, Sméagol!’
The Lord of the Rings Book Four, Chapter 3: The Black Gate is Closed
Page 640 (Houghton Mifflin Harcourt; Single Volume 50th Anniversary Edition)
After Gollum attacks him on the way up Mount Doom, Frodo tells Gollum:
‘Begone, and trouble me no more! If you touch me ever again, you shall be cast yourself into the Fire of Doom.’
The Lord of the Rings Book Six, Chapter 3: Mount Doom
Page 944 (Houghton Mifflin Harcourt; Single Volume 50th Anniversary Edition)
The next time Gollum touches Frodo, he bites the Ring from his finger and falls into the fire in the Cracks of Doom. Coincidences are rarely what they seem in Tolkien's writing, and I think we can consider Frodo's words to be foresighted.
Tolkien says this of the nature of Ainur, comparing the glory of King Elú Thingol at his peak to the inhabitants of the West .
In Beleriand King Thingol upon his throne was as the lords of the Maiar, whose power is at rest, whose joy is as an air that they breathe in all their days, whose thought flows in a tide untroubled from the heights to the deeps.
-Page 59,The Silmarillion
So we can expect that as one of the Ainur who had participated in the Song of Creation, Gandalf had great knowledge of the workings of the Arda and some limited foreknowledge according to his stature on Eru's grand plan for Eä and of the events that were to come to pass on Middle Earth.
However late into the third age when the Maiar who would later become the Five Istari were called upon to journey into Middle Earth to lead the resistance against the growing power of Sauron, they were required to give up most of their Maia powers and take up mortal forms to dwell among the inhabitants of Middle Earth. Undoubtedly though their wisdom was far superior to that of an ordinary mortal, their taking up mortal forms had come at the cost of a surrender of their natural Maia state of mind.
It was resolved to send out three emissaries to Middle-earth. 'Who would go? For they must be mighty, peers of Sauron, but must forgo might, and clothe themselves in flesh so as to treat on equality and win the trust of Elves and Men. But this would imperil them, dimming their wisdom and knowledge, and confusing them with fears, cares, and weariness coming from the flesh.' But two only came forward: Curumo, who was chosen by Aulë, and Alatar, who was sent by Oromë. Then Manwë asked, where was Olórin? And Olórin... asked what Manwë would have of him. Manwë replied that he wished Olórin to go as the third messenger to Middle-earth... But Olórin replied that he was too weak for such a task, and that he feared Sauron. Then Manwë said that was all the more reason why he should go...
-Page 393, Unfinished Tales
Thus we can conclude that the wisdom and foresight of Gandalf while being vastly superior to all but the greatest among the Men and Elves was still greatly diminished when he came to Middle Earth. His ability to see into the future would have correspondingly been limited by the constraints placed on his earthly mind.
