Saruman was strong enough to defeat Gandalf the Grey, but upon banishment from Orthanc and his taking over of the Shire he seemed like nothing more than a normal old man. He apparently retained his powers of persuasion (somewhat) but his great fighting strength seemed to all be gone.
Why did Saruman lose all or most of his power?
I don't think the books ever went into much detail, but from this...
Book III, ch.10:
‘Behold, I am not Gandalf the Grey, whom you betrayed. I am Gandalf the White, who has returned from death. You have no colour now, and I cast you from the order and from the Council.’
He raised his hand, and spoke slowly in a clear cold voice. ‘Saruman, your staff is broken.’ There was a crack, and the staff split asunder in Saruman’s hand, and the head of it fell down at Gandalf’s feet. ‘Go!’ said Gandalf. With a cry Saruman fell back and crawled away.
..and this...
Book 6, Ch.8:
A sudden light broke on Frodo. ‘Sharkey!’ he cried. Saruman laughed.
‘So you have heard the name, have you? All my people used to call me that in Isengard, I believe. A sign of affection, possibly. But evidently you did not expect to see me here.’
‘I did not,’ said Frodo. ‘But I might have guessed. A little mischief in a mean way: Gandalf warned me that you were still capable of it.’
‘Quite capable,’ said Saruman, ‘and more than a little...
...it would seem that Gandalf removed Sarman's power. At least, Saruman's greatest capital-P Powers; Saruman still has some abilities, but nothing beyond the reach of any "ordinary" (i.e. non-Maiar) being in Middle-earth. Could Saruman have redeemed himself? Maybe,...
Book III, Ch.10:
‘He will have guessed, surely?[re. Treebeard]’ said Merry, ‘Were they likely to end any other way?’
‘Not likely,’ answered Gandalf, ‘though they came to the balance of a hair. But I had reasons for trying; some merciful and some less so. First Saruman was shown that the power of his voice was waning...Then I gave him a last choice and a fair one: to renounce both Mordor and his private schemes, and make amends by helping us in our need. He knows our need, none better. Great service he could have rendered. But he has chosen to withhold it, and keep the power of Orthanc...He lives now in terror of the shadow of Mordor, and yet he still dreams of riding the storm. Unhappy fool! He will be devoured, if the power of the East stretches out its arms to Isengard.
...but Saruman was given a last choice, and he still chose "the dark side".
Did Gandalf remove Saruman's power himself, or was it a Higher Power (the Valar or Ilúvatar themself)? I think it depends on how you want to look at it. An executioner acts on the order of the Monarch/Judge/whatever, with the "Power" of the throne/gang/society/legal system behind them, but it's still the executioner that pulls the lever/chops the head/pushes the syringe. If Saruman had shown any redeeming qualities, Gandalf could have chosen to give him a second chance. But Saruman refused the offer, so Gandalf chose to break him. In the end, it's always the choice of the person with their finger on the trigger. Which is a very Roman Catholic point-of-view, but as Tolkien was a Catholic I think that's how it was meant to be read.
@joe-l answer is pretty much correct, I just want to add that Saruman didn't lose his powers as a result of being banished from the tower (being in the tower or in control of the tower didn't give him any power). His staff being broken didn't deprave him of his powers either, it was only a symbolic act.
Gandalf indeed is the one who took his powers away, but how was he able to do that? Or how did he have the authority to do so? Well, after he died while fighting the Balrog, he had a "meeting" with Eru and Eru "turned" Gandalf into what Saruman should have been: The White.
TLDR: Saruman lost his powers because Gandalf removed them with the authority granted by Eru.
Gandalf received his power to break Saruman's staff from the same authority that brought him back from the dead, and turned him from Grey to White.
Maiar/Valar have shown their capacity for their 'spirit' to be diminished when they defy the One (Morgoth being stuck in a physical body, Sauron's inability to take one).
Throughout his writing, Tolkien had a very keen sense of "moral rightness" (consistent with a WWI veteran). Gandalf was clearly the judge, under Illuvatar... and JRR is Illuvatar.
The banishment from Orthanc was symbolic of his ruination, but it was Gandalf himself who removed Saruman's power by revoking his Eru-given authority on The Council of Five.
The banishment from Orthanc was not directly connected to his loss of power, although you could argue that his previous power was being augmented by magical artifacts he had within the tower. Rather, the power loss was connected to his removal from the role as The White Wizard.
Each of the wizards was granted a role within Middle-earth, with Saruman being the leader and most powerful. The wizards - collectively called The Council of Five - are granted their power and purpose from Eru himself, and are on Middle-earth for the express purpose of helping mankind to prevent the rise of Sauron and his forces. By colluding with Sauron instead, Saruman has betrayed that purpose.
Gandalf himself states this right before breaking the staff:
Behold, I am not Gandalf the Grey, whom you betrayed. I am Gandalf the White, who has returned from death. You have no colour now, and I cast you from the order and from the Council.
The Lord of the Rings, book III; The Two Towers, chapter 10: "The Voice of Saruman"
This is exactly what happens - brought back to life by Eru himself, Gandalf has been chosen to replace Saruman as the leader and most powerful wizard, which means that he now has the authority and power to banish Saruman from the council. The key phrase in understanding why Saruman loses most of his power is the final line:
You have no colour now, and I cast you from the order and from the Council.
He then breaks Saruman's staff, which is a symbol of his power and station. He also revokes his ownership of Orthanc, which was a gift to Saruman from Gondor - also a symbol of his station.
So, essentially, at that point Saruman is no more powerful than any other human or elvish sorcerer would be - having no innate power and limited to his knowledge of alchemy & such.
Lots of great answers, although very speculative, and not very canon. Although Gandalf the White was brought back with more power than Gandalf the Grey, he was not given any more power than his natural abilities, and Olorin (Gandalf) the Maiar spirit in Valinor never had the power to take away the power of any Maiar, especially not a powerful spirit such as Curumo (Saruman). The native power and memory of the wizards were diminished by the Valar when they were sent to Middle Earth. Much of Gandalf's native powers were returned to him as Gandalf the White by Eru Illuvatar, but no more than his native power.
How did Saruman lose his powers? Well, no need to speculate. Gandalf provides that answer:
"But I rode to the foot of Orthanc, and came to the stair of Saruman and there he met me and led me up to his high chamber. He wore a ring on his finger."
And later, Saruman states:
"For I am Saruman the Wise, Saruman Ring-maker, Saruman of Many Colours!"
In brief, Saruman spent many years studying the lore of the Eregion Elven smiths and the teachings of Sauron in order to create a ring of power. The creator must put much of his innate power into the ring to create it. Such did Saruman. Unfortunately for Saruman, the ring was created using the teachings of Sauron, and like the other rings of power, its power was destroyed with the destruction of the One Ring. Most of Saruman's power was lost forever. Saruman speaks to this in his exchange with Gandalf and Galadriel during their chance encounter on the road after the destruction of Sauron and the One Ring.
In the end, it was the actions of Saruman that led to his loss of power. Gandalf simply broke his staff as a symbolic gesture of being removed from the Counsel.
