Most Prime Ministers are removed directly or indirectly by General elections and referendums. In the post war Conservative party only two Prime Ministers have been removed by the party despite being big election winners, Margaret Thatcher and Boris Johnson
Edward Heath was a one term PM. He used a lot of political capital forcing the U.K. into the EEC against the wishes of a substantial portion of his party. He imposed a ludicrous range of controls on the economy to try to control inflation and picked a fight with the miners he was unlikely to win. Electors despatched him.
Mrs Thatcher won three great victories in a row. The experience of government and the evolving EU power grab changed her from Europhile to Eurosceptic. She was brought down by MPs in her own party led by Europhiles.
John Major won an election in 1992 and ploughed on with the deeply damaging European Exchange Rate mechanism policy which duly wrecked the economy. The party foolishly kept him and he went down to a predictable defeat. So comprehensively had he trashed the Conservative reputation for economic competence that he suffered the biggest post war defeat of any Conservative leader in 1997. It was only when Labour did worse by the economy 10 years later that the Conservatives had a chance to win again in 2010.
David Cameron won a good victory in 2015. Misunderstanding the importance of the EU referendum to the win, he backed the wrong side in the popular vote he had granted on the EU and lost. He accepted his defeat and resigned.
Mrs May fought and lost an election in 2017 and then persisted in backing a Brexit sell out to the EU. When she stubbornly insisted on her poor deal for a third time and lost the Commons vote she had to go.
Boris Johnson won a great victory in 2019 by backing Brexit fully.He was then brought down by a range of different MP groups for a range of different reasons.
The common thread seems to be the Eurosceptic PMs were more successful in elections with the public but more vulnerable to party dissent.The more popular PMs faced a much more vitriolic barrage of criticism from Opposition parties, the BBC and the rest of the Establishment. Two of the Europhile PMs, Theresa May and David Cameron were brought down directly by the EU issue, and John Major indirectly as it was his support for the European economic policy that did the damage. The party ratings fell when we were evicted from the Exchange Rate Mechanism and could see the damage, and never rose again in the following 4 years.