Mr Coulton told the Daily Liberal he predicts the move could happen within three weeks.

"If the party loses confidence in the prime minister they will ask Rudd to take the leadership," he said.

Although former prime minister Rudd adamantly dismissed leadership ambitions, Mr Coulton thought otherwise.

"He said he wasn't seeking it which leaves the door open a bit, as he won't challenge but will be asked to lead," he said.

Mr Coulton said the Prime Minister would not easily give up the leadership battle.

"Julia Gillard is a very tough fighter and she won't walk away from it," he said.

Mr Coulton thought it would make no difference who led the party to the election. He believed people would see through the change of leaders and Labor policies would remain the same.

The recent disastrous Nielsen poll gave a nine point lead to opposition leader Tony Abbott as preferred prime minister over Julia Gillard who fell five points.

"The poll was not a complete surprise," Mr Coulton said.

"People have lost confidence in the government because they have a clear lack of direction and are fed-up."

It would be "childish and foolish" to jump for joy and believe the opposition would automatically win, he said.

He suggested anything could happen between now and the September election but the federal opposition had a "fairly good chance" of forming government.

Mr Coulton said Greens leader Christine Milne tried to distance her party from Labor as it ended the formal alliance which propped the federal government

."When push comes to shove, the Greens support every legislation Labor brought," he said.