Shadow attorney-general Andrew Wallace has described growing speculation over a challenge to Opposition Leader Sussan Ley’s leadership this week by right-wing aspirant Angus Taylor as “hypothetical”, “rumour” and “innuendo”.

“At the moment, there is no challenge, and I’m getting on with my job as the shadow attorney-general to prosecute our political agenda and to ensure that this government does what it should be doing, and that is keeping Australians safe,” Wallace told journalists at Parliament House.

“Sussan Ley is our elected leader. She is the first woman leader of the Liberal Party. And were she to be deposed, I can almost guarantee you there’ll be people out there saying, ‘Oh, you know, you guys have just politically assassinated the first female leader of the Liberal Party’,” Wallace said.

Wallace said Ley had done “a good job in very difficult circumstances”, but did not criticise senator Jane Hume for this morning saying the party was being led astray.

“We have a very strong tradition in our party room, and that is that people are free to voice any concerns or issues that they have … We do believe in the ability to be able to have your say. We do believe in the ability to be able to even cross the floor if you’re a backbencher. So I would encourage my colleagues to speak freely,” Wallace said.

Speaking earlier to Sky News, Wallace said he backed Ley’s leadership and that she needed clear air to lead the party.