Anthony Albanese has labelled Pauline Hanson nasty and spiteful over resurfaced comments from the One Nation leader criticising his mother for living in public housing for decades.

Hanson also faced further a political threat from Liberal MP Andrew Hastie – who vowed to go “all the way” in his fight with her – as he suggested One Nation MPs Barnaby Joyce and David Farley were likely feeling buyers’ remorse for joining the party.

On Sunday, Albanese said he was “stunned” by Hanson’s comments, made in a February episode of Karl Stefanovic’s podcast, in which the One Nation leader criticised the length of time his mother lived in public housing. Hanson had also labelled the prime minister “pathetic” for repeatedly referencing his housing commission upbringing.

Albanese said he was “proud” of his “battler” mother, who died in 2002.

“I tell you what’s weakness: nastiness and spitefulness, and I am stunned at these comments,” he told reporters.

“But I’ll make this point: if Pauline Hanson wants to criticise me, then that’s fine. But my mother was way out of Pauline Hanson’s league. Way out.”

The trade minister, Don Farrell, also lambasted Hanson over the “appalling” comments and called on the One Nation leader apologise.

“There’s a certain bitterness in One Nation’s approach to the world that I find quite un-Australian, actually. I think Ms Hanson should apologise,” Farrell told News24.

Hanson has been approached for comment.

In the podcast, Hanson referred to Albanese’s comments about his mother living in public housing “for 65 years” and said she believed it was “not on”.

• Sign up for Guardian Australia’s Politics, really newsletter here

“Housing commission houses should not be there for people to live in them for 30, 40 [years] – and he said his mother lived in the same one she was born in for 65 years,” Hanson had said. “Sorry, that is not on, to me.”

“It should be limited time, you have been given a hand up, move it on to the next ones coming through.”

Hastie vows to go ‘all the way’

It came as Hastie again accused Hanson of being a “lying coward”, after she continued to refuse to take down an animated cartoon that described him as a “traitor”. This came after he was subpoenaed to give evidence during Ben Roberts-Smith’s failed defamation trial.

“She’s still a lying coward. I’m not backing down,” Hastie told News24 on Sunday. “I put the challenge to her. She could have accepted that she’d got it wrong and pulled it down, but she refused to do so, and she has doubled down.

“I’ll go all the way with One Nation if they want … she started a fight, and I intend to win it.”

He also said he had previously tried to develop a relationship with Hanson, but that she is “not the person I thought she was”.

Hastie accepted Joyce’s apology to him over the cartoon video on Friday, who claimed he had not seen it. But the Liberal MP claimed his colleague, and One Nation MP David Farley, were likely having “buyers’ remorse” over joining the party.

“The whole party centres on her. I think this is why Barnaby Joyce and, I reckon, David Farley are probably having buyers’ remorse. They’re thinking, what did I get myself into?” Hastie said.

“I’ll say this about Barnaby Joyce. I feel sorry for the bloke, all right? He left a party that made him deputy prime minister twice, to jump in with this rabble, there’s no principles, they’re all over the shop, and he’s out there every week cleaning up after them.”

Joyce told Guardian Australia the assumption was “not correct” and defended his party.

“My fear is that the Coalition tread water rather than find land, and lack the strength to make difficult decisions. Politics is not a beauty pageant nor a reality tv show and we get more and more of that,” Joyce said.

“I am in One Nation because I am deeply concerned that the political status quo has become lazy and then weak.”

Farley has been approached for comment.

Hastie has been under attack from One Nation politically for several months, with the party announcing it would target his West Australian seat of Canning. The state seat of Secret Harbour, which One Nation won in a byelection last month, overlaps his federal electorate.