John Key, the New Zealand prime minister and leader of the National party, has resigned in a decision that has taken the country by surprise.

Key held a media conference in Wellington on 5 December 2016, at 12:45PM local time, after informing the cabinet of his decision.

His resignation will be effective from 12 December 2016, when National MPs will meet to select a new leader.

Key said he would vote for his deputy prime minister, Bill English, if he put his name forward. English briefly led the party to its worst-ever electoral defeat in the 2002 election, but has since served as a successful finance minister.

Key is widely regarded as one of the most popular prime ministers in New Zealand’s history. He was first elected in 2008, and recently marked his 10-year anniversary as leader of the National party.

He said stepping down was the hardest decision he had ever had to make, but there was “no way” he could have served a full fourth term.

This felt like the right time to go, he told reporters. “Sometimes you’ve got to make hard decisions to make right decisions,” he said, adding it was an opportunity to refresh the National party’s leadership of the country and hopefully clinch a fourth term.

“I think one of the reasons governments fail at that fourth-term hurdle is leaders don’t want to leave, everyone says ‘I’ve seen this before’. This is the chance to demonstrate newness about us.”

He said he had a “pretty long discussion” about standing for a fourth term with his wife, Bronagh Key. “I don’t feel comfortable looking down the barrel of the camera and not being honest … On a family basis, I don’t think I could commit much longer than the next election.”

He denied that Bronagh had given him an ultimatum, but said his leadership had come at a cost to his family. “It’s been a decade of a lot of long, lonely nights for her and it’s the right time for me to come home.”

His children, Max and Stephie, had suffered an “extraordinary level of intrusion” along with the opportunities that came with being New Zealand’s “first family”.

In the past year, Key has had to answer to questions about his son’s burgeoning career as a DJ and social media personality. Spending more time with his family was a major factor in his decision to stand down, he said.

Key said he believed the mark of a good prime minister was one who left the country “in better shape than they found it”.

“Over time others will judge whether I’ve done that. All I can say is I gave it everything I had. I have left nothing in the tank.”

One of his regrets was not getting the Trans-Pacific Partnership “over the line”, as well as his failed bid to change the flag.

He said he considered his economic management of the country as a mark of success, even though New Zealand had weathered some crises during his government, including the Canterbury earthquake and the Pike River mining disaster. “Very few countries are in the financial position we’re in.”

But Winston Peters, the leader of the New Zealand First party, said Key had consistently misled the public about the state of the economy, and his resignation showed he was “unable to muddy the waters any more”.

Jennifer Lees-Marshment, an associate professor in politics and international relations at Auckland University, said: “Recently the normally high-performing Key brand has been undermined by denying a growing housing and traffic crisis as people in New Zealand’s biggest city, Auckland, struggle to find a place to live and to travel to work.

“The Key brand has become disconnected and he has increasingly appeared to be someone who doesn’t understand what it is like to be an ordinary New Zealander any more.”

Key said he would remain in parliament long enough to avoid a byelection, but would step down as an MP before the next election. He said he stood down “hoping and believing New Zealand had been well served by the government I led”. He plans to remain in Auckland after leaving politics, and enter the commercial sector.

English, who has Key’s backing to replace him, praised the prime minister’s “intelligence, optimism and integrity” and said he would be “judged by history as one of New Zealand’s greatest leaders”. He said the National caucus would “now consider the implications of the prime minister’s decision”, and that Key “left behind a united team with plenty of talent”.

He told reporters at a press conference that he learned from Key he was considering leaving in September. English had not yet decided whether to stand for leadership, but would not rule it out.