"The decision was made in view of the allegations and speculations raised in relation to the recording, which can cast doubt on the integrity and credibility of the country's judicial system," Najib told a news conference.
The video has touched a nerve in the judiciary, whose reputation has been under question since the late 1980s, when the head of the Supreme Court was removed from office after a clash between then premier Mahathir Mohamad and the judiciary.
Mahathir also introduced constitutional changes in 1988 that limited judicial power and, critics say, effectively ensured that government decisions could be protected from legal challenge.
Najib said a three-person, independent panel would be formed to investigate the video, but two opposition parties labelled the move as inadequate and echoed the legal fraternity's call last week for a royal commission of inquiry.
"There should be a full-scale royal commission of inquiry with a credible membership and go into all aspects of the tape," said Lim Kit Siang, leader of the Democratic Action Party.
"We cannot expect a panel appointed by the cabinet do a proper, free and independent investigation," said Wan Azizah Ismail, head of Parti Keadilan Rakyat.
Unlike other types of inquiry, a royal commission has special powers to subpoena evidence and witnesses. The commission, empowered by a 1950 law, reports directly to the King instead of the prime minister.
Malaysia's Bar Council, which says it represents all of the country's roughly 12,000 lawyers, said it would proceed with a planned protest march on Wednesday in the administrative capital of Putrajaya to press its call for a royal commission of inquiry.
"It is hoped that the remit of this panel will extend beyond merely probing the authenticity of the video, although it is recognised that this is an essential first step," Bar President Ambiga Sreenevasan said in a statement.
Malaysia's de facto opposition leader, Anwar Ibrahim, released the video last Wednesday and said it showed a senior lawyer speaking by phone with a judge in 2002.
The video, which he handed to media and said would also be sent to anti-corruption investigators, shows the lawyer talking on a mobile phone. It does not show who was on the other end of the line, nor does the lawyer clearly identify the other party.
Anwar said he had received the video recently from a source he declined to identify.
Reuters tried to contact the lawyer immediately after the video's release for comment and left a message on his mobile phone, but the call has still not been returned.
Malaysia's de facto justice minister has been quoted in the local press as saying that the judge named by Anwar as being the other party in the conversation has denied any involvement.
Najib said the new panel would be chaired by former Chief Justice of Malaya Haidar Mohamad Noor and also include the head of Malaysia's military national-service regime, Lee Lam Thye, and a former Court of Appeal judge, Mahadev Shankar.
"It will be a full inquiry in the sense we will launch the necessary investigation, analyse evidence and make the necessary conclusion. They have to analyse whether the recording is authentic, and everything relating to the recording," he said.
He said the panel's findings would be submited to government within weeks, well before year-end, and then be made public.
"They will do whatever is necessary," he said when asked if the panel would question the lawyer in the video and the judge alleged to have taken part in the conversation.
"They will get the full support of various government agencies. I will be meeting this panel in the next few days."
Malaysia's Human Rights Society backed calls for a royal commission of inquiry on Tuesday, saying the video, if true, undermined the entire judicial system.
(Additional reporting by Clarence Fernandez and Mark Bendeich)