The case of the 74-year-old, who died while serving an eight-year prison sentence for indecent assault, was scrutinised by the Court of Appeal last month.
The court was told the PR guru denied “until his death” that he had sexually assaulted young women.
But, in a ruling on Tuesday, Lady Justice Rafferty described the case against Clifford as “strong”.
Dismissing an appeal against his conviction, she said: “Nothing we heard came anywhere near imperilling the safety of his conviction.”
Before suffering heart failure in December 2017, Clifford won the right for his fight to overturn his conviction to be heard at the Court of Appeal.
His daughter Louise continued the challenge on her late father’s behalf.
Clifford, from Hersham in Surrey, was jailed in May 2014 after being convicted of a string of indecent assaults carried out between 1977 and 1984.
He branded his accusers “fantasists” and denied the charges, but was convicted at London’s Southwark Crown Court.
He was sentenced to a total of eight years on eight counts of indecent assault relating to four victims.
Sentencing Clifford after his trial, Judge Anthony Leonard said his personality and position in the public eye were the reasons his crimes were not revealed earlier.
The former celebrity agent vowed to clear his name, claiming he was wrongly convicted.
An application by Crown lawyers for their £6,408 costs of the appeal, made against Louise Clifford, will be dealt with at a later date.
Kim Harrison, of law firm Slater and Gordon, who represented five victims in civil cases against Clifford, said: “Hopefully, this loss will put an end to efforts to clear Clifford’s name and finally give his victims the chance to put these terrible events behind them.
“These proceedings had yet again forced them to confront the abuse he subjected them to and caused them significant distress.
“Clifford always insisted his accusers were ‘fantasists’ in a cynical attempt to bully and deter them from pursuing justice.
“The truth is Clifford was a predatory sex abuser – that much has been shown time and time again.”