The league had rejected the 17-year contract, saying the longest deal in league history violated its salary cap.
The NHL Players Association filed a grievance against the league. A hearing was held last week and arbiter Richard Bloch sided with the league when he issued his ruling.
"We want to thank arbitrator Bloch for his prompt resolution of a complex issue," NHL deputy commissioner Bill Daly said in a statement. "His ruling is consistent with the league's view of the manner in which the collective bargaining agreement should deal with contracts that circumvent the salary cap."
The decision put Kovalchuk back on the market as an unrestricted free agent, and the high-scoring Russian already was talking to the Devils again.
"While we do not currently have a contract with Ilya Kovalchuk, discussions have resumed and we are hopeful that a contract will be reached that meets with the principles in arbitrator Bloch's award and the NHL's approval," New Jersey president and general manager Lou Lamoriello said.
Kovalchuk was hockey's biggest prize in free agency this year with 338 goals and 304 assists in 642 career games. He thought he had a lucrative deal with the Devils, but now he has to work out a new one.
Lamoriello noted Bloch's ruling indicated neither the team nor Kovalchuk operated in bad faith and both parties believed the contract was compliant with the collective bargaining agreement.
NHL spokesman Frank Brown said it was too early to say whether the league would take punitive action against the Devils. New Jersey can be fined or lose draft picks for signing Kovalchuk to a contract that circumvented league rules.
The players' association said it was disappointed with the ruling, which it was reviewing. It had no further comment.
Kovalchuk and the Devils agreed to the deal July 19. The next day, the league determined the contract was illegal because years of low salary at the end lowered the cap hit.
The union filed a grievance July 26.