Authorities say two other men were pulled from the water Thursday afternoon and survived.
The bodies of the drowned men were also recovered.
Capt. Bruce Norris of the Ventura County sheriff's department says the victims are 17, 19 and 21.
He says a 17-year-old boy and 27-year-old man survived.
It wasn't immediately clear if the men were related.
The wave struck just before 2 p.m. at Mugu Rock, a rocky outcrop that has appeared in the background of many TV shows, movies and car commercials.