A Jewish American, Davis was from a working class Jewish family and grew up in Brooklyn, New York. He was named Clive by his mother, who was a great fan of a British actor with that name. Davis graduated from New York University`s College of Arts and Sciences in 1953 and received a scholarship to Harvard Law School. Davis graduated and practiced law in a small firm which folded, then moved on to the firm of Rosenman, Colin, Kaye, Petschek and Freund, which had CBS Records as a client. Davis was then hired by the legal department of CBS subsidiary Columbia Records.
He discovered a passion for music which led him up the ranks of Columbia and CBS. In 1967, he became president of CBS Records; at almost the same time, he became a convert to the newest generation of folk rock and rock and roll. That year he attended the Monterey Pop Festival and was inspired by what he saw as the future of music. He immediately signed Janis Joplin and Big Brother & the Holding Company, and Columbia went on to sign Electric Flag, Santana, Bruce Springsteen, Chicago, Billy Joel, Blood, Sweat & Tears, and Pink Floyd. The company, which had previously avoided rock music, doubled its market share in three years. In 1972, Davis also signed the group Earth, Wind & Fire to CBS Records.
Amid government investigations of financial irregularities in the record industry, CBS, nervous about losing its licenses, fired Davis amid charges that he had used company money for personal expenses — allegations he still denies.
In 1974, he founded Arista Records, naming it after his secondary school honor society. This label has one of the most diverse lineups in the record industry. It has been home to Barry Manilow, Dionne Warwick, Whitney Houston, Monica, Exposé, Angie Aparo,Sarah McLachlan, Annie Lennox, Kenny G, Puffy Combs, Aretha Franklin, Toni Braxton, Air Supply, Ace of Base, The Alpha Band, The Grateful Dead, TLC, Willie Nile, and Patti Smith, among others. In the 1970s, Arista also had an extensive jazz line, most notably its Freedom imprint, concentrating on contemporary, sometimes avant-garde, musicians and widely praised reissues from the legendary Savoy label.
Davis was parodied in part by Patrick Macnee, in the 1984 movie This is Spinal Tap. On January 28, 1997, Davis received a star on the Hollywood Walk of Fame.
In 2000, Davis ran into disagreements with new Arista owners BMG (a member of the Bertelsmann group) over the way he ran the label. Reports point to his age (late sixties at the time), lack of a succession plan, and free-spending habits as points of contention. Once released from Arista, he founded yet another independent record label, J Records. Bertelsmann even invested $150,000,000 (USD) in the company. In building his new company, some of his first signings were recording artists Luther Vandross and Alicia Keys. That same year, Davis was inducted into the Rock and Roll Hall of Fame in the non-performers branch.
Three years later, at the beginning of 2003, BMG wooed Davis back into the fold, naming him head of RCA Records. The following year in 2004, Davis was promoted to Chairman and CEO of BMG North America by Rolf Schmidt-Holtz.
In 2003, Davis donated money to the Tisch School of the Arts at New York University (NYU) to create the Clive Davis Department of Recorded Music. The undergraduate program, the first of its kind in the country, recognizes the creative record producer as an artist in his or her own right and musical recording itself as a creative medium.