Coleman was born in Austin, Texas to Mary Wharton Johns and Melvin Randolph Coleman. He entered the Virginia Military Institute in 1949, then studied law at the University of Texas before turning to acting.

Though a capable character actor with a wide range, and more than 60 movies to his credit, Coleman is usually typecast as a smarmy, selfish, patronizing, self-absorbed bigot, usually an authority figure of some sort, most famously as a powerful, chauvinistic boss. His fate in these types of roles was cemented with his performances in roles such as Franklin Hart, Jr. in 1980's Nine to Five, director Ron Carlisle in 1982's Tootsie and the earnest John McKittrick in 1983's WarGames.

He is also known for his lead roles in the TV cult classics, Buffalo Bill and The Slap Maxwell Story, which asked audiences to embrace his own charisma and comic timing (and good writing), as compensation for his characters' lack of character...and abundance of personality.

Coleman voice-acted in the Disney series Recess as Principal Prickly.

Coleman has been married twice. He was married to Ann Courtney Harrell from 1957 to 1959. He had three children with actress Jean Hale, whom he was married to from 1961 to 1983. He has four children: Meghan, Kelly, Randy, and Quincy. His daughter, Quincy, is a recording artist whose first album, "Also Known as Mary", was released in September 2003. His son, Randy, fronted the alternative/rock band Zoo Story. A gifted vocalist and songwriter, Randy has also recorded a solo album.