With a grant to study at the Boston Museum School of Fine Arts, he became a disc jockey on WBCN-FM, hosting a show called The All-Night House Party. The show touched on many musical bases, and reflected Wolf's own broad musical interests. While still in college in Boston, Wolf joined his first musical group, comprised of fellow art students. They played blues music, and later got to meet and tour with their heroes like John Lee Hooker and Muddy Waters. In 1967, he formed the group that would go on to become the J. Geils Band, which began playing clubs around New England. Their first big break was a chance to play at the Fillmore East in New York City, and they became known for their marathon live performances, with Wolf establishing a reputation as a particularly dynamic frontman. The group was signed to Atlantic Records by producer/impresario Jerry Wexler, and toured constantly over the next couple of years, performing as special guests with the Rolling Stones throughout the U.S. and Europe. At one of these concerts, Wolf met actress Faye Dunaway, whom he later married for a short time.

In 1983, the group was at the height of its popularity, and had gone 17 years without a personnel change. Finally, the bandmembers went their separate ways and Wolf went on to produce numerous film soundtracks and run art exhibits of his original paintings. In 1984, he released his first solo album, Lights Out, followed in 1987 by Come as You Are, which spurred the hit single of the same name. In between albums he worked on duets with Mick Jagger and Aretha Franklin, who recruited him specifically for her Who's Zoomin' Who album. In 1989, after a six-month songwriting retreat in Nashville, he recorded his third solo album, Up to No Good, which appeared the following year. In 1994, Wolf assembled a group of musicians and began playing clubs as a way to test out newer material on live audiences. It was the live feeling he so successfully captured on Long Line, his 1996 Reprise release. While on tour in 1997, Wolf met producer Kenny White, and together the pair recorded Fool's Parade, which was released in 1998 on Mercury. It was selected by Rolling Stone magazine as one of the "50 most influential records of the decade." In 1998, he toured with the Royal Soul Review, a star-studded gathering of soul artists including Lloyd Price, Chuck Jackson, Sam Moore, Ben E. King, Irma Thomas, Percy Sledge, Gene Chandler, and Jerry Butler. Soon after, Wolf was asked to record with blues great Little Milton. They worked at the famed Muscle Shoals Sound Studios and are featured on the Malaco album Welcome to Little Milton.

In 1999, J. Geils reunited for an end-of-the-century tour. Wolf reentered the studio with White and cut Sleepless for the Artemis label in 2002. This set, with guests including Mick Jagger, Keith Richards, and Steve Earle, was almost universally regarded as Wolf's finest solo moment and garnered -- again from Rolling Stone -- the honor of being "one of the greatest 500 albums of all time." In 2005, the J. Geils Band reunited once more for a special charity event to benefit the Cam Neely Cancer Foundation, Denis Leary's Firefighters Association of New England, and Michael J. Fox's Parkinson's Disease Research Foundation. The band later reunited in 2009 for a short series of sold-out shows across the U.S.; Wolf claims they were more than likely their very last. He finally re-emerged from his own recording silence in 2010 with Midnight Souvenirs on Verve, co-produced with White; the pair enlisted duet help from Shelby Lynne, Neko Case, and one of Wolf's true heroes, Merle Haggard.