Born in London in 1947, Lytton studied drums privately from the age of 16 and played dance music before his introduction to jazz. He performed with many of the London jazz establishment from 1966 to 1969 while also taking tabla lessons from P.R. Desai. Around 1969, Lytton began his initial forays into freely improvised music, forming a duo with the soon-to-be legendary multi-reedist Evan Parker, who played not only saxophones but also various homemade instruments including the "lyttonophone." Bassist Barry Guy eventually linked up with Parker and Lytton, and the Evan Parker Trio was born. Lytton was a founding member of the London Musicians' Cooperative and collaborated with various other artists on the London improvisers' scene from 1970 until 1975. He also developed a strong relationship with Aachen-based percussionist Paul Lovens, and in 1976 helped found the Aachen Musicians' Cooperative.

Lytton has since performed in Europe, the United States, Canada, and Japan as a soloist and in various combinations with other improvising musicians. He continues to work with Parker and Guy, and is a member of Evan Parker's Electro-Acoustic Ensemble, the London Jazz Composers' Orchestra, and the King Übü Orchestra. Lytton appears on over 40 recordings on various labels including Island, Incus, Po Torch, FMP, Intakt, ECM, Leo, CIMP, and Rastacan. In 1999, Lytton toured in the United States with Chicago saxophonist/clarinetist Ken Vandermark and bassist Kent Kessler; he appears with Vandermark on the duo CD English Suites, released that year on Wobbly Rail. During the 1990s, Lytton also performed on two Evan Parker Trio CDs featuring guest pianist Marilyn Crispell: Natives and Aliens in 1996, and After Appleby in 1999 (both on the Leo label).