Butterfield was born and raised in Wilson, North Carolina, in a prominent African American family. He is the son of Addie Lourine Davis and George Kenneth Butterfield, both of whom were mixed race Americans. His father was an immigrant from Bermuda.