James Caldwell was born in Newbury County, South Carolina in 1951, the son of teenage sharecroppers. During his early years Caldwell was raised by an aunt in a rural setting while his parents tried to carve out a new life for the family in New York. While attending high school in Park Slope, Brooklyn, he worked as a grocery bagger as well as a bike messenger. After spending three years in Germany while in the Army, Caldwell began working as a car salesman - first in Texas and then back in Brooklyn, on Bensonhurst's 86th Street. After developing a reputation for being active in his home neighborhood of Crown Heights, Brooklyn, he was elected president of the 77th Precinct Community Council. A consistent presence in local politics, he also served as president of the non-profit Brooklyn United for Innovative Local Development (BUILD), which played a pivotal role in the construction of the Barclay's Center by endorsing builder Bruce Ratner's controversial development of Atlantic Yards.