The Association for the Benefit of Colored Orphans was founded in 1836 and was originally located on Fifth Avenue between 43rd and 44th Streets in Manhattan. In 1884, the institution was renamed the Colored Orphan Asylum and Association for the Benefit of Colored Orphans; sometime after 1944, the name was again changed, this time to the Riverdale Children's Association. The institution was also relocated to Riverdale-on-Hudson.

The Asylum was among the earliest organizations in the country to provide housing, training, and employment specifically for African-American orphans. In the late 1880s, the Asylum adopted the "cottage-home" system, in which residents of varying ages lived in small groups under the supervision of a matron. The children in each cottage performed domestic chores. The system was thought to promote a less institutional atmosphere.

During the Draft Riots of July 14, 1863, the Colored Orphan Asylum was attacked by a mob, whose size was estimated by the New York Times at several hundred, mostly women and children. At that time, the Asylum housed some 600 to 800 homeless children in a large four story building surrounded by grounds and gardens. The crowd plundered the Asylum, even looting donated baby clothes, then set fire to the first floor despite the pleas of administrators. The building burned to the ground.