Francis (Frank) Bannerman VI was born in Dundee, Scotland on March 24, 1851. At the age of three he and his family moved to the United States, where they settled in Brooklyn. Soon after, Bannerman's father began a business of re-selling items purchased at Naval auctions. When his father joined the Union Army during the Civil War, Francis carried on and gradually grew the business by collecting large quantities of scrap. Once the Civil War ended, Bannerman officially founded his own company, Francis Bannerman Co., at the age of 14. It was during a business trip to Ireland a few years later, in 1872, that Bannerman met and married his wife, Helen Boyce.

In 1900 Bannerman purchased Polopel Island, a small island located in the Hudson River just north of West Point, NY, with the aim to turn it into a safe storage location for his large store of military surplus that had now outgrown his warehouse space in Brooklyn. Patriotic to his heritage, Bannerman designed plans to have a unique, Scottish-style estate constructed on the island to serve both his business and personal needs. Construction began in 1900 and continued steadily through the time of Bannerman's death in 1918. Buildings constructed included an arsenal, a superintendent's house, a workshop, a powder house, an ice house, breakwaters, turrets and a grand personal residence and gardens for the Bannermans. Proud of his island home, Francis attempted to officially rename the island "Bannerman Island," but never gained consent to do so. During their time living on the island, a number of unfortunate incidents befell the Bannermans, including a large explosion at the powder house in 1920 that caused minor injuries and a fair amount of damage, and frequent lightning strikes due to the castle's numerous flag poles.

In 1918 two of Francis Bannerman's sons, Frank Jr. and David Boyce, became partners in the business and changed its name to Francis Bannerman Sons. When Francis passed away later that year, his two sons carried on and continued to grow the business. In 1946, following the death of Frank Jr., Francis' grandson Charles S. Bannerman became a partner and incorporated the business. In 1959 Charles, then President, closed the company's storefront that had long been on Broadway and moved the business office to Blue Point, Long Island. The family sold the company to James F. Hogan, previously the company's secretary, in the late 1960s.

In 1967 the Bannermans sold Polopel Island to The People of the State of New York, and on July 1, 1968 it was placed under the supervision of the Taconic State Park Commission. Unfortunately, a fire during the night of August 8, 1969 severely damaged all of the buildings. Today the island and the ruins remaining of the buildings are maintained by a non-profit agency associated with the New York State Office of Parks, Recreation and Historic Preservation.