The quarry on the east side of the Island was in operation beginning in 1850, when labor for the quarry was provided by state prisoners housed in the prison hulk Alban, anchored just off the island. Seventeen prisoners overpowered a guard and escaped from the hulk in 1852 – the ensuing public outcry had an influence on the decision to erect a new state prison, San Quentin, a few years later.
Stone from the Angel Island quarry operation was used for the construction of the new fortress on Alcatraz in 1854, and for the erection of the Navy base at Mare Island in 1857. Later, stone from the island was used in the California Bank building at Sansome and California Streets in San Francisco. The civilian manager of the quarry built a house on the island for the quarry supervisor, but in 1867, the civilian operation ended when General McDowell took the quarry under military control.
The Army used the quarry to produce stone that was used in construction at Fort Point, the San Francisco Presidio, and on Angel Island itself. Military prisoners from Alcatraz were used for quarry labor by the army, particularly after 1870 when the number of Army prisoners on Alcatraz increased. Men working at the quarry were brought to Angel Island by boat in the morning, and returned to Alcatraz in the evening.
The last known use of stone from this excavation was in 1922, when stone from the quarry was used in the Richmond breakwater. Early sketches of the area around Quarry Point show a substantial hill, more than one hundred feet high, on the site of the quarry. By the time the quarry operations ceased the hill was completely gone – it had been quarried away.