Caves are formed by the dissolution of limestone. Rainwater picks up carbon dioxide from the air and as it percolates through the soil, which turns into a weak acid. This slowly dissolves out the limestone along the joints, bedding planes and fractures, some of which become enlarged enough to form caves.
