In 1980, inspired by his father-in-law, sculptor Arnold Geissbuhler, potter/sculptor Harry Holl, in alliance with his friend artist/lawyer Roy Freed, proposed the creation of an art museum which would both display and preserve the works of generations of artists associated with Cape Cod, and the Islands. They assembled a small group of artists, educators and community activists and the Scargo Lake Museum was incorporated on January 28, 1981.

In 1982, the name was changed to Cape Museum of Fine Arts and a Charter Membership Drive was begun. By 1985 there were nearly 1000 members. A rented storefront in Theater Marketplace on Route 6A in Dennis Village provided the first home for the fledgling museum. To facilitate growth in a permanent location, in 1984, the CMFA Trustees signed an agreement with the Raymond Moore Foundation to lease an acre of land on the grounds of the Cape Playhouse to “erect a fine arts museum for the Cape.” In 1986 the Davenport West family donated a building to permanently house a growing collection of fine art. A $300,000 campaign entitled Museum on the Move was established to fund the construction of a foundation and to move the building from South Harwich to the grounds of the Cape Playhouse. When the building was moved to Dennis in September of 1987, it was divided into 11 pieces and required seven trucks to make the journey across the Cape, traveling Route 6, to Old Bass River Road and Route 6A to reach the Playhouse grounds.

From 1987-1990, the museum continued to hold exhibits and events in its Theater Marketplace location while the Davenport West building was placed on its new foundation and reconstructed to house the Cape Museum of Fine Arts. While several of the planned galleries and amenities remained unfinished, the new facility finally opened in 1990. Giant steps forward were taken in the mid-1990s when the Trustees, under the inspired leadership of President Joe Signore, successfully launched a three-phase capital campaign. This led to the CMFA becoming a year-round, state-of-the-art facility with the completion of a major renovation and addition in the summer of 2001, which included seven climate-controlled galleries, an auditorium, sculpture garden, museum shop, administrative offices, conference room, library, an art storage area for the collection, and an elevator serving all floors. The third stage of the capital campaign, the construction of the Weny Education Center, was completed in the summer of 2003. Attached to the museum on the north side of the building, the Weny Education Center has a separate entrance. Its large, open studio space provides room for classes and workshops for children and adults as well as space for special exhibitions.