The building, built in the mid-eighteenth century, initially housed the military arsenal.

Designed by the duke Francesco III d'Este to house the General House of the Poors, was renovated by Pietro Termanini in 1769. In 1788 it was turned into the House of the Crafts and into the building were set up works and factories. After the restoration, the building housed a military hospital, a prison and accommodation of officers; in 1828 received the Lapidary Museum set up by the duke Francesco IV of Austria-Este.

Italian unification marked a new stage in the history of the building, that was partially purchased by the Public Administration in 1881, in order to reunite the art and bibliographic collections left by the Este to the town, and the civic and cultural institutions.

On the ground floor of the Palace Museum, in the northern courtyard, there is a reception for all the cultural institutions hosted into the building, that is:
- Historical Archives
- Estense Library 
- Municipal Museum of Art
- Municipal Museum of Archaeology and Ethnology
- Este Gallery
- Public Library of Art "Luigi Poletti".