Smart Manufacturing Systems (SMS) are envisioned to contain highly-automated and IT-driven production systems. To address the complexity that arises in such systems, a standard and holistic model for describing the activities and their inter-relationships is needed. We analyse cases illustrating problems that occur when the design of a new factory or the improvement of an existing factory is performed without a holistic consideration of inter-related activities. Based on the analysis, we derive requirements to develop a Factory Design and Improvement (FDI) activity model. In essence, FDI is a reference activity model that encompasses a range of manufacturing system activities for designing and improving a factory during its initial development and also its operational phases. The FDI model shows not only the dependency between activities and manufacturing control levels but also the pieces of information and software functions each activity relies on. We envision that the availability of these pieces of information in digital form to integrate across the software functions will increase the agility of factory design and improvement projects. Therefore, our future work lies on developing standards for exchanging such information.