Background: Owing to the policy of extramuralization of care in most Western countries older people with severe mental illness have to rely more and more on informal caregivers for daily care. Caregivers themselves are often aged, and although caregiving implies an impact on daily life that exceeds the boundaries of usual informal care, the impact differs across caregivers. Some caregivers seem to suffer more than others, and the differences cannot be fully understood by factors currently known to exacerbate the burden of caregiving. In order to help caregivers reduce the impact of caregiving it is important to gain a deeper understanding of factors influencing the burden and its impact on the caregiver’s life. With this in mind, the aim of the study is to explore and understand differences in the impact of long-term caregiving on the quality of life of caregivers who look after older adults with severe mental illness. Methods: A qualitative, associative, inductive strategy and continuous simultaneous coding were used to interpret the data of 19 semi-structured interviews. Results: We identified an underlying psychological factor “perceived freedom of choice” which explains the gross differences in impact, leading to a definition of two main types of caregivers. Depending on how people perceive freedom of choice to provide care, the consequences of caregiving can be characterized as a process of gain (type 1) or loss (type 2). Four influential factors deepen the impact of caregiving for the type 2 caregivers, and two subtypes are identified for this category. Consequences of caregiving are most readily seen in a deteriorating quality of the relationship with the care recipient and in the psychosocial well-being of the caregiver. Conclusions: The concept of freedom of choice adds to our understanding of the differences and explains the variation in impact on the caregivers’ life. The type 1 caregiver generally experiences gain whereas type 2 generally experiences loss, which puts the latter group typically at risk of becoming overloaded. Whether people perceive that they have freedom of choice in caregiving is an important consideration in evaluating the type of intervention needed to support caregivers.