Suppose someone damages your prized possession. How should they respond? If they are blameworthy for doing so, then they should take responsibility for what they have done. But what if they aren’t blameworthy for doing so? Or what if it isn’t clear whether they are blameworthy for doing so? Should they still take responsibility for what they’ve done? According toThe Concern Account (Mason 2019; Capes 2019; Piovarchy 2020; Sliwa, forthcoming), we have reason, and perhaps even a duty, to apologise and in so doing take responsibility for (at least some) blameless harms because we ought to demonstrate concern for others. While recent work largely focuses on the positive aspects of taking responsibility, in this paper, I investigate the darkside of taking responsibility. I first outline an account of taking responsibility. I then argue that just as there are concern-based reasons to take responsibility for blameless harms, there are concern-based reasons for not accepting our attempts to take responsibility for such harms. I then argue that taking responsibility can lead to various forms of narrative disempowerment –that is, different ways we can lose control over our life stories. I consider: (1) negative social identity categorization, (2) diminished narrative setting power, and (3) increased negative attention. Finally, I identify the underlying problem with taking responsibility for blameless harms: it makes us vulnerable to others such that they can establish our public self– that is, socially shared account of our character or real self.