The relevance of extracellular matrix (ECM) remodeling is reported in white adipose tissue (AT) and obesity-related dysfunctions, but little is known about the importance of ECM remodeling in brown AT (BAT) function. Here, we show that a time course of high-fat diet (HFD) feeding progressively impairs diet-induced thermogenesis concomitantly with the development of fibro-inflammation in BAT. Higher markers of fibro-inflammation are associated with lower cold-induced BAT activity in humans. Similarly, when mice are housed at thermoneutrality, inactivated BAT features fibro-inflammation. We validate the pathophysiological relevance of BAT ECM remodeling in response to temperature challenges and HFD using a model of a primary defect in the collagen turnover mediated by partial ablation of the Pepd prolidase. Pepd-heterozygous mice display exacerbated dysfunction and BAT fibro-inflammation at thermoneutrality and in HFD. Our findings show the relevance of ECM remodeling in BAT activation and provide a mechanism for BAT dysfunction in obesity.