We investigate a quantum dot (QD) system coupled to a vibrational environment with a super-Ohmic spectral density and weakly coupled to a leaky cavity mode, a model relevant for semiconductor-based single-photon sources. The phonon coupling induces dephasing and broadens the absorption and emission line shapes, while the weakly coupled cavity mode leads to effective driving of the QD. To capture non-Markovian effects, we use non-Markovian quantum state diffusion and its extension the hierarchy of pure states to compute multitime correlation functions underlying absorption and resonance fluorescence spectra. We present numerical results for the absorption spectra at strong phonon coupling and finite temperature as well as for resonance fluorescence spectra at varying phonon coupling strengths and temperatures and analyze the visibility of the resonance fluorescence spectra to provide insights into how phonon coupling and thermal effects influence the spectral features.