Respiratory rate (RR) is a vital physiological indicator for assessing pulmonary and cardiovascular health. Conventional methods for estimating RR require direct contact with the skin and are unsuitable for long-term or unobtrusive use. Remote photoplethysmography (rPPG) enables contactless RR estimation from standard RGB video, but its performance remains limited compared to contact photoplethysmography (PPG), which provides richer and more stable RR-related information. To address this gap, we present a novel rPPGbased RR estimation method (PulseGuide-RR) that distills rich temporal-spectral knowledge from contact PPG to guide feature learning in rPPG. It extracts Eulerian (color-based) and Lagrangian (motion-based) rPPG signals and employs hierarchical attention-based fusion to first integrate temporal and spectral features within each modality and then combine the two modalities for final RR estimation. A composite loss with knowledge-distillation and data-fidelity terms ensures effective feature transfer. We also introduce a new dataset (DIL-RR) with synchronized PPG, face video, and ground-truth RR to support future research. Experiments show that PulseGuide-RR outperforms existing methods on both our dataset and publicly available datasets, substantially narrowing the performance gap between PPG and rPPG-based RR estimation. DIL-RR can be accessed via https://github.com/TrishnaSaikia/DIL-RR.git.