We present six years of optical monitoring of the black hole (BH) candidate X-ray binary Swift J1357.2–0933, during and since its discovery outburst in 2011. On these long timescales, the quiescent light curve is dominated by high amplitude, short-term (seconds–days) variability spanning ∼2 mag, with an increasing trend of the mean flux from 2012 to 2017 that is steeper than in any other X-ray binary found to date (0.17 mag yr^‑1). We detected the initial optical rise of the 2017 outburst of Swift J1357.2–0933, and we report that the outburst began between 2017 April 1 and 6. Such a steep optical flux rise preceding an outburst is expected according to disk instability models, but the high amplitude variability in quiescence is not. Previous studies have shown that the quiescent spectral, polarimetric, and rapid variability properties of Swift J1357.2–0933 are consistent with synchrotron emission from a weak compact jet. We find that a variable optical/infrared spectrum is responsible for the brightening: a steep, red spectrum before and soon after the 2011 outburst evolves to a brighter, flatter spectrum since 2013. The evolving spectrum appears to be due to the jet spectral break shifting from the infrared in 2012 to the optical in 2013, then back to the infrared by 2016–2017 while the optical remains relatively bright. Swift J1357.2–0933 is a valuable source to study BH jet physics at very low accretion rates and is possibly the only quiescent source in which the optical jet properties can be regularly monitored.