Observations made using TESS revealed a sample of low-mass stars which show a periodic modulation on a period <0.2 d. Surprisingly, many of these ultra fast rotating (UFR) stars showed no evidence of flare activity which would be expected from such rapidly rotating stars. We present results from a spectroscopic survey of UFRs using the Nordic Optical Telescope to search for radial velocity variations which could reveal evidence for binarity. Our sample of 29 sources have a photometric period between 0.1-0.2 d, cover spectral classes of M0-4V, and show no evidence for flares. We detect only one source with clear radial velocity shifts, with another two having Gaia RUWE values which suggests they are binaries. Further observations reveal the former star possibly contains a brown dwarf companion with a mass of M2 > 58 MJup and probability P(M2 < 90 MJup) = 50 per cent. There is no evidence for the companion in our spectra, strengthening the case for a brown dwarf companion. We also examine the folded TESS light curves of all our targets, finding at least two are eclipsing binaries and one which has been contaminated by a spatially nearby δ Sct star. We estimate that around 1/4 of our targets may have been contaminated by short period variable stars. However, the majority of our targets are consistent with being single, low mass stars whose variability is due to starspots. We outline the possible reasons why they are not flare active despite being such rapid rotators.