In order to improve the toughness and reduce polymerization shrinkage of traditional bisphenol A-glycidyl methacrylate (Bis-GMA)/triethylene glycol dimethacrylate (TEGDMA) based dental resin system, a hyperbranched thiol oligomer (HMDI-6SH) was synthesized via thiol-isocyanate click reaction using pentaerythritol tetra(3-mercaptopropionate (PETA) and dicyclohexylmethane 4,4'-diisocyanate (HMDI) as raw materials. Then HMDI-6SH was mixed with 1,3,5-Triallyl-1,3,5-Triazine-2,4,6(1H,3H,5H)-Trione (TTT) to prepare thiol-ene monomer systems, which were added into Bis-GMA/TEGDMA resins with different mass ratio from 10 wt% to 40 wt% to serve as anti-shrinking and toughening agent. The physicochemical properties of these thiol-ene-methacrylate ternary resins including functional groups conversion, volumetric shrinkage, flexural properties, water sorption, and water solubility were evaluated. The results showed that the incorporation of HMDI/TTT monomer systems into Bis-GMA/TEGDMA based resin could improve C=C double bond conversion from 62.1% to 82.8% and reduced volumetric shrinkage from 8.53% to 4.92%. When the mass fraction of HMDI/TTT monomer systems in the resins was no more than 20 wt%, the flexural strength of the resin was higher or comparable to Bis-GMA/TEGDMA based resins (p > 0.05). The toughness (it was measured from the stress-strain curves of three-point bending test) of the resins was improved. Water sorption and water solubility tests showed that the hydrophobicity of resin was enhanced with increasing the content of thioester moiety in resin.