Aims. This is a continuation of our study of early-type binaries with high-precision broadband polarimetry. Here we present the results of our observations of the massive O+O-type binary system HD 165052 located in the very young open cluster NGC 6530. Our aim was to investigate the variations in linear polarization in this system with time and obtain an independent estimate of the orbital period from polarization data. By fitting phase-locked variations in Stokes parameters q and u, it is possible to obtain independent estimates of the orbit inclination, i, orientation, Ω, and the direction of rotation.

Methods. We employed the DiPol-2 polarimeter in combination with the remotely controlled 60 cm KVA and Tohoku T60 telescopes. Linear polarization measurements of HD 165052 were obtained in the B, V, and R passbands with an accuracy better than similar to 0.01%. The Lomb-Scargle method of period search was applied to the acquired polarization data to identify periodic signals. To study interstellar polarization in NGC 6530, we observed 25 cluster stars located in close proximity to the binary.

Results. Our observations reveal clear periodic variations in the Stokes parameters in all three passbands with amplitudes of similar to 0.1%. A period search applied to polarization data detected the strongest (and unambiguous) periodic signal at 1.47755 +/- 0.005 d, which corresponds to half of the orbital period, 2.95510 +/- 0.005 d. Our independent period search, performed on polarization data, supports a shorter orbital period for HD 165052 (as determined from radial velocity measurements done in the past) and rejects the longer value obtained very recently. In the observed polarization variations, the second terms of the Fourier harmonics clearly dominate, suggesting that light scattering material is symmetrically distributed with respect to the orbital plane. We conclude that the most probable polarization mechanism is electron scattering in the interacting stellar winds. To evaluate the effect of nonperiodic noise on the orbital parameters derived from the analysis of periodic variations in polarization, we ran rigorous Monte Carlo simulations. We thereby derived our best estimate of the orbital inclination, i, as 55 degrees + 5 degrees/-55 degrees, and Ω as 148 degrees(328 degrees) + 20 degrees/-22 degrees averaged over the B, V, and R passbands. These values provide direct and unambiguous evidence that nonperiodic noise, if present, significantly reduced the accuracy of orbital parameters obtained from high-precision polarization data for the low-inclination systems. Using the values of the polarization variability amplitude, we estimated the mass-loss rate for the whole system to be M = 2.46 x 10^-7 +/- 2.47 x 10^-8 M-circle dot yr(-1). The direction of the binary system rotation on the plane of the sky is clockwise. Our observations of HD 165052 neighbor stars reveal a complex behavior of interstellar polarization in the cluster NGC 6530.