We analyse spin-up/spin-down of the neutron star in Be-X-ray binary system GX 304-1 observed by Swift/X-ray telescope (XRT) and Fermi/gamma-ray burst monitor (GBM) instruments in the period of the source activity from 2010 April to 2013 January and discuss possible mechanisms of angular momentum transfer to/from the neutron star. We argue that the neutron star spin-down at quiescent states of the source with an X-ray luminosity of L-x similar to 10(34) erg s(-1) between a series of Type I outbursts and spin-up during the outbursts can be explained by quasi-spherical settling accretion on to the neutron star. The outbursts occur near the neutron star periastron passages, where the density is enhanced due to the presence of an equatorial Be-disc tilted to the orbital plane. We also propose an explanation to the counterintuitive smaller spin-up rate observed at higher luminosity in a double-peak Type I outburst due to lower value of the specific angular momentum of matter captured from the quasi-spherical wind from the Be-star by the neutron star moving in an elliptical orbit with eccentricity e >= 0.5.