This paper addresses the issue on estimating semiparametric time series models specified by their conditional mean and conditional variance. We stress the importance of using joint restrictions on the mean and variance. This leads to take into account the covariance between the mean and the variance and the variance of the variance, that is the skewness and kurtosis. We establish the direct links between the usual parametric estimation methods, namely the QMLE, the GMM and the M-estimation. The usual univariate QMLE is, under non-normality, less efficient than the optimal GMM estimator. However, the bivariate QMLE based on the dependent variable and its square is as efficient as the optimal GMM one. A Monte Carlo analysis confirms the relevance of our approach, in particular the importance of skewness.