The article presents the control of the nonlinear dynamics of the cantilever-sample system of an atomic force microscope (AFM) by the combination of Q-control and model reference adaptive control, when the AFM operates in contact mode. In this mode the AFM is always in contact with the sample, being able to measure the topographic characteristics for different materials and structures at a nanometric scale. For this task, the AFM uses a cantilever with a micro tip at one end that explores the surface of the sample during scanning. During this process, the closed loop feedback control keeps the acting force on the cantilever beam constant, where the error between the reference and the output of the plant is equivalent to the topography of the sample. We know that the nonlinear dynamics of the cantilever beam system is complex, due to the different types of nonlinear forces that act. In the contact mode the interaction force is described by the modified Hertz model when the cantilever-sample distance is less than 0.2 nm. Here we use an approximate model of the interaction force to reduce the complexity of the model, which depends on the Q factor. The proposed method combine the adaptive control with the control Q, where the control Q allows to reduce the force of beam interaction cantilever-sample, reducing the probability of damage in the sample and in the micro tip due to permanent contact. The Q control is incorporated to the proposed method through the design of the reference model and also a design formula for the effective Q factor is obtained. As a result we have that the proposed control method is stable, showing good performance for different surfaces and reference inputs. The stability of the system is proved by the second Lyapunov method. To show the effectiveness of the proposed method a variety of simulations are presented. The proposed method is totally general and can be applied to any nonlinear complex system.