Rehabilitation is the process of helping people regain or improve lost or impaired function due to injury, illness, or disease. To assist in tracking the progress of patients undergoing rehabilitation, this paper proposes a lightweight graph-based deep-learning model for the automatic assessment of physical rehabilitation exercises. The model takes as input the 3D skeleton sequence of a patient performing a movement and outputs a continuous quality score, as a means for patient supervision that could complement or even substitute the need for ordinary clinical exams. Two graph convolutional networks (GCNs) are sequentially employed to learn spatial and temporal features, the first learning key joint relationships per exercise category and the second exploiting frame correlation to focus on relevant parts of the input sequence. Furthermore, in order to enhance the significance of the scores derived from testing, we propose implementing a classification phase. This phase enables the regression model to produce scores exclusively for sequences specifically tailored to each exercise, which further ensures that an input sequence is assessed only if it corresponds to a complete movement demonstration. The evaluation of the proposed approach on the publicly available KIMORE and UI-PRMD datasets shows that our approach outperforms the state-of-the-art in terms of quality score prediction as well as in terms of efficiency. Our project page is available online.