In this paper, we present an impressive face recognition model, which represents a robust improvement over the original MobileNetv2. Our model introduces the Receptive Field Block (RFB) to prevent any loss of facial feature information, expands the perceptual field, and implementing multi-scale feature fusion to enhance the model's feature extraction capability. Moreover, we have incorporated Coordinate Attention (CA) into the RFB to enhance recognition accuracy within the lightweight network. The proposed model is named CA_RFB_MobileNetv2. Our experimental results from eight public datasets demonstrate that the recognition accuracy rate of the proposed CA_RFB_MobileNetv2 model is either greater than or equal to that of MobileNetv2. In one of the eight datasets, the recognition accuracy of CA_RFB_MobileNetv2 was slightly reduced by 0.18% compared to FaceNet. However, it offers a significant advantage, a 2.3 times reduction in processing time per image and an 8.8 times decrease in the number of parameters used. Finally, our proposed model was used in a face recognition system, achieving an impressive accuracy of 97.5% with a low false acceptance rate of 2% when tested on 200 randomly selected face images from the Labeled Faces in the Wild dataset.