It is easy to find computed tomography (CT) images that include metals such as implants, bone plates, and bone shafts. These metals replacing body parts, cause serious artifacts in the CT image originated by x-ray beam-hardening. Traditionally, CT physics-based image processing techniques were empirically applied to reduce metal artifacts (MAs). Recently, with the great success of deep learning, many studies on metal artifact reduction (MAR) using convolutional neural networks (CNNs) have been introduced. Most of them commonly meet a challenge to obtain ground truth images for MAR in clinical practice. Thus, for effective MAR without ground truth images, we propose a residual MA model for unsupervised deep learning scheme in CT images. In the 1st stage, MAs are extracted by CT physics-inspired residual model, which is enabled by the residual learning scheme. The result of the 1st stage is fed into the input for the 2nd stage, where artifacts that are not reduced enough in the 1st stage are further removed by the contrastive learning scheme. Then, the networks in the 2nd stage can easily recognize the original structures due to primary artifact reduction in the 1st stage and can properly refine the image. Our model was validated on three datasets. The results show that the proposed model outperforms other MAR models, preserving both original body and metal structures while reducing MAs effectively. We hope that this unsupervised learning model can contribute to good achievements in the MAR field while overcoming the limitations of data construction.