Despite their promising application, current Computer-Aided Detection (CAD) systems face difficulties, especially in the detection of malignant masses –a major mammographic sign for breast cancer. One of the main problems is the large number of false positives prompted, which is a critical issue in screening programs where the number of normal cases is considerably large. A crucial determinant for this problem is the dependence of the CAD output on the single pixel-based locations initially detected. To refine the initial detection step, in this paper, we propose a novel approach by considering the context information between the neighbouring pixel features and classes for every initially detected suspicious location. Our modelling scheme is based on the Conditional Random Field technique and the mammographic features extracted by image processing techniques. In experimental study, we demonstrated the practical application of the approach and we compared its performance to that of a previously developed CAD system. The results demonstrated the superiority of the context modelling in terms of significantly improved accuracy without increase in computation efforts.