Pleural invasion of lung cancer should be evaluated based on the relationship between the tumor and the pleura. The pleural surface or the pleura itself can be delineated by ultrasonography based on the differences between the acoustic impedance of the soft tissue, pleural fluid, and the alveolar air. The range resolution of ultrasonography for pleural imaging was determined by using chest wall specimens and silicone membrane phantom. Ultrasonographic grading (uP-grading) from uP0-uP3 was defined, which comparable to the macroscopic classification of pleural invasion factor (P factor) defined by the Japan Lung Cancer Society. This uP grading was tested by two readings with 21 observers, using an ultrasonographic library of 64 surgically confirmed lung cancer cases. Receiver Operating Characteristic (ROC) analysis and Bias to Variance Characteristic (BVC) analysis were used to analyze the results. At the first session, images were interpreted by observer's criteria based on the Japan Lung Cancer Society's macroscopic classification. A sensitivity of 85.7%, a specificity of 68.9%, an accuracy of 72.5%, a positive predictive value of 45.5%, and a negative predictive value of 94.6% were obtained at the second operating point of ROC analysis. In the second session, the images were interpreted according to uP-grading. A sensitivity of 87.1%, a specificity of 74.9%, an accuracy of 77.5%, a positive predictive value of 52.2%, and a negative predictive value of 95.4% were obtained at the second operating point of ROC analysis. The fluctuation range of each observer decreased on the ROC graph in the second reading session. The standard deviation in each case also decreased on the BVC graph in the second reading session.