One of the most quoted paper on this subject is Cohen, below. Not without controversy though. This page contains a nice review of the pros and cons of the method.

Cohen J. Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin. 70:213-20, 1968.

Tags: research methodology