In the context of object-oriented video coding, the encoding of segmentation maps defined by contour networks is particularly critical. In this paper, we present a lossy contour network encoding algorithm where both the rate distortion contour encoding based on maximum operator and the prediction error for the current frame based on quadratic motion model are combined into a optimal polygon contour network compression scheme. The bit rate for the contour network can be further reduced by about 20% in comparison with that in the optimal polygonal boundary encoding scheme using maximum operator in the rate distortion sense.