SelectQ: Calibration Data Selection for Post-Training Quantization

Post-training quantization (PTQ) can reduce the memory footprint and latency of deep model inference while still preserving the accuracy of model, with only a small unlabeled calibration set and without the retraining on full training set. To calibrate a quantized model, current PTQ methods usually randomly select some unlabeled data from the training set as calibration data. However, we show the random data selection would result in performance instability and degradation due to the activation distribution mismatch. In this paper, we attempt to solve the crucial task on appropriate calibration data selection, and propose a novel one-shot calibration data selection method termed SelectQ, which selects specific data for calibration via dynamic clustering. The setting of our SelectQ uses the statistic information of activation and performs layer-wise clustering to learn an activation distribution on training set. For that purpose, a new metric called Knowledge Distance is proposed to calculate the distances of the activation statistics to centroids. Finally, after calibration with the selected data, quantization noise can be alleviated by mitigating the distribution mismatch within activations. Extensive experiments on ImageNet dataset show that our SelectQ increases the Top-1 accuracy of ResNet18 over 15% in 4-bit quantization, compared to randomly sampled calibration data. It′s noteworthy that SelectQ does not involve both the backward propagation and batch normalization parameters, which means that it has fewer limitations in practical applications.