This paper presents a framework for analyzing the changes in agricultural labor productivity with regards to the structural, land intensity, and land productivity effects. This approach allows for the residual-free decomposition of data from different levels of aggregation. The logarithmic mean Divisia index was applied for the analysis and a data envelopment analysis model was constructed to identify potential gains in agricultural labor productivity due to the optimization of input use and output production. The proposed approach was applied to the case of China over the period of 1997–2017. Province-level data were used to identify the major driving factors behind agricultural labor productivity change. Land productivity change appeared to be the major source of agricultural labor productivity gains in China. The structural change was rather negligible, suggesting that the reallocation of the agricultural labor force did not add to the agricultural labor productivity growth in China. A frontier analysis indicated that agricultural labor productivity could increase by some 45% on average in case full technical efficiency is achieved.