Abstract:
Taking the Chongqing Mesoscale Numerical Prediction model as an example, the authors designed and developed a parallel transmission program for large data of the mesoscale numerical prediction model to improve transmission efficiency.The large dataset the Chongqing mesoscale model produced has been transferred to the data management server for every day processing and analysis. The one-core serial program with scp was currently used to complete the task. Because it took more time and could not make full use of cpu cores and network bandwidth, this program could not meet the demand of timely use of Chongqing Weather Forecasting. The parallel transmission program takes process pool as main architecture, and adopts Rsync instead of scp to improve the data transmission speed. Meanwhile the model data is divided on the basis of data features, and the best data partition strategy would be chosen by comparison. The results showed that the parallel transmission program with the best data partition strategy can greatly reduce the data transmission time, and effectively improve the availability of cpu cores and network bandwidth.