machine learning pipeline cloud resource incident
Context: We own a core business machine learning pipeline that runs weekly. The pipeline generates customer-campaign-content matching across 10+million customers, 200+ campaigns. The pipeline also pass the final selected allocation to the downstream system to send out the campaign send Issue Summary: The pipeline has to finish within 1 day due to the data availability timing and requirement of the downstream system. The majority of the pipeline is running on GKE preemptible machine. The parallel batch scoring step in the pipeline uses ~1000 containers/pods which requires ~66 x 64cpu 416ram machines. In one of our production run, we notice that only 1 machine get allocated, which means the runtime will increase from 20min to 30+hours. This will not meet the deadline requirement. Root Cause and Remediation: The root cause is due to use of preemptible machine machine type availability resource allocation base on machine type Preemptible machine We use the preemptible machines t...