Quick Problem Recap
You have a service. In order to operate, service depends on database.
Once database schema gets an update, you’d also want to deploy new version of your service.
There are different approaches to solve this problem. During my experience I saw these solutions:
- run migrations before app starts - what if you want to start 100 replicas at the same time?
- run migrations before app starts as an initContainer - does it really solves anything more than above one?
- separate job
So far separate job approach works the best. But what happens if your migrations fails, will new version of service work with semi-broken schema?
Below I’ll show you how to make consistent zero-downtime database migrations and service deployments in one go.
apiVersion: batch/v1 kind: Job metadata: name: migrations-$VERSION labels: app: migrations spec: backoffLimit: 1 template: metadata: labels: app: migrations spec: containers: - name: migrations image: myrepo/migrations:$VERSION imagePullPolicy: Always env: - name: DBHOST value: dbhost.local - name: DBUSER value: user - name: DBPASS value: pass - name: DBNAME value: database restartPolicy: Never
Indicates the number of times a job should be retried if it fails.
If the migrations fail on the second attempt, the Job will fail completely and the new version of the application won’t run.
Indicates if the pod should be restarted in case of failure.
This is separate from job restart.
Testing the job
VERSION=1.2.3 envsubst < job.yaml | kubectl apply -f - ➜ ~ k get job -l app=migrations NAME COMPLETIONS DURATION AGE migrations-1.2.3 1/1 4s 4s ➜ ~ k get pod -l app=migrations NAME READY STATUS RESTARTS AGE migrations-1.2.3-mxj2h 0/1 Completed 0 12s
Using init container to delay pod startup
We will include container with
k8s-wait-for script as an initContainer in all serivces which depend on database schema migrations.
apiVersion: apps/v1 kind: Deployment metadata: name: service spec: template: spec: # The init containers initContainers: - name: wait-for-job image: groundnuty/k8s-wait-for:v1.3 imagePullPolicy: Always args: - job - migrations-$VERSION
Once you have updated your deployment definition, apply to test:
VERSION=1.2.3 envsubst < deploy.yaml | kubectl apply -f -
This might not be obvious, but you’ll need to allow serviceaccount to get job statuses by creating role and rolebinding.
--- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: job-status-reader namespace: $NAMESPACE rules: - apiGroups: - batch resources: - jobs verbs: - get --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: job-status-reader namespace: $NAMESPACE roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: job-status-reader subjects: - kind: ServiceAccount name: $SERVICE_ACCOUNT namespace: $NAMESPACE
Congratz! That’s your first fully completed zero-downtime database migration and deployment with Kubernetes!