Upgrade gitlab-ce in docker and run a PostgreSQL migration • July 5, 2020
Tags: Configuration

Prerequisites

• Of course you need a running and working gitlab-ce docker instance.
• Same free disk space as your gitlab instance + docker image is using right now (a little less should work too).
• Backup all your data!

To get gitlab running in production, I use a customized compose file:

docker-compose.yml

version: '3.3'
services:
  web:
    image: 'gitlab/gitlab-ce:latest'
    restart: always
    hostname: 'gitlab.example.org'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://gitlab.example.org'
        letsencrypt['enabled'] = false
    dns:
      - 8.8.8.8
      - 1.1.1.1
    ports:
      - '8180:80'
      - '8143:443'
    volumes:
      - '/srv/gitlab/config:/etc/gitlab'
      - '/srv/gitlab/logs:/var/log/gitlab'
      - '/srv/gitlab/data:/var/opt/gitlab'
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 4000M
        reservations:
          cpus: '0.1'
          memory: 500M

Start container using docker-compose in compatibility mode: docker-compose --compatibility up -d

My default way to upgrade gitlab-ce in docker

docker pull gitlab/gitlab-ce:latest
docker-compose down
docker image prune -f
docker-compose --compatibility up -d

I recently upgraded my gitlab-ce docker container to the latest version 13.1.2, coming from 12.9.10.

Now unfortunately gitlabs PostgreSQL no longer works :(

Error: The data directory was initialized by PostgreSQL version 10, which is not compatible with this version 11.7.

I found out, that it was possible to migrate the database from version 10 to 11 with the old gitlab version, but going back to this doesn´t work. All parts, except the database, were migrated to the new version. So, how to fix it?

Upgrade PostgreSQL 10 to 11 within a gitlab-ce 13.1.2 container

Using docker-compose to start our container will start gitlab in it. Because of the database failure, the container restart every minute. We don´t want this for our migration work. So we start it manually and access the bash in it.

Please replace (at least) the hostname and the other options with your custom configuration!

docker run --rm \
--hostname gitlab.example.org \
--publish 8143:443 --publish 8180:80 \
--env GITLAB_OMNIBUS_CONFIG="external_url 'https://gitlab.example.org'; letsencrypt['enabled'] = false" \
--name gitlab_custom \
--volume /srv/gitlab/config:/etc/gitlab \
--volume /srv/gitlab/logs:/var/log/gitlab \
--volume /srv/gitlab/data:/var/opt/gitlab \
-it gitlab/gitlab-ce:latest /bin/bash

Now we are in the docker container.

First of all run the default entry script to check that it really doesn`t work!

/assets/wrapper

If there is a failure in the Recipe: gitlab::database_migrations section and at least the error message Error executing action 'run' on resource 'bash[migrate gitlab-rails database]', continue to migrate the database.

We need to install the default (old and new) PostgreSQL versions.

apt update
apt install postgresql-10 postgresql-11

Create the destination folder for the migrated database and update the directory owner to gitlab-psql. We assume, the 'old' (version 10) database data is located at /var/opt/gitlab/postgresql/data.

mkdir -p /tmp/11/data
chown -R gitlab-psql /tmp/11 /var/opt/gitlab/postgresql/data

Now we need to initialize the new database structure.

sudo -u gitlab-psql /usr/lib/postgresql/11/bin/initdb -D /tmp/11/data

Take a look at /var/opt/gitlab/postgresql/data/postgresql.conf for the right port (and set it with -p).

Before running the real migration, we should check the access and schema of the database. In case of errors, this command can be run multiple times.

sudo -u gitlab-psql /usr/lib/postgresql/11/bin/pg_upgrade -D /tmp/11/data -d /var/opt/gitlab/postgresql/data/ -v -b /usr/lib/postgresql/10/bin -B /usr/lib/postgresql/11/bin/ -p 5432 --check

If there was an error during the check, you may need to update some other directory permissions.

If the check was successful, do the database migration with the following command. This can take some time, depending on the size of your gitlab instance.

sudo -u gitlab-psql /usr/lib/postgresql/11/bin/pg_upgrade -D /tmp/11/data -d /var/opt/gitlab/postgresql/data/ -v -b /usr/lib/postgresql/10/bin -B /usr/lib/postgresql/11/bin/ -p 5432

Stop all gitlab services, backup the old database and move the migrated data to the original path.

gitlab-ctl stop
rm -f /var/opt/gitlab/postgresql/.s.PGSQL.5432*
mv /var/opt/gitlab/postgresql/data /var/opt/gitlab/postgresql/data_old_10
mv /tmp/11/data /var/opt/gitlab/postgresql/data

Now run the default entry script to finish the built-in gitlab migration.

/assets/wrapper

If it works (some log output appears after loading the gitlab parts and of course the website works), you can press Ctrl+C, and enter exit to leave and remove the current container.

If there were still database errors, ... oops.

If there is an error like this,

PG::ConnectionBad: could not connect to server: No such file or directory
          |                Is the server running locally and accepting
          |                connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?

make sure no postgresql process is running, remove the socket file and try again.

After that, leave the container.

Start your gitlab container the way you start it by default, e.g. using docker-compose up like me:

docker-compose --compatibility up -d

---

I hope this short explanation helped someone who had a database problem similar to mine after a gitlab upgrade.