Upgrading GitLab with external PostgreSQL

     

If you’ve migrated from the internal PostgreSQL instance to an external one, you’ll see something like this, when trying to update:

[root@git ~]# yum update
Loaded plugins: fastestmirror, post-transaction-actions
Loading mirror speeds from cached hostfile
 * base: ftp.foobar.org
 * extras: ftp.foobar.org
 * updates: ftp.foobar.org
gitlab_gitlab-ce/x86_64/signature                                                                                                                                                                                     |  836 B  00:00:00
gitlab_gitlab-ce/x86_64/signature                                                                                                                                                                                     | 1.0 kB  00:00:00 !!!
gitlab_gitlab-ce-source/signature                                                                                                                                                                                     |  836 B  00:00:00
gitlab_gitlab-ce-source/signature                                                                                                                                                                                     |  951 B  00:00:00 !!!
Resolving Dependencies
--> Running transaction check
---> Package gitlab-ce.x86_64 0:10.8.4-ce.0.el7 will be updated
---> Package gitlab-ce.x86_64 0:11.0.0-ce.0.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================================================================================================
 Package                                                Arch                                                Version                                                      Repository                                                     Size
=============================================================================================================================================================================================================================================
Updating:
 gitlab-ce                                              x86_64                                              11.0.0-ce.0.el7                                              gitlab_gitlab-ce                                              414 M

Transaction Summary
=============================================================================================================================================================================================================================================
Upgrade  1 Package

Total download size: 414 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
gitlab-ce-11.0.0-ce.0.el7.x86_64.rpm                                                                                                                                                                                  | 414 MB  00:00:13
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
gitlab preinstall:
gitlab preinstall: This node does not appear to be running a database
gitlab preinstall: Skipping version check, if you think this is an error exit now
gitlab preinstall:
gitlab preinstall: Automatically backing up only the GitLab SQL database (excluding everything else!)
Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... pg_dump: server version: 10.4; pg_dump version: 9.6.8
pg_dump: aborting because of server version mismatch
Backup failed
[FAILED]
gitlab preinstall:
gitlab preinstall: Backup failed! If you want to skip this backup, run the following command and
gitlab preinstall: try again:
gitlab preinstall:
gitlab preinstall:   sudo touch /etc/gitlab/skip-auto-migrations
gitlab preinstall:
error: %pre(gitlab-ce-11.0.0-ce.0.el7.x86_64) scriptlet failed, exit status 1
Error in PREIN scriptlet in rpm package gitlab-ce-11.0.0-ce.0.el7.x86_64
  Verifying  : gitlab-ce-11.0.0-ce.0.el7.x86_64                                                                                                                                                                                          1/2
gitlab-ce-10.8.4-ce.0.el7.x86_64 was supposed to be removed but is not!
  Verifying  : gitlab-ce-10.8.4-ce.0.el7.x86_64                                                                                                                                                                                          2/2

Failed:
  gitlab-ce.x86_64 0:10.8.4-ce.0.el7                                                                                    gitlab-ce.x86_64 0:11.0.0-ce.0.el7

Complete!

Oops. Now you might as well touch the mentioned file, but as the filename suggests, that will skip any migrations, so in that case you’ll get an 500 error. gitlab-ctl reconfigure will perform the DB upgrade, but that won’t work if GitLab is stopped. In that case, you get this error:

Errno::ENOENT: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket

So after all, the perfect sequence is this:

# make sure gitlab is running
gitlab-ctl start

# make sure yum update won't attempt to backup the internal db
touch /etc/gitlab/skip-auto-reconfigure

# update the gitlab package
yum update

# perform the migrations manually
gitlab-ctl reconfigure

# reconfigure should do this already
#gitlab-rake db:migrate

# restart gitlab
gitlab-ctl restart

Now you should be upgraded to the latest and greatest version of GitLab without any errors.

If you use yum-cron to install updates automatically, this won’t be perfect though. Eventually GitLab will get updated, and then it won’t work properly until you reconfigure it. But post-transaction-actions is for the rescue:

yum install yum-plugin-post-transaction-actions
echo 'gitlab-ce:any:/bin/gitlab-ctl reconfigure' > /etc/yum/post-actions/gitlab-ce.action

This tells yum to run gitlab-ctl reconfigure everytime the gitlab-ce package is updated, so at this point, you should be fine.

Cheers!

References: