We have automated the rebuilding of Debian Stretch PHP7.0 packages and the provisioning of an APT repository to enable easy updates, using GitLab CI / CD and Ansible, because we found that our shared hosting web servers were hitting the default PHP maximum number of file handles, following is a write up of how we did this.

The PHP packages we have built are available at deb.webarch.net, where you can also find instructions for installing them.

The Problem

The problem with the maximum number of file descriptors being reached was found when trying to send emails using SMTP from InvoicePlane via PHP sockets: 

A PHP Error was encountered

Severity: Warning

Message: stream_select(): You MUST recompile PHP with a 
larger value of FD_SETSIZE. It is set to 1024, but you have 
descriptors numbered at least as high as 1056. 
--enable-fd-setsize=2048 is recommended, but you may want to 
set it to equal the maximum number of open files supported by 
your system, in order to avoid seeing this error again at a 
later date.

Filename: src/SMTP.php

Line Number: 1124

As a result we could no longer send out invoices, the immediate way we addressed this was to migrate the invoice site to another server which was hosting fewer sites, but that was clearly not the best answer.

The Solution

We didn’t want to rebuild the Debian Stretch PHP7.0 packages once and then find that a security update required us to manually repeat the whole process, we wanted to automate everything in order that it would be very quick and easy to build and provision updated PHP packages within a couple of hours of any security updates being made available. We also wanted a solution that could be used, if necessary, when Debian Buster is released later this year.

The solution we have come up, which takes 90 minutes to run from start to finish, is partly based on an example on the GitLab blog, but, as is generally the case with these things, if you don’t write most of the code yourself you find that you don’t really understand what it all does and if you don’t understand what all the code does then not in a position to debug things and fix them when they go wrong, so we started from scratch, but we found guidance from blog posts like this and wiki pages like this and, of course, had lots of help from StackExchange.

GitLab CI / CD

We have a GitLab server available for the use of our members at git.coop (our co-op is open to clients and investors to join) and we also have it set to spin up Debian Stretch Docker containers via GitLab CI / CD so it made sense to make use of this existing infrastructure.

We configured GitLab CI / CD to install Ansible and write a GPG private key and SSH secret key to the Docker container and then hand over to Ansible to continue the more complex parts of the process.

Ansible

We could have just used GitLab CI / CD and perhaps some Bash scripts to automate everything but, since we love Ansible, it made sense to use the .gitlab-ci file to setup the Docker container up so it could then be passed over to Ansible for the next steps.

Once the Docker container has been bootstrapped the Ansible playbook takes over and sets a number of variables such as the number of sockets we want to build PHP to support and then runs three roles, build, test and deploy.

The first, role builds the packages by downloading and installing the Debian PHP source packages and the libraries needed to recompile them, it then installs these packages and then rebuilds them. Before it does that however it creates a directory for the source code and one for the APT repo layout and it creates the front page for the APT repo and a .htaccess file to configure Apache to correctly serve the APT repo. It uses Reprepro to build a APT repository in the container and it GPG signs the packages. This role does the bulk of the complicated work, if it completes without an error it hands over to the test role.

The second role installs the PHP packages, to test that they install without errors and then runs a Bash script to check that the value of --enable-fd-setsize matches the value specified. If these tests pass then it hands over to the last role.

The third role does the deployment, it copies the repository layout, packages and front page to one of our static shared hosting accounts which is available at deb.webarch.net using SSH (even our cheapest static shared hosting accounts have optional SSH access via a chroot, which makes them ideal for CI / CD integration).

Update: 11th February 2019

The initial set of PHP7.0 Debian Stretch packages that were rebuilt, as documented above, using --enable-fd-setsize=2048 wouldn’t support more than 1024 sockets, it turned out that the header files also needed editing and rebuilding the Apache 2 and OpenSSL packages was also needed plus it was discovered that the number of sockets that could be opened in the Docker container that is used to build and test the packages is limited to 1024 somewhere — when the packages are installed on a Xen virtual server they can then open more than 1024 sockets, however it still doesn’t appear to solve the initial problem with PHP Mailer…