An incomplete inventory of ways to share code among Drupal initegrators

Whenever a team starts working on a new Drupal-based website, there's an inevitable discussion on how to organize collaboration. Three questions come up regularly:

  1. How and when to use Features, Features Extra and Features Override modules
  2. How to organize production, testing and development environments and where to develop — on a shared server or locally
  3. How to share source code.

Out of the three, the question of the source code is the most contentious. One reason is that while everyone and their friends are already on git, most of the teams that implement Drupal websites do not really need a version control system where developers can cherry-pick and merge changes or analyze bugs by navigating commit histories. Instead, teams are usually interested in incremental backup systems where each team member can be sure that he can roll back his own and other people's changes until everything works again.

Below is the inventory of ways to share source code in Drupal projects.

Incremental backup

Incremental backup with drush

One way of ensuring incremental backups without the overhead of git or other version control system is to use drush. Drush keeps a backup of previous module versions in ~/drush-backups — there is enough info to revert manually to the previously known good state. This setup is ideal for small projects with a handful of custom modules that can be kept in their own git repositories.

When in doubt, don't put your Drupal project in git. Use drush and its build-in backup mechanism.

Ansible playbook for a smallish and very simple Drupal cluster

The cluster runs on apt-based systems. It is designed for high-availability: failure of one server is not critical. However, there is no automatic failover configuration. Instead, manual recovery is possible within minutes.

  • Load balancers run varnish. Its configuration file takes into account the context_breakpoint cookie that's used to implement responsive delivery. The same server also has memcached.
  • Application servers run nginx and a recent php5-fpm through unix sockets. There's also drush. The filesystem is shared through glusterfs.
  • MariaDb is configured in master-slave mode on database servers. They also run Apache Solr.

All servers set up exim to work as smarthost, sending mails through gmail. In practice, gmail limits outgoing emails to a few thousands per day, so it is better to replace it by a dedicated solution such as Mailchimp. There's also newrelic for server monitoring on all servers.

The playbook assumes that all servers have public IPs on eth0 and sit in the private network on eth1.

For the rest, check the code in github.

Googless Calendar

If the only thing that keeps you from closing your Google Account is Calendar, here's the solution: setup your own CalDav server. I've chosen radicale. The setup is easy, but pam support is broken in Debian wheezy, so I had to fix it the following way:
.
# get the pypi installer
apt-get install python-stdeb
patch <<EOF
--- /usr/bin/pypi-install.ori   2014-05-11 21:32:24.884512975 +0200
+++ /usr/bin/pypi-install       2014-05-10 20:23:34.427058833 +0200
@@ -16,7 +16,7 @@
 
 USER_AGENT = 'pypi-install/0.6.0+git ( http://github.com/astraw/stdeb )'
 
-def find_tar_gz(package_name, pypi_url = 'http://python.org/pypi',verbose=0):
+def find_tar_gz(package_name, pypi_url = 'http://pypi.python.org/pypi',verbose=0):
     transport = xmlrpclib.Transport()
     transport.user_agent = USER_AGENT
     pypi = xmlrpclib.ServerProxy(pypi_url, transport=transport)
EOF
# install the latest pam library from pypi
pypi-install  pam
patch <<EOF--- /usr/lib/python2.7/dist-packages/radicale/acl/PAM.py.ori    2014-05-11 21:35:36.441065840 +0200
+++ /usr/lib/python2.7/dist-packages/radicale/acl/PAM.py        2014-05-10 20:27:12.771722350 +0200
@@ -50,7 +50,7 @@
 
     # Check whether the group exists
     try:
-        members = grp.getgrnam(GROUP_MEMBERSHIP)
+        members = grp.getgrnam(GROUP_MEMBERSHIP)[3]
     except KeyError:
         log.LOGGER.debug(
             "The PAM membership required group (%s) doesn't exist" %
EOF
apt-mark hold python-pam
apt-get install radicale
Tags: 

Recruiter with level SPECIALIST

Here's a an email that I just received from a Belgian recruiter (name changed):
On Fri, Apr 11, 2014 at 04:26:33AM -0400, Abdullah Rahmoyan wrote:
>
> For a well known financial institution, I am currently looking
> for a senior Java developer to take part in a 6 months project.
>
>
> The requested skills are:
>
> - BACK-END AND FRONT-END with level SPECIALIST
>
> - DEVELOPMENT KNOWLEDGE with level SPECIALIST
> - BUILD TOOLS, WEB SERVERS with level SPECIALIST
> - AND DATABASES with level SPECIALIST
> - DESIGN REST SERVICES with level EXPERT
> - AND API'S with level EXPERT
> - JAVA EE6, JAVASCRIPT with level SPECIALIST
> - ANGULARJS,POSTRESQL with level SPECIALIST
> - FULL DETAIS TO BE SENT with level SPECIALIST
> - BY EMAIL with level SPECIALIST

A small firewall of China

There's been quite a few spam{bots,turks} lately passing through Drupal Captcha , reCAPTHA, Honeypot module… After a bit of research, I decided to block China from accessing my site. It turned out to be easy:

# install geoip filter into iptables
apt-get install xtables-addons-common
# download the existing geoip database
/usr/lib/xtables-addons/xt_geoip_dl
# set up csv parser in perl
apt-get install libtext-csv-xs-perl
# create a directory for geoip filter's database
mkdir /usr/share/xt_geoip
# build the database
/usr/lib/xtables-addons/xt_geoip_build  -D /usr/share/xt_geoip  *.csv
# load the geoip filter module
modprobe xt_geoip
# block China
iptables -A INPUT -m geoip --src-cc CN -j DROP

PROFIT!!!

P.S. Next in sight is US. US an CH account for 97% of Drupal spam.

Hosting Drupal on bare metal vs. cloud (Acquia)

We hosted Drupal websites at Hetzner for a few years. While it's unbeatable on price, it requires a skilled Linux sysadmin, which weights on personnel costs. Our guesstimate was that we'll pay a third more for a Drupal hosting, but our sysadmin costs would go down by ⅔. Add in non-material considerations, such as lower personnel turnover risks and better DDoS protection, and alternatives to Hetzner start to look almost attractive.

So, we decided to check Drupal cloud hosting solutions and asked for quotes from Pantheon and Acquia. Pantheon was less expensive and had more features. It also edged out Acquia on technology by using Linux containers instead of Amazon Web Services as underlying infrastructure. Unfortunately, Pantheon servers were located near Chicago, while most of our readers are from Europe, so we had no choice but to go for Acquia.

Notes on setting up Debian with ChromeOS kernel on a SD card plugged into Samsung ARM Chromebook

Two main sources of my inspiration were this post by Daniel P. Berrangé and this very detailed HOWTO on Debian User Forums. There's also a great, albeit a bit stripped down rootfs from Vassilis Laganakos for those who don't want to bother using debootstrap. Here's my script to run in root shell on ChromeOS to put the signed kernel onto the SD card:
echo "console=tty1 debug verbose root=/dev/mmcblk1p3 rootwait rw lsm.module_locking=0" > /tmp/config
dd if=/dev/mmcblk0p2 of=/tmp/oldblob
vbutil_kernel --repack /tmp/newkern --keyblock /usr/share/vboot/devkeys/kernel.keyblock --version 1 \
  --signprivate /usr/share/vboot/devkeys/kernel_data_key.vbprivk --config=/tmp/config --oldblob /tmp/oldblob
dd if=/tmp/newkern of=/dev/mmcblk1p1
dd if=/tmp/newkern of=/dev/mmcblk1p2
cgpt add -i 1 -S 1 -T 5 -P 10 -l KERN-A /dev/mmcblk1
cgpt add -i 2 -S 1 -T 5 -P 5 -l KERN-B /dev/mmcblk1
Note the use of --oldblob option -- newer ChromeOS does now mount /boot anymore, but HOWTOs out there still suggest to point to the kernel using --vmlinuz . Also, lsm.module_locking=0 is important. Without it, you won't be able to load kernel modules in Debian.

How to backup to an external disk once it is plugged in

How it should work:

  • insert a disk with a partition labelled as backup
  • wait until the beep
  • remove the disk

To label an ext3/ext4 disk, use

e2label device [newlabel]

Then, create an udev rule that runs your script when you insert a disk with a partition named backup:

echo 'KERNEL=="sd*", ENV{ID_FS_LABEL}=="backup", RUN+="/usr/local/bin/backup.sh"' 
   >> /etc/udev/rules.d/99-backup.rules

your backup script can be as simple as this:

#!/bin/sh
/bin/mount /dev/disk/by-label/backup /media/backup && \
  /usr/bin/rsync -r /home/* /media/backup && \
  /bin/umount /media/backup && \
  /usr/bin/beep

Watch out, $PATH is not set, you shoud use absolute paths everywhere.

To make it all work, reload udev rules with

udevadm control --reload-rules

Convert a git repository from submodules to subtrees

Git submodules and git subrees are well explained in the git-scm book. Here's a small script that automatically converts the former to the later:
cat .gitmodules |while read i
do
  if [[ $i == \[submodule* ]]; then
    mpath=$(echo $i | cut -d\" -f2)
    read i; read i;
    murl=$(echo $i|cut -d\  -f3)
    mcommit=`eval "git submodule status ${mpath} |cut -d\  -f2"`
    mname=$(basename $mpath)
    echo -e "$name\t$mpath\t$murl\t$mcommit"
    git submodule deinit $mpath
    git rm -r --cached $mpath
    rm -rf $mpath
    git remote add $mname $murl
    git fetch $mname
    git branch _$mname $mcommit
    git read-tree --prefix=$mpath/ -u _$mname
fi
done
git rm .gitmodules

Pages