How to coerce your team into creating branches and tags while using Subversion

Remember the standard structure of Subversion repositories? The one that you create with mkdir project/{trunk,tags,branches}? I now figured why people create so few branches and tags in this configuration. Because they checkout at project/trunk level and not at project level by fear of getting essentially the same code multiple times. And if you are at project/trunk, you can't really work with project/branches or project/tags easily.

But there's a solution! Use the --depth and --set-depth options to svn checkout and svn update commands. For instance, when checking out a repository, do it in two steps. First, checkout only the {trunk, tags, branches} folders, but nothing below them:

svn co --depth immediates http://example.com/svn/project

then, change to project/trunk and get the rest of the codebase from trunk:

cd project/trunk
svn up --set-depth infinity

See how it helps? You can now cherry-pick only the branches you want. And get rid of them by setting depth back to immediates

Processing csv reports from your KBC Online Banking: just use awk, dude!

KBC exports bank statements in an awkward format. For instance, there's no structured field for the correspondent's bank account number — this information is lumped together with the description field. Fortunately, awk can come to your rescue

Here is for instance the code that prints your balance

BEGIN {
 FS=";" # this is the field separator
 RS="\r" # linefeed is the record separator, they are probably using AS400 still
}
{ if (NF == 0) next # there's usually some garbage at the end of the file 

  total += gensub(/,/, "", "g",$9) # sum up the total for each record
}
END {
  print "TOTAL", gensub(/(.+)(..)/, "\\1,\\2", "g", total) # print the total
}

Want to know how much you spent on gas? Here is the code to do just that:

BEGIN {
 FS=";" # this is the field separator
 RS="\r" # linefeed is the record separator, they are probably using AS400 still
}
{  if ($7 ~ /PAIEMENT CARBURANT/)  $11="GAS"  sub(/,/, "", $9)
  totals[$11] += gensub(/,/, "", "g",$9)
}
END {
  for (n in totals) {
  print n, gensub(/(.+)(..)/, "\\1,\\2", "g", totals[n]);
}

Facebook says, Ukraine is 81.25% Russian-speaking

Try not to make it political or bring in the discussion on civil war. I was just playing around with Facebook stats for advertisers and found that 81.3% of Ukrainians use Facebook in Russian. Anyone who published ads on Facebook can check it themselves in the Ad Manager. BTW, 92.86% of Belarusians use Facebook in Russian. And there's no "Belarusian" among language choices there, so the rest should be covered by English and a fraction of a percent for Polish.
Tags: 

An incomplete inventory of ways to share code among Drupal initegrators

Whenever a team starts working on a new Drupal-based website, there's an inevitable discussion on how to organize collaboration. Three questions come up regularly:

  1. How and when to use Features, Features Extra and Features Override modules
  2. How to organize production, testing and development environments and where to develop — on a shared server or locally
  3. How to share source code.

Out of the three, the question of the source code is the most contentious. One reason is that while everyone and their friends are already on git, most of the teams that implement Drupal websites do not really need a version control system where developers can cherry-pick and merge changes or analyze bugs by navigating commit histories. Instead, teams are usually interested in incremental backup systems where each team member can be sure that he can roll back his own and other people's changes until everything works again.

Below is the inventory of ways to share source code in Drupal projects.

Incremental backup

Incremental backup with drush

One way of ensuring incremental backups without the overhead of git or other version control system is to use drush. Drush keeps a backup of previous module versions in ~/drush-backups — there is enough info to revert manually to the previously known good state. This setup is ideal for small projects with a handful of custom modules that can be kept in their own git repositories.

When in doubt, don't put your Drupal project in git. Use drush and its build-in backup mechanism.

Ansible playbook for a smallish and very simple Drupal cluster

The cluster runs on apt-based systems. It is designed for high-availability: failure of one server is not critical. However, there is no automatic failover configuration. Instead, manual recovery is possible within minutes.

  • Load balancers run varnish. Its configuration file takes into account the context_breakpoint cookie that's used to implement responsive delivery. The same server also has memcached.
  • Application servers run nginx and a recent php5-fpm through unix sockets. There's also drush. The filesystem is shared through glusterfs.
  • MariaDb is configured in master-slave mode on database servers. They also run Apache Solr.

All servers set up exim to work as smarthost, sending mails through gmail. In practice, gmail limits outgoing emails to a few thousands per day, so it is better to replace it by a dedicated solution such as Mailchimp. There's also newrelic for server monitoring on all servers.

The playbook assumes that all servers have public IPs on eth0 and sit in the private network on eth1.

For the rest, check the code in github.

Googless Calendar

If the only thing that keeps you from closing your Google Account is Calendar, here's the solution: setup your own CalDav server. I've chosen radicale. The setup is easy, but pam support is broken in Debian wheezy, so I had to fix it the following way:
.
# get the pypi installer
apt-get install python-stdeb
patch <<EOF
--- /usr/bin/pypi-install.ori   2014-05-11 21:32:24.884512975 +0200
+++ /usr/bin/pypi-install       2014-05-10 20:23:34.427058833 +0200
@@ -16,7 +16,7 @@
 
 USER_AGENT = 'pypi-install/0.6.0+git ( http://github.com/astraw/stdeb )'
 
-def find_tar_gz(package_name, pypi_url = 'http://python.org/pypi',verbose=0):
+def find_tar_gz(package_name, pypi_url = 'http://pypi.python.org/pypi',verbose=0):
     transport = xmlrpclib.Transport()
     transport.user_agent = USER_AGENT
     pypi = xmlrpclib.ServerProxy(pypi_url, transport=transport)
EOF
# install the latest pam library from pypi
pypi-install  pam
patch <<EOF--- /usr/lib/python2.7/dist-packages/radicale/acl/PAM.py.ori    2014-05-11 21:35:36.441065840 +0200
+++ /usr/lib/python2.7/dist-packages/radicale/acl/PAM.py        2014-05-10 20:27:12.771722350 +0200
@@ -50,7 +50,7 @@
 
     # Check whether the group exists
     try:
-        members = grp.getgrnam(GROUP_MEMBERSHIP)
+        members = grp.getgrnam(GROUP_MEMBERSHIP)[3]
     except KeyError:
         log.LOGGER.debug(
             "The PAM membership required group (%s) doesn't exist" %
EOF
apt-mark hold python-pam
apt-get install radicale
Tags: 

Recruiter with level SPECIALIST

Here's a an email that I just received from a Belgian recruiter (name changed):
On Fri, Apr 11, 2014 at 04:26:33AM -0400, Abdullah Rahmoyan wrote:
>
> For a well known financial institution, I am currently looking
> for a senior Java developer to take part in a 6 months project.
>
>
> The requested skills are:
>
> - BACK-END AND FRONT-END with level SPECIALIST
>
> - DEVELOPMENT KNOWLEDGE with level SPECIALIST
> - BUILD TOOLS, WEB SERVERS with level SPECIALIST
> - AND DATABASES with level SPECIALIST
> - DESIGN REST SERVICES with level EXPERT
> - AND API'S with level EXPERT
> - JAVA EE6, JAVASCRIPT with level SPECIALIST
> - ANGULARJS,POSTRESQL with level SPECIALIST
> - FULL DETAIS TO BE SENT with level SPECIALIST
> - BY EMAIL with level SPECIALIST

A small firewall of China

There's been quite a few spam{bots,turks} lately passing through Drupal Captcha , reCAPTHA, Honeypot module… After a bit of research, I decided to block China from accessing my site. It turned out to be easy:

# install geoip filter into iptables
apt-get install xtables-addons-common
# download the existing geoip database
/usr/lib/xtables-addons/xt_geoip_dl
# set up csv parser in perl
apt-get install libtext-csv-xs-perl
# create a directory for geoip filter's database
mkdir /usr/share/xt_geoip
# build the database
/usr/lib/xtables-addons/xt_geoip_build  -D /usr/share/xt_geoip  *.csv
# load the geoip filter module
modprobe xt_geoip
# block China
iptables -A INPUT -m geoip --src-cc CN -j DROP

PROFIT!!!

P.S. Next in sight is US. US an CH account for 97% of Drupal spam.

Hosting Drupal on bare metal vs. cloud (Acquia)

We hosted Drupal websites at Hetzner for a few years. While it's unbeatable on price, it requires a skilled Linux sysadmin, which weights on personnel costs. Our guesstimate was that we'll pay a third more for a Drupal hosting, but our sysadmin costs would go down by ⅔. Add in non-material considerations, such as lower personnel turnover risks and better DDoS protection, and alternatives to Hetzner start to look almost attractive.

So, we decided to check Drupal cloud hosting solutions and asked for quotes from Pantheon and Acquia. Pantheon was less expensive and had more features. It also edged out Acquia on technology by using Linux containers instead of Amazon Web Services as underlying infrastructure. Unfortunately, Pantheon servers were located near Chicago, while most of our readers are from Europe, so we had no choice but to go for Acquia.

Pages