Software

Q: What do a group of lawyers and a group of Drupal tech geeks have in common?  

What follows is part one in a series of posts on web performance that I've wanted to write for quite some time. In this series of posts I'll not only be talking about optimizing web performance generally, but also providing specific guidance for speeding up Drupal sites.

Although I'm not a web performance specialist or expert, I have taken a keen interest in the topic in my work as a frontend developer building responsive websites. I love building fast sites and have gained some experience over the years getting Drupal to shed some its inherent sluggishness. 

As a way of systematically tackling what can be a complex subject, we'll use the results of a test from WebPageTest.org, a Google-sponsored tool that provides very in-depth information about the performance of a site in nice, easily digestible chunks.

How Fast Is Fast Enough?

Before we get into the details of web performance we should first stop to ask how fast a site should be in order to qualify as "fast". Here are some research results courtesy of Radware that might help bring things into focus:

• 64% of smartphone users expect pages to load in less than 4 seconds.
• The performance poverty line (i.e. the plateau at which your website’s load time ceases to matter because you’ve hit close to rock bottom in terms of business metrics) for most sites is around 8 seconds.

More guidance comes courtesy of Ilya Grigorek of Google. In his recent presentation at the Velocity conference, he cited research that indicates a target page render time should be 1000ms - or one second  - to avoid "context switching" among users.

Basically, if it takes a page longer than one second to render, you risk losing the attention of the user. If it takes longer than eight seconds for a page to render, it's similar in terms of business metrics (conversions, sales, etc) as if it took 30 seconds or a minute.

If one second sounds impossibly ambitious, there is further research showing that a load time of three seconds or less is probably OK. 

The bottom line: your pages should load in under three seconds on desktop, and under 4 seconds on a mobile.

Pretty harsh reality check, huh? Let's see what can be done to get our sites whipped into shape.

Test Results for this Analysis

In order for us to have a practical example for our discussion, I ran the Friendly Machine site through WebPageTest. Here are the results (click to enlarge image):

I recently completed a refresh of the design of this site with a lot of attention focused on keeping things fast. My target page load time was one second, so I was happy when the results consistently came in below that.

Let's start our analysis by looking at the first number in the above table - under the heading "Load Time". You'll see the value is 0.662 seconds. That's pretty darn good, but if you scan across the table you may see something on the far right that's a bit confusing - a Fully Loaded Time of 0.761 seconds.

So what's the difference between Load Time and Fully Loaded Time?

Load Time is calculated from the time when the user started navigating to the page until the Document Complete event is fired. The Document Complete event is fired by the browser once the page has completed loading.

The Fully Loaded time, on the other hand, also includes any metrics up until there is no network activity for two seconds. Most of the time this means watching for things being loaded by JavaScript in the background.

First Byte Time = Backend Performance

Whenever talk turns to web performance, it seems a lot of folks immediately start thinking of what's happening on the server. Although it's a very important piece of the puzzle, as we walk through this analysis, you'll see that most web performance issues actually reside on the frontend.

Before we get ahead of ourselves, though, let's return to the results from our test and look at the next metric in our table, First Byte Time (highlighted in blue below) which tells us about performance on the server.

This First Byte Time represents the time from when a user began navigating to the page until the first bit of the server response arrived at the browser. The target time for this on WebPageTest is a meager 87 ms!

This metric is also represented as the first in the series of letter grades you see at the top of the test results. You'll notice Friendly Machine got an "A" and I really wish I could take credit for it, but the truth is my host Pantheon - and the awesome backend performance they provide - are responsible for this metric scoring well.

Backend Drupal Performance

Let's pause here and talk specifically about backend Drupal performance. How would we address this metric if it hadn't scored well? This topic can get pretty deep, so we'll only review the most popular options, but they'll still be able to do wonders for improving this key metric if your site is not performing well.

Let's start by discussing server resources (with a brief, tangential mini-rant about shared hosting).

If you want a fast Drupal website, you really shouldn't be on a shared host, period. Although many of them will claim to be Drupal specialists, very few of them actually are. One giveaway is the number for PHP memory limit.

Although this number doesn't directly impact performance, it can break your site if it's too low and is also useful for smoking out hosts that don't know Drupal. You can find this number at admin/reports/status and it will look something like the image below.

You can see on my site this number is at 256 megabytes and this is most likely where you want it, although if you have a simple site without Views or Panels, then 128M might work. If it's set at 64M, then it's too low and this is often what you'll find with shared hosting arrangements. 

Another issue with shared hosting - and one that does impact performance - is that your website is on a server with perhaps hundreds of other sites. If one of those sites gets hit with a large spike in traffic or some other issue, it can affect all the sites on that server as it gobbles up the available resources.

Perhaps the biggest issue with shared hosting, however, is that advanced caching using tools like Memcached and Varnish are rarely, if ever, available. And when it comes to Drupal backend performance, caching is critical. The best you'll probably be able to do with regard to caching on shared hosting is using the Boost module, which we'll talk about in the next section.

To ensure that server resources aren't an issue for your website, consider either a managed VPS or a Drupal host like Pantheon, both of which start at around $25 per month. Pantheon is what I recommend for small to medium sized sites because your site will scale better with them and they offer tremendous value, although they are a great fit for enterprise clients as well. If you have a bigger budget, Acquia or BlackMesh might fit the bill.

Sure, these options cost more than the $7 per month the cheap hosts offer, but they will provide a professional level of service that will more than pay for itself over time.

Caching for Drupal Websites

We said caching was critical, so here are five of the most important caching solutions for a Drupal website:

1. Drupal's built-in caching
2. Boost module
3. Memcached
4. Varnish
5. Views caching

There are other options, of course, but these five cover most of the ground. Let's briefly go through them one at a time.

Drupal's Built-in Caching

Most of a Drupal site is stored in the database - nodes, information about blocks, etc. - and enabling the default caching will store the results of these database queries so that they aren't executed every time a page is requested. Enabling these settings alone can have a big impact on performance, particularly if your pages use a lot of views. This one is kind of a no-brainer.

Boost Module

The Boost module is pretty great. It works very well in tandem with Drupal caching, but it requires some additional configuration. What you end up with after you have the module installed and configured is a caching system that stores the output of your Drupal site as static HTML pages. This takes PHP processing out of the equation, leading to another nice bump in performance.

Memcached

Memcached can speed up dynamic applications (like Drupal) by storing objects in memory. With Boost and Drupal caching, the data being cached is stored on the server's hard drive. With memcached, it's being stored in memory, something that greatly speeds up the response time for a request. Memcached works great in conjunction with both Boost and Drupal caching.

Varnish

Varnish is an HTTP accelerator that, similar to memcached, stores data in memory. It's capable of serving pages much faster than Apache (the most common web server for Drupal sites). It can also be used in conjunction with memcached, although it's often the case that they are not used together and other advanced caching methods are instead implemented alongside Varnish.

Views Caching

Another type of database caching is Views caching. Views is a very popular, but rather resource intensive, Drupal module. Implementing Views caching can give your site a nice additional performance boost by possibly removing a few database queries from the build process.

To set views caching, go to your view. On the right hand side, under Advanced > Other, you'll see a link for Caching. Just go in and set a value - an hour is usually a good default - for each view on your site.

Wrapping Up Part One

Wow, long post and all we've really covered so far is backend performance and caching! This discussion hasn't been comprehensive by any means, but it does provide a great start.

Next time we'll start digging into frontend performance, the area where most of our performance issues reside. What should be obvious so far is that web performance is a subject that is both deep and wide, but also critically important to building successful websites.

If you have any comments about this post, you may politely leave them below.

Drupal Web Performance

Doing some initial prototyping work on the Comstack module I hit this question without a clear answer. For clarity here's a chunk of the schema structure for a Message Type (exportable entity).

/**
* Implements hook_schema().
*/
function comstack_schema() {
  $schema = array();

  $schema['comstack_message_type'] = array(
    'description' => 'Stores information about all defined {comstack_message} types.',
    'fields' => array(
      'id' => array(
        'type' => 'serial',
        'not null' => TRUE,
        'description' => 'Primary Key: Unique {comstack_message} type ID.',
      ),
    ...
    'delivery_methods' => array(
        'type' => 'text',
        'not null' => FALSE,
        'size' => 'big',
        'serialize' => TRUE,
        'description' => 'A serialized array of allowed send methods for this type.',
      ),

Following the instructions on how to create an exportable Entity as over here on d.o https://www.drupal.org/node/1021526 and here https://www.drupal.org/node/1021576

The second link has the following code which is the submit process where the form values are wrapped up and a new entity put together for you before saving.

/**
* Form API submit callback for the type form.
*/
function profile2_type_form_submit(&$form, &$form_state) {
  $profile_type = entity_ui_form_submit_build_entity($form, $form_state);
  ...

So how do we construct a form which will allow for arbitrary array structures? Like this (in your form function)!

  $form['delivery_methods'] = array(
    '#title' => t('Delivery methods to allow'),
    '#type' => 'checkboxes',
    '#required' => TRUE,
    '#options' => $delivery_methods,
    '#default_value' => isset($comstack_message_type->delivery_methods) ? $comstack_message_type->delivery_methods : array(),
    '#tree' => TRUE,
  );

It's the #tree bit there that does it. Here's an explanation from the Form API documentation page which is marked as archived but still useful https://www.drupal.org/node/48643.

When we set fieldset value to TRUE we create the form:

<?php
$form['colors'] = array(
'#type' => 'fieldset',
'#title' => t('Choose a color'),
'#collapsible' => FALSE,
'#tree' => TRUE,
);
$form['colors']['green'] = array(
'#type' => 'checkbox',
'#title' => t('Green'),
'#default_value' => $node->green,
'#required' => FALSE,
);

and this is how they are inserted or updated in a db_query:

<?php
function example_insert($node){
  db_query("INSERT INTO {example} (nid, question, green, blue) VALUES (%d,'%s', %d, %d)", $node->nid, $node->title, $node->colors['green'], $node->colors['blue']);
}

Any questions? Leave them in the comments :]

I am sitting here at DrupalCamp Asheville2014. I took a break and hung out in the BoF room and decided to compose this list of ideas on how Drupal Event attendees can engage the event.

I'd love to hear your comments below!

1. Know The Session
   • What is this session about? Is it a show-n-tell of a module? Is it a case study of a website? If you are consciously aware of what to expect, then you are prepared to take what you hear and frame it within the context of the session topic. This is important for the attendee because not every element of the presentation will relate directly to the session topic. If the speaker needs to lay down some groundwork for a few minutes, it is important for you to remember what the overall topic is so that you don’t get lost in the weeds.
2. Know The Session Speaker
   • Check out their Drupal.org Profile or their profile on the Event website. Get a sense of their background and perspective. This is also helpful if you ask questions at their session. You can ask questions that you know relate to elements of their background.
3. Ask Questions At The Sessions
   • The number of questions at any given session will vary. But when there are none it can be a tad awkward. Then after the session, you might still see ppl walk up and ask questions.
   • I encourage you to fill in that silence with some immediate questions that come to mind. The speakers really really appreciate the questions.
4. Engage Social Media
   • Tweet about the event. Maybe tweet about each session you attend and provide a link to the session description and invoke the speaker’s twitter name as well. 
   • Take pictures and post them wherever you post your pics.
   • Use the Hashtag if the event has one.
   • Do you blog? Blog about the event and what you liked.
   • The event organizers and spekers REALLY appreciate the media exposure.
   • Don’t forget that many Drupal events publish their videos online so you can catch the ones you missed or revisit the one you liked.
5. Hang Out
   • Don’t feel like you have to attend a session in every timeslot. Feel free to hang out near the coffee tables or registration tables or in Birds of a Feather rooms. Wherever you see people hanging out, join them!
6. Join A Stranger For Lunch
   • In general, the Drupal Community is a VERY social bunch. When it’s time to sit down and eat, it is also a good time to make some new friends. To the extent that you are comfortable with it, you can learn a lot by asking ppl how the event is going and what they do with Drupal.
7. Get Swag
   • Walk around the sponsor’s booths and look for swag. These sponsors often DO NOT want to take that stuff back to the office. Sometimes you find some pretty useful things like shirts, pens, thumb drives, fold-up cloth flying disks, hackysacks, yo-yos, puzzles, keychains, etc.
8. Talk To The Sponsors
   • I’ve never seen a sponsor bite or hard-sell a passerby at their booth! :-)
   • You may be amazed at what you will learn by reading the signs, looking at any literature on their table, and actually talking to the representative. 
9. Fill Out Any Feedback Forms
   • Not ever event has feedback forms, but more and more are using them.
   • Forms may be available per-class, and for the event in general.
   • The organizers REALLY appreciate ALL comments.
   • As you might expect, the negative ones get more attention, so don’t hold back about the audio/video comments, or the need for more beginner topics, or how difficult it was to get to the venue from the airport, etc.
   • They really want to hear this!
10. THANK The Organizers!
    • If you know their faces, be sure to thank them personally for their hard work organizing the speakers, the facilities, the meals, the WiFi, etc.
    • Be sure to tweet and post about it as well when you leave.

Drupal Planet

View the discussion thread.

I've mentored a number of students in 2013 and 2014 for Debian and Ganglia and most of the companies I've worked with have run internships and graduate programs from time to time. GSoC 2014 has just finished and with all the excitement, many students are already asking what they can do to prepare and become selected in 2015.

My own observation is that the more time the organization has to get to know the student, the more confident they can be selecting that student. Furthermore, the more time that the student has spent getting to know the free software community, the more easily they can complete GSoC.

Here I present a list of things that students can do to maximize their chance of selection and career opportunities at the same time. These tips are useful for people applying for GSoC itself and related programs such as GNOME's Outreach Program for Women or graduate placements in companies.

Disclaimers

There is no guarantee that Google will run the program again in 2015 or any future year.

There is no guarantee that any organization or mentor (including myself) will be involved until the official list of organizations is published by Google.

Do not follow the advice of web sites that invite you to send pizza or anything else of value to prospective mentors.

Following the steps in this page doesn't guarantee selection. That said, people who do follow these steps are much more likely to be considered and interviewed than somebody who hasn't done any of the things in this list.

Understand what free software really is

You may hear terms like free software and open source software used interchangeably.

They don't mean exactly the same thing and many people use the term free software for the wrong things. Not all open source projects meet the definition of free software. Those that don't, usually as a result of deficiencies in their licenses, are fundamentally incompatible with the majority of software that does use approved licenses.

Google Summer of Code is about both writing and publishing your code and it is also about community. It is fundamental that you know the basics of licensing and how to choose a free license that empowers the community to collaborate on your code.

Please read up on this topic early on and come back and review this from time to time. The The GNU Project / Free Software Foundation have excellent resources to help you understand what a free software license is and how it works to maximize community collaboration.

Don't look for shortcuts

There is no shortcut to GSoC selection and there is no shortcut to GSoC completion.

The student stipend (USD $5,500 in 2014) is not paid to students unless they complete a minimum amount of valid code. This means that even if a student did find some shortcut to selection, it is unlikely they would be paid without completing meaningful work.

If you are the right candidate for GSoC, you will not need a shortcut anyway. Are you the sort of person who can't leave a coding problem until you really feel it is fixed, even if you keep going all night? Have you ever woken up in the night with a dream about writing code still in your head? Do you become irritated by tedious or repetitive tasks and often think of ways to write code to eliminate such tasks? Does your family get cross with you because you take your laptop to Christmas dinner or some other significant occasion and start coding? If some of these statements summarize the way you think or feel you are probably a natural fit for GSoC.

An opportunity money can't buy

The GSoC stipend will not make you rich. It is intended to make sure you have enough money to survive through the summer and focus on your project. Professional developers make this much money in a week in leading business centers like New York, London and Singapore. When you get to that stage in 3-5 years, you will not even remember exactly how much you made during internships.

GSoC gives you an edge over other internships because it involves publicly promoting your work. Many companies still try to hide the potential of their best recruits for fear they will be poached or that they will be able to demand higher salaries. Everything you complete in GSoC is intended to be published and you get full credit for it. Imagine an amateur musician getting the opportunity to perform on the main stage at a rock festival. This is how the free software community works.

Having a portfolio of free software that you have created or collaborated on and a wide network of professional contacts that you develop before, during and after GSoC will continue to pay you back for years. While other graduates are being screened through group interviews and testing days run by employers, people with a track record in a free software project often find they go straight to the final interview round.

Register your domain name and make a permanent email address

Free software is all about community and collaboration. Register your own domain name as this will become a focal point for your work and for people to get to know you as you become part of the community.

This is sound advice for anybody working in IT, not just programmers. It gives the impression that you are confident and have a long term interest in a technology career.

Choosing the provider: as a minimum, you want a provider that offers DNS management, static web site hosting, email forwarding and XMPP services all linked to your domain. You do not need to choose the provider that is linked to your internet connection at home and that is often not the best choice anyway. The XMPP foundation maintains a list of providers known to support XMPP.

Create an email address within your domain name. The most basic domain hosting providers will let you forward the email address to a webmail or university email account of your choice. Configure your webmail to send replies using your personalized email address in the From header.

Update your ~/.gitconfig file to use your personalized email address in your Git commits.

Create a web site and blog

Start writing a blog. Host it using your domain name.

Some people blog every day, other people just blog once every two or three months.

Create links from your web site to your other profiles, such as a Github profile page. This helps re-inforce the pages/profiles that are genuinely related to you and avoid confusion with the pages of other developers.

Many mentors are keen to see their students writing a weekly report on a blog during GSoC so starting a blog now gives you a head start. Mentors look at blogs during the selection process to try and gain insight into which topics a student is most suitable for.

Create a profile on Github

Github is one of the most widely used software development web sites. Github makes it quick and easy for you to publish your work and collaborate on the work of other people. Create an account today and get in the habbit of forking other projects, improving them, committing your changes and pushing the work back into your Github account.

Github will quickly build a profile of your commits and this allows mentors to see and understand your interests and your strengths.

In your Github profile, add a link to your web site/blog and make sure the email address you are using for Git commits (in the ~/.gitconfig file) is based on your personal domain.

Start using PGP

Pretty Good Privacy (PGP) is the industry standard in protecting your identity online. All serious free software projects use PGP to sign tags in Git, to sign official emails and to sign official release files.

The most common way to start using PGP is with the GnuPG (GNU Privacy Guard) utility. It is installed by the package manager on most Linux systems.

When you create your own PGP key, use the email address involving your domain name. This is the most permanent and stable solution.

Print your key fingerprint using the gpg-key2ps command, it is in the signing-party package on most Linux systems. Keep copies of the fingerprint slips with you.

This is what my own PGP fingerprint slip looks like. You can also print the key fingerprint on a business card for a more professional look.

Using PGP, it is recommend that you sign any important messages you send but you do not have to encrypt the messages you send, especially if some of the people you send messages to (like family and friends) do not yet have the PGP software to decrypt them.

If using the Thunderbird (Icedove) email client from Mozilla, you can easily send signed messages and validate the messages you receive using the Enigmail plugin.

Get your PGP key signed

Once you have a PGP key, you will need to find other developers to sign it. For people I mentor personally in GSoC, I'm keen to see that you try and find another Debian Developer in your area to sign your key as early as possible.

Free software events

Try and find all the free software events in your area in the months between now and the end of the next Google Summer of Code season. Aim to attend at least two of them before GSoC.

Look closely at the schedules and find out about the individual speakers, the companies and the free software projects that are participating. For events that span more than one day, find out about the dinners, pub nights and other social parts of the event.

Try and identify people who will attend the event who have been GSoC mentors or who intend to be. Contact them before the event, if you are keen to work on something in their domain they may be able to make time to discuss it with you in person.

Take your PGP fingerprint slips. Even if you don't participate in a formal key-signing party at the event, you will still find some developers to sign your PGP key individually. You must take a photo ID document (such as your passport) for the other developer to check the name on your fingerprint but you do not give them a copy of the ID document.

Events come in all shapes and sizes. FOSDEM is an example of one of the bigger events in Europe, linux.conf.au is a similarly large event in Australia. There are many, many more local events such as the Debian France mini-DebConf in Lyon, 2015. Many events are either free or free for students but please check carefully if there is a requirement to register before attending.

On your blog, discuss which events you are attending and which sessions interest you. Write a blog during or after the event too, including photos.

Quantcast generously hosted the Ganglia community meeting in San Francisco, October 2013. We had a wild time in their offices with mini-scooters, burgers, beers and the Ganglia book. That's me on the pink mini-scooter and Bernard Li, one of the other Ganglia GSoC 2014 admins is on the right.

Install Linux

GSoC is fundamentally about free software. Linux is to free software what a tree is to the forest. Using Linux every day on your personal computer dramatically increases your ability to interact with the free software community and increases the number of potential GSoC projects that you can participate in.

This is not to say that people using Mac OS or Windows are unwelcome. I have worked with some great developers who were not Linux users. Linux gives you an edge though and the best time to gain that edge is now, while you are a student and well before you apply for GSoC.

If you must run Windows for some applications used in your course, it will run just fine in a virtual machine using Virtual Box, a free software solution for desktop virtualization. Use Linux as the primary operating system.

Here are links to download ISO DVD (and CD) images for some of the main Linux distributions:

• Debian
• Fedora
• Ubuntu

If you are nervous about getting started with Linux, install it on a spare PC or in a virtual machine before you install it on your main PC or laptop. Linux is much less demanding on the hardware than Windows so you can easily run it on a machine that is 5-10 years old. Having just 4GB of RAM and 20GB of hard disk is usually more than enough for a basic graphical desktop environment although having better hardware makes it faster.

Your experiences installing and running Linux, especially if it requires some special effort to make it work with some of your hardware, make interesting topics for your blog.

Decide which technologies you know best

Personally, I have mentored students working with C, C++, Java, Python and JavaScript/HTML5.

In a GSoC program, you will typically do most of your work in just one of these languages.

From the outset, decide which language you will focus on and do everything you can to improve your competence with that language. For example, if you have already used Java in most of your course, plan on using Java in GSoC and make sure you read Effective Java (2nd Edition) by Joshua Bloch.

Decide which themes appeal to you

Find a topic that has long-term appeal for you. Maybe the topic relates to your course or maybe you already know what type of company you would like to work in.

Here is a list of some topics and some of the relevant software projects:

• System administration, servers and networking: consider projects involving monitoring, automation, packaging. Ganglia is a great community to get involved with and you will encounter the Ganglia software in many large companies and academic/research networks. Contributing to a Linux distribution like Debian or Fedora packaging is another great way to get into system administration.
• Desktop and user interface: consider projects involving window managers and desktop tools or adding to the user interface of just about any other software.
• Big data and data science: this can apply to just about any other theme. For example, data science techniques are frequently used now to improve system administration.
• Business and accounting: consider accounting, CRM and ERP software.
• Finance and trading: consider projects like R, market data software like OpenMAMA and connectivity software (Apache Camel)
• Real-time communication (RTC), VoIP, webcam and chat: look at the JSCommunicator or the Jitsi project
• Web (JavaScript, HTML5): look at the JSCommunicator

Before the GSoC application process begins, you should aim to learn as much as possible about the theme you prefer and also gain practical experience using the software relating to that theme. For example, if you are attracted to the business and accounting theme, install the PostBooks suite and get to know it. Maybe you know somebody who runs a small business: help them to upgrade to PostBooks and use it to prepare some reports.

Make something

Make some small project, less than two week's work, to demonstrate your skills. It is important to make something that somebody will use for a practical purpose, this will help you gain experience communicating with other users through Github.

For an example, see the servlet Juliana Louback created for fixing phone numbers in December 2013. It has since been used as part of the Lumicall web site and Juliana was selected for a GSoC 2014 project with Debian.

There is no better way to demonstrate to a prospective mentor that you are ready for GSoC than by completing and publishing some small project like this yourself. If you don't have any immediate project ideas, many developers will also be able to give you tips on small projects like this that you can attempt, just come and ask us on one of the mailing lists.

Ideally, the project will be something that you would use anyway even if you do not end up participating in GSoC. Such projects are the most motivating and rewarding and usually end up becoming an example of your best work. To continue the example of somebody with a preference for business and accounting software, a small project you might create is a plugin or extension for PostBooks.

Getting to know prospective mentors

Many web sites provide useful information about the developers who contribute to free software projects. Some of these developers may be willing to be a GSoC mentor.

For example, look through some of the following:

• Planet / Blog aggregation sites: these sites all have links to the blogs of many developers. They are useful sources of information about events and also finding out who works on what.
  • planet.debian.org
  • planet.fedoraproject.org
  • planet.ubuntu.com
  • planet.drupal.org
  • planet.postgresql.org
• Developer profile pages. Many projects publish a page about each developer and the packages, modules or other components he/she is responsible for. Look through these lists for areas of mutual interest.
  • my package list in Debian
  • my package list in Fedora
• Developer github profiles. Github makes it easy to see what projects a developer has contributed to. To see many of my own projects, browse through the history at my own Github profile

Getting on the mentor's shortlist

Once you have identified projects that are interesting to you and developers who work on those projects, it is important to get yourself on the developer's shortlist.

Basically, the shortlist is a list of all students who the developer believes can complete the project. If I feel that a student is unlikely to complete a project or if I don't have enough information to judge a student's probability of success, that student will not be on my shortlist.

If I don't have any student on my shortlist, then a project will not go ahead at all. If there are multiple students on the shortlist, then I will be looking more closely at each of them to try and work out who is the best match.

One way to get a developer's attention is to look at bug reports they have created. Github makes it easy to see complaints or bug reports they have made about their own projects or other projects they depend on. Another way to do this is to search through their code for strings like FIXME and TODO. Projects with standalone bug trackers like the Debian bug tracker also provide an easy way to search for bug reports that a specific person has created or commented on.

Once you find some relevant bug reports, email the developer. Ask if anybody else is working on those issues. Try and start with an issue that is particularly easy and where the solution is interesting for you. This will help you learn to compile and test the program before you try to fix any more complicated bugs. It may even be something you can work on as part of your academic program.

Find successful projects from the previous year

Contact organizations and ask them which GSoC projects were most successful. In many organizations, you can find the past students' project plans and their final reports published on the web. Read through the plans submitted by the students who were chosen. Then read through the final reports by the same students and see how they compare to the original plans.

Start building your project proposal now

Don't wait for the application period to begin. Start writing a project proposal now.

When writing a proposal, it is important to include several things:

• Think big: what is the goal at the end of the project? Does your work help the greater good in some way, such as increasing the market share of Linux on the desktop?
• Details: what are specific challenges? What tools will you use?
• Time management: what will you do each week? Are there weeks where you will not work on GSoC due to vacation or other events? These things are permitted but they must be in your plan if you know them in advance. If an accident or death in the family cut a week out of your GSoC project, which work would you skip and would your project still be useful without that? Having two weeks of flexible time in your plan makes it more resilient against interruptions.
• Communication: are you on mailing lists, IRC and XMPP chat? Will you make a weekly report on your blog?
• Users: who will benefit from your work?
• Testing: who will test and validate your work throughout the project? Ideally, this should involve more than just the mentor.

If your project plan is good enough, could you put it on Kickstarter or another crowdfunding site? This is a good test of whether or not a project is going to be supported by a GSoC mentor.

Learn about packaging and distributing software

Packaging is a vital part of the free software lifecycle. It is very easy to upload a project to Github but it takes more effort to have it become an official package in systems like Debian, Fedora and Ubuntu.

Packaging and the communities around Linux distributions help you reach out to users of your software and get valuable feedback and new contributors. This boosts the impact of your work.

To start with, you may want to help the maintainer of an existing package. Debian packaging teams are existing communities that work in a team and welcome new contributors. The Debian Mentors initiative is another great starting place. In the Fedora world, the place to start may be in one of the Special Interest Groups (SIGs).

Think from the mentor's perspective

After the application deadline, mentors have just 2 or 3 weeks to choose the students. This is actually not a lot of time to be certain if a particular student is capable of completing a project. If the student has a published history of free software activity, the mentor feels a lot more confident about choosing the student.

Some mentors have more than one good student while other mentors receive no applications from capable students. In this situation, it is very common for mentors to send each other details of students who may be suitable. Once again, if a student has a good Github profile and a blog, it is much easier for mentors to try and match that student with another project.

Conclusion

Getting into the world of software engineering is much like joining any other profession or even joining a new hobby or sporting activity. If you run, you probably have various types of shoe and a running watch and you may even spend a couple of nights at the track each week. If you enjoy playing a musical instrument, you probably have a collection of sheet music, accessories for your instrument and you may even aspire to build a recording studio in your garage (or you probably know somebody else who already did that).

The things listed on this page will not just help you walk the walk and talk the talk of a software developer, they will put you on a track to being one of the leaders. If you look over the profiles of other software developers on the Internet, you will find they are doing most of the things on this page already. Even if you are not selected for GSoC at all or decide not to apply, working through the steps on this page will help you clarify your own ideas about your career and help you make new friends in the software engineering community.

These are some metrics based on the extensive performance tuning that I've had to do as part of the ELMS Learning Network project.  Trying to do a ton of systems on limited resources can be challenging but fortunately there are tons of experts in and outside the drupal community to help make this hapen.

Ever since the Drupal community started holding national regional conferences, dubbed DrupalCons and DrupalCamps, there has been a perception that design, usability and “theming” (i.e., managing HTML, CSS and JS output) were afterthoughts. And, to be honest, that has been true for many camps and conferences – while developers were geeking out on advanced Views API usage and performance tuning, those looking for a design focus were often left with introductory theming sessions.

I had the opportunity to play with Search API filters to modify Solr searches lately as we had to implement a complex set of filtering rules for a Drupal project. This becomes necessary because Views filters don’t easily support our conditions, and you really don’t want to rely on the node access system because your goal is to alter the result set instead of forbidding access altogether. Using SearchApiQuery can be tricky at first, but with practice, one can get the hand of using it effectively.

First, the easiest way to modify these searches is to implement hook_search_api_query_alter() in one of your modules. Since we use Features heavily, we usually have a search focused Features module, and write any related code within the appropriate feature’s .module file. This hook has one parameter, which is the SearchApiQuery object itself.

/** * Implements hook_search_api_query_alter(). */ function my_module_search_api_query_alter($query) { }

Remember: we don’t need to pass objects by reference, since objects are always passed by reference, this means we also don’t need to return anything from this hook implementation. The SearchApiQuery object provides several operations (presented in its documentation) to modify its search query, but I’m just going to focus on two for today.

$filter = $query->createFilter($conjunction);

…where $conjunction is a string with a value of either an AND or OR condition. This creates a new SearchApiQueryFilter object that can be added back to the $query later, after we’ve added conditions to it.

$query->filter($filter);

Once we have a filter object, we can add conditions, or other filters to it:

$filter->condition('field_name', 'value'); $filter->filter($filter);

As any programmer knows, nesting conditions can be simplified into its basic parts. So the following statement

if ($this == $that && ($here > $there || $foo == $bar))

…can be broken down into five evaluations (working backwards allows us to resolve the child conditions before the parents):

1. $foo == $bar

2. $here > $there

3. Result of evaluation 1 || Result of evaluation 2

4. $this == $that

5. Result of evaluation 4 && Result of evaluation 3

Working with Search API filters is no different than constructing the above nested If statement.

First, we want to create a base filter for our evaluation. Here I am using an example that the node should be published, but this would typically be already added by your view or use of node access integration by checking the “Node access” option sat admin/config/search/search_api/index/[index name]/workflow.

$base_filter = $query->createFilter('AND'); $base_filter->condition('status', 1);

We use AND since that is what we outlined. To create additional sub filters, we draw from the same query object.

$subfilter = $query->createFilter('OR');

Conditions a straightforward. Just pass the field and the value to filter by. This defaults to == but you do get additional comparisons for integer values. Refer to the API link above for a complete list of options.

$subfilter->condition('field_my_field', 'value'); $subfilter->condition('field_other_field', 'value2');

Now to add this sub filter to the base filter, just pass it to filter().

$base_filter->filter($subfilter);

and then add the base filter back to the query object,

$query->filter($base_filter);

That’s all there is to working with the SearchApiQuery interface. The complete example looks something like this:

/** * Implements hook_search_api_query_alter(). */ function my_module_search_api_query_alter($query) { $base_filter = $query->createFilter('AND'); $base_filter->condition('status', 1); $subfilter = $query->createFilter('OR'); $subfilter->condition('field_my_field', 'value'); $subfilter->condition('field_other_field', 'value2'); $base_filter->filter($subfilter); $query->filter($base_filter); }

Implementing your filters this way should really be a last resort. The other option that should be considered first is the node access system, which can define node-level permissions that can get indexed in services like Solr.

When we launched the new version of ThinkShout.com last spring, something glaring was missing. That little something is what companies like Pantheon and Acquia have worked so hard to solve for more complex Drupal sites, namely a deployment workflow making it dead simple to deploy changes to your site and preview them before publishing to a production server. At the time of launch, we had some rudimentary tools in place, namely a set of Rake tasks to build the site and deploy to separate staging and production environments.

This worked fine for the uber geeks among us, who had a full ruby stack running and were proficient using git and running terminal commands. But for those less technically inclined, not so good. Not to mention, the lack of automation meant lots of room for errror. The talented team at Development Seed created Jekyll hook, a node based app that listens for notifications from GitHub then builds and deploys the site based on a number of configuration options or customizations to the build script. That seemed like a good solution, and we even started work on our own fork of the project. It was moving along nicely, and we had it running on Heroku, which largely eliminated the need for maintaining a server. Our customizations allowed us to deploy to S3 using the powerful s3_website gem and deploy to different buckets depending on the branch being committed. Still, this solution required a good deal more care and maintenance than a typical site hosted on Pantheon or Aquia and lacked any built in visual status or notifications.

Around the same time, I received a great tip while attending CapitalCamp: use Travis CI to test, build, and deploy the site. This was such a such a simple and great idea that I couldn't help but slap myself on the head for not thinking of it sooner. Travis is one of the leading continuous integration platforms with tight GitHub integration. It's free for open source projects and charges a modest monthly fee for private ones. It's also dead simple to configure, comes with loads of built in features, and requires little to no ongoing maintenance. While I knew of Travis, you can't help but see those nice "build passing" images on all your favorite open source projects, ![Build Passing](https://api.travis-ci.org/travis-ci/travis-web.svg?branch=master, I didn't realize just how powerful it can be. Some highlights include:

• Supports all the major platforms including PHP, Ruby, Node, Python, and Java.
• Lots of major databases and service are avaialble, E.g., MySql, PostgreSQL, Redis, Memcache, etc.
• Has built in notifications via email, IRC, and other popular services
• Can run your test suites and report back the status
• Built in deployment to a number of platforms such as Heroku and Amazon, in addition to your own server.

The secret to all this Travis magic lies in a .travis.yml file located in the project root. For ThinkShout.com, it looks something like this:

language: ruby rvm: 2.0.0 script: "./_scripts/travis_build.sh" branches: only: - master - live env: global: - secure: ... - secure: ... notifications: hipchat: rooms: secure: ...

I won't go through this line by line, there's great documentation for that, but basically this tells Travis:

• That we need an environment running Ruby 2.0.0.
• To execute ./_scripts/travis_build.sh for our build.
• To only trigger the build on the master and live branches.
• Triggers a notification in our HipChat project room.

The build scrip is very simple as well:

#!/bin/bash if [[ $TRAVIS_BRANCH == 'master' ]] ; then bundle exec rake stage elif [[ $TRAVIS_BRANCH == 'live' ]] ; then bundle exec rake publish else echo 'Invalid branch. You can only deploy from master and live.' exit 1 fi

While we could put script commands directly into .travis.yml, having a bash script affords us some additional flexibility, in our case to deploy to different S3 buckets based on the branch being committed to.

When all is said and done, we have a simple automated deployment workflow, as illustrate below

Now the ThinkShout.com deployment workflow goes something like this:

1. Make a commit to the master branch. This can be done directly in GitHub, using Prose.io, or the old fashioned way in your own working copy. Note that new features are done in feature branches, which do not trigger a build, and are eventually merged into master for review.
2. The changes are pushed to our staging site for review within a couple minutes.
3. When everything looks good, a pull request is opened comparing master to live.
4. After any final discussions are complete, the pull request is merged and the code is pushed to the production S3 bucket.

That's it, done. No Ruby stack, no Jekyll build or compass compile, no worrying about S3 access keys. We're excited to refine this workflow further, including adding automated tests using PhantomJS, and put it to a real test for an upcoming site launch for a client. Stay tuned!

In this episode the Drupalize.Me team talks about the new video experience we have on the site. We start off by explaining why we made the changes, and what changes we made. We completely updated both the video player and the video pages, and the entire video delivery system, which gave us the ability to offer new features, like toggling playback speed. Most of the episode focuses on the timeline and process we used to tackle this monumental change. Join us to get a glimpse behind the scenes at Drupalize.Me.

• Customize Panels Styles with CTools Style Plugins
• InVision App

Lewis Nyman has been discussing the difficulties of being a DrupalCon track chair and how we can make it easier for session submitters to collect their speaking history using a service like joind.in or Lanyrd.

A trial page has been set up for DrupalCon Amsterdam on joind.in, you can sign up to attend and test the experience here - https://joind.in/event/view/2355

Get involved in the discussion on groups.drupal.org.

Is your Content Management System (CMS) really working for you? In this eBook, Solutions Architect Paul McKibben takes a lighthearted look at common issues you might find in an aging CMS:

We had a pretty good turnout here at Pantheon HQ for a Headless Drupal themed SF Drupal User’s Group:

Good crowd at SFDUG for #headlessDrupal pic.twitter.com/55miCJdzSl

— Josh Koenig (@outlandishjosh) August 19, 2014

The excitement is clear. So what's next?

Put Your Code Where Your Mouth Is

In any place where there's much excitement, there also tends to be a lot of discussion about what could or should be done. My suggestion is that we focus on creating real-world implementations with various JS frameworks and other API consuming clients.

To that end, I'm putting my code where my mouth is. Here's a working demo site:

I've placed the rough demo code online at GitHub, so you can set up your own. Feel free to experiment with different implementations, or help out with the listed TODOs. Pull requests are welcome!

Next Steps

My ultimate vision is that we have a commonly accepted goal for demonstration implementations, and we find people who want to work on them for a number of popular JS MVC frameworks. It would also be interesting to have people do straight API implementations in raw Python or Ruby, or even just using Curl.

The end result would be something similar to TodoMVC, which helps people evaluate and embrace different JS frameworks. Having a repository of implementations would definitely speed up the process of attracting front-end developers to think about Drupal as a back-end for future projects.

At the same time, by focusing on practical/working implementations, we can much more effectively provide input and guidance to the continued core (and contrib) development. There's no need for Drupal engineers to build Headless in a vacuum.

Stay tuned for more updates as we see the outcomes of the community response. Based on what I've seen online and at the SF DUG meetup, I think some really exciting energy will come out of this.

Blog Categories: Engineering Tweet

Drupal has included basic site search functionality since its first public release. Search administration was added in Drupal 2.0.0 in 2001, and search quality, relevance, and customization was improved dramatically throughout the Drupal 4.x series, especially in Drupal 4.7.0. Drupal's built-in search provides decent database-backed search, but offers a minimal set of features, and slows down dramatically as the size of a Drupal site grows beyond thousands of nodes.

In the mid-2000s, when most custom search solutions were relatively niche products, and the Google Search Appliance dominated the field of large-scale custom search, Yonik Seeley started working on Solr for CNet Networks. Solr was designed to work with Lucene, and offered fast indexing, extremely fast search, and as time went on, other helpful features like distributed search and geospatial search. Once the project was open-sourced and released under the Apache Software Foundation's umbrella in 2006, the search engine became one of the most popular engines for customized and more performant site search.

As an aside, I am writing this series of blog posts from the perspective of a Drupal developer who has worked with large-scale, highly customized Solr search for Mercy (example), and with a variety of small-to-medium sites who are using Hosted Apache Solr, a service I've been running as part of Midwestern Mac since early 2011.

Timeline of Apache Solr and Drupal Solr Integration

If you can't view the timeline, please click through and read this article on Midwestern Mac's website directly.

A brief history of Apache Solr Search and Search API Solr

Only two years after Apache Solr was released, the first module that integrated Solr with Drupal, Apache Solr Search, was created. Originally, the module was written for Drupal 5.x, but it has been actively maintained for many years and was ported to Drupal 6 and 7, with some relatively major rewrites and modifications to keep the module up to date, easy to use, and integrated with all of Apache Solr's new features over time. As Solr gained popularity, many Drupal sites started switching from using core search or the Views module to using Apache Solr.

As part of a client project we recently had the requirement of displaying an image field's default image in the node add form, before the user had uploaded a picture. This seemed simple enough to implement but thinking about it I realized this was quite a generic feature request and that such a module might prove useful for others. Rather by accident I even noticed that Drupal.org itself employs the same feature on user profile pages.

Thus, I hereby present the Image widget default image module, which does just that: If you have provided a default image for an image field and the user has not yet uploaded an image for that field, it displays that default image as a preview. Not many bells or whistle, but it works. (I can claim that, because it comes with tests. ) If it does not work for you or if you have any questions please leave a comment or open an issue on Drupal.org.

Weitere Bilder: 

Episode Number: 163

In part 2 of the multiple views series you will learn how to add the jQuery needed to switch between multiple classes. By having the ability to use multiple classes, we will (in part 3) be able to use CSS to change the look and feel of the same view.

Here is the jQuery code used to switch between grid and list view:

Tags: DrupalViewsDrupal 7Theme DevelopmentDrupal PlanetJavascriptJQuery

Over 30 people attended this wildly successful training at DrupalCon Austin. Now is your chance to attend this training at DrupalCon Amsterdam!

In this course, we cut past the evangelism that exists around Agile, and instead focus on real-world practical training that you can put into action.

The course is delivered using the Agile Scrum techniques it teaches. At the start, delegates see the backlog of requirements that the product owner (a trainer) has developed for the course, and the prioritization of those requirements. The course then progresses in one-hour periods of work called sprints, working through training modules from the top of the backlog.

Part way through the morning, delegates are ready to take over as the product owners. They will take responsibility for specifying the requirements for the course, based on the needs and interests of the delegates in the room, re-prioritise them, and even add completely new requirements. Our trainers will respond to these changes by creating new training modules on the fly based on real project experience, to provide the highest possible value to the delegates.

Through this approach we demonstrate and explain the processes of Agile many times, and we also demonstrate their value, and delegates leave with a real insight into how they could apply agile, and handle some of the challenges they may have faced.

The trainers are highly experienced Agile coaches, who mentor teams at Wunderkraut (known as WunderRoot in the UK), as well as consulting with large clients about ensuring successful delivery of their projects.

Meet the Trainers from Wunderkraut

Steve Parks (steveparks), UK Managing Director
Vesa Palmu (wesku), CEO
Roel De Meester (demeester_roel), CTO Benelux
Florian Huber (fuber), Project Manager, Scrum Master

Attend this Drupal Training

This training will be held on Monday, 29 September from 09:00-17:00 at the Amsterdam RAI during DrupalCon Amsterdam. The cost of attending this training is €400 and includes training materials, meals and coffee breaks. A DrupalCon ticket is not required to register to attend this event.

Our training courses are designed to be small enough to provide attendees plenty of one-on-one time with the instructor, but large enough that they are a good use of the instructor's time. Each training course must meet its minimum sign-up number by 5 September in order for the course to take place. You can help to ensure your training course takes place by registering before this date and asking friends and colleagues to attend.

Register today

This year’s Twin Cities DrupalCamp had no shortage of new faces, quality sessions, trainings, and after parties. Most of my time was spent in frontend sessions and talking with folks. Being that I live in Minneapolis, this camp is especially rewarding from a hometown Drupal represent kind of perspective. Below are some of my favorite sessions and camp highlights.

Community Drupal

More and more often I am asked, when putting together a design Drupal for a website, what is the importance of designing above the fold and whether or not that today’s users will scroll to read content.

In my experience as an organization leader one of the most important tools in my toolbox has always been my personal values. It's been my experience that even when data points in one direction, and best practices say you should approach the problem in this way, it's always been my values that help me make the best decisions as an individual. And the truth is, there are very rarely any decisions that are 100% clear cut. In almost every decision, some amount of judegment is required. 

That's why I believe so strongly in defining values for the organizations I work for. When everyone is working from a shared sense of values, we're making decisions - even big giant judgement calls - from the same perspective. To that end, we spent some time this year working with the board and staff to develop a values statement for the Drupal Association.

We started in a board retreat, brainstorming the implicit (though not documented) values of the Association and the larger Drupal Community. The board ranked their favorites, and then we created a committee of board and staff to draft some language. Those initial values statements were vetted by both the entire Association staff and the full board, and then additional edits were made. Here's the result of that process:

The Drupal Association shares the values of our community, our staff, and open source projects:

• TEAMWORK: We add value to the Drupal community by helping each other solve problems to create quality human and digital experiences.
• COMMUNICATION: We value communication. We seek community participation. We are open and transparent.
• ACTION: We act decisively and proactively, embracing what we learn from both our successes and our mistakes.
• RESPECT: We respect and value inclusivity in our global community and strive to recognize, understand, and respond to its needs.
• FUN: We create environments that embrace humor resulting in fun, positive, supportive and safe interactions.

To be clear, these are the values we're defining for our staff. We're not trying to impose these values on the larger community. However, we do hope they reflect the values you feel are important in the larger Drupal community as well. We also want to recognize that writing down the words is one thing, and living up to them is something else. We intend to live these values in all our work. 

Now it's your turn. The values are set by the board and staff, but we want to make sure we know what you think.

Flickr photo: Howard Lake