CivicActions has been working with Google's “Make the Web Faster” project team to make some (last minute) improvements that make Drupal 7 faster.
What do we mean by “faster” - this word can have many meanings, but here we simply mean the response time for end users, also sometimes called front-end performance: after a user presses an appropriate mouse/keyboard button, how long does it take for pages to be generated, download and display for a user. This response time metric can also be applied to AJAX and other “in browser” interactions - however this is distinct from (for example) server resource usage, scalability or availability concerns.
Most work to improve end-user response times breaks down into several broad strategies:
- Reducing the time taken to generate/serve requests
- Reducing the number of requests needed to display the page
- Reducing the size of requests
- Reducing the time it takes for browsers to parse, render or interact with downloaded pages (not discussed here)
This post will give a brief overview of each topic, the state of Drupal 7, and some of the recent/ongoing work with some benchmarks of these changes. A summary of this is available as a guest post over on the Google Code blog.
Reducing the time taken to generate/serve requests
It is fairly hard in Drupal core to make radical, end-user scale improvements in the page generation time - assuming there aren't any huge blunders that everyone has missed. Of course this work is still very valuable in terms of scalability – a 5ms improvement won't be noticed by end-users (on it's own), but adds up quickly on a server dealing with millions of requests.
The exception to this is high level caching, such as page caching – this can often shave off a significant portion of the page generation time, especially for anonymous users. Drupal 7 has made some major improvements in both the native caching layer, such as easier pluggability (e.g. for memcache integration). Another change enables a site to be configured to deliver HTTP headers that allow anonymous users to cache entire pages locally (for most pages: when no PHP session is needed) as well as improve compatibility with reverse proxies and content distribution networks (CDNs) which can cache requests in multiple geographic locations.
Reducing the number of requests needed to display the page
Before a page can be rendered most of the page assets need to be downloaded – each HTTP request needs to travel from the browser to the server, and then back to the client. This time taken for this round trip is known as network latency – this is often in the order of 100-200 ms, occasionally less, but sometimes significantly more (mobile/3G connections can be more like 600 ms) and unlike bandwidth it is not showing signs of improving anytime soon. Some of this is due to the intrinsic speed of light, some of this is due to network delays, and some due to connection/parsing overheads. Browsers can request several page assets simultaneously (often 6 or so), but it is common to max-out the limit of simultaneous connections and cause queuing. The browser may also need to make several requests to determine a complete list of all the assets it needs. Given that it is not uncommon for unoptimized web pages to include many dozens of assets, the effect of latency on total page load times can be in the order of seconds.
The bottom line of all this is that you generally want to limit the number of HTTP requests per-page as much as possible. There are several ways of doing this:
Reduce the number of assets in your pages
In other words, just take away some page assets completely. Certainly something worth considering when working on the design of your sites. However, there is clearly some limits to this approach – there is a need to balance site performance with a clients need for branding, and the users need for usable sites that convey more than just text.
Allow clients to cache and expire assets locally
This one is really important – without proper cache headers with page assets many client/server combinations will check the validity of their cache for each individual asset (normally receiving a “304 Not Modified” response). While this saves the extra time needed to re-download the assets the latency for each of these requests is still present and can really slow down page loads.
Drupal's default .htaccess file includes rules to tell clients that all page assets should be expired only after 2 weeks (by default) – this means that clients can use these resources for this time without checking that the cached version is valid, which avoids these requests on subsequent pages. However, for these rules to function, mod_expires Apache module needs to be enabled. This should be enabled by default on most Red Hat based distributions, on Debian or Ubuntu you can enable it by running “sudo a2enmod expires” and reloading Apache as directed. For web servers other than Apache you should read the documentation to see how to enable cache expiration headers for these assets.
Aggregate multiple assets into a single request
While this worked quite well it had several places where it needed improvement.
Enable aggregation by default
One issue is that aggregation is disabled by default in the interface. This means that sites built by inexperienced Drupal users are much less likely to have aggregation enabled, even when the site is launched. When aggregation is disabled this can add anything from 0-1.5 seconds to page loads, depending on the number of CSS/JS requests and visitor connection speeds/latencies. This might seem simple to fix - just change the default - however it is not quite as easy as that. While we want all visitors to Drupal sites to have the best experience possible, we also need to ensure that Drupal is an accessible platform for brand new site builders and developers. It is quite normal when learning a new platform to spend time experimenting, and so it is useful for Drupal to stay as “hackable” as possible - for example, if you edit some CSS file you can see the effects without needing to dig in settings pages.
There is a simple solution that gets the best of both worlds: if we include the modification time (“mtime”) of each file in with the list of filenames we hash to generate the aggregate filename then we will make the aggregate file automatically refresh whenever a source file is edited. We could then have aggregation enabled by default - patch is in progress that does just this. One complexity that needs some consideration is that there is a slight decrease in the server side performance (a few ms) in checking all the files mtimes. While pretty minimal this is something that large, busy sites would want to disable, so an option to allow aggregation without checking mtimes is necessary
Prevent aggregates with duplicate contents
For example a user visits 3 pages:
- Page X sends an aggregate of files a, b, c and d
- Page Y sends an aggregate of files a, b, c and e
- Page Z sends an aggregate of files a, c, d and e
In this case the user has downloaded the code for file “a” 3 times, “b”, “c” and “e” twice over. Files “b”, “d” and “e” have been added conditionally. Not only is this a waste of bandwidth, but it breaks the native browser ability to cache things properly. This is not unlikely to occur, even with skilled core developers - some tests show that even in Drupal 7 core alone we were generating over 3MB (yes, megabytes!) of duplicated CSS/JS code in aggregate files across different page loads, and over 30 additional HTTP requests. Add a few contributed modules and this issue would be even larger.
To demonstrate an improved approach, lets say that file “b” and “d” are files that regular visitors would likely need over the course of their visit, and file “e” is actually a file than only administrators would ever need, or only appears in site pages that visitors would rarely use.
- Page X sends an aggregate of files a, b, c and d
- Page Y sends an aggregate of files a, b, c and d, file e sent individually
- Page Z sends an aggregate of files a, b, c and d, file e sent individually
The fix then involves several elements:
- Setting preprocess to default to FALSE - this is the safe default.
- Checking that all files that typical visitors would need (on a typical visit) are preprocessed, and added unconditionally - normally in the modules hook_init() implementation.
- Checking that all other files (for administrators or non-frequently visited used site pages) are not set to preprocess and are added conditionally - only when required for a specific page.
- Fix the documentation so that the usage of this is clear to module developers.
Consider CSS sprites
Reducing the size of requests
This can be done by reducing the amount of code, compressing the code using format specific compression, or using general purpose compression at the http level.
Front-end performance is a critical area for increasing the usability and effectiveness of web sites, and hence also the tools used to build the web, of which Drupal is now a major contender. One of the many statistics available on this topic shows that a 1-second delay in page load time equals 11% fewer page views, a 16% decrease in customer satisfaction, and 7% loss in conversions. Several of the patch benchmarks highlighted in this post each have the potential to avoid 1 second (or more) of page load delay.
Improving the front-end performance of Drupal can be a complex task, balancing different use cases and users - trying to find the best overall solution. Dozens of contributors have contributed to this effort - including: Khalid Baheyeldin (kbahey), Owen Barton, Alex Bronstein (effulgentsia), Mike Carper (mikeytown2), Daniel F. Kudwien (sun) (who also helped review this post) and David Rothstein.
Going forward, no doubt many performance optimizations will continue to appear and be refined in the contributed module/theme space, as well as site building best practices and documentation. Getting Drupal 7 to final release is of course critical to bring the above improvements to the wider web. In Drupal 8 we will hopefully see further improvements in the CSS/JS file aggregation system, increased high-level caching effectiveness and hopefully more tools to help site builders reduce file sizes. Whatever your skill set contributions are always welcome - get involved!
Owen Barton is Director of Engineering at CivicActions. He has been developing elegant solutions in Drupal for over 12 years and is widely credited with building one of the most reputable and experienced Drupal engineering teams on the planet.