Planet Drupal

What is Governance?

This is the first of a set of blog posts on governance written for the Future of Drupal Governance core conversation at Drupalcon Denver - hope to see you there. The series continues with How Do Open Source Communities Organize Themselves?, Drupal Governance and The Future of Drupal Governance. (presentation video), (Governance project)

What is "governance?" Governance just a fancy word encompassing all the things we do to organize ourselves, make plans and decisions together, get things done, and resolve conflicts. It can be very loose (as is most of our governance in the Drupal community) or very structured. The key, of course, is whether it accomplishes what we want it to accomplish in the contexts we need it.

Why do organizations (and countries, and families) need good governance?

  • Consistency: Governance can provide consistency of organizational behavior, so that when you take an action, you have a reasonable expectation what the result will be. In the context of a national government, a criminal must expect to be punished, and a hard-worker must expect to be rewarded. In the context of a family, a child must know that what is rewarded one day will be treated the same the next day. And in the context of an open source community, it means knowing that if you invest yourself in solving a problem that you have a reasonable expectation of a return on your investment.

  • Accomplishing shared objectives: When different people work together, they organize themselves somehow to accomplish shared objectives. The more effectively they do this, the better their outcomes. This again is good governance.

  • Conflict Resolution: Without some kind of governance, it can be pretty hard to authoritatively resolve conflicts. Of course, the governance can be in the form of shared values held by a community, but what happens when one or more of the community members in conflict haven't actually absorbed those shared values, or when those values don't actually address the conflict in question? In that case, we have to have some way to take some kind of step outside the conflict to an agreed-upon technique or authority, or it won't get solved.

The following posts in this series will be on governance in other open-source communities, the current Drupal community governance model, and why and how might we want to change Drupal's governance structure. But you don't have to wait! Feel free to start the conversation in the comments here.

For more on governance definitions see the in-depth Wikipedia article, but for more down-to-earth studies of open-source communities, read Jono Bacon's The Art of Community (free download at the bottom of that page). Chapter 8 is about governance, and chapter 9 is about conflict resolution. Please do not refer to the US ("insert your country here") political process for inspiration.

git clone --reference Considered Harmful

Just over a year ago I wrote a blog post explaining how to use git clone --reference to speed up git clones. That technique then went on to become a drush option (if $options['cache'] = TRUE and you're using git for drush downloads)

Well, I repent. git clone --reference should not be used IMO unless you really really understand three things:

  1. You really need to speed up clones and other fetches.
  2. You know that the cache directory will never be destroyed (.drush/cache/git in the case of drush managing the caching)
  3. You will never copy your working directory some place it cannot reach the cache directory (like scp'ing it to a remote machine)

Unless you actually know #2 and #3, you can be really hurt. I know, I've done it to myself several times now.

When you use git clone --reference, it doesn't actually copy git objects into your local repository; instead it references them in your "cache". So if somehow your cache directory is damaged (or deleted as I did again today) all your repositories are broken. I finally codified how to recover from this damage, so that will follow below.

First, here's what happens if you delete your cache directory:

$ git status
error: object directory /Users/rfay/.drush/cache/git/aes.git/objects does not exist; check .git/objects/info/alternates.
fatal: bad object HEAD
fatal: 'git status --porcelain' failed in submodule sites/all/modules/aes

Note that this is only the first error message. Every submodule will have this problem. What is happening is that .git/objects/info/alternates points to the cache directory.

I know of no way to recover from this (aside from re-cloning or otherwise rebuilding your working directory) unless you actually have access to the cache. In my case, I recovered the cache from backup (thank you Time Machine) and then learned how to remove the dependency on the cache. And of course, I'm going to turn git caching off in drush.

Detaching alternates (cached objects) from your git repositories

  1. Get the cache directory back where it belongs, from backup or whatever.
  2. For each repository
    • git repack -a
    • rm .git/objects/info/alternates

In a submodule situation, you can do this with an approach like this:

git repack -a
git submodule foreach git repack -a
find $(find . -name .git) -name alternates # Make sure this finds only the right files
find $(find . -name .git) -name alternates | xargs rm  # Remove the alternates files

NOW when you have verified that all your repositories are free of the evil cache, you can remove the cache directory and turn off drush caching.

But please, if you're using drush and the package_handler git_drupalorg... turn $options['cache'] = FALSE.

How Do Open Source Communities Govern Themselves?

This is the second in a series on governance and Drupal in preparation for the Drupalcon Denver Core Conversation on The Future of Drupal Governance. The initial article started off with What is Governance; Read all, Governance project on drupal.org, Drupalcon presentation video.

The fact that world-wide open source communities can organize themselves to collaborate is one of the great treasures of the world's recent history. If you're pessimistic about the world's future and about governance in general, you should really take some time to look at all these great open source projects and their incredible success. Drupal is an example right there with the Linux kernel project, Wikipedia, Debian, Ubuntu, KDE, and dozens of others. I think they give us great reason for optimism. The fact that communities like the Drupal community can thrive is a huge reason for optimism about the future!

The various communities thrive with a whole variety of different governance mechanisms. Some, like Drupal and KDE, have nearly no formal governance process, and rely completely on community norms and culture. Others, like Debian, have an extremely formalized governance process. And there are lots and lots of variants.

  • Drupal is a huge community with a Benevolent Dictator for Life (Dries Buytaert) who is exceptionally gentle, community-oriented, and generally non-dictatorial. With the exception of issues where he is directly involved, almost all governance is by a loosely defined consensus model, where involved parties seek to find middle ground. Although very often successful, this can also lead to a committee mentality where disagreeing parties can prevent anything ever being done, or can result in design-by-committee. As do most other projects, Drupal has a parallel funding organization, the Drupal Association, which deals with money and events. It has its own governance, but in general does not have anything to do with core or contrib development, or with community governance. Drupal has the Drupal Code of Conduct, which spells out hoped-for community behavior, but the DCOC has no teeth in terms of resolving problems or making decisions.
  • Debian (the project that produces the operating system and packaging system Ubuntu Linux is built on) is the most striking for its careful organization. They have a Social Contract, a Constitution, and a completely specified technique for electing leaders. Wikipedia short version. Although the Debian community of contributors is probably small compared to Drupal. they're exceptionally clear about what they're doing. You can go to their site and figure out in a moment who has influence in what area. I don't have personal experience with the Debian community (although I very much appreciate and benefit from the project), but I chatted at length with anarcat, a distinguished member of both Debian and Drupal, and he is very positive about Debian's governance and its future.
  • The KDE project is at the far opposite end of the spectrum from Debian. (KDE is one popular graphical UI for the Linux family of operating systems.) KDE has almost no governance - they don't even have a designated leader. They have just two documents, on Project Management and concerning their Code of Conduct, which like Drupal's is derived directly from the Ubuntu Code of Conduct. The one body they have is a Community Working Group, which is essentially a conflict resolution group for when arguments get out of hand. And it has no formal power, just the magic of people trying to resolve conflicts. KDE does have a "Money organization", the analogue of the Drupal Association, which handles events and the like. It seems that nearly every open source group has a parallel organization to manage money and events. But KDE, despite being a successful and established community, has no Benevolent Dictator, no voting, no constitution; essentially, it has only its culture: "If you are going to do the work, you get to decide how it will be done." KDE's entire governance structure fits into one short page.
  • Wikipedia, the world-famous encyclopedia of everything, has a Benevolent Dictator for Life (Jimmy Wales, the founder) but is perhaps most noted for its emphasis on conflict resolution. With articles on a zillion controversial topics and people, and with editing open to nearly anyone nearly any time, they have a highly refined view of what consensus is and spend an enormous amount of energy on it. Wikipedia/Wikimedia has something of a Constitution, And Wales, as the founder, has an impressive but limited array of powers.. However, the most striking thing to me about Wikipedia is its capability to resolve conflicts in its editorial process even when there are millions of articles to care for.
  • Ubuntu (probably the most visible Linux OS distribution) has both a Benevolent Dictator for Life (BDFL) in Mark Shuttleworth and a backing commercial entity in Canonical. However, it has a huge, structured, and thriving community of contributors. Although Shuttleworth has certain distinguishing powers, and although Canonical as its commercial backer has specific capabilities to influence the direction of the project, a huge community and an enormous amount of community organization underlies a very large project that is not just a fan club for Canonical. The Ubuntu Technical Board makes feature decisions and a Community Council is empowered with non-technical governance. Ubuntu's Code of Conduct has inspired many imitators, including KDE and Drupal, but unlike theirs it cites specific ways to resolve conflict, pointing into the significant governance structure.
  • Wordpress.org, a famous competitor to Drupal in the CMS space, may be quite similar to Drupal on the surface, but has a very different leadership and organization. It has a BDFL/founder, Matt Mullenweg, who is also the head of the most important Wordpress company, Automattic. It has an impressive worldwide corps of contributors, including committers who are not employed by Automattic. The most obvious differences between Wordpress and Drupal seem to be that Matt is very involved in decision making at a number of levels, and Wordpress has an explicit organization chart, explaining who has what authority and what commit privilege. (Contrary to popular belief, Automattic is not a fully controlling backer of Wordpress.org; Wordpress.org is an open-source project, with a vibrant community which includes lots of people outside Automattic.)
  • The Linux Kernel project is unique in that it has a BDFL and strong, centralized leadership. As I understand it, Linus Torvalds as the founder and BDFL sets the agenda and also appoints strong lieutenants, so most issues do not require his attention until they're ready for commit. Anybody can submit a patch to the mailing list, but getting it accepted is very much a matter of working with the chain of command.

It seems to me that the key external differentiators between these projects and their governance are:
* BDFL or strong, semi-permanent leader (Drupal, Ubuntu, Linux kernel, Wikipedia)
* Top-down leadership (Wordpress, LInux Kernel)
* Amount of governance structure (org chart, method for selecting leaders)
* Membership distinction for some contributors (Debian, Ubuntu, KDE)
* Backing commercial organization (Ubuntu, Wordpress.org)
* Participation style (open, like Drupal, or more limited, like Linux Kernel)
* Explicit governance (like Debian) or culture-only based governance (like KDE and Drupal, mostly)

The bottom line is there is certainly no one right model. The purpose of governance is to build and sustain the community so it can accomplish community objectives and attract and keep contributors. And governance is something that typically evolves over time.

I'm well aware that this simple (minded) comparison of open-source projects is incomplete and inaccurate and will gladly improve it based on your comments, so please help out that way in the comments.

Resources and Acknowledgments

  • Thanks to Lydia Pintscher (@Nightrose) of KDE who graciously took more than an hour of her time to explain the inner magic of KDE.
  • Thanks to anarcat, a distinguished member of both Debian and Drupal communities, who did the same explaining Debian on IRC. See also * Scientific study of Debian Governance
  • Jono Bacon's excellent The Art of Community explains at length in the chapter on governance how communities can and do organize, and goes into depth about how Ubuntu organizes itself. (free download at the bottom of that page).
  • Jane Wells' presentation on how decisions get made in the Wordpress project does away with many myths about Wordpress, but also makes clear that Wordpress has a more top-down leadership style than Drupal does.
  • David Eaves is well-known in the Drupal community and well-known for his thinking about governance, metrics, and community organization. Thanks to Dave for taking the time to chat about governance both good and bad.

Drupal's Governance

This is the third in a series on governance and Drupal in preparation for the Drupalcon Denver Core Conversation on The Future of Drupal Governance. The first article discussed What is Governance and the second How Do Open Source Communities Govern Themselves?. Read all, Governance project on drupal.org, Drupalcon presentation video.

Drupal's governance structure is so lightly defined that when I searched for Drupal governance the primary article I found was this one on drupalmyths.com denying that Drupal's governance is poorly defined. Certainly nothing showed up on drupal.org. Of course, these things can use words other than the formal word "governance", but there really isn't much written about this topic.

Here is my description of Drupal's governance. I am certainly interested in your reaction to it in the comments:

  • Dries as Benevolent Dictator has the right to lead core development and make final decisions about any number of issues, but mostly key core development decisions.
    • He is expected to separate his business interests from "what the community needs".
    • He is not expected to intervene in the vast majority of community issues.
  • Core, contrib, documentation, infrastructure disagreements are handled almost exclusively through a "Do-ocracy" and a "consensus" model. In other words, if you think something is important, you bring it up. If you're willing to work on it, you may have the power to make it happen. But you still have to step over the "consensus" barrier, which means that 1) you have to get somebody else to care about your issue in the first place and 2) you might have to use your persuasive skills to get people to agree about your method.
  • The Drupal Association handles money and Drupalcons. The DA does have formal and explicit governance structures, and takes these very seriously. However, except when it intervenes by choosing to fund a development or similar initiative in the community, it has little impact on the community's day-to-day governance, which is Do-ocracy and Consensus.

Great Things About Drupal's Current Governance

  • In a do-ocracy, you can get things done just because they need to be done. That gives a special pre-emptive power to nearly everyone.
  • It's worked for a long time, and we have a great community.

Problems with Drupal's Current Governance (and risks of not changing it)

  • We often lack a strategic focus in attacking problems (or features). Whatever someone wants to do and has the energy to do (and doesn't get vetoed in the "consensus" process) ends up being what gets done. These tasks unfortunately may not the most strategically important tasks for our community or the Drupal product.
  • Lack of predictability: Because there is not a clear decision-making structure, it's difficult to know whether a given initiative can be successful at any level, even if expert technical resources are applied in adequate quantity.
  • When consensus doesn't come easily, a number of negative outcomes may result:
    • Nothing gets done at all.
    • Feelings are hurt, potential contributors leave the field.
    • Getting something done takes vastly more energy than it should have.
    • The issue can take on the aspect of (in webchick's words) a smoking crater. Smoking craters of destruction are not good for our community. Any of us can list a pile of frustrating issues still mouldering under the smoke of bikeshedding.
  • We have no (almost) structure for properly resolving conflicts. Unlike many of our sister projects, we have absolutely no mechanism for mediating or escalating conflicts to a resolution point. The only thing we have is the big hammer of a Dries decision, which happens very rarely.
  • Prized contributors can waste time and energy on initiatives that never go anywhere or are vetoed by lack of consensus.
  • People we care about can walk away from the community** or from important areas of development just because there is too much heat there.

Risks Of Building More Governance Structures For Drupal

  • Increased formalization (one obvious path) might offend some community members for philosophical reasons.
  • Increased formalization might cause community members to become more passive ("They" are in charge of this, so I don't need to get involved.)
  • What we have has worked so far. Changing it might have unforeseen consequences.

I'm sure you can read between the lines: I think we should deliberately build more explicit governance structures into our community, and that we should do it without bikeshedding :-)

Again, I welcome your comments both here and at the Core Conversation Wednesday, 21 March, at 2:15pm in Denver.

Resources

The Future of Drupal Governance: Resources and Next Steps

Thanks to all of you for your enthusiastic and thoughtful participation in the Future of Drupal Governance core conversation at Drupalcon Denver. This post will provide resources for the conversation and attempt to summarize the ideas raised and actions taken at the core conversation, as well as to provide a path forward for us.

Resources

Review

The first part of the presentation restated the ideas that have been discussed in this blog series: what governance is, how Drupal and other open source communities do their governance, what the issues/problems with Drupal's governance are, and potential paths forward with Drupal's governance.

The key initiative discussed in the presentation was "How do we add more structure and effectiveness to Drupal's governance?" In other words, can we develop techniques and processes to deal with policy and decision-making at the micro and macro levels? For instance, can we:

  • Make sure that blocked issues (that are important) get unblocked?
  • Provide a way to do decision-making and conflict resolution at the technical and community levels (Ubuntu has committees for these two areas; even KDE has one for community issues.)
  • Make sure that important projects and changes get tackled even if there isn't full community consensus?
  • Make sure that effective policies are developed openly and presented explicitly rather than by being quietly implemented by fiat?
  • Provide a way for our community to adapt its governance going forward. How do we implement new policies or change our governance?
  • Guard and enhance our open community culture while doing these things?

There have already been some suggestions and conversation on these various issues in the Governance project issue queue.

Dries has stated clearly that he considers this a project worth working on, and would like to see us develop a process to deal with it. He would like to have us develop new processes and structure before Drupalcon Munich, which means we need to work on this carefully and probably have an in-person sprint to finalize a number of recommendations to the community by June.

Proposal

We're going to have to figure out a process and conversation to accomplish this. I propose a "Governance and Policy Steering Working Group". This would be a temporary working group, dissolved by Drupalcon Munich, charged with proposing improvements to the Drupal Community's governance and policy management. In my opinion, this should be composed mostly of trusted leaders within the Drupal community, but should also include a lesser-known contributor or two, and might include someone from outside the community. It would be best to have Dries active in this group, but at the very least we need his buy-in and oversight.

Proposed initial areas of endeavor:

  • As a pilot project, develop new techniques for improving the productivity of the core issue queue. (Possible techniques: resolve blocked issues, delegate authority to resolve, grant early approval to specific issues.)
  • Define authority structure for initiatives.
  • Define a method for creating, publishing, and maintaining policies in the Drupal Community.
  • Define a structure and process for technical and community conflict resolution.
  • Define a structure and process for changing and improving Drupal Community governance.

Sprint after Community Leadership Summit

The general idea would be for this group to be active in exploring these areas of endeavor through online activities from now through or July. In the summer, an on-location sprint would finalize proposals, most likely July 16-17 in Portland Oregon, just after the Community Leadership Summit and during OSCON.

Proposed mechanism for choosing members

This is such an important process, because we must have community buy-in, and we could lose it if this were to look like an insider coup against the community.

My suggestion:

  • Dries directly picks two members (and participates himself).
  • Community suggestions (including self-nominations) are considered in the Governance issue queue; webchick and rfay could choose three based on community input (and perhaps participate themselves).

Working style

Obviously, in keeping with Drupal traditions, this group should work openly. However, it's just not going to please everybody, and of course it's crucial that it be able to make some choices, even if they end up not pleasing everybody. It would be a tragedy for an initiative aimed at resolving decision-making capabilities to be killed by a bikeshed.

How to proceed

I opened Form a governance working group in the Governance project issue queue. Please follow up with specific suggestions there. If you have non-working-group reactions to this article, you can leave them as comments here, but please if you want to suggest members or heatedly discuss the very idea of this, proposal, do it over in the issue queue.

Why I Donated To BackdropCMS

I've been a rabid supporter of Drupal; Of course anybody who knows me has watched my enthusiasm wear thin for various reasons, including simple burnout mostly, but also a number of frustrations with lack of progress in the community, especially with simple things like solving the support vacuum. However, I do not have an agenda against Drupal in any way, and as a professional programmer, I wouldn't be surprised if the core changes being introduced in Drupal 8 are a step in the right direction for future maintainability and sustainability. (Disclaimer: I have mostly ignored all of the changes and have only a rudimentary understanding of them. I wore myself out on Drupal 7 and swore off Drupal 8.)

Technical debt: But in my personal life I've been using Drupal for almost a decade now. And like everybody else I was sucker enough to say "I know how to do that" when my friends, family or organizations I was a member of had a problem that required a website. Oops. So now I have probably a couple of dozen websites that I am the de facto maintainer for. Already I've been through major version updates for many of them. And for a simple brochure site it's sometimes not too difficult. But for a more complex site with lots of contrib modules, it can be daunting. And then for a site that has custom code it can be overwhelming. The site I've been supporting since 2005, Warmshowers.org, (still on D6) has nearly 40,000 members, serves a vital role, and has a budget of about $3000/year, mostly spent on hosting. So what am I going to do about that when D8 comes along? Even upgrading to D7 will take probably more than a couple hundred hours of work, lots of custom code. D8 is unfathomable.

The forkers of Backdrop have promised to slow down the rate of API change, and to be specifically attractive to small sites without a perpetual budget for rewrites. I need that very much, whether or not I invest in learning Drupal 8, which may be loads of fun. I really need a simpler upgrade path, and one that won't force me to redo everything every couple of years. If I help somebody with a website, I no longer want to have to say "Oh and by the way, you need to budget at least $5000 for an upgrade every couple of years." I really never want to say that again. I need this. Or I have to get help breaking my addiction to helping people with web communication problems.

Oh, but the real reason I donated is that Backdropcms.org says on the top of the page "Don't Panic" in large, friendly letters.

Tunneling HTTP to Debug or Develop an External Webservice Call

Summary: This post provides a way to develop/debug in a web environment where you don't have control of the calling party and your environment must be available on the global internet.

It can be difficult to debug/develop a handler for an incoming web service when you don't have control of the caller. For example, I was developing a handler for Mandrill's inbound webhooks, where Mandrill calls your site when an email comes in. So in this case

  • The receiving website must be accessible on the internet so that Mandrill can call it.
  • Mandrill alone is hitting the site. I could debug with an imitation POST, but then that wouldn't be the real thing would it?

There are a couple of ways to approach this:

  • Set up a full debugging environment on an internet-connected host (For example, set up a full dev environment possibly with GUI and PHPStorm for example, on an internet host. This is easy enough to do but kind of ugly.
  • Debug with the application on the server but your debugger local using dbgp. This is do-able and an alternate approach to what's described here. It's a bit more awkward, but great for when the issues or problems can't be recreated on your local machine. It probably also requires most of the strategies shown here, but tunneling the debugger connection instead of tunneling the actual HTTP request.
  • Use printf-style debugging, outputting logs of everything that happens. This works, but you have little control and it's really ugly and you don't have enough info about what's going on.

A nicer way with a full local debugger

I use the wonderful PHPStorm IDE with debugger (which they provide free to open-source projects like Drupal and Warmshowers.org!), but this technique should work for any IDE. You do have to already know how to set up your system for debugging.

What you need:

  • A host on the internet. Any machine which is reachable directly on the internet. My instructions will explain how to do it with a linux machine. This really can be any machine, it's going to be used only to proxy a port to your local machine. You could even use a production machine without doing any damage. You do have to have control of the sshd configuration (/etc/ssh/sshd_config or equivalent). And you have to make sure that no firewall is blocking the port you choose to use (my example uses port 8888.)
  • Your local IDE/project set up for debugging with Xdebug. (I'm sure you can do this with any other IDE and with Zend Debugger as well, but I will only show for PHPStorm and xdebug.)


  1. Make sure you can do step-debugging on your local machine. Use the classic technique where you create a debug configuration for your project rather than the new PHPStorm "Listen" button. (I imagine there's a way to do this with the listen button.) Test this by setting a breakpoint and running your project and making sure it stops on the breakpoint. If you don't already know how to do step-debugging, this would be a good time to learn. It's a critical skill for any developer. Resources: (PHPStorm instructions, Drupal.org article, PHPStorm debugging article) (Images of my PHPStorm web application configuration are below.)
  2. On your internet-connected machine, configure sshd to set "GatewayPorts yes". (The default is "no"). This is configured in /etc/ssh/sshd_config on both DebianUbuntu and RHEL/CentOS distributions.). After changing the configuration, restart sshd with "sudo service ssh restart"
  3. Use ssh to create a tunnel which will bring a port on your internet-available machine to your local machine: ssh -R :8888:localhost:80 example.com will bring traffic on port 8888 of the internet-connected "example.com" to port 80 on localhost (where your debugging environment is). Essentially, once you've done this, all traffic hitting http://example.com:8888 will actually land on your local machine on port 80. (This won't work if you're blocking port 8888 on the firewall of your internet-connected host, of course.)
  4. Try accessing your internet-connected machine with a browser to demonstrate that traffic is routed through. With my example, you would now be able to hit http://example.com:8888 and the dev site on you local machine would appear (slowly, perhaps). If you don't have access to your local dev site now using this technique, you aren't ready to continue.
  5. Test debugging by accessing your external site. Start debugging on PHPStorm using the configuration you created earlier. The console will show "Waiting for incoming connection with ide key '15247'". Your IDE key will be different of course. With a breakpoint set, visit your external internet-connected URL like this: htttp://example.com:8888?XDEBUG_SESSION_START=15247- you should see your IDE take control and stop at your breakpoint.
  6. Now configure the external service with the URL it is to hit when some event happens AND with the XDEBUG_SESSION_START query string. I was working with the Mandrill Incoming project, so my inbound URL looked like this: http://example.com:8888/services/rest/mandrill_events?XDEBUG_SESSION_START=15247
  7. Whatever you have to do to cause an event to happen, make it happen. In my case, this meant sending an email via Mandrill which was to be delivered to my web service.
  8. Voila, your debugger opens and lets you step through the web service call.


PHPStorm Server Configuration

PHPStorm Web Application Configuration

PHPStorm "Waiting for connection"

Remote Drupal/PHP Debugging with Xdebug and PHPStorm

Step-debugging is one of the key skills for any developer, and it can be baffling. When you start trying to control a remote webserver with an IDE running on your local workstation though, it gets even more complicated with the network problems. This article will deal directly with Xdebug as the debugging engine on the PHP end and Jetbrains PHPStorm 7.x as the IDE on the workstation side, but these techniques will work for other IDEs like Eclipse or Komodo, and they'll work for the Zend debugging engine as well.

The Basics: How Remote Debugging Works

The first thing that you absolutely must understand is the sequence of events and how they're initiated. If you have xdebug configured in your PHP configuration (I'll show how later), then when PHP starts executing:

  1. If the PHP session is started with a URL like http://example.com/?XDEBUG_SESSION_START=PHPSTORM (or it was earlier, creating a cookie), then Xdebug within the PHP execution will attempt to make a connection to an IDE, which defaults to be at localhost, port 9000. (If it can't make a connection, it just moves on.)
  2. If the IDE is listening, it gives instructions to Xdebug about breakpoints and such.
  3. When a breakpoint is encountered, Xdebug notifies the IDE on this connection.

You'll notice that the remote machine is attempting to make a connection to an IDE that is, in our situation, on another machine. Unfortunately with actual server configurations, there is usually no way that an outgoing TCP connection can get to your local workstation. If it can you can just configure that in your php.ini or xdebug.ini by setting xdebug.remote_host and you won't have to do all the things we're about to do. But for now we're going to assume you're trying to debug a machine that is out on the internet that cannot directly access your workstation.

To make this happen we'll do three things:

  • Configure the remote server for xdebug
  • Tunnel the remote Xdebug connection to our local machine via ssh
  • Configure PHPStorm on the local machine

Remote Server Configuration

On the remote server:

  1. Make sure the php5-xdebug package or its equivalent is installed. On Debian/Ubuntu this is sudo apt-get install php5-xdebug. On RedHat/Fedora/CentOS this may be sudo yum install php53u-pecl-xdebug depending on the repositories and PHP version you are working with.
  2. Configure the PHP ini file for the xdebug extension. Debian/Ubuntu: /etc/php5/conf.d/xdebug.ini or /etc/php5/conf.d/20-xdebug.ini, RedHat/Fedora/Centos: /etc/php.d/xdebug.ini. You'll need: xdebug.remote_enable=1 in there. The default host to connect to is localhost, and that's the way we're going to leave it, since we're going to tunnel/proxy the connection to our local machine. (If you cannot use port 9000 because it's already in play, you'll have to also configure xdebug.remote_port here and on your local IDE. But we're not going to go there right now.)
  3. Restart Apache or whatever webserver you're using. (If you're using php-fpm, restart it instead). Check for errors of course.
  4. Visit the phpinfo of the remote machine with a web browser. On drupal you can just visit /admin/reports/status/php to get the full output. You should see "with Xdebug v2.2.3, Copyright" … in there. If you see it, you can scroll down to the xdebug section and check that the config is as you set it, especially remote_enable and remote_host (which should be localhost).
  5. Check to see that port 9000 is not already in use (especially by php-fpm if you use that). You can do this by making a connection using the venerable telnet tool. telnet localhost 9000 on the remote machine (install it if you don't already have it by installing the telnet package). If you can make a connection to this point, before proxying the port, then things aren't going to work because something else is there on port 9000. You'll need to find out what it is and work around it one way or another. (To find out what process is using it, try lsof | grep 9000. If you don't have lsof, install it.)

(Had you been working on a remote machine that had direct access to your workstation, you could have set xdebug.remote_host to your workstation's hostname or IP address and not have to do the ssh tunneling/proxying that we'll do in the next step.)

Proxying/Tunneling Your Debugger Connection

Now we need to bring port 9000 to our local machine. What we're doing here is putting a "feeler" on the remote host that will listen on port 9000 and bring anything that happens there to our local machine on port 9000.

ssh -R 9000:localhost:9000 some_user_account@www.example.com

You'll need to re-establish this connection any time the tunnel is interrupted due to network connectivity or a remote or local host reboot, etc.

Set Up The Local IDE

On your local machine you need these things:

  1. A matching set of code to what's on the remote machine. This can be a remote mount via sshfs or other techniques or a copy, git checkout, etc. Of course if you have a copy you can get very confused by having the wrong code presented to you at a breakpoint. We'll demonstrate sshfs and git checkout below.
  2. PHPStorm opened to a project with the code which matches the remote machine and with the "Zero Configuration Debugging" button turned on (listening).

So:

  • Mount the remote code to your local machine or check out a copy locally that exactly matches. For example: If you have sshfs configured on your workstation, sshfs some_user_account@www.example.com:/var/www/myproject /tmp/myprojectwould mount the remote code into /tmp/myproject (which you must create ahead of time). If you don't have sshfs yet, google it to figure out how to configure it. It's nice and it's easy. If you want to run a copy, do something like cd /tmp; git clone git://github.com/example/myproject.git to get an identical codebase. If you prefer, PHPStorm has code syncing capabilities that can maintain a copy for you, but I find it cumbersome.
  • Run PHPStorm and choose File->Open Directory (or open an already existing project, of course). Open the directory where your local copy or mount is.
  • Click the "Start Listen for PHP Debug Connections" button on the top toolbar to turn on listening.
  • At this point you should be able to 'telnet localhost 9000' on your local workstation and get a connection. If you click to turn off listening, you should not get a connection. PHPStorm not listening PHPStorm listening
  • On the remote machine the same behavior should now be proxied. telnet localhost 9000 should result in a connection when PHPStorm is listening and the ssh proxy is working, and it should result in no connection when PHPStorm is not listening.

Debug!

If all has gone well,

  1. Set a breakpoint early in your PHP project. For example, in Drupal set a breakpoint on the first line of index.php.
  2. Visit your remote website in the browser with "?XDEBUG_SESSION_START=PHPSTORM" at the end of the URL. For example, http://example.com?XDEBUG_SESSION_START=PHPSTORM. (Note: As far as I can tell, with this configuration it doesn't matter what XDEBUG_SESSION_START is set to. PHPSTORM is just a placeholder here because all connections are going to the same place.)
  3. PHPStorm should pop up and let you start debugging.

Troubleshooting

Check these things:

  • Make sure that on the server you can use telnet to connect to port 9000 when PHPStorm is listening, and not when it's not.
  • Make sure you have a breakpoint set and PHPStorm is listening.
  • Make sure Xdebug is configured by visiting phpinfo any way you want to.

Resources

  • The Xdebug site, xdebug.org, is easy, once you understand what's going on and why. It has full configuration instructions. Send a little (or big) donation while you're there. Derick Rethans has been maintaining Xdebug faithfully for years and years.
  • PHPStorm is available for a demo download and there are many instructions and blog posts on "Zero-configuration debugging", which essentially means that PHPStorm will listen for an accept anything, as I've demonstrated here.
  • You can also debug command-line php activity on a remote server using nearly the same technique. See my related article on remote command-line debugging. It basically involves doing what we've done here and setting the environment variables XDEBUG_CONFIG="idekey=PHPSTORM" and PHP_IDE_CONFIG="serverName=yourservername", where yourservername is the name of a PHPStorm "server" you've set up to map remote to local paths for your project.
  • More than one person can debug on a remote server at the same time, but it requires a dbgp proxy to be running on the remote server which will accept connections and connect to the correct local workstation based on the IDEKEY. Configuring PhpStorm, XDebug, and DBGp Proxy Settings for Remote Debugging with Multiple Users explains this quite nicely and explains where to obtain a proxy (pydbgpproxy) that will do the work required.

Remote Command-Line debugging with PHPStorm for PHP/Drupal (including drush)

Introduction

XDebug with PHPStorm can do step-debugging on remote sessions started from the command line on a remote machine. You just have to set up a couple of environment variables, map the remote code to the local code that PHPStorm has at its disposal, and tunnel the xdebug connection to your workstation.

Note: If you just want to debug a PHP script (or drush command) on the local machine, that's much easier. Just enter PHPStorm's Run/Debug configuration and create a new "PHP Script" configuration.

Overview of Setup

  • We'll create a PHPStorm project that contains all the code we want to debug on the remote machine. This can be done via source control with matching code, by mounting the remote directory to your local machine, or any way you want.
  • Create a mapping from server side directories to PHPStorm-side code (A "Server" configuration in PHPStorm)
  • Use environment variables on the remote machine to tell xdebug what to do with the debugging session
  • Tunnel the Xdebug TCP connection if necessary.
  • Make PHPStorm listen for a connection
  • Create a breakpoint somewhere early in the execution path
  • Run the command-line tool on the remote server.

Step-by-Step

  1. On the remote server install xdebug and set xdebug.remote_enable=1 In your xdebug.ini (or php.ini). For complete details see Remote Drupal/PHP Debugging with Xdebug and PHPStorm.
  2. Open your project/directory in PHPStorm; it must have exactly the same code as is deployed on the remote server. (You can optionally mount the remote source locally and open it in PHPStorm using sshfs or any other technique you want, see notes below.)
  3. If you're debugging drush, you probably need to copy it into your project (you don't have to add it to source control). PHPStorm is quite insistent that executing code must be found in the project.
  4. Create a debug configuration and a "Server" configuration in your project. The Server configuration is used to map code locations from the server to your PHPStorm code. Run->Edit Configurations, Create PHP Web App, Create a server, give the server a name. Click "Use path mappings" and configure the mappings from your project to the remote server's code. (See PHPstorm server configuration)
  5. If your remote server cannot directly create a tcp connection to your workstation, you'll have to tunnel port 9000 to your local machine. ssh -R 9000:localhost:9000 your_account@remote_server.example.com - For more details and debugging, see
  6. Click the "Listen for PHP Debug Connections" button in PHPStorm. I call this the "unconditional listen" button, because it makes PHPStorm listen on port 9000 and accept any incoming connection, no matter what the IDE key. See Remote Drupal/PHP Debugging with Xdebug and PHPStorm
  7. In PHPStorm, set a breakpoint somewhere that your PHP script is guaranteed to hit early in its execution. For example, if you're debugging most drush actions, you could put a breakpoint on the first line of drupal_bootstrap() in includes/bootstrap.inc.
  8. If the computer is not reachable from the server, you will need to tunnel the connection from the server to your workstation. ssh -R 9000:localhost:9000 some_user_account@www.example.com For more details and debugging, Remote Drupal/PHP Debugging with Xdebug and PHPStorm
  9. In your command-line shell session on the remote server set the environment variable XDEBUG_CONFIG. For example, export XDEBUG_CONFIG="idekey=PHPSTORM remote_host=172.16.1.1 remote_port=9000" (Note that port 9000 is the default both for xdebug and for PHPStorm.) If you're tunneling the connection then remote_host must be 127.0.0.1. If you're not tunneling, it must be the reachable IP address of the machine where PHPStorm is listening.
  10. export PHP_IDE_CONFIG="serverName=yourservername" - the serverName is the name of the "server" you configured in PHPStorm above, which does the mapping of remote to local paths.
  11. On the command line run the command you want to run. For example drush cc all or php /root/drush/drush.php status
  12. If all goes well you'll stop at the breakpoint. You can step-debug to your heart's content and see variables, etc.

Drush+Drupal-Specific Hints

  • I've had the best luck actually copying drush into my codebase so that its mappings and Drupal's mappings can be in the same directory.
  • Once you have the environment variables set you can either use the "drush" command (which must be in your path) or use "php /path/to/drush/drush.php" with your drush options. Make sure you're using the drush that's mapped as a part of your project.

Notes and resources

  • We set the xdebug.remote_host in the XDEBUG_CONFIG environment variable; it could also have been configured in the xdebug.ini as xdebug.remote_host=myworkstation.example.com. (Note that the default is "localhost", so if you're tunneling the connection you don't actually have to set it.)
  • Full details about remote debugging on xdebug.org
  • Debugging: Make sure that only PHPStorm is listening on port 9000. If something else (most notoriously php-fpm) is listening there, you'll have to sort it out. PHPStorm is happy to listen on another port, see the preferences, and you'll need to change your environment variable to something like export XDEBUG_CONFIG="idekey=PHPSTORM remote_host=172.16.1.1 remote_port=9999
  • You may want to use sshfs or some other technique to mount the remote code to your local machine. sshfs your_account@server.example.com:~/drush /tmp/drush would mount the contents of drush in the remote home directory to /tmp/drush (which must already exist) on your local machine. It must be writeable for PHPStorm to work with it as a project, so you'll have to work that out.
  • The article that taught me everything I know about this is Command-line xdebug on a remote server. Thanks!

Pages

Subscribe to Planet Drupal