DeployMint: A Staging and Deployment system for WordPress

Exec Summary: Today I’m launching a Beta open source project called DeployMint. I’m using it on WordPress installations where WordPress is being used as a CMS. It runs as a WordPress plugin and allows for staging and deployment of WordPress sites along with robust version control and zero down-time during deployments. It uses the Git version control system to store site snapshots in a safe and space efficient way. It also takes a “belt and braces” approach and provides an emergency back-out system separate to Git in case a deployment fails. You can download the latest version of DeployMint and see a video demo at the DeployMint project page on Google Code.

Full blog entry:

My company is busy moving to using WordPress as a CMS and I wanted a way to instantly deploy several new pages of content or an entire site and have dev and staging sites to test new ideas. I also wanted version control, instant deployment and an emergency back-out system.

I also needed comments to be preserved on the live site so that if I deploy a new version, the existing comments on the live site stay where they should be and only page or post content is updated.

So I created a WordPress plugin called DeployMint.

DeployMint runs under WordPress MU. You create as many subdomains as you would like, for example:

  • development.example.com
  • staging.example.com
  • example.com (your live site)
  1. Then you design  your entire site with themes, pages, content on development.example.com.
  2. Once you’re done, you take a snapshot of development.example.com and deploy that snapshot to staging.example.com.
  3. Your client reviews the new site on staging.example.com and suggests changes.
  4. You make the changes on development.example.com, take a new snapshot and deploy that snapshot to staging.example.com
  5. Once your client is happy, you take a snapshot of staging.example.com and deploy it to example.com, your live site.

Here is a video showing the basic functionality of DeployMint. DeployMint is installed on this blog and I use it to test out new themes and design changes. It works as well on a blog or when WordPress is being used as a CMS. In this video I take a snapshot of my live site (this blog) and deploy it to my staging site staging.markmaunder.com. Then I make a minor modification, I re-snapshot the staging blog and deploy that snapshot to the live site.

DeployMint is space efficient because it uses Git to store snapshots. It also makes a full copy of your entire WordPress database including all your WordPress MU sites every time you deploy. Because these require more space, you can choose how many of these full backups you want to keep. If things go awry with your database or deployment for some reason, you have an emergency backout system that will restore your WordPress MU installation to the state it was in before your previous deployment.

Behind the scenes, DeployMint (DM) works as follows:

  • To install DeployMint you need to create a data directory that is not under your web root, but is writable by your web server.
  • When you create a new project, a new Git repository is created.
  • When you take a snapshot, DM dumps all tables belonging to the blog you snapshotted into individual files.
  • Those files are checked into a ‘Git’ repository. DM uses git for storage because it’s space efficient and robust.
  • Every snapshot you create is a new branch in the repository and only the changes are stored.
  • When you deploy using DM, it simply checks out the branch you want to deploy and imports it into a temporary database.
  • In that temp database, we merge all existing comments on your site into the site we’re about to deploy.
  • DM also modifies any hostnames it needs to, to reflect the site we’re about to deploy to’s hostname.
  • Before deployment, DM takes a full backup of your entire WordPress MU database including all sites and stores this for emergencies in case you need to back-out your changes.
  • These backups take up more disk space than snapshots, so you can choose how many of them you want to keep and DM auto-deletes the oldest ones first.
  • Then a rename is done which takes a few hundredths of a second to replace your old database with the new database we’re deploying.
  • And you’re live with your new site!

Please post a comment below if you have any features suggestions or comments. Thanks.

87 thoughts on “DeployMint: A Staging and Deployment system for WordPress

  1. Hi Mark, I’m wondering if its possible to create multiple dev environments unique for each person adding content (like: dev1.example.com, dev2.example.com, dev3.example.com) and then have all of those users using the dev only be able to post to the staging.example.com site? Ideally only one or two people would be able to login to WP from the staging site and push to production from there. Is this doable? Thanks.

  2. I have a need for this plugin for a client of mine running 2 sites on with an up-to-date WP Multisite install.
    Question is, will it still work with WP version 3.8 ?
    I read somewhere you’re seeking to add much of Deploymint’s features to Wordfence.
    I have that plugin running on several of my sites, including this one. Should I wait until it incorporates deploymint, or will it be more than a few months before its ready?
    Thanks

    • The issue is when deploying your snapshot to another site, it overwrites the option name of wp_*_roles with the site you are deploying from.

      You have to look in the wp_*_options table and update it with the correct name.

  3. Nice tool! I have made some updates to it base on our company use case scenarios. We wanted Admins to have the ability to do deployments but only see their sites they have access to and projects that they created. I have added that functionality. I still reserve the “Emergency Revert” and “Option” pages for Super Admin. If anyone is interested in my updates I can send it to them or post it to bitbucket or github.

    When I find some more time I will be updating the tool to deploy to different db’s so you can separate the environments.

  4. This is simply great idea. This should be the final destination for the person who is in search of deployment and staging in wordpress. The way you illustrated this is really nice.

  5. This is a great idea. How does it handle WP core updates? How do you safely install and test the new version of WP w/ out risking the production install?

    Is there a method for pushing files only? For example, I need to test a bug so I edit posts as part of my testing. I don’t want those changes pushed to production. I only want to push the file changes.

  6. Wow. Great idea. I’ve got some plans for some great sites in the works, and have been wondering how I could go about creating a development area and then sync it up with the live version.

    One question I might have is – can the plugin handle developing in one multisite installation and then moving over to another multisite install – being on totally different domains? I’ve had a hard painful lesson with moving multisite across domains that I would rather not repeat, and this might be one possibility for me.

    Thanks!
    Scott

  7. I love love love this idea. This would be perfect for us if it wasn’t MU. Have you made any progress on a deployment plugin for no MU sites? If not, do you know of any options out there that are similar?

  8. Hey guys i know that’s probably a stupid question but i couldn’t solve it in any way

    I’m trying to specify a Path to a data directory for DeployMint. Using filezilla i created the public_html/try but it keeps returning me this message.

    The directory ‘/public_html/try/’ specified as the data directory doesn’t exist.

    What im i doing wrong? (hosted on: hetzner)

  9. Hi Mark,
    I recently found and installed your excellent plugin, DeployMint. Everything was going smoothly until I ran into a snag just now…
    My Options don’t seem to actually save (even though the message says “successfully updated”). I have confirmed the GIT, MYSQL, and MYSQLDUMP paths with my host…they weren’t positive about the data directory, though. This directory is the only one I’ve tried that doesn’t return a [The directory '/X/Y/Z' specified as the data directory doesn't exist.] error.\, so I think it is more of an issue with the Save Options function…

    Do you have any idea what would be causing this? If so, I would greatly appreciate your input!
    -DR

    • OK, so I found how to fix the above issue. See the following steps:

      1. As David S said, go to http://code.google.com/p/deploymint/source/browse/deploymintClass.php?spec=svne20a501c87ed355113e408cd8992ff782c310e78&r=e20a501c87ed355113e408cd8992ff782c310e78 and grab the 3 mysql Create arguments (listed below).
      a.
      create table if not exists dep_options (
      name varchar(100) NOT NULL PRIMARY KEY,
      val varchar(255) default ”
      ) default charset=utf8
      b.
      create table if not exists dep_projects (
      id int UNSIGNED NOT NULL auto_increment PRIMARY KEY,
      ctime int UNSIGNED NOT NULL,
      name varchar(100) NOT NULL,
      dir varchar(120) NOT NULL,
      deleted tinyint UNSIGNED default 0
      ) default charset=utf8
      c.
      create table if not exists dep_members (
      blog_id int UNSIGNED NOT NULL,
      project_id int UNSIGNED NOT NULL,
      deleted tinyint UNSIGNED default 0,
      KEY k1(blog_id, project_id)
      ) default charset=utf8

      2. Open your PHP MyAdmin, browse to your wordpress table’s SQL tab; then run each of the 3 aforementioned commands.

      3. After confirming that the dep_members, dep_options, and dep_projects tables now exist, return to your DeployMint Options (in your WP dashboard) and enter all the paths again… (this time the save should work)..

      • So I figured out the above issue and added a new project (with the site and 2 subsites I want to include).

        Everything was going great, until I went to click on the project link (to start making snapshots).

        I get the following error:

        Warning: call_user_func_array() [function.call-user-func-array]: First argument is expected to be a valid callback, ‘deploymint::projectMenu1′ was given in /home/[...]/public_html/wp-includes/plugin.php on line 405

        Does anyone have any idea why this might be happening?

        I looked at plugin.php and the 405th line is:

        call_user_func_array($the_['function'], array_slice($args, 0, (int) $the_['accepted_args']));

        …but I’m not sure that’s very helpful…
        Mark? Thoughts?!?
        Many thanks,
        -DR

        • I found the section of the code that runs a loop to add submenu(s) for all of the projects a user has created [located in the DeployMint options as (a) submenu(s) of Manage Projects].
          Here it is:

          add_menu_page( “DeployMint”, “DeployMint”, ‘manage_network’, ‘DeployMint’, ‘deploymint::deploymintMenu’, WP_PLUGIN_URL . ‘/DeployMint/images/deployMintIcon.png’);
          $projects = $wpdb->get_results($wpdb->prepare(“select id, name from dep_projects where deleted=0″), ARRAY_A);
          for($i = 0; $i < sizeof($projects); $i++){
          add_submenu_page("DeployMint", "Proj: " . $projects[$i]['name'], "Proj: " . $projects[$i]['name'], "manage_network", "DeployMintProj" . $projects[$i]['id'], 'deploymint::projectMenu' . $projects[$i]['id']);

          ….but I'm not sure what function this calls once clicked… Hopefully Mark still supports/cares about his plugin! It's too awesome to let it fall by the wayside.

  10. Hi – any help would be greatly appreciated with this problem. The options that I am setting through the Options page for DeployMint appear to be saving correctly (at least with respect to the AJAX response), but when I reload the page all of the settings are empty. Any ideas? DeployMint seems like it would be really useful if I could get it working.

  11. Hello
    Can anyone tell me what version of PHP I need? I get the following error…..
    [07-Nov-2011 02:56:12] PHP Warning: call_user_func_array() [function.call-user-func-array]: First argument is expected to be a valid callback, ‘deploymint::projectMenu1′ was given in /home/virtual/public_html/wp-includes/plugin.php on line 405

  12. hi there, Mark

    in some near future the issue “convert my MU blog into single blog” will be solved, and maybe you’re ‘the man’ who’s gonna do it :o[)

    but right now, my issue is not being able to install…
    at first, it says “options successfully saved”, but then the Projects tab gives the red alert…

    WP debug is dumping this

    Notice: Undefined index: git in /home/user/domain.com/wp-content/plugins/DeployMint/deploymintClass.php on line 68

    host says everything is ok in their side….
    ¿any hints?

    cheers,

    • Also I’d recommend adding a nonce to your “checkPerms()”. Its recommended when making ajax request.

      One more, WordPress has a pretty nicely fromated wp_die() function, just saying if your dieing anyway go out in style ;D

  13. This plugin is going to be a lifesaver if I can get it work! Thank you so much for developing it!

    Two questions for you:

    First, I keep getting this error message:

    An error occured updating your options.

    But there’s no explanation about the error. I’ve doublechecked with my web host and they say I’ve entered all the paths correctly. More information would be helpful for troubleshooting.

    Also, there’s a typo. The correct spelling is occurred.

    Second, and this *may* be the cause of the first issue, my web host won’t give me access to the dumps directory but they sent me this message:

    However, after further review, I don’t think the plugin is asking for the location of previous dumps. According to the main site, “When you take a snapshot, DM dumps all tables belonging to the blog you snapshotted into individual files.”

    From that line, I believe the DeployMint plugin is asking you for a location where it can place dumps when you take snapshots, rather than asking you for the location of previous dumps. So you should be able to enter any directory you’d like to have DeployMint place the database dumps in and it should allow you to finish the installation.

    Is that the case? If so, does it matter what I name the directory?

    Many thanks!
    ~KC

  14. Hi Mark, this Pools great, can’t wait to try. One question regarding users. When new users register in production while one is editing staging, will the newly registered users be deleted when applying the staging snapshot to production? Thanks for Insight, Jan

    • After installing I understood that wordpress multisite shares the users, so yes, registered users will be shared across staging/production.

      Great plug-in!

  15. Hi there. Great idea and nicely done. Thanks.

    FYI on a multi-user VPS (I’m using WHM/CPanel) you may need to grant additional privileges to the database user for the deploy snapshot to work. I haven’t narrowed it down completely, but it works with SHOW DATABASES and LOCK TABLES Administration permissions as well as all Data and Structure ones.

    FYI the error is like this: I have installed WordPress under one of the VPS user accounts (user) with a specific mySQL user called user_wp.

    When I come to deploy a snapshot, I get the following error:
    We encountered an error importing the data files from snapshot deployed.20110908. The error was: ERROR 1044 (42000): Access denied for user ‘user_wp’@’localhost’ to database ‘deptmp__131546081699′

    When I grant user_wp the privileges mentioned to the whole mySQL database the deploy works. This is, I guess, a bit of a security issue, and you need to log in as root to make the global changes in phpMyAdmin.

    Hope that helps,

    Tom

  16. Mark, I run MU with a local school district for teachers/staff to have their own blogs and am about to use it as the CMS for the main site as well as the subdomains. Here’s long explanation for a short question:

    I’ll have a main site for the district with each school & department on their own subdomain (eg, http://www.school.com, MS.school.com, HR.school.com, etc). Using dev.school.com, I’ll take a snapshot of MS & deploy to dev. Then deploy the new dev version to live MS once completed. Correct so far?

    Now here’s my question: if I were to do all that and now I want to snapshot HR & deploy to dev, can I assume it will overwrite the MS-dev version without any issues? I can just use the one dev subdomain for every sub-/domain I have, correct?

    Wonderful idea for a plugin, by the way :)

  17. wow, excellent plugin! would be really nice if it worked with separate domains. I usually only move the site to the client’s server once they’ve paid in full :)
    but it should be great for plugin and theme dev

  18. Would DeployMint work if I’m developing a site for a client on my server (development.xxxxxx.com) and planning to migrate to the client’s server (yyyyy.com) once she approves what’s on my server?

    • Sarah that’s a good question and unfortunately no. It’s designed to work on a multi-user wordpress installation with subdomains where you’re developing on development.example.com and deploying to example.com on the same machine and wordpress installation but configured as a separate site under wordpress multi-user.

      However, I know yours is a common use-case so I’ll give it some thought…

      • I can definitely second this. Unfortunately I don’t want to risk doing dev work in the same installation/host environment/server as production.

        However, I must congratulate you on a wonderfully creative solution, I know loads of people have been wondering how to do this for a long time. Wow, just wow. :)

      • I would like to add my vote (and any potential assistance I might be able to offer) for this functionality, as well.

        I can think of a few different use-cases (all of which I am trying to handle somewhat manually now):

        1. I have one set up using multisite (actually multinetwork, using Networks+ from Ron & Andrea Rennick) where we have a development server, a staging server and a production server. All three are completely separate VMs, and all three use completely separate MySQL database servers. It would be awesome to be able to snapshot the info from one of our servers and deploy it (even if the deployment has to happen manually) to one of the other servers.

        The biggest issue we face right now when trying to deploy between environments (nothing is live, yet, so it’s not something we’ve had to face in earnest, yet) is the idea of replacing all of the URLs (including ones that are inside of serialized data).

        2. Deploying a single site from my personal multisite installation to a client’s standard install of WordPress.

        3. Deploying a full multisite installation from my personal server to a client’s server.

        Granted, scenarios 1 & 3 are very similar, but would probably require a slightly different approach (especially because, in scenario 1, we are using multi-network; which brings with it the use of domain mapping).

        Any thoughts you can offer would be great. Thanks for starting this project, Mark.

          • Thanks, Jon. We looked at RAMP a while back, but that looked like it would kind of require us to do all of our content authoring in the staging environment.

            We’re planning to use the staging environment strictly to test code/theme changes that we make in development before pushing it to production.

  19. As a recent convert to Git this looks awesome.

    I currently use git for deployment from local environment to my staging server using this setup from Joe Maller http://joemaller.com/990/a-web-focused-git-workflow/ however I don’t trust it (yet) for deployment to live sites and my live sites are each on there own separate server.

    This looks great for the case though where you staging and live are on the same server, but unless I’m missing it you can’t push to a separate server right?

    • Hi Jon. DeployMint is designed to run on a single web server, but the database server can be a separate machine – also a single database server. This configuration i.e. one web server and one db server, scales up to tens of millions of visits per day provided it’s well configured and a fast dedicated (non VPS) machine.

      • I understand, and I hope you didn’t read that as a critic of the plugin, I’d rather see it be perfect for one workflow than to accomodate several workflows poorly :) However, commenting the code a little would be welcomed.

        Anyway, I’m sure many people work that way (similar to you) and this is an awesome solution for them. Personally, I deal with many smaller sites and my workflow tends to be develop local, stage on my dev server, then deploy to client’s server. The client’s server is often just shared hosting.

  20. I got a question about this project in the WordPress support forums.

    I assume you mean WordPress multisite when you say “MU”. As I understand it, MU has been dead and buried with the release of Version 3 of WordPress.

    Just wanted to clarify as I am, quite honestly, confused.

    • You’re referring to the fact that MU was merged into WordPress core as part of the version 3 release.

      When I refer to MU, I’m referring to the multi-user functionality of WordPress. This is not enabled by default and DeployMint requires that it be enabled to work. DeployMint then lets you snapshot sites within WordPress MU.

  21. Hi mark,

    i really like the idea of DeployMint – especially regarding nearly zero downtime on deployment. I’ve got two questions regarding use of DeployMint:

    1) Is it possible to exclude certain post types from being overwritten on deployment? We’ve got a client with a massive news section, updated up to 20 times a day. So if we’d fork on work on that dev-site, we would pretty sure overwrite new content, would we?

    2) The site mentioned above already uses MU for the main site and 2 big parts of it. How would that work with DeployMint? Would it just seamlessly integrate into all them three MU-Sites or would it just work on the main Blog?

    Thank you in advance for a short answer.

    Cheers,

    Christian

    • Hi Christian,

      Deploymint is designed to work with WordPress MU only. You take a snapshot of one of the MU blogs and deploy it to another.

      It only takes complete snapshots of a blog or site. So regarding question 1, deploymint doesn’t support merging work from different sources right now. You need to take a snapshot, modify it and then redeploy it.

      What you could do is take a snapshot, deploy it to a development blog, have everyone work on that and when everyone is happy with the way the preview looks, deploy it to the live site.

      Hope that helps.

      • Thanks for your quick reply. That does help on the hand, but also keeps me from working with it. I just cannot explain to our client, to work on the one backend while we’re working on the fork, but using the regular backend while we’re not.

        I’ll have a look into your code and see if i can modify it to my needs. Thank you anyway!

  22. I often have to make WordPress blogs for different keyworded domains that all have the same theme and plugins. Would this help me to, for example, setup multiple blogs on multiple domains without going through the tedious process I now do?

    Also, I do a lot of split testing (a/b tests) and I wonder if this could be useful somehow.

    • Jack that’s an interesting use case. You could snapshot the primary blog and deploy it to 10 other blogs and make slight changes by hand on each.

      AB testing really relies on running different page or content alternatives (test A/B/C) on the same page on the same site, so not sure how this would help there.

  23. Can this integrate with existing MU sites? example.com is already running a site as well as many subdomains. How could I use this tool in that situation? If I had aaa.example.com, would you just make aaa-staging.example.com?

    • Patrick, yes it will integrate with existing MU sites. In fact it’s designed to. Just install it for the network, activate it on any blog and take a snapshot of one of the MU blogs and deploy it to another.

      If something goes wrong, use the emergency backout feature to restore your entire database to it’s original state.

  24. ” Every snapshot you create is a new branch in the repository and only the changes are stored.”

    In git, you don’t store diffs but the entire tree as it looked when you commited. Is there something different going on here? Git does pack objects together to be space efficient though, but you always store all of it.

    Great tool, btw! I’ve been happily using WP to back FB apps but this would be so useful for secure versioning. Cheers!

    • Thanks Gursimran, I don’t know much about git internals. Here’s a section from the git book:

      http://book.git-scm.com/7_how_git_stores_objects.html

      The other format for object storage is the packfile. Since Git stores each version of each file as a seperate object, it can get pretty inefficient. Imagine having a file several thousand lines long and changing a single line. Git will store the second file in it’s entirety, which is a great big waste of space.

      In order to save that space, Git utilizes the packfile. This is a format where Git will only save the part that has changed in the second file, with a pointer to the file it is similar to.
      When objects are written to disk, it is often in the loose format, since that format is less expensive to access. However, eventually you’ll want to save the space by packing up the objects – this is done with the git gc command. It will use a rather complicated heuristic to determine which files are likely most similar and base the deltas off that analysis. There can be multiple packfiles, they can be repacked if neccesary (git repack) or unpacked back into loose files (git unpack-objects) relatively easily.

      Git will also write out an index file for each packfile that is much smaller and contains offsets into the packfile to more quickly find specific objects by sha.

      I’m not sure if I’m using a packed format, so this might be an improvement for DeployMint. Thanks for the input!!

  25. Mark: this looks really great, can’t wait to try it out. I can see several situations where this would be useful to me and my business. Very nice work, and excellent post.

  26. have you had any luck doing this with non MU/Network sites? If you have some ideas on how to do this, I’d love to talk to you more, and perhaps engage your free time.

    • Vid it was a tough call to choose not to support non MU sites initially. I can probably add this in a future version, but the reason I chose to only support MU is so that you can install theme’s and plugins on the MU server and have it available for dev/staging and live blogs without having to transfer the files across to a different WP installation. It radically reduces the complexity and the only thing you need to worry about deploying when you’re on MU is data.

      • I agree, but the MU stuff doesn’t work for us, when we have thousands of sites, this is a really cool project, I hope you don’t mind if we use some inspiration from this, if you put it on Github, we’d probably just fork it and send you pull requests.. large scale deployments of WP will be the next big thing for 2012, glad to see you under an early start.. we’re slowly releasing our code as well.

        • Go wild. :-) Make sure your guys use proc_open() to execute git or you’ll get some hard to diagnose bugs related to file descriptors and the git binary.

          • thanks for the tip :). we’re doing a lot of stuff using queues and ruby, rather than letting people do eval, or proc_open in production environments, since we don’t necessarily have control of what plugins people install.

            But, I will keep this in mind.

    • Thanks Ted. Grab the plugin, set it up on a WordPress MU install and give it a whirl, I’d love to hear what you think. Drop me an email or comment with questions/suggestions.

  27. Are you going to go the whole hog with any future versions and be able to deploy from remote git repos? I have been thinking of building something similar for a while as I have multiple clients on multiple domains and would be quicker to deploy automatically or at least on a click of a button.

    And is a good idea as loads of big companies are now using wordpress as it does everything out the box.

      • There are many free and paid GIT hosting services of high quality out there. My project handles its own hosting, but we might opt for a hosted service rather than wrapping our heads around a new VCS just for the sake of web-testing. Remote repos would be a major feature. I can actually see a hosted service targeted specifically at WordPress growing up from that:

        ‘Safely deploy your latest changes on a test site. When ready, apply your changes to your live site with the click of a button” – Quite the offering! VaultPress for instance might be interested in this functionality.

    • WordPress is one of the world most popular CMS’s. According to Google trends, it’s overtaking Drupal and Joomla for CMS related queries: http://bit.ly/rfJdBC

      If you’re running a low traffic site, it’s easy to just edit the site in real-time and if one or two visitors run into a dead link, then no big deal. But if you need a way to have a site in a dev or staging area while you edit it and then instantly deploy it to a live site, there aren’t many tools out there to do this. So DeployMint solves this problem.

      It’s targeted at high traffic sites that use WP as a CMS with larger teams and the need for revision control.

    • @Rick,

      Other open source projects that label themselves CMS have even more need for this than WordPress, i.e. Drupal. It’s one of the reasons I switched to WordPress from Drupal.

      -Mike

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Notify me of followup comments via e-mail. You can also subscribe without commenting.