GitHub Continuous deployment to a Raspberry Pi

raspberry-logo

So, Im still playing with the Raspberry Pi while working on the wackcoon project (more to come on that) and I have the need to work in a group.  Instead of just using SCP to get files over to my device, I wanted to be able to have my Raspberry Pi update whenever there was an update to the master branch of the node project I was working on.  This would facility being able to work with a remote team (where the pi is in my network) and theoretically, if I had multiple wackcoon devices (although I have not thought about how to implement that yet.

 

To accomplish this, we will need the following.

  • A GitHub hook on the repository
  • A node app on the pi listening for messages from GitHub
  • A way for the pi to be reached from the internet

Github hook

The first thing we want to do is set up a hook on our github repository.  To do this, go to your GitHub repository click on the settings tab –> Webhooks & services –> Add webhookpideploy1

 

In the Add webhook screen we need to add just a few things :

  • Payload URL –  This is where it will send the information to whenever events happen on your master branch.  I will show you how I set this up in the next step.
  • Content type – You can select application/json (what I am using) or  application/x-www-form-urlencoded
  • Secret (not required) – this is a secret you supply to verify on the sender
  • Events – Which events you want to trigger this hook. The default is the push event and is what I am using for this example but there are many to choose from depending on your needs

(NOTE:  To learn about webhooks in depth, check out the documentation https://developer.github.com/webhooks/ )

Fill out this information and click on Add webhook

pideploy2

 

A node app on the Pi

The next step is to set up an application on the pi that will receive the event.  You can see what I have set up by looking at the wackcoon-hook project

https://github.com/DanielEgan/wackcoon-hook

All of the code is in the app.js file (check project for full code) .  It is a node site using express that has a route for a post on /payload.   So for instance, in this app it will be listening on http://localhost:5000/payload  (yes… localhost, we will get to that in a minute)

pideploy3

 

 

 

 

so lets look inside my express route for post.

pideploy4

The first thing I do is look through the JSON that the webhook sent me for a couple of pieces of information (there is a ton, I will post an example at the bottom of this post, or look in the docs)  I am just pulling out who did the push, and what repository they pushed to.

console.log(req.body.pusher.name + just pushed to + req.body.repository.name);

Next, after writing a simple message to the console, I am using a core node module called child_process  https://nodejs.org/api/child_process.html that will allow me to run commands on the machine it is running on.

I call git, the folder I want to run it on, the command, and any flags i want.  Since I might be using SCP to send files over the the Pi when testing, I hard reset it, clean it, pull, and then do a npm install in case any modules were added ( an added step could be to look through the JSON to see if the package.json was modified before running that last one, but I am not doing that), and finally running tsc since we are using typescript in the project.

	// reset any changes that have been made locally
	exec('git -C ~/projects/wackcoon-device reset --hard', execCallback);

	// and ditch any files that have been added locally too
	exec('git -C ~/projects/wackcoon-device clean -df', execCallback);

	// now pull down the latest
	exec('git -C ~/projects/wackcoon-device pull -f', execCallback);

	// and npm install with --production
	exec('npm -C ~/projects/wackcoon-device install --production', execCallback);

	// and run tsc
	exec('tsc', execCallback);

Obviously, make sure you run this project on your pi using  node app.js from the terminal

Again, you can see see the entire project looking at the wackcoon-hook project

https://github.com/DanielEgan/wackcoon-hook

A way for the pi to be reached from the internet

Since this site is running on localhost, we need a way for GitHub to access this site. There are quite a number of ways for you to accomplish this. You could use port forwarding on your router to make it accessible to the outside world.  Or you could use one of two dev tools that I have used.

localtunnel – http://localtunnel.me/  localtunnel is an npm module that will run on your pi (or other devices ) that allows you to set up a “tunnel” from your local port to a mapped url that you can use point git hub to.   This is an opensource free npm module. I have used it but it is a bit flaky, it will shut down periodically and you have to re-run it.  If you need help setting this up, here is a great tutorial to help you.  http://thisdavej.com/make-your-raspberry-pi-web-server-available-on-the-internet-with-node-js/

ngrok – https://ngrok.com/ –  This is a free teir, paid tier serves that, in there words helps you create “Secure tunnels to localhost” to answer the question ”I want to expose a local server behind a NAT or firewall to the internet.”  If you need help with this, you can check out this tutorial http://www.instructables.com/id/Raspberry-Pi-online-SSH-easy-way/?ALLSTEPS

That’s it, that is all you need.  Now you can work as a team and have them push to your master branch on GitHub and it will automatically push it to your pi.  If you have any questions fell free to ask them in the comments.

 

#############################

As promised, here is the JSON that is sent from the webhook

########## Headers ##########

Request URL: https://wackcoon.localtunnel.me/payload
Request method: POST
content-type: application/json
Expect:
User-Agent: GitHub-Hookshot/0b0c52f
X-GitHub-Delivery: 7507b280-318c-11e6-9322-eaaef242aa6c
X-GitHub-Event: push

######################
############ Body #########

Webhook – https://wackcoon.localtunnel.me/payload

  • {
      "ref": "refs/heads/master",
      "before": "3d23741be4de283f6bbfc10634c6b6ca909b8efc",
      "after": "0035745ae9ef481ac1477da1296b5253a2d22c0a",
      "created": false,
      "deleted": false,
      "forced": false,
      "base_ref": null,
      "compare": "https://github.com/DanielEgan/wackcoon-device/compare/3d23741be4de...0035745ae9ef",
      "commits": [
        {
          "id": "0035745ae9ef481ac1477da1296b5253a2d22c0a",
          "tree_id": "5d2e67e1235f175b93332125379924814cc357f3",
          "distinct": true,
          "message": "swapping out resemblejs for node-resemple-js",
          "timestamp": "2016-06-15T10:56:04-07:00",
          "url": "https://github.com/DanielEgan/wackcoon-device/commit/0035745ae9ef481ac1477da1296b5253a2d22c0a",
          "author": {
            "name": "Daniel Egan",
            "email": "daniel.egan@microsoft.com"
          },
          "committer": {
            "name": "Daniel Egan",
            "email": "daniel.egan@microsoft.com"
          },
          "added": [
    
          ],
          "removed": [
    
          ],
          "modified": [
            "index.js",
            "index.js.map",
            "index.ts",
            "package.json"
          ]
        }
      ],
      "head_commit": {
        "id": "0035745ae9ef481ac1477da1296b5253a2d22c0a",
        "tree_id": "5d2e67e1235f175b93332125379924814cc357f3",
        "distinct": true,
        "message": "swapping out resemblejs for node-resemple-js",
        "timestamp": "2016-06-15T10:56:04-07:00",
        "url": "https://github.com/DanielEgan/wackcoon-device/commit/0035745ae9ef481ac1477da1296b5253a2d22c0a",
        "author": {
          "name": "Daniel Egan",
          "email": "daniel.egan@microsoft.com"
        },
        "committer": {
          "name": "Daniel Egan",
          "email": "daniel.egan@microsoft.com"
        },
        "added": [
    
        ],
        "removed": [
    
        ],
        "modified": [
          "index.js",
          "index.js.map",
          "index.ts",
          "package.json"
        ]
      },
      "repository": {
        "id": 60712114,
        "name": "wackcoon-device",
        "full_name": "DanielEgan/wackcoon-device",
        "owner": {
          "name": "DanielEgan",
          "email": "daniel.egan@live.com"
        },
        "private": false,
        "html_url": "https://github.com/DanielEgan/wackcoon-device",
        "description": "",
        "fork": false,
        "url": "https://github.com/DanielEgan/wackcoon-device",
        "forks_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/forks",
        "keys_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/keys{/key_id}",
        "collaborators_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/collaborators{/collaborator}",
        "teams_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/teams",
        "hooks_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/hooks",
        "issue_events_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/issues/events{/number}",
        "events_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/events",
        "assignees_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/assignees{/user}",
        "branches_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/branches{/branch}",
        "tags_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/tags",
        "blobs_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/git/blobs{/sha}",
        "git_tags_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/git/tags{/sha}",
        "git_refs_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/git/refs{/sha}",
        "trees_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/git/trees{/sha}",
        "statuses_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/statuses/{sha}",
        "languages_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/languages",
        "stargazers_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/stargazers",
        "contributors_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/contributors",
        "subscribers_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/subscribers",
        "subscription_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/subscription",
        "commits_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/commits{/sha}",
        "git_commits_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/git/commits{/sha}",
        "comments_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/comments{/number}",
        "issue_comment_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/issues/comments{/number}",
        "contents_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/contents/{+path}",
        "compare_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/compare/{base}...{head}",
        "merges_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/merges",
        "archive_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/{archive_format}{/ref}",
        "downloads_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/downloads",
        "issues_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/issues{/number}",
        "pulls_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/pulls{/number}",
        "milestones_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/milestones{/number}",
        "notifications_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/notifications{?since,all,participating}",
        "labels_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/labels{/name}",
        "releases_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/releases{/id}",
        "deployments_url": "https://api.github.com/repos/DanielEgan/wackcoon-device/deployments",
        "created_at": 1465402669,
        "updated_at": "2016-06-08T16:26:28Z",
        "pushed_at": 1466013392,
        "git_url": "git://github.com/DanielEgan/wackcoon-device.git",
        "ssh_url": "git@github.com:DanielEgan/wackcoon-device.git",
        "clone_url": "https://github.com/DanielEgan/wackcoon-device.git",
        "svn_url": "https://github.com/DanielEgan/wackcoon-device",
        "homepage": "",
        "size": 212,
        "stargazers_count": 0,
        "watchers_count": 0,
        "language": "TypeScript",
        "has_issues": true,
        "has_downloads": true,
        "has_wiki": true,
        "has_pages": false,
        "forks_count": 0,
        "mirror_url": null,
        "open_issues_count": 0,
        "forks": 0,
        "open_issues": 0,
        "watchers": 0,
        "default_branch": "master",
        "stargazers": 0,
        "master_branch": "master"
      },
      "pusher": {
        "name": "DanielEgan",
        "email": "daniel.egan@live.com"
      },
      "sender": {
        "login": "DanielEgan",
        "id": 6954533,
        "avatar_url": "https://avatars.githubusercontent.com/u/6954533?v=3",
        "gravatar_id": "",
        "url": "https://api.github.com/users/DanielEgan",
        "html_url": "https://github.com/DanielEgan",
        "followers_url": "https://api.github.com/users/DanielEgan/followers",
        "following_url": "https://api.github.com/users/DanielEgan/following{/other_user}",
        "gists_url": "https://api.github.com/users/DanielEgan/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/DanielEgan/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/DanielEgan/subscriptions",
        "organizations_url": "https://api.github.com/users/DanielEgan/orgs",
        "repos_url": "https://api.github.com/users/DanielEgan/repos",
        "events_url": "https://api.github.com/users/DanielEgan/events{/privacy}",
        "received_events_url": "https://api.github.com/users/DanielEgan/received_events",
        "type": "User",
        "site_admin": false
      }
    }