Creating and Providing HipChat Integrations with Atlassian Connect, Nodejs and Express
August 18th, 2015 by Micha KopsHipChat is Atlassian’s alternative to Slack and its solution to team collaboration chats. Atlassian Connect offers developer tools to bootstrap applications, connect to Atlassian’s cloud products with easy and in combination with HipChat’s REST APIs allows us to write integrations for such a chat server in no time.
In the following tutorial I’d like to show how to write an integration within a few steps using Atlassian Connect, Node.js and Express and how to connect the integration to a HipChat server.
Finally on the one hand I’m going to explain how to speed up local development with ngrok, an in-memory database and nodemon for automatic application restarts and on the other hand I’m going to demonstrate how to configure the application for production, running with a Redis key-value store on Heroku.
Contents
- Prerequisites
- What we're going to build
- Communication Flow Explained
- Atlassian Connect Express
- Writing our Integration
- Running the Server Application, Tunnelling with ngrok
- Automatic Application Restart with Nodemon
- Integration Installation in HipChat
- Heroku Configuration and Deployment
- Tutorial Sources
- Resources
Prerequisites
The only thing we need is having Node.js and the NPM package manager installed (you might want to install ngrok, too but it is optional).
$ node --version v0.10.22 $ npm --version 1.3.14
What we’re going to build
We’re going to build an integration for HipChat that allows us to search for blog articles from this blog, www.hascode.com by entering a command in the following syntax into a HipChat room where the integration is enabled: /hascode tag.
Afterwards, a list of links to matching blog articles should be displayed in the room. If there are more then 10 search hits, a link to display all search results should be displayed rather than polluting the chat room with a long list of articles.
The screenshot above should give us a good impression of the interaction of the integration within HipChat.
The following, hopefully not too confusing mind-map demonstrates the technology stack used to build the integration and to communicate with the chat server.
Integrations are not deployed within the HipChat server but must be run as a separate server instance. HipChat is only configured to call this remote instance on specific events, called web-hooks.
That’s why our approach of building an integration is to create a server application with Nodejs and Express here.
Communication Flow Explained
So what is happening when we install our integration to a HipChat room or run our blog article search?
The following simplified sequence diagram tries to explain the process flow here:
- When the integration is installed into a room, HipChat reads the plug-in descriptor file and scans for web-hooks, permissions and install hooks.
- HipChat sends a request to our remote application with a specific JSON structure that contains important information like the OAuth2 credentials.
- If this happens, our remote application stores this information in a database and responds with a success message and status code
- HipChat stores the plug-in meta-data, web-hooks etc and confirms the successful installation to the requesting user
- When now in a chat-room the blog article search is triggered by a user, HipChat forwards the event to our remote application
- The remote application now sends a search request to the blog, parses the result, generates a view-friendly message and sends the message to the chat-room using the REST-API and OAuth2
Atlassian Connect Express
Atlassian Connect Express is a toolkit that eases the process of creating Atlassian Connect addons for us providing utility and helper libraries to bootstrap new projects, configure Express.js based web applications and connect to an existing HipChat instance using Oauth2 and a REST client.
Installation with NPM
The atlassian-connect command line tool is installed using NPM:
npm i -g atlas-connect
Creating a new Connect Project
We’re now ready to create a new connect project using the command line interface:
$ atlas-connect new -t hipchat hipchat-integration-tutorial Downloading template [===================] 100% 0.0s hipchat-integration-tutorial/ hipchat-integration-tutorial/.gitignore hipchat-integration-tutorial/Procfile [..] hipchat-integration-tutorial has been created. `cd hipchat-integration-tutorial` then run `npm install` to install all dependencies. For information on what to do next, visit: http://j.mp/get-started-with-atlassian-connect-express.
Afterwards we have a full configured project and may take a first look at our application. We may remove the handlebar templates (*.hbs), addon.css and addon.js as we won’t need them in the following tutorial.
Finally our initial directory structure should look similar to this:
. ├── app.js ├── atlassian-connect.json ├── config.json ├── lib │ └── hipchat.js ├── package.json ├── Procfile ├── public │ ├── css │ │ └── addon.css │ └── js │ └── addon.js ├── README.md ├── routes │ └── index.js └── views ├── config.hbs └── layout.hbs
Dependency Management with NPM
The dependencies our application needs are specified in the package.json file in the root directory – this is what ours looks like:
{ "name": "com.hascode.hc.blogsearch", "version": "1.0.0", "private": true, "scripts": { "start": "node app.js" }, "dependencies": { "express": "^3.4.8", "express-hbs": "*", "static-expiry": ">=0.0.10", "request": "^2.33.0", "rsvp": "^3.0.3", "atlassian-connect-express": "^1.0.0-beta2", "atlassian-connect-express-redis": "^0.1.3", "atlassian-connect-express-hipchat": "^0.2.0", "lodash": "*" } }
To install the dependencies in our project directory, we’re running the following command:
$ npm install
When npm has finished downloading the internet, we can see that our dependencies have been saved in the node_modules directory.
Writing our Integration
Now we’re entering the fun zone: implementing our application..
Project Configuration
This is our config.json and it contains our configuration for the two stages development and production (explained in detail in the following chapter).
{ "development": { [..] }, "production": { [..] } }
Development Configuration, ngrok and In-memory Database
This is part of our development stage configuration, the most important settings here are:
- we’re setting a ngrok tunnel URL as base URL (we don’t really need to because we’re going to pass this information from the command line)
- the port for our application is set to 3000
- for persistence we’re using jugglingdb with an in-memory database (we might also use another database e.g. sqlite here, we just needed to set the type to “sqlite” and add the sqlite dependency to our package.json)
- searchHost, searchBasePath and maxResultsDisplayed are settings that we’re going to use to control the behaviour of our search application
{ "development": { "localBaseUrl": "https://d6f97a22.ngrok.io", "usePublicKey": false, "watch": false, "port": 3000, "maxTokenAge": 86400, "store": { "adapter": "jugglingdb", "type": "memory" }, "searchHost": "www.hascode.com", "searchBasePath": "/wp-content/byTag.php?tag=", "maxResultsDisplayed": 10 } [..] }
Production Configuration for the Cloud and Redis
The production configuration differs from the development configuration:
- the port is specified by a placeholder variable (later set by Cloud provider)
- the localBaseUrl points to the Heroku app URL
- for persistence we’re using a scalable Redis database now, the connection URL again is set using a placeholder (see chapter Heroku setup)
{ [..] "production": { "usePublicKey": false, "port": "$PORT", "localBaseUrl": "https://hascode-tagsearch.herokuapp.com", "store": { "adapter": "redis", "url": "$REDIS_URL" }, "whitelist": [ "*.hipchat.com" ], "searchHost": "www.hascode.com", "searchBasePath": "/wp-content/byTag.php?tag=", "maxResultsDisplayed": 10 } }
In addition to this, we’re adding the following Procfile to tell our cloud provider how to start our application
web: node app.js
Plugin Descriptor
The plug-in descriptor is used to install the integration into the HipChat instance and to display requested permissions and web-hooks to establish interaction between HipChat and our application. First of all we’re adding some description about our plug-in and its vendor by specifying name, description, vendor etc.
More interesting is the capabilities section as here in the node
- hipchatApiConsumer we’re requesting concrete permissions that are granted to our integration upon install – for our plug-in all we need here is the permission to send notifications as we want to post the result of our blog search into the room.
- installable we’re specifying a callback URL that is called when someone installs our plug-in or removes it from a HipChat instance. Upon install we’re receiving the OAuth2 credentials and the clientId that we need to store to enable further communication between our application and the remote HipChat instance.
- webhook we’re specifying a callback URL (web-hook) that is called when a message is posted to a room that begins with “/hascode“.
{ "name": "hasCode.com Blog Search", "description": "Search the hasCode.com Blog for articles for a given tag using /hascode tag", "key": "com.hascode.hc.blogsearch", "vendor": { "name": "Micha Kops", "url": "http://www.micha-kops.com" }, "links": { "self": "{{localBaseUrl}}/atlassian-connect.json", "homepage": "{{localBaseUrl}}/atlassian-connect.json" }, "capabilities": { "hipchatApiConsumer": { "scopes": [ "send_notification" ] }, "installable": { "callbackUrl": "{{localBaseUrl}}/installable" }, "webhook": { "url": "{{localBaseUrl}}/message", "pattern": "^/hascode", "event": "room_message", "name": "Blog Search Command" } } }
Generated Base Application
Just for the sake of completeness is this the generated app.js that bootstraps the web application, sets up stuff .. and luckily for us, we don’t need to change a single line of code here.
// This is the entry point for your add-on, creating and configuring // your add-on HTTP server // [Express](http://expressjs.com/) is your friend -- it's the underlying // web framework that `atlassian-connect-express` uses var express = require('express'); // You need to load `atlassian-connect-express` to use her godly powers var ac = require('atlassian-connect-express'); process.env.PWD = process.env.PWD || process.cwd(); // Fix expiry on Windows :( // Static expiry middleware to help serve static resources efficiently var expiry = require('static-expiry'); // We use [Handlebars](http://handlebarsjs.com/) as our view engine // via [express-hbs](https://npmjs.org/package/express-hbs) var hbs = require('express-hbs'); // We also need a few stock Node modules var http = require('http'); var path = require('path'); var os = require('os'); // Let's use Redis to store our data ac.store.register('redis', require('atlassian-connect-express-redis')); // Anything in ./public is served up as static content var staticDir = path.join(__dirname, 'public'); // Anything in ./views are HBS templates var viewsDir = __dirname + '/views'; // Your routes live here; this is the C in MVC var routes = require('./routes'); // Bootstrap Express var app = express(); // Bootstrap the `atlassian-connect-express` library var addon = ac(app); // You can set this in `config.js` var port = addon.config.port(); // Declares the environment to use in `config.js` var devEnv = app.get('env') == 'development'; // Load the HipChat AC compat layer var hipchat = require('atlassian-connect-express-hipchat')(addon, app); // The following settings applies to all environments app.set('port', port); // Configure the Handlebars view engine app.engine('hbs', hbs.express3({partialsDir: viewsDir})); app.set('view engine', 'hbs'); app.set('views', viewsDir); // Declare any Express [middleware](http://expressjs.com/api.html#middleware) you'd like to use here app.use(express.favicon()); // Log requests, using an appropriate formatter by env app.use(express.logger(devEnv ? 'dev' : 'default')); // Include stock request parsers app.use(express.urlencoded()); app.use(express.json()); app.use(express.cookieParser()); // Gzip responses when appropriate app.use(express.compress()); // Enable the ACE global middleware (populates res.locals with add-on related stuff) app.use(addon.middleware()); // Enable static resource fingerprinting for far future expires caching in production app.use(expiry(app, {dir: staticDir, debug: devEnv})); // Add an hbs helper to fingerprint static resource urls hbs.registerHelper('furl', function (url) { return app.locals.furl(url); }); // Mount the add-on's routes app.use(app.router); // Mount the static resource dir app.use(express.static(staticDir)); // Show nicer errors when in dev mode if (devEnv) app.use(express.errorHandler()); // Wire up your routes using the express and `atlassian-connect-express` objects routes(app, addon); // Boot the damn thing http.createServer(app).listen(port, function () { console.log() console.log('Add-on server running at ' + (addon.config.localBaseUrl() || ('http://' + (os.hostname()) + ':' + port))); // Enables auto registration/de-registration of add-ons into a host in dev mode if (devEnv) addon.register(); });
Express Routes
This is our index.js that contains the core functionality of our application and the routes mapped to our web application.
The imported hipchat client layer eases the communication with the remote HipChat instance and its RESTful webservices (and its authentication mechanism).
We’re specifying the following routes:
- /message receives the blog search events triggered in HipChat when the /hascode command is used
- /installable receives install and uninstall events (POST or DELETE)
- other requests are redirected to the plugin descriptor file
When we receive a search event, we’re sending a new request to the blog search, we’re parsing the JSON response and generate a HTML list for display in the HipChat room.
If the amount of search hits exceeds the given limit of 10, we’re displaying the first 10 search results and additionally display a link to view all search results on the blog.
var http = require('http'); module.exports = function (app, addon) { var hipchat = require('../lib/hipchat')(addon); var options = { host: addon.config.searchHost(), path: '/', method: 'GET', headers: { 'Content-Type': 'application/json' } }; // Root route. This route will serve the `addon.json` unless a homepage URL is // specified in `addon.json`. app.get('/', function (req, res) { // Use content-type negotiation to choose the best way to respond res.format({ // If the request content-type is text-html, it will decide which to serve up 'text/html': function () { res.redirect(addon.descriptor.links.homepage); }, // This logic is here to make sure that the `addon.json` is always // served up when requested by the host 'application/json': function () { res.redirect('/atlassian-connect.json'); } }); } ); // This is an example route to handle an incoming webhook app.post('/message', addon.authenticate(), function (req, res) { var term = req.body.item.message.message.split(' ')[1]; console.log('searching for given term: ' + term); var message = "<a href=\"https://www.hascode.com/\">hasCode.com</a> blog articles for given term: <b>"" + term + ""</b>"; options.path = addon.config.searchBasePath() + term; var clientRequest = http.request(options, function (clientResponse) { var output = ''; clientResponse.setEncoding('utf8'); clientResponse.on('data', function (chunk) { output += chunk; }); clientResponse.on('end', function () { var hits = JSON.parse(output); var hitsNum = hits.length; console.log(hitsNum + ' results for term ' + term + ' found, max-results set to: ' + addon.config.maxResultsDisplayed()); message += ' (' + hitsNum + ' hit/s) <ul>'; message += hits.slice(0, addon.config.maxResultsDisplayed()).reduce(function (prev, cur) { return createMessage(prev) + createMessage(cur); }, ""); message += '</ul>'; if (hits.length > addon.config.maxResultsDisplayed()) { message += '<b><a href="https://www.hascode.com/tag/' + term + '">Show all ' + hitsNum + ' results for "' + term + '"</a></b>' } hipchat.sendMessage(req.clientInfo, req.context.item.room.id, message) .then(function (data) { res.send(200); }); }); }); clientRequest.end(); } ); // Notify the room that the add-on was installed addon.on('installed', function (clientKey, clientInfo, req) { hipchat.sendMessage(clientInfo, req.body.roomId, 'The ' + addon.descriptor.name + ' add-on has been installed in this room'); }); // Clean up clients when uninstalled addon.on('uninstalled', function (id) { addon.settings.client.keys(id + ':*', function (err, rep) { rep.forEach(function (k) { addon.logger.info('Removing key:', k); addon.settings.client.del(k); }); }); }); function createMessage(hit) { if (typeof hit === 'string') { return hit; } return '<li><a href=\"' + hit.url + '\">' + hit.title + '</a></li>'; } };
Running the Server Application, Tunnelling with ngrok
Ngrok is a nice tool that allows us to export our application running on localhost to the internet using a SSH tunnel.
The only downside of the free version is, that we’re getting a random sub-domain assigned.
In the following example, we’re tunnelling our application running on port 3000 to the outside world:
$ ngrok http 3000 ngrok by @inconshreveable (Ctrl+C to quit) Tunnel Status online Version 2.0.19/2.0.19 Web Interface http://127.0.0.1:4040 Forwarding http://db6b58f8.ngrok.io -> localhost:3000 Forwarding https://db6b58f8.ngrok.io -> localhost:3000 Connections ttl opn rt1 rt5 p50 p90 0 0 0.00 0.00 0.00 0.00
When now starting our application, we might want to override its base-URL by adding the following environment variable set to our ngrok sub-domain:
AC_LOCAL_BASE_URL=https://<subdomain>.ngrok.com node app.js
Automatic Application Restart with Nodemon
During development, we don’t want to reboot our application every-time we’re changing the application logic.
Luckily for us, there is nodemon to help us here: nodemon scans the project directory for file changes and reboots the application if changes were detected.
Nodemon is installed in no time using npm:
sudo npm install -g nodemon
Now when running our application with nodemon it is restarted automatically when a file has been changed e.g.:
$ AC_LOCAL_BASE_URL=https://c569b888.ngrok.io nodemon app.js 3 Aug 19:29:12 - [nodemon] v1.4.0 3 Aug 19:29:12 - [nodemon] to restart at any time, enter `rs` 3 Aug 19:29:12 - [nodemon] watching: *.* 3 Aug 19:29:12 - [nodemon] starting `node app.js` Add-on server running at https://c569b888.ngrok.io Auto registration not available with HipChat add-ons Initialized memory storage adapter 3 Aug 19:30:40 - [nodemon] restarting due to changes... 3 Aug 19:30:40 - [nodemon] starting `node app.js`
Integration Installation in HipChat
Finally our application is ready for integration into a concrete HipChat instance. This is done within a few steps that I’d like to demonstrate here.
Adding the Integration via Plugin Descriptor URL
We may add an integration for a room having sufficient permissions by clicking “Integrations” in the operation menu.
We’re now entering the URL to the deployment descriptor exported by our application (I have used address of my Heroku app – more about Heroku configuration later..).
Confirm Plugin Installation
HipChat loads the descriptor and displays the following confirmation with additional information about the plug-in vendor and required permissions.
As we can see, the only permission we’re requesting is the permission to send room notifications.
Finally HipChat displays an overview of the plug-in and the affected chat-rooms:
Our Integration in Action
First of all we should be able to view the response from the on-install hook that confirms that the plug-in was successfully installed to this room.
And as we can see in the following screenshot, we’re now able to search for blog articles by typing /hascode keyword #yeah :)
Heroku Configuration and Deployment
Our application is not bound for deployment and provision by a specific provider but to give a concrete example, I’d like to demonstrate the configuration and deployment for Heroku.
As a preparation all we need is a Heroku account and a free slot for a Nodejs application ;)
Deployment using Git
We just need to add our application’s Heroku git repository and push our changes and our application is launched automatically:
$ git push heroku master [..] remote: -----> Node.js app detected remote: -----> Creating runtime environment [..] remote: -----> Installing binaries remote: engines.node (package.json): unspecified remote: engines.npm (package.json): unspecified (use default) remote: remote: Resolving node version (latest stable) via semver.io... remote: Downloading and installing node 0.12.7... remote: Using default npm version: 2.11.3 remote: remote: -----> Restoring cache remote: Loading 1 from cacheDirectories (default): remote: - node_modules remote: remote: -----> Building dependencies remote: Pruning any extraneous modules remote: Installing node modules (package.json) remote: remote: -----> Caching build remote: Clearing previous node cache remote: Saving 1 cacheDirectories (default): remote: - node_modules remote: remote: -----> Build succeeded! remote: ├── atlassian-connect-express@1.0.4 remote: ├── atlassian-connect-express-hipchat@0.2.3 remote: ├── atlassian-connect-express-redis@0.1.3 remote: ├── express@3.21.1 remote: ├── express-hbs@0.8.4 remote: ├── lodash@3.10.0 remote: ├── request@2.60.0 remote: ├── rsvp@3.0.20 remote: └── static-expiry@0.0.11 remote: remote: -----> Discovering process types remote: Procfile declares types -> web remote: remote: -----> Compressing... done, 22.1MB remote: -----> Launching... done, v7 remote: https://nameofmyapp.herokuapp.com/ deployed to Heroku remote: remote: Verifying deploy.... done. To https://git.heroku.com/nameofmyapp.git f7ae602..439be85 master -> master
Adding the Redis Plugin
First, we’re adding the Heroku Redis Plugin to our application – either by using the administration panel in the browser or in the command line using the following command:
heroku addons:create heroku-redis:blogsearch-clients
We may verify that it is installed using the cli:
$ heroku addons | grep REDIS REDIS thinking-busily-3739 hascode-tagsearch
In addition we might want to verify that the connection URL to the Redis database is present in our environment:
$ heroku config|grep REDIS REDIS_URL: redis://h:1111111111111111111111111111.compute-1.amazonaws.com:10859
Setting the Production Stage
Again we have the choice to set variables on the Heroku dashboard or using the command-line interface. We’re setting the stage NODE_ENV to production using the following command:
$ heroku config:set NODE_ENV=production Setting config vars and restarting hascode-tagsearch... done, v6 NODE_ENV: production
The following screenshot demonstrates how the config variables are display in the Heroku app settings:
Viewing Application Logs
We’re able to view our application logs using the following command (snapshot taken during the installation of the integration into a HipChat room):
$ heroku logs --tail 2015-07-23T20:17:33.597314+00:00 heroku[web.1]: Starting process with command `node index.js` 2015-07-23T20:17:34.785515+00:00 app[web.1]: Detected 512 MB available memory, 512 MB limit per process (WEB_MEMORY) 2015-07-23T20:17:34.785541+00:00 app[web.1]: Recommending WEB_CONCURRENCY=1 2015-07-23T20:17:35.083154+00:00 app[web.1]: Node app is running on port 37451 [..] 2015-07-30T08:40:10.210809+00:00 heroku[router]: at=info method=GET path="/atlassian-connect.json" host=hascode-tagsearch.herokuapp.com request_id=a0bebd4f-0965-4d0d-b987-8bc496bff28e fwd="54.197.167.216" dyno=web.1 connect=1ms service=5ms status=200 bytes=907 2015-07-30T08:40:10.208892+00:00 app[web.1]: ::ffff:10.181.95.28 - - [30/Jul/2015:08:40:10 +0000] "GET /atlassian-connect.json HTTP/1.1" 200 680 "-" "-" 2015-07-30T08:40:10.602429+00:00 heroku[router]: at=info method=POST path="/installable" host=hascode-tagsearch.herokuapp.com request_id=bba03b73-4cd6-4129-85b1-6969446159e7 fwd="54.197.167.216" dyno=web.1 connect=2ms service=240ms status=204 bytes=142 2015-07-30T08:40:10.600201+00:00 app[web.1]: Saved tenant details for 37052e9e-b215-4000-a5b6-b471a0008f3e to database 2015-07-30T08:40:10.600208+00:00 app[web.1]: { clientKey: 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeee', 2015-07-30T08:40:10.600209+00:00 app[web.1]: oauthSecret: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', 2015-07-30T08:40:10.600211+00:00 app[web.1]: capabilitiesUrl: 'https://api.hipchat.com/v2/capabilities', 2015-07-30T08:40:10.600213+00:00 app[web.1]: capabilitiesDoc: 2015-07-30T08:40:10.600214+00:00 app[web.1]: { capabilities: { hipchatApiProvider: [Object], oauth2Provider: [Object] }, 2015-07-30T08:40:10.600216+00:00 app[web.1]: description: 'Group chat and IM built for teams', 2015-07-30T08:40:10.600217+00:00 app[web.1]: key: 'hipchat', 2015-07-30T08:40:10.600219+00:00 app[web.1]: links: 2015-07-30T08:40:10.600221+00:00 app[web.1]: { api: 'https://api.hipchat.com/v2', 2015-07-30T08:40:10.600222+00:00 app[web.1]: homepage: 'https://www.hipchat.com', 2015-07-30T08:40:10.600224+00:00 app[web.1]: self: 'https://api.hipchat.com/v2/capabilities' }, 2015-07-30T08:40:10.600225+00:00 app[web.1]: name: 'HipChat', 2015-07-30T08:40:10.600226+00:00 app[web.1]: vendor: { name: 'Atlassian', url: 'http://atlassian.com' } }, 2015-07-30T08:40:10.600228+00:00 app[web.1]: roomId: 123456, 2015-07-30T08:40:10.600229+00:00 app[web.1]: groupId: 654321, 2015-07-30T08:40:10.600231+00:00 app[web.1]: groupName: 'hasCode.com' } 2015-07-30T08:40:10.601568+00:00 app[web.1]: ::ffff:10.171.92.168 - - [30/Jul/2015:08:40:10 +0000] "POST /installable HTTP/1.1" 204 - "-" "-"
Another nice add-on to aggregate and visualize our logs is the Heroku Papertrails plugin – it’s limited but for free!
Tutorial Sources
Please feel free to download the tutorial sources from my Bitbucket repository, fork it there or clone it using Git:
git clone https://bitbucket.org/hascode/hipchat-integration-tutorial.git
Resources
- Atlassian HipChat
- Atlassian Developer Documentation: Atlassian Connect
- Atlassian Connect Express on Bitbucket
- Heroku Website
- ngrok Website
- Nodejs Project Site
- npm Website
- Express Website
- nodemon Website
- Heroku Redis Plugin Site
Tags: Atlassian, atlassian-connect, chat, database, db, express, heroku, hipchat, inmemory, integration, javascript, js, jugglingdb, ngrok, node, nodejs, nodemon, nosql, npm, oath, oauth2, redis, rest, slack, staging, webhook