Using Sails JS to Create an IMAP Email API
Ahoy! There be buried treasure and email ahead.
Ok, so mostly email but hopefully, you will treasure this guide.
I recently discovered SailsJS when I had the crazy idea to create an email newsletter API service for my startup. The idea is simple to aggregate content into a single source and send it out to my readers. I needed something easy to pick up and fast to develop.
SailsJS was a breeze to learn so I set sail with IMAP in hand but quickly found deployment to be an anchor. So hang tight and I’ll guide you through my struggles at sea in this sails js tutorial.
Setting Sail
Before we leave port we’ll need to set up our sails project so go ahead and run either command to install Sails globally.
# yarnyarn global add sails# npmnpm install sails -g
Setting up a new project is as easy as:
sails new my-email-api
For this tutorial, we’ll choose the empty template. This will take a moment to install dependencies but once it’s done you can:
cd my-email-apisails lift
Sea for yourself at localhost:1337
Note: Sails uses npm for the initial install if you’re using npm for this tutorial this is fine but if you end up using yarn just delete the package-lock.json file.
Setting up your Email (The IMAP server)
You remember that old tongue twister; ‘She sends emails by the seashore’
Or something like that…
Hopefully, your domain already has its own email. If you do then you just need to find your IMAP settings and you can skip this step.
Still here? Ok, Next I’ll show you how to set up an email account with ZOHO. I chose ZOHO because they were a cheap option but still have plenty of features.
For this to work you will have to sign up for the $1 a month subscription plan. This will give us access to IMAP connectivity in ZOHO. If you have a free option for this please let me know in the comments but for me, $12 a year isn’t too much to ask.
You can sign up here:
This guide assumes that you have your own domain so if you don’t go ahead and set one up through ZOHO here or with another host.
With a domain in hand go ahead and fill out the requested information in the next few forms.
You’ll need to configure and confirm various things here is a brief list with links to guides for each step:
Email and Other Mysteries (What is IMAP?)
The aforementioned acronym stands for Internet Message Access Protocol and it’s basically just a fancy way of interacting with emails programmatically.
If you didn’t set up an account with ZOHO then this section will be different for you. You’ll need to find the IMAP settings for your email provider, In ZOHO it’s located at https://mail.zoho.com/zm/#settings/all/mailaccounts
Under the IMAP tab click the access check button and hit save. Below the check button, you can find the IMAP server configuration we’ll need that in a moment.
Now back in the project install the imap-simple library:
# yarnyard add imap-simple# npm npm install imap-simple
This is a wrapper around the node-imap client for node js that makes using it easier. Now get ready to create your first API endpoint.
We’ll also want to do some basic parsing on our emails so add this mail parser library as well.
# yarnyarn add mailparser# npmnpm install mailparser
Burying Treasure (API route for Emails)
Our app is just a homepage for Sails js documentation at this point. We need an endpoint to call and get our emails so run this command.
sails generate action mail/newsletter
This will make a standalone action in api/controllers/mail called newsletter.js that will handle requests to get items out of our newsletter box in ZOHO.
If you go to /mail/newsletter you’ll get a 404 page but why?
Well by default Sails makes you define your own routes in config/routes.js but we can skip this step by defining blueprints for our actions.
In config/blueprints uncomment the actions line and change it to true:
actions: true,
If you restart your sails lift process and return you should see the message “OK” displayed at the top of the page. Before we make changes to this file there are a few more things we have to set up first.
Add the dotenv library to the project:
# yarnyarn add dotenv# npm npm install dotenv
Then require it in the app.js file at the root of your project:
// app.jstry {...require('dotenv').config();}
Now add a lift command in your package.json file:
"scripts": { "lift": "node app.js --port=8080", ...}
Now add a .env file at the root of your project:
NODE_TLS_REJECT_UNAUTHORIZED='0'IMAP_USER=you@yourdomain.comIMAP_PASSWORD=YourZohoEmailPasswordIMAP_HOST=imappro.zoho.comIMAP_PORT=993IMAP_TLS=true
This will be the IMAP configuration you found earlier most likely these settings will be the same but use whatever settings you have if they differ.
The setting at the top will allow our endpoint to be accessed from a non-secure channel since we’re just testing this in development we don’t need to be using HTTPS.
Finally, Stop the current lift process and run the new one:
# yarnyarn lift# npmnpm run lift
Make sure this time you visit localhost at the new port: localhost:8080
Now you can replace the newsletter.js file with the following code:
// newsletter.jsconst imap = require('imap-simple');const { simpleParser } = require('mailparser');const { find } = require('lodash');module.exports = {friendlyName: 'Newsletter',description: 'Newsletter mail.',inputs: {},exits: {error: {message: 'Error!'},success: {data: null,message: 'success!'}},fn: async function (_inputs, exits) {try {const result = [];var config = {imap: {user: process.env.IMAP_USER,password: process.env.IMAP_PASSWORD,host: process.env.IMAP_HOST,port: Number.parseInt(process.env.IMAP_PORT),tls: JSON.parse(process.env.IMAP_TLS)}};const connection = await imap.connect(config);await connection.openBox('NEWSLETTER');const messages = await connection.search(['1:5'], { bodies: ['HEADER', 'TEXT']});for (const item of messages) {const all = find(item.parts, { 'which': 'TEXT' });const header = find(item.parts, { which: 'HEADER' });const id = item.attributes.uid;const idHeader = 'Imap-Id: '+id+'\r\n';const { html, textAsHtml, text } = await simpleParser(idHeader+all.body);result.push({ html, id, textAsHtml, text, from: header.body.from });}exits.success({ message: 'success', data: result });} catch (err) {exits.error({ message: err.message });}}};
What we’re doing here is opening a connection to our mailbox and opening the email newsletter box. If you haven’t already go ahead and subscribe to at least one newsletter using your ZOHO email so you can test this out. Most newsletters will be automatically recognized and moved into this folder.
Once we’ve opened the box we get all the messages and parse their bodies. We’ll return back basic things like the ‘id’, ‘text’, and ‘from’ fields. Once it’s all said and done you should see your messages displayed as JSON in the browser when you visit mail/newsletter.
Marooned (Azure DevOps)
This is great and all but we can’t actually use this API until it’s deployed to the ocean that is the internet. Unfortunately, there seemed to be limited options for deployment and very few guides.
I ended up going with azure for my hosting and had issues during the setup process. You’re going to need git set up on your machine for this next part I won’t cover that here but there are plenty of guides.
In the project initialize a git repo and do the initial commit:
git init git add -A git commit -m "Initial Commit"
We’ll be using the azure client to set up our pipeline you can download it here:
You need to have an Azure subscription so go ahead and create one and log in:
az login
Then create the app group and set it to default:
az group create --name sailsGroup --location westusaz configure --defaults group=sailsGroup location=westus
Now create the actual app service:
az webapp up --name appName --logs --launch-browser
This will take a few minutes but once that’s done it should launch your new azure app in a new tab.
Now, set up your user level deployment using new credentials specific to this deployment (Not your azure credentials):
az webapp deployment user set --user-name <username> --password <password>
And retrieve the azure git endpoint:
az webapp deployment source config-local-git --name appName
This should output something like this in the terminal:
{ "url": "https://userName@testsails.scm.azurewebsites.net/appName.git"}
You’ll use this output to set the azure remote branch:
git remote add azure https://userName@testsails.scm.azurewebsites.net/appName.git
It’s the URL minus the username. Finally, push your code to the new remote:
git push azure master
You’ll be prompted for the password you made earlier.
Edit: 3/11/2021
I ran into some issues with authentication using the remote without the userName. If you got an authentication error during the last step and you didn’t get the option to sign in then add the remote with the userName.
git remote set-url azure https://userName@testsails.scm.azurewebsites.net/appName.git
Once that’s done you should be able to refresh the browser window that was opened earlier and see… an error.
Here there be monsters (Errors and more fun)
This was the part that tripped me up. We aren’t actually done with configuration yet.
For one the container is looking for our app at port 8080. If you recall we changed this in our lift command but the start command that runs our entry point at app.js doesn’t use this port. To configure it update the port value in config/env/production.js.
port: 8080,hookTimeout: 30000
While we’re in this file there are a few more steps to get our app ready for a production deployment:
// config/env/production.js// In the session settingscookie: {secure: true,maxAge: 24 * 60 * 60 * 1000, // 24 hours},
And in the security section of production.js we need to set up our cors settings:
cors: {allowOrigins: ['https://your-requesting-site',]},
Just put any of the sites where you’ll be making API requests from here. And we’ll also be including any sites we list here in the config/sockets.js file.
onlyAllowOrigins: ['https://your-requesting-site']
I have to credit this source for the grunt configuration steps coming up.
This guide explains to downgrade grunt in your package.json to 1.0.0 and to remove the sails-hook-grunt dependency altogether.
This is how your dependencies should look after you’re done.
"dependencies": {"@sailshq/connect-redis": "^3.2.1","@sailshq/lodash": "^3.10.3","@sailshq/socket.io-redis": "^5.2.0","dotenv": "^8.2.0","grunt": "1.0.0","imap-simple": "^5.0.0","mailparser": "^3.1.0","sails": "^1.4.0","sails-hook-orm": "^3.0.1","sails-hook-sockets": "^2.0.0"}
Of course, after making these changes you’ll have to reinstall your packages. These were the magic settings I had to use to get my app running.
After you’ve made all these changes and committed them to the master branch go ahead and push your changes to Azure.
git push azure master
You should now be able to refresh your app in the browser and see the default Sails js welcome page, if not you may have to rerun the azure webapp up command from earlier. If you run into troubled waters you can run:
az webapp log tail --name appName
This will give you more detailed logs for your deployment. As with everything else we’re done in Azure this can be viewed in the Azure console online.
Message in a bottle (Requesting our Emails)
With all the fun stuff out of the way… Now, all we need to do is actually get our emails.
The only thing we’re missing is to add the environment variables we declared in the app to our production app settings. To do this you’ll be using the Azure portal and go to your new app service. From there navigate the sidebar until you find Configuration under the settings section.
Click the new application setting button and add each of the name/value pairs we used locally. Make sure that the name you use is the same casing, for this guide, it’s all uppercase.
This may not be necessary but I went ahead and set the TLS value back to ‘1’ in these settings since our deployed app now is using HTTPS.
NODE_TLS_REJECT_UNAUTHORIZED='1'
Don’t forget to hit save at the top once you’re done. With those settings in place go to the URL of your Azure app. If you don’t know what it is you can find it via the Overview section located at the top of the sidebar panel.
Navigate to the endpoint we made by appending /mail/newsletter to the URL and you should see a JSON list of our emails, congratulations!
Conclusion
This has been my journey with SailsJS from building an endpoint to deployment. As a next step, I recommend you add security to the endpoint you made so that not just anyone can go to your app to read your emails. You can find out more about how to secure your API with policies in SailsJS here.
And if you found this sails js tutorial helpful please sign up for the email newsletter where I will be using the API I’m showcasing here.
And you can follow me on Twitter for more tutorials like this one @SquashBugler
So have you had smooth sailing with SailsJS or have you been marooned? Let me know in the comments and happy sailing!
Other Posts
