Category Archives: NodeJS

Setting up PostgreSQL (9.6) on Debian Stretch

I have been trying to setup a home network of connected devices for… lets just say a long time! I have made some progress in recent days and the next step is to start using some sort of backing store to store some values.

While MongoDB works well with the Nodejs stack, it isn’t well supported on Raspbian because Raspbian is 32bit and MongoDB stopped supporting 32 bit fully due to data size restrictions (biggest DB you can have is 2GB). Raspbian Stretch does provide an updated MongoDB but it is still a few (minor versions) short of the shiniest.

However using Jessie Backports I was able to install PostgreSQL 9.6 (the shiniest version). Having installed it, I had no idea what to do next. Here’s my journey to get PostgreSQL to a place where I can start using it.

My dev environment’s base OS is OSX and I run my DB in a VM hosted by VirtualBox. So I have setup a Debian Stretch VM called dev-db before I start with PostgreSQL installation. I am trying to setup a parallel runtime environment on a Raspberry Pi as well so if there are any exceptions to these steps I’ll document them for the Raspbian as well.

Installing on Debian/Raspbian Stretch

Both the distros have the required repositories in the apt list, out of the box, so all you have to do is

sudo apt-get install postgresql-9.6

This will create a PostgreSQL default user called ‘postgres’.

sudo service start postgres

Setting up your User Account

PostgreSQL databases are usually tied to a user. Out of the box, PostgreSQL users are not the same as *nix account users, however, if you a common name between your *nix account and PostgreSQL account things become much easier. So lets assume account I am using to log in to Debian stretch is called ‘dbuser’. It is in the sudoers list and I logged in to the terminal/desktop using it.

First step is to change user from ‘dbuser’ to ‘postgres’ that PostgreSQL created during installation.

sudo -i -u postgres

This is the admin account for postgresql, in a production environment, it is a good idea to protect this with a password at least.

Next step is to tell PostgreSQL, we have a ‘dbuser’ account that we want to be able to use for create databases in PostgreSQL.

postgres@dev-db: ~$ createuser dbuser --pwprompt

It will ask for password, I entered the same password as my *nix account but that may not be a best practice.

Finally create a db for the dbuser

postgres@dev-db: ~$ createdb dbuser

Now log out of the postgres account and go back to dbuser account

postgres@dev-db: ~$ exit
dbuser@dev-db: ~$

Start psql client and you should get logged into the pi database using the PLSQL

dbuser@dev-db: ~$ psql
psql (9.6.4)
Type "help" for help.
pi=>

Enter \q to exit psql.

We have setup PostgreSQL and have established local client access

Enabling remote access

Default PostgreSQL setup enables local client access only. In Dev environment I like to access my VM or PI using a client on my local desktop/laptop. This needs a few additional steps.

Step 1: Open up firewall on VM or Raspberry Pi hosting the DB Server

sudo iptables -A INPUT -s 192.168.0.21 -p tcp --destination-port 5432 -m state --state NEW,ESTABLISHED -j ACCEPT
sudo iptables -A OUTPUT -s 192.168.0.21 -p tcp --destination-port 5432 -m state --state NEW,ESTABLISHED -j ACCEPT

Step 2: Edit PostgreSQL config to allow remote access

Edit the pg_hba.conf file

sudo nano /etc/postgresql/9.6/main/pg_hba.conf

Scroll to the bottom and add the following line

host all all 192.168.0.21/24 trust

Edit the file postgresql.conf

sudo nano /etc/postgresql/9.6/main/postgresql.conf

Find the setting ‘listen_address’ and set its value to the IP address of the VM/Raspberry Pi. This line maybe set commented (starts with #), so uncomment it first.

listen_addresses = '192.168.0.21'

Restart PostgreSQL server

sudo service postgresql restart

You should now be able to access if from a Remote Client.

I use the standard pgAdmin 4 and the connection settings were as follows

Screen Shot 2017-09-10 at 5.48.34 PM

And done!

Up next compiling and setting up Redis on the Pi πŸ™‚

Advertisements
Tagged , , ,

Getting started with NodeJS – Part 1: Fumbling around

I’ve been meaning to try out NodeJS for a while now, and finally got around to doing it over the last few days. I thought I would share my experience as I go along.

Update: Those ‘few days ago’ are actually a couple of months now πŸ˜‰

I have used NodeJS as a (build) tool to help me ‘compile’ front-end scripts, that involves taking dev source code and minifiying it into cache-busted deployable code. I use gulp for it and it works pretty okay. Fact is, while writing the gulp script I got pretty interested in NodeJS.

Also given the fact that ASP.NET vNext is pretty much going the ‘Node way’, I thought I should know what the real deal is, before I muck around with ASP.NET vNext.

So here is my first go at building ‘something’ using NodeJS as a platform as opposed to just a Dev/Build tool. The article expects you to have heard of NodeJS and NPM (Node Package Manager, something like Nuget but runs off the command line and available to both Windows and *nix). If you have never used either of them, it’s fine.

Environment

OS

Debian 8 (Jessie).

My readers using Windows fear not, you can use NodeJS on Windows using nearly the same steps, so if you get stuck just let me know and I’ll try and help.

NodeJS:

node --version

v4.2.1

npm --version

2.14.7

Side note on upgrading Node in Debian: If you read my previous article I had mentioned Jessie comes with a Node package by default, but it’s a rather old one. I un-installed that one using

sudo apt-get remove nodejs

Thereafter I followed the step outlined on Nodejs.org . Reproduced here

curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -

sudo apt-get install -y nodejs

This basically downloads the latest package from the official node repository and installs it.

Windows Users: Just get the latest Node installers from nodejs.org and rock on! NPM is installed as a part of Node.
OSX Users: You guys are all good, just install Node and the rest of the commands should be all the same.

IDE/Editor

Well, you could use anything you want, starting with Visual Studio full on, to Visual Studio Code, or any other IDE/Editor that suits your fancy. I am using Atom by Github. I am new to Atom as well, so there might be some moments when the experienced Atom user in you wince at my noobish-ness.

The Project

Well, I want to figure out what it takes to use QuillJS wrapped in a KO Component and then save the text in component into Azure Blob Storage. Simple right ;-). The project is called ParchmentScroll. Why? Well you use quills to write Scrolls on Parchment paper… πŸ˜‰ πŸ˜‰ πŸ˜‰

Oh, BTW, QuillJS is a really cool JavaScript library for add Rich Text capabilities to your Web Application. It was open sourced by Sales Force and is available under a permissive BSD license.

So lets get started, but before that lets try wrapping our head around ‘Server-side’ JavaScript.

JavaScript… umm… TypeScript everywhere (client-side and server-side)

You either love JavaScript or loathe it! I have made my peace with it and I kind of like its dynamic quirkiness. After I started using TypeScript I like JavaScript even better.

Anyway, traditionally we all know how to use JavaScript in the browser. But Node JS takes JavaScript and runs it through Google’s V8 engine on the server so you can actually write HTTP services in JavaScript. So you can have a HTML page hosted on IIS, NGINX, Apache or wherever, do an AJAX post to your NodeJS application that you’ve written in JavaScript and send back a response. To put things in contrast with the .NET world, think of writing Web API services, but instead of writing ApiControllers in C# you get to write it in JavaScript.. err TypeScript!!!

Down to some code… err well… kind of

Since I am using Atom, a lot of the steps I describe here, to setup a blank project, will look long drawn, when compared to Visual Studio’s File->New Project->Wizard->Done.

So lets get started.

Open a terminal.

Select/Create your favourite project folder and navigate to it. Mine is at

cd /home/sumitkm/myprojects/demo/parchmentscroll

Packages, their managers and Node JS

The NodeJS ecosystem thrives on a huge repository of third party libraries that are distributed as packages. Packages inherit the idea of Linux packages. They bundle self contained units of code/binaries that can be installed and updated using their respective package managers.

In this project I have used three package managers

1. The Node Package Manager aka npm – This is Node’s default package manager and is used to distribute all node packages, binaries and extensions. Fun fact, you use npm to install other package managers :-). So npm is the alpha dog of package managers in Node and is installed by default with Node. Node packages are mostly used for installing dependencies that you will use on the server side. For client side Script/style dependencies you use the next package manager – Bower.

2. Bower – The front-end package manager. Bower installs front-end dependencies that are mostly distributable versions of the libraries or frameworks that you will use e.g. KnockoutJS, RequireJS, QuillJS etc. To get started with Bower you first need to install it using npm at the global npm repository location as follows.

Please note if you are are not the administrator but have sudo-er rights, you need to prepend sudo to every shell command unless I say you don’t need one explicitly.

npm install bower -g

3. TSD – The TypeScript Definitions package manager. While the good thing about TypeScript is it provides better code management due to type enforcement at compile time, the flip side is you need TypeDefinitions for existing libraries written in ES3-4. DefinitelyTyped is a nice Open Source repository of TypeDefinitions that users have contributed as they have used existing libraries. While creating a TypeDefinition is relatively easy, its good to have a leg up with the existing libraries. So we install TSD a package manager that helps you retrieve type definitions for libraries you will use from the DefinitielyTyped repository

npm install tsd -g

We start by initializing an NPM ‘project’. This creates a package.json file which has the list of dependencies as well as details like the Project Name, version, Git repository, Author name, License information etc.

npm init

(don’t use sudo here)

This will present you with a series of prompts where you provide the requested details and it will in turn scaffold a package.json file for you. I provided the following details:

This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.
See `npm help json` for definitive documentation on these fields
and exactly what they do.
Use `npm install  --save` afterwards to install a package and
save it as a dependency in the package.json file.
Press ^C at any time to quit.
name: (parchmentscroll) 
version: (1.0.0) 
description: A blogging platform built using Node, QuillJS and TypeScript
entry point: (index.js) 
test command: 
git repository: https://github.com/sumitkm/parchmentscrolldemo/
keywords: QuillJS, NodeJS, TypeScript
author: Sumit Kumar Maitra
license: (ISC) MIT
About to write to /home/sumitkm/myprojects/demo/parchmentscroll/package.json:
{
  "name": "parchmentscroll",
  "version": "1.0.0",
  "description": "A blogging platform built using Node, QuillJS and TypeScript",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/sumitkm/parchmentscrolldemo/"
  },
  "keywords": [
    "QuillJS",
    "NodeJS",
    "TypeScript"
  ],
  "author": "Sumit Kumar Maitra",
  "license": "MIT"
}
Is this ok? (yes) yes

If you do ls now, you’ll see that a package.json file exists in the folder.

Here on, you have to decide your project structure. There are lots of sensible defaults, you can look up on the net. I am trying out one that I feel comfortable with. I may change it as we go along and build the project.

Since there are no csproj files or equivalent (package.json is a distant cousin, more like a .sln file than anything else), I am going to create top level folders as projects. So I create two main folders

(no sudo required)

mkdir www

mkdir server

Next will initialize the typescript definition file tsconfig.json
(no sudo required)

tsc --init

This creates a TypeScript config file that helps TypeScript compiler with location of the ts files in the project and other configuration items. If you open the file in an editor you’ll see the default:

{
"compilerOptions": {
"module": "commonjs",
"target": "es3",
"noImplicitAny": false,
"outDir": "built",
"rootDir": ".",
"sourceMap": false
},
"exclude": [
"node_modules"
]
}

 

The compiler options are same ones available via command line. I tend to remove the outDir attribute completely. This results in the .js files being generated in the same folder as the .ts file. This fits better with my deploy script that we’ll see sometime in the future.

The exclude array tells typescript compiler which folder it shouldn’t look at. Currently only node_modules is excluded.

The final tsconfig.json file we are starting with is

 

"compilerOptions": 
 {
  "module": "commonjs",
  "target": "es3",
  "noImplicitAny": false,
  "rootDir": ".",
  "sourceMap": true
 },
 "exclude": [
   "node_modules"
  ]
 }

This completes our ‘File->New Project’. Here on we’ll get on with some real code.

Application layout in a little more details

In the previous section we created two folder server and www as our two ‘projects’. The server folder will be root folder for all the server side logic and the www folder will hold whatever resources the browser needs to serve up the web page. So folder names basically help us with a mental segregation of what goes where.

 

Node JS can open ports and server content on ports if you want it to. But we don’t want to go that low level. Instead we’ll get help from a framework called Express JS to the low level stuff of opening/listening to ports, parsing requests, sending back responses etc. Basically we’ll use ExpressJS to bootstrap the application. The handy bit is, Express can serve up static files as well, so we’ll use the same framework to host the front-end and handle backend request/responses.

Down to some code, finally!

Getting started with ExpressJS

Setting up anything in node basically means npm –install. Express is no different. In the ‘parchmentscroll’ folder using the following

npm install express --save 

Save tells npm to update the package.json file with this particular dependency. So when you get sources on to a new folder all you have to do is npm install and all dependencies listed in package.json will be installed for you.

Time to start up Atom in the parchmentscroll folder enter

(no sudo required)

atom .

This should launch atom with the following layout

new-project-atom-editor
Under the server folder create a folder called app

Add app.ts file under app folder. This is going to be our entry point into the application.

But before we start writing code, we need a little more ‘configuration’ to do.

Back to the console in the parchmentscroll folder we’ll use the tsd package manager to install typescript definitions for node itself
(no sudo required)

tsd query node --action install --save

This tells tsd to look for the node type definition and if found install it and save it to tsd.json

Similarly we install typedefinitions for ExpressJS as well

ts query express --action install --save

Next we’ll install a couple of npm modules that Express JS uses for parsing a request body and serving up static files.

npm install body-parser

npm install serve-static

We also need the typescript definition for these two, so invoke tsd again –

tsd query --action install serve-static --save --

tsd query --action install body-parser --save --resolve

Note the — resolve flag that we’ve used in the above two commands. This tells tsd to resolve sub-dependencies of the library and get their type-definitions as well. You’ll note both use another dependency call mime that gets installed automatically.

Back to Atom in app.ts paste the following code and save the file.

/// <reference path="../../typings/tsd.d.ts"/>
import * as express from "express";
var app = express();
var bodyParser = require('body-parser');
app.use(bodyParser.json()); // for parsing application/json
app.use(bodyParser.urlencoded({ extended: true })); // for parsing application/x-www-form-urlencoded
app.use(express.static('www/'));
var server = app.listen(3001, () =>
{
 var host = server.address().address;
 var port = server.address().port;
 console.log('Example app listening at http://%s:%s', host, port);
});

– This code initializes express.
– Initializes an instance of the body parser module and sets it up to handle HTTP request body of type application/json

– Sets up the bodyParser module to handle parsing of url encoded HTTP requests

– Sets up express to handle static files in the ‘www’ folder (which is currently empty).

– Finally it setup up the express instance to listen to port 3001 and once the server starts print out a console message.
With the code setup, switch back to the terminal and in the parchmentscroll folder run tsc.

tsc 

The code should compile silently and come back with no messages.

Next we try to run the app using the following command

node .

The . tells Node to use the package.json to start up. However you’ll get an error at this point.

Error: Cannot find module '/home/sumitkm/myprojects/demo/parchmentscroll'
at Function.Module._resolveFilename (module.js:338:15)
at Function.Module._load (module.js:289:25)
at Function.Module.runMain (module.js:467:10)
at startup (node.js:136:18)
at node.js:963:3

This because when we setup our package.json we said index.js was our ‘main’ file. Problem, easily fixed, switch to Atom and open package.json.

Set the “main” attribute to “server/app/app.js” instead of the initial “index.js”.

Save the file and flip back to the terminal. Run node . Again

node .

This time you should see a message like the following:

Example app listening at http://:::3001

If you open your browser and go to localhost:3001/ you’ll get a message saying β€œCan’t GET /”

So switch back to Atom and add a file under www called
index.html.

Add a bit of hello world markup

<!DOCUMENT>
<body>
Hello Node JS
</body>

Save the file.

Refresh the browser and voila!

 

first-node-js-hello-world
Phew! Lot of work for a Hello World!

To sum up…

That may have seemed a lot of work up-front but all of it can be automated and scaffold-ed if we wanted to. Open source tool chains are much lighter weight when compared to enterprise apps like Visual Studio. However, they give you a lot more freedom to mix and match and hey all of them are actually free without you signing away your keystrokes in some EULA.

We have not even scratched the surface on NodeJS yet. In the next part, I’ll jump straight into more real life Node JS concepts like routing and middleware and show how to build front-end clients as well as HTTP services using it.

To be continued… (oh and wish you all a Happy new 2016)!

Tagged ,
%d bloggers like this: