Introduction to Node.js / A beginners guide to Node.js and NPM

Node.js & NPM

Sometimes it is very difficult to learn a new programming language because we generally don?t know where to begin. Node.js is nothing but JavaScript running on the server and it?s super awesome. In this article, we will learn how to set it up and understand some key concepts. By the end of this article, you will be to writing applications using Node.js.

(Source: Background(https://bit.ly/2HoyhFL), Logo(https://bit.ly/2NsWTB0))

Before going directly into Node.js, we need to understand JavaScript first.

Introduction to JavaScript

JavaScript is dynamically typed single-threaded interpreted languages for the Web. That means if you are doing web development, you can use this language to perform some operating on the web page, like running some JavaScript code when a button is clicked by the user.

JavaScript is a dynamically typed language which means a variable can hold any data type like String or Number in its lifetime and JavaScript interpreter won?t complain about it. It?s single-threaded which means your JavaScript code runs synchronously or sequentially line by line. It?s interpreted which means you don?t need to compile your JavaScript code.

JavaScript is interactive, which means you can directly feed JavaScript code to the interpreter and it will be executed immediately. You can try this by opening DevTools in the browser (in Chrome, press command + option + i) or right-click anywhere on the page and click inspect. Then go to the console tab, here you can type any valid JavaScript code and press enter to run it. Use shift + enter to add a new-line in your code.

(interacting with JavaScript interpreter)

Every browser ships a JavaScript Interpreter also called a JavaScript Engine. V8 is the JavaScript engine designed by Google and used in the Google Chrome browser while SpiderMonkey is a JavaScript engine developed by Mozilla for their Firefox browser.

Since JavaScript engine designed by every browser is different, ECMA standardizes features of JavaScript. This standard is known as ECMAScript (pronounced as ek-ma-script). Whenever ECMA adds a feature to this JavaScript standard, the browser has to add it in their JavaScript engine to stay in the competition (though this process is very slow).

JavaScript is a very easy language to learn and fun to write. Every year, new features are added to ECMAScript which brings JavaScript one more step closer to dominate the planet. The latest major revision of JavaScript is ES6 or ECMAScript 6 or ECMAScript 2015 which has dumped a ton of features to make it more fun to code in. At the moment, JavaScript supports the OOP paradigm very well and can be used in functional programming as well.

Mozilla is an open-source foundation that documents JavaScript very well on their developer documentation AKA Mozilla Developer Network or MDN. It is one of the top online destinations to learn JavaScript, though there are other online resources as well. If you want to take a look at the JavaScript specifications and learn simple tutorials, visit MDN Documentation.

JavaScript is sometimes abbreviated as JS or .js/.JS (dot J S) to state that an entity is related to JavaScript, like Node.js or ReactJS or AngularJS. But in no ways, JavaScript is related to Java, or so you think ?. If you need a history lesson about JavaScript and its evolution, watch this amazing video.

https://www.youtube.com/watch?v=Sh6lK57Cuk4

Assuming that you are familiar with JavaScript and gained a good amount of knowledge, we can move forward. But if you don?t know JavaScript at all, learn basic JavaScript from the MDN documentation I explained earlier. Because learning about Node.js without knowledge of JavaScript is like understanding web development without HTML.

What is server-side JavaScript?

JavaScript is a single-threaded language, it knows how to get things done one at a time. It can?t do asynchronous tasks or run JavaScript code in multiple threads for efficiency. It just knows about JavaScript as defined in ECMAScript specification and nothing more.

Since JavaScript is used on the web, it needs to be secure. Hence, using JavaScript, you can?t access the computer it is running on, like File System, IO, Networking, etc. and neither ECMAScript has specifications for that.

So it is up to the browser vendors to extend JavaScript engine with APIs that can do other things. For example, DOM API is responsible to print an HTML code into actual pixels on the screen, I have explained this process in my Medium article. Also, the XMLHttpRequest API gives us the ability to send network requests to fetch data from a remote server in the background.

These sorts of APIs are responsible to perform other operations that JavaScript is not designed to perform. These APIs are provided by the browser and they are called Web APIs. These APIs are written in some Low-Level languages like C or C++ but their interface is made available through JavaScript.

These Web APIs sometimes do their job in separate thread allowing other JavaScript code to run normally while the job is running in the background. Once the job is done, it then informs the main JavaScript thread.

So through JavaScript, you are executing C++ code and returning result back to the JavaScript, that doesn?t sound so difficult.

For example, setTimeout(callback, delay) function is not part of ECMAScript specification, it is provided by the browser to perform an asynchronous operation. The callback function is executed in the main JavaScript thread once delay milliseconds has elapsed.

(an oversimplification of how JavaScript runs in a browser)

So far we know that JavaScript is essential in a browser. But if you think about JavaScript engine alone, it can exist anywhere. You can take the V8 JavaScript engine and install it on your computer (let?s call it as a server). With some fiddling, you can feed JavaScript code to it and it will run that code for you and may return the result. In theory, it looks pretty simple.

The concept of server-side JavaScript comes from this simple idea. You can take any JavaScript engine, wrap inside an application that gives a clean interface to take the user?s JavaScript code and execute it in the JavaScript engine. You can also provide APIs to perform operations like File System IO, Networking, etc. which do not run on JavaScript engine.

(an oversimplification of how JavaScript runs on a server)

Ryan Dahl took this idea and made Node.js.

To understand more about how a JavaScript engine works in a browser, you should read my article on How JavaScript engine works in browser and Node. This article also explain concept of Web APIs in depth.

How Node.js works?

Sometimes, Node.js is also called simply Node or node.

Node.js is a framework that contains the V8 JavaScript engine, the standard library of packages, and some binaries. In reality, it is more complex than that as explained in the below diagram (follow the link for more details).

(Source: Stackoverflow)

Like Web APIs in the browser, Node.js has a standard library that contains JavaScript packages (we will learn about packages later) which may also provide an interface to low-level APIs. For example, Node.js comes with fs package which contains readFile function among many. This function reads the file on the disk of the machine and returns file content back.

Most of these packages contain code written in a low-level programming language to communicate with the device, like for file system access. These packages export JavaScript functions and other types to run this code. Since JavaScript can not talk to C++ or some other language, Node.js has to create a binding to facilitate this communication. The process to create such packages is very tricky, but it is explained here in-depth.

Node.js uses different threads to perform low-level non-JavaScript time-taking operations. This way, our JavaScript is not blocked while a time taking operation like reading a file is in progress. Since these operations are running in the background once initiated, we need a confirmation or a callback when the operation is finished. This callback is a JavaScript function that will execute as soon as the operation is finished.

const fs = require(‘fs’); // fs is built-in packagefs.readFile(‘/path/to/file.txt’, function(error, data){ // if error is not empty, show error log // read data and do something with it});

The Node.js architecture is very complex and made of different parts as seen in the earlier diagram. It also contains an event loop that facilitates the execution of these callback functions. You should watch the below video on the event loop, though it is in the context of the browser but things are pretty similar in Node.js as well. This will clear your remaining doubts.

Installing Node.js

You should install Node.js from their official website at nodejs.org. If you are using Windows, Mac OS, or Linux, you can get precompiled binaries and installers. The best way to go is by using an installer.

When you install Node.js, you get node and npm binaries added to your path. That means, now you can use node and npm command. We will talk about npm later, but for now, let?s focus on the node.

Using -v or –version flag, we can check the version of the Node installed. The latest node will have the latest V8 engine, hence latest JavaScript features. If you need the flexibility to change Node version at any given time, in that case, you should not install Node.js using above method. Instead, you should use Node Version Manager or NVM.

Here is the list of Node.js releases with V8 engine versions.

Like we saw in DevTools of the browser, using the simple command node will open a JavaScript interpreter in the terminal. This way we can run some simple JavaScript code to amuse yourself.

(node interpreter)

Running JavaScript code with Node.js

Now that we have a good understanding of what Node.js is and how JavaScript engine works, we can start messing with it.

Using an interpreter we can perform some basic arithmetics and other basic stuff. But most of the time, your actual JavaScript code will be in a .js file. Instead of giving one line at a time to the interpreter, we need to give whole file content at once. We can do that by using node /path/to/file.js command.

(hello-world.js)

In the above example, we have created hello-world.js file and it contains hello function. This function is executed after it was defined in the same file. Using the terminal, we executed node ./hello-world.js command. node will pick up hello-world.js file from the directory where the terminal is opened and run it using the V8 JavaScript engine.

Since node can only execute .js files, adding .js extension to the file path is optional. If we provide a directory path instead of file path to node, Node.js will try to resolve index.js file inside that directory.

(executing index.js in the current directory with Node)

Importing scripts in the program

Normally our application is broken down to different parts. For example, if a set of functions are used over and over, we would like them to be contained in a separate file and import that file wherever those functions are to be used.

Node.js supports this functionality natively. When you import a file inside another file, that file is called a module. Node.js uses the CommonJS module system syntax to import modules and packages.

Though ES6+ supports new module system, it is yet to be implemented in Node.

Let?s create a lib directory and place math.js file inside it. This file contains some common math functions like add and multiply. We will import this file inside calculate.js file situated in the main directory. Hence math.js is a module, since we are importing it and not executing it directly with Node.

To import a module, we need to use require function provided by the Node.js and available globally everywhere in the program. This function takes a relative or an absolute path of the module file and returns what module is exporting. Hence, we need to save it inside a variable.

// calculate.jsvar math = require( ‘./lib/math.js’ ); // `.js` is optional

require() statement sometimes also called as import statement.

Now let?s take a look at math.js file. We need to provide AKA export some values to the require() function call. This is done using exports variable. This variable is also globally available everywhere.

This variable is actually an object which is empty {} at the beginning. When we import this module using require function, this is the object require() call will return.

// calculate.jsvar math = require( ‘./lib/math’ );console.log(math); // {}

Since we know that exports is an object that will be exported from the module, we can stuff it with whatever we want. As objects go, an object is a collection of key-value pairs. So let?s add some math functions to it.

// lib/math.js// add `add` function to `exports`exports.add = function( num1, num2 ) { return num1 + num2};

From math module, we are exporting add function which returns the sum of the two numbers (arguments). Let?s see what math variable looks like.

// calculate.jsvar math = require( ‘./lib/math’ );console.log(math);{ add: [Function] }

It shows that math variable is an object that contains add key which has a Function value. Let?s execute that function and see the result.

// calculate.jsvar math = require( ‘./lib/math’ );// add 1 + 2var result = math.add(1, 2);console.log( result ); // 3

What the heck is module.exports then?

I kind of skipped over this part so that you can understand module import with ease. I have a simple question, what if my math module exports only one function like add but I don?t want to export it inside an object. This is the only function my module is exporting, so I want require() call to return this function only so that I can start using it like below.

// calculate.jsvar math = require( ‘./lib/math’ );// add 1 + 2var result = math(1, 2); // math is a functionconsole.log( result ); // 3

This is where module global variable comes into the picture. Like exports, module is also globally available everywhere. module is an object and it contains information about module (auto injected by Node.js in key-value pairs). The important key in this object we should know about is exports.

exports variable inside a module points to exports property on the module object, as you can prove in the below test.

// lib/math.jsconsole.log(exports === module.exports); // true

That means when we were setting exports.add, we were actually setting module.exports.add. So if we want our module to export only one function, we can just assign module.exports to that function.

We could say that since exports and module.exports is the same, why not just set exports to the function. The reason is how objects are handled in JavaScript. Read this answer to explore this topic in details.

// lib/math.js// export function onlymodule.exports = function( num1, num2 ) { return num1 + num2};

If your module import path is a directory, then require function will resolve index.js file inside it. Using this feature, you can have multiple .js files in a directory that contains different exports and you can import them inside index.js to exports them again from a single point.

??? lib/ ??? index.js (import `math` and `graph` and export them) ??? math.js ??? graph.js

This way, the importer does not need to target individual module files in a directory. The importer can just point to index.js file.

// lib/math.jsexports.add = function( num1, num2 ) { return num1 + num2};// lib/index.jsvar math = require(‘./math’);exports.add = math.add// calculate.jsconst lib = require(‘./lib’); // points to ‘./lib/index.js’lib.add(1, 2) // 3

In Node.js, a module or a package is loaded only once (per thread or session) even when you require() them multiple times in the program. Once loaded, it will be cached by the Node for performance enhancement.

There are other tricks with CommonJS module system and sometimes we also need to be careful. Read this article to understand more about imports.

Packages in Node.js

A package is nothing but a directory that contains a bunch of modules. Like for example lib in our previous example can be called a package but not quite yet. The most important feature about a node package is that, from anywhere in the program, we should be able to import it, without providing a relative or an absolute path.

Well, that sounds absurd. If our .js files are nested, the import path will also change. Let?s say that, we have a src directory and it contains compute.js. If we need to import lib package, the import path will be ../lib.

??? lib/| ??? index.js| ??? math.js??? src/ ??? compute.js // require( ‘../lib’) ??? deep/ ??? nested.js // require( ‘../../lib’)

As we nest our files deeper, the import path is very difficult to track. What would be easy is instead of the relative path, we would just use lib and Node.js just finds the path to that package for us.

// src/compute.jsconst lib = require(‘lib’); // points to ‘../lib/index.js’lib.add(1, 2) // 3

This might sound like a fantasy but it is actually very real. Node.js can do this for us, just that we need to create node_modules directory and clone lib directory inside it. This way, when we call require(‘lib’), it will point to the lib directory inside node_modules. Now, lib is a package.

You might wonder, why they are called node_modules and not node_packages? In a conventional sense, a module is a file and package is a collection of modules. But when it comes to require function, they are ambiguous. Hence, let?s stick to a common name, module. But normally, when people say node module, it is a package inside node_modules directory.

But the real question is, how does Node knows where the node_modules directory is. The answer is, it doesn?t. When we import a package, it searches that package inside node_modules directory of the current file path (where import statement is written). If it doesn?t find node_modules directory or the package directory, it performs a similar search in the parent directory.

This continues until the last directory in the file system is reached. If it doesn?t find the package, it throws Error: Cannot find module ?lib? error.

Let?s create node_modules directory in our project and clone lib directory there. From src/compute.js, we will call the add function as before.

(package introduction)

In the above example, we are importing math.js file from lib package. If we just use require(‘lib’), Node.js will point to index.js file inside lib package directory. If math.js is missing, require will try to resolve math/index.js file treating math as a directory.

Even though packages seems easy, their management if done manually is very difficult. Like what if we needed a 3rd-party package? I mean, should we clone the remote source code and put it inside node_module directory manually? Do you know how hard that would be for multiple people in the team? And whenever package version changes, it would be a mess to update.

This is where NPM comes into the picture.

This whole modules and packages theory might sound familiar to you if you are a python developer. But you don?t need __init__.py like file in Node.js ?

What is NPM?

When we installed Node.js, we also got npm command. NPM or Node Package Manager is the default package manager for Node.js. A role of a package manager is to download and install remote package, with ease.

BTW, we can also install packages from a local directory.

Node.js has a wide community that develop good packages for everybody to use. For example, lodash is the package that is used widely. This package contains useful utility functions like add in our earlier example. These utility functions are very well documented on their official website.

When we have a remote package in our project, it is called as a dependency since our project depends on it. We need to keep track of our dependencies or at least list them down somewhere. We list all our dependencies inside a package.json file which is a JSON file that contains some information about our project and dependencies it needs.

This file is essential for NPM. To create this file using npm, use npm init command. This command will ask you some question to fill project-specific data in package.json and eventually create package.json file. You can bypass the questions using -y flag.

(Initializing package.json)

Note dependencies section in the package.json file, it is empty at the moment. This means, our project at the moment does not depend on any of the remote packages.

To install a package, we use npm install <packagename> command. For example, to install lodash package, we use npm install –save lodash command. Using –save flag, we can make entry of this package inside dependencies section of the package.json file.

This command will do the following things.

1. At first, it searches for this package on registry.npm.com which contains the database of all packages. You can see the documentation of a package by visiting https://www.npmjs.com/package/<packagename> URL.
2. Then it downloads the compressed zip (or tar) file that contains all the source code of the package. If a version of the package was not specified in the command, it will download the latest version.
3. Then it adds the package entry to the dependencies section of package.json with the version of the package. If the entry of the package already exists, it will just override the version of the package downloaded.
4. Then it creates node_modules folder in the same directory if it doesn?t already exist.
5. Then it will copy all the files from the downloaded compressed file in the directory with the name of the package inside node_modules.

Let?s actually install lodash package and see how node_modules and package.json file looks like after the install.

(npm install — save lodash)

From the above example, npm installed the version 4.17.15 of the lodash. This type of version number system is called as Semantic Versioning. We can also specify a specific version of a package to install using the command like npm install –save [email protected].

Now that we have installed lodash, let?s use its _.toUpper function to change the case of a string. But first, we need to import the package.

// src/transform.jsvar lodash = require( ‘lodash’ );var result = lodash.toUpper( ‘hello world!’ );console.log( result ); // HELLO WORLD!

When we run this file using the command node src/transform.js, we get HELLO WORLD! printed in the terminal.

When you install a package, NPM also creates package-lock.json file if not already present. This file contains a list of dependency packages with their versions that your project has installed as well as dependencies of those packages (because a package might use other packages and so on). This file including package.json should be tracked by your VCS while node_module directory should be ignored (reasons explained later).

NPM and package management is far more sophisticated (and for good) than this but we will discuss it later in details.

Built-in Packages AKA Built-in Modules

Node.js ships with a collection of built-in packages called as a Node Standard Library. These packages are essential to perform low-level operations like File System I/O and Networking. We do not have to install them using NPM.

Since these packages contain code in a low-level programming language tailored to a specific version of Node.js, they have to be shipped as a part of the installation process. Here is a list of built-in modules in Node.js.

These packages do not exist on disk like lodash. They are compiled into low-level or intermediate stuff (explained here) but their sources are listed here.

fs package is used to perform File System operations like file read and write while path package is used to resolve a file or directory path on the system. Let?s use these packages to demonstrate a cool example.

??? res| ??? hello-world.txt??? fs-example.js

According to the above project structure, we have hello-world.txt file which contains Hello World! text. Using fs-example.js, we want to read the text in the file and log in to the console.

(Sample `fs` and `path` module introduction)

In the above program, we imported the fs and path built-in modules.

When we require(name) a package, Node.js first searches for the package name in the built-in packages. If it doesn?t find it in the standard library, then it searches for it in node_modules as explained earlier.

__dirname is a globally available variable that resolves to the absolute path of the current file on the disk. path.resolve function takes multiple path segments and joins them. This is a safe way to create an absolute path of a file on the disk, as Windows and Unix systems use different path delimiter.

You can also use a relative path like var filePath = ?./res/hello-world.txt?; in the above example but path.resolve is safer. process is a globally available object that contains information about environment variables and current process context in general. Unlike __dirname, process.cwd() function returns the path of the directory from where the node command was executed in the terminal (or the current directory in the terminal).

Let?s talk about the example in detail. Inside fs-example.js file, we imported built-in modules fs and path. Then we constructed a filePath which points to hello-world.txt on the disk.

fs.readFile function takes below arguments in series

1. filePath: A absolute or relative path to the file we are trying to read.
2. options: An object that contains a configuration about reading. In the above example, we set encoding to utf-8 which converts binary data to Text format. This will convert file content to Text.
3. callback: Sync file read operation using readFile function is asynchronous, we need a callback function to execute when the file is read completely. This function will receive read error (if any) as the first argument and file data as the second argument.

Notice the console log. The first -end-of-the-program- statement got printed as fs.readFile was reading the file in the background. Once file reading was completed, the callback function was called.

Node.js can also perform synchronous (blocking) operations. For example, using fs.readFileSync, we can block the JavaScript thread until the file is read. This way, we can make sure, JavaScript code run sequentially.

(Reading a file synchronously)

Creating an HTTP server in Node.js

Node.js can do anything, literally anything. Node.js has built-in http module as well as https module to create an HTTP/HTTPS server. But their implementation is kind of hard.

ExpressJS is a 3rd-party package that wraps the built-in http module and provide a cleaner interface to create an HTTP server. This package is listed on NPM registry under express name. Let?s create a basic HTTP server.

First, we need to install express package using NPM.

npm install –save express

Then we will import this package and create a basic HTTP server inside server.js file. We will follow their startup documentation.

(Sample express HTTP server)

When we executed server.js file with node, ExpressJS will lock the Node process as we want the server to be running forever. Now that server is running on the port 9000, we can open the browser and access URL http://localhost:9000/ which will execute the .get callback.

(http://localhost:9000/)

This was just a basic example of how we can create an HTTP server in Node.js. But with express.js, we can create more complex servers which can send HTML file content using response.sendFile(filePath) function or send JSON using response.json(object) function. We can also create an endpoint that serves static files like images from the disk using middlewares.

To stop the server, we need to stop the locked Node.js process. We can do that by pressing ctrl+c in the terminal. But what if we need to actually run the server on the production literally forever. In that case, we can?t have terminal open for years. This is where the process managers comes in.

PM2 is one of the best process managers that can run a Node process in the background. When you install it, it will give you a clean command-line interface to start a Node.js process and PM2 will monitor it.

Execute a Bash command from Node.js

If you make Node.js your life and want to do everything from Node, then this topic is very important. Let?s say that from a JavaScript program, you want to execute a BASH command. A Bash command would be echo Hello World!. You can try this command in the terminal and it will print Hello World!.

Node.js provides a built-in child_process command to run Bash command in a separate process. child_process.exec function takes a Bash command and executes it. It takes an optional callback function to execute (with some process information) when the process is terminated.

Let?s create a echo.js file that executes echo Hello World! Bash command.

(child_process example)

In the above program, the callback function to child_process.exec receives the standard output of the program.

child_process module can do many things, like execute a Bash file using child_process.execFile function. It also supports synchronous variants to run a bash command synchronously, like child_process.execSync function.

How to ship your code to Production?

Now that we have a good understanding of Node.js and NPM, we can move forward to this most important topic.

package.json and package-lock.json files are very important as they contain information about dependencies our project has. Those dependencies are stored inside node_modules directory by the NPM.

So if our project is managed using a VCS like Git then should we commit all our code? The answer is, ? NOOOOOOOOOOO. node_modules directory can be very large as it contains deeply nested dependencies. Hence it should be ignored by the VCS. Use .gitignore file to do that.

# .gitignorenode_modules

But then when your buddy takes the clone or a pull of the project, he/she won’t get node_modules. Nothing to worry about here because NPM can take care of that.

When we used npm install <packagename> command for the first time, NPM created node_modules directory and install packagename package. Using npm install command (without a package name), NPM will look at pakage.json to install all the dependencies listed inside it.

Actually, since NPM v.5, npm install command looks at package-lock.json command to install the dependencies since it contains the exact versions of the packages and their dependencies (which were installed by the developer of the project). This minimizes the conflict of versions between the development machine and production machine.

By ignoring node_modules, we are actually saving a lot of time and bandwidth of transferring the project from a development machine to production.

You can find the codebase of the code used in this article on my GitHub repository.

Other important topics

If you have gone through every single point I have explained in this article and understood it well, you are a GOOD Node.js developer. But to take this greatness to a higher level, there are other important topics you must explore.

Topics:

1. What are the shortcut commands of NPM? For example, npm i -S is equivalent to npm install –save.
2. What are dependencies, devDependencies, peerDependencies in package.json and what are the differences?
3. How to update a Node.js package? What is the difference between ~ and ^ (and other symbols) in package version?
4. What is the purpose of scripts object in package.json?
5. What is the purpose of main field in package.json?
6. What are the global NPM modules? How to install and use one?
7. How to write your own NPM module and publish it?
8. What are ES6 JavaScript features? How to use it in the Node.js?
9. What is TypeScript and what are the Transpilers?

(thatisuday.com / GitHub / Twitter/ StackOverflow / Instagram)