This article introduces EchoVault, its features, use cases, and the motivations behind its development.

What is EchoVault?

EchoVault is an embeddable in-memory datastore tailored for Go applications. It provides a RESP-compatible interface over TCP while offering the flexibility to be embedded directly into applications.

By doing so, EchoVault aims to replace traditional in-memory data stores like Redis in some use cases, offering similar functionalities with enhanced integration for Go.

Why EchoVault was Built

My initial motivations for building EchoVault were:

• Embedded Flexibility: Traditional in-memory data stores like Redis are external services that require separate management and deployment. EchoVault eliminates this overhead by allowing developers to embed the data store directly into their applications, streamlining the deployment process.

• Go Ecosystem Integration: EchoVault is designed primarily for Go, ensuring seamless integration and optimal performance within Go applications. However, it also maintains RESP compatibility over TCP, making it compatible with existing Redis clients such as redis-cli and Jedis.

• Redis License Changes: EchoVault is open-source (licensed under Apache 2.0). We’re committed to staying on this fully open-source model and not open-core.

Features Provided by EchoVault

EchoVault is packed with features that make it a robust choice for in-memory data storage:

• TLS and mTLS Support: Secure communication with support for multiple server and client Root Certificate Authorities (RootCAs). Replication Cluster Support: Utilizes the RAFT algorithm for replication and clustering.

• Access Control Layer: Provides user authentication and authorization to secure data access.

• Distributed Pub/Sub Functionality: Supports publish/subscribe to channels and patterns for real-time data processing.

• Data Structures: We support a variety of data structures, including sets, sorted sets, hashes, lists, and more. We are continuing to add more data structures and commands within them.

• Persistence Layer: This layer ensures data durability with snapshot and append-only file persistence. The AOF files are RESP compatible but not yet fully Redis compatible.

• Key Eviction Policies: Implements various strategies for key eviction to manage memory usage. EchoVault has passive and active key eviction. With passive key eviction, expired keys are not evicted until the next time they are accessed. With active eviction, EchoVault will proactively delete keys that are expired.

• Command Extensions: Allows runtime extension of commands via shared object files and an embedded API.

The feature list continues to expand as we’re in the early stages of development. We have some more cool features lined up for the future, including:

• Streams.

• Extensions with Lua modules.

• Sharding.

Architecture of EchoVault

EchoVault supports both standalone and clustered deployments.

Standalone Mode

In standalone mode, EchoVault operates a single (standalone) instance. This is the easiest way to run EchoVault. You can run a standalone instance from the embedded library or start it as its own process that accepts TCP connections. The embedded instance can also accept TCP connections, allowing you to communicate with your Go process from a TCP client.

RAFT Replication Cluster Mode

For applications requiring strong consistency and fault tolerance, EchoVault supports a RAFT-based replication cluster mode.

RAFT is a consensus algorithm that ensures data consistency across distributed systems. In this mode, multiple EchoVault instances can form a cluster, providing data replication and ensuring that data remains consistent even in the event of node failures.

Key Features of RAFT Cluster Mode:

• Fault Tolerance: Ensures data availability even if some nodes in the cluster fail.

• Consistency: Guarantees that all nodes in the cluster have the same data.

• Scalability: Allows for horizontal scaling by adding more nodes to the cluster.

You can run EchoVault clusters even in embedded modes. This means that your application instances can communicate with each other through the EchoVault layer without having to deploy a third service. This is ideal for use cases such as session management across a cluster of an application’s instances.

Use Cases for EchoVault

EchoVault's versatility makes it suitable for a wide range of applications. Here are some potential use cases:

• In-Memory Caching Scenario: An e-commerce website needs to cache product details and user session information to improve performance. Solution: Use EchoVault to cache frequently accessed data to speed up response times. Benefits: Faster page loads, improved user experience, and reduced database strain.

• Service Discovery Scenario: A microservices architecture requires dynamic service discovery for inter-service communication. Solution: Store service endpoints in EchoVault, allowing services to discover and communicate with each other efficiently. Benefits: Simplified service discovery and enhanced communication efficiency.

• Session Management Scenario: A distributed web application must manage user sessions across multiple instances. Solution: Use EchoVault embedded cluster to store session data, ensuring consistency and accessibility across all application instances. Benefits: Consistent session management across the application’s cluster without deploying a secondary service like Redis.

• Real-Time Analytics Scenario: A financial trading platform requires real-time analytics and monitoring of trade data. Solution: Store and process real-time trade data in EchoVault, leveraging its sorted sets and pub/sub capabilities for tracking and analysis. Benefits: Real-time data processing and faster analytics.

• Distributed Task Queue Scenario: A backend system needs to manage and distribute tasks to multiple worker nodes. Solution: Implement a task queue using EchoVault's list data structure, where tasks are pushed onto the list and worker nodes pop tasks for processing. Benefits: Efficient task distribution and scalability.

• Feature Flags and Configuration Management Scenario: A SaaS application needs to dynamically enable or disable features and manage configuration settings without redeploying the application. Solution: Store feature flags and configuration settings in EchoVault. Benefits: Dynamic configuration management and reduced downtime.

• Rate Limiting and Throttling Scenario: An API gateway needs to enforce rate limits and throttle requests to prevent abuse and ensure fair usage. Solution: Implement rate limiting using EchoVault's in-memory capabilities to track request counts and enforce limits in real-time. Benefits: Effective rate limiting and improved API reliability.

• Leader Election Scenario: A cluster of distributed services needs to elect a leader to coordinate tasks. Solution: Rely on EchoVault's embedded distributed cluster’s leader election to ensure that only one application instance is designated as the leader. Benefits: Reliable leader election and improved coordination without implementing it yourself at the application level.

Conclusion

EchoVault is an ambitious project. In these early stages, we expect many kinks to have to be ironed out and many features to be added. However, we welcome this challenge and would love to get the Go community involved in the project.

If this article has triggered your curiosity, or you’re interested in contributing to a project like this, check out our GitHub and drop us a star! We’re always open to more feedback and contributions!

When I'm building an application on AWS infrastructure, I prefer using Cognito user pools due to its seamless integration with other AWS services such as API Gateway authorization.

Cognito user pools also allow you to set up social login for your users. This is very convenient in a world where most people have become accustomed to authentication via existing accounts such as Google, Facebook, Apple, etc.

In this article, I will be covering how to integrate social login in your Next.js applications through Cognito. We will essentially allow our users to log in using their social accounts while using Cognito as a proxy auth server.

We will do this while bypassing the Cognito-hosted UI for a seamless auth experience for the user.

There are a few benefits to using this approach instead of manually handling social logins ourselves:

1. We can leverage Cognito user pools and all its rich features.

2. In our Next.js application, we only have to deal with 1 auth provider, Cognito.

3. We have the freedom to design our social sign-in buttons as we please, without being shackled by Cognito's hosted UI.

4. We do not have to manage a database of users ourselves.

Prerequisites

This article will not go over the process of setting up your Cognito user pool. Before you implement the steps in this article, make sure you have:

1. A Cognito user pool.

2. An app client set up for your user pool.

3. Social login setup for your app client. You don't need to set up all the social providers, just one of them is enough. You can always plug more providers in later.

4. A domain for your user pool. It does not need to be a custom domain. The default domain provided by aws is sufficient.

Our auth structure should roughly look like this:

Setting up NextAuth

In the Next.js application, first, install the NextAuth library.

npm install next-auth

Next, set up the NextAuth configurations in /api/auth/[...nextauth].ts.

import { TokenSet } from "next-auth";
import NextAuth from "next-auth/next";
import { Provider } from "next-auth/providers";

const { 
  NEXTAUTH_URL,
  COGNITO_REGION,
  COGNITO_DOMAIN,
  COGNITO_CLIENT_ID,
  COGNITO_USER_POOL_ID,
  COGNITO_CLIENT_SECRET,
} = process.env;

type TProvider = "Amazon" | "Apple" | "Facebook" | "Google";

function getProvider(provider: TProvider): Provider {
   /*
      Provider generation function to avoid repeating ourselved when
      declaring providers in the authOptions below.
   */
  return {
    // e.g. cognito_google | cognito_facebook
    id: `cognito_${provider.toLowerCase()}`,  

    // e.g. CognitoGoogle | CognitoFacebook
    name: `Cognito${provider}`,

    type: "oauth",

    // The id of the app client configured in the user pool.
    clientId: COGNITO_CLIENT_ID,

    // The app client secret.
    clientSecret: COGNITO_CLIENT_SECRET,

    wellKnown: `https://cognito-idp.${COGNITO_REGION}.amazonaws.com/${COGNITO_USER_POOL_ID}/.well-known/openid-configuration`,

    // Authorization endpoint configuration
    authorization: {
      url: `${COGNITO_DOMAIN}/oauth2/authorize`,
      params: {
        response_type: "code",
        client_id: COGNITO_CLIENT_ID,
        identity_provider: provider,
        redirect_uri: `${NEXTAUTH_URL}/api/auth/callback/cognito_${provider.toLowerCase()}`
      }
    },

    // Token endpoint configuration
    token: {
      url: `${COGNITO_DOMAIN}/oauth2/token`,
      params: {
        grant_type: "authorization_code",
        client_id: COGNITO_CLIENT_ID,
        client_secret: COGNITO_CLIENT_SECRET,
        redirect_uri: `${NEXTAUTH_URL}/api/auth/callback/cognito_${provider.toLowerCase()}`
      }
    },

    // userInfo endpoint configuration
    userinfo: {
      url: `${COGNITO_DOMAIN}/oauth2/userInfo`,
    },

    // Profile to return after successcul auth.
    // You can do some transformation on your profile object here.
    profile: function(profile) {
      return {
        id: profile.sub,
        ...profile
      }
    }
  }
}

export const authOptions = {

  /*
    Generate the providers for each of our social login.
    Adding a new OIDC provider is a simple as adding the name of the         provider as displayed in the Cognito user pool to the TProvider type
and to this array.
  */
  providers: [
    ...["Amazon", "Apple", "Facebook", "Google"].map((provider: TProvider) => getProvider(provider)),
  ],

  callbacks: {
    async signIn({ user, account, profile }) {
      // Return true to allow sign in and false to block sign in.
      return true;
    },
    async redirect({ url, baseUrl }){
      // Return the url to redirect to after successful sign in.
      return baseUrl;
    },
    async jwt({ token, account, profile, user }){
      // Retrieve jwt tokens
      if (account) {
        // account is provided upon the inital auth
        return {
          ...token,
          accessToken: account.access_token,
          idToken: account.id_token,
        }
      }
    },
    async session({ session, token }) {
      /* 
         Forward tokens to client in case you need to make authorized
         API calls to an AWS service directly from the front end.
      */
      session.accessToken = token.accessToken;
      session.idToken = token.idToken;
      return session;
    }
  }
};

export default NextAuth(authOptions);

As you can see, we're configuring a basic OAuth2.0 flow. You can read more about Cognito's OIDC endpoints here. This flow will work with any OIDC provider that you configure in your user pool.

Callback enhancement

Right now there's a scenario we're not handling. When authenticating in this way, Cognito will return a long-lasting refresh token. A refresh token is used to request new access and id tokens when said tokens have expired.

This allows us to keep the user logged in for a long time without forcing them to sign in every time their tokens expire. We will only prompt the user to sign in again once the refresh token itself has expired or has been invalidated in some other way.

To support this functionality, we need to update the code in the jwt and session callback functions.

First, let's update the jwt callback function:

async jwt({ token, account, profile, user }){
  if (account) {
    // This is an initial login, set JWT tokens.
    return {
      ...token,
      accessToken: account.access_token,
      idToken: account.id_token,
      refreshToken: account.refresh_token,
      expiresAt: account.expires_at,
      tokenType: 'Bearer'
    }
  }
  if (Date.now() < token.expiresAt) {
    // Access/Id token are still valid, return them as is.
    return token;
  }
  // Access/Id tokens have expired, retrieve new tokens using the 
  // refresh token
  try {
    const response = await fetch(`${COGNITO_DOMAIN}/oauth2/token`, {
      headers: { 
        "Content-Type": "application/x-www-form-urlencoded"
      },
      body: new URLSearchParams({
        client_id: COGNITO_CLIENT_ID,
        client_secret: COGNITO_CLIENT_SECRET,
        grant_type: "refresh_token",
        refresh_token: token.refreshToken
      }),
      method: "POST"
    })

    const tokens: TokenSet = await response.json();

    if (!response.ok) throw tokens;

    return {
      ...token,
      accessToken: tokens.access_token,
      idToken: tokens.id_token,
      expiresAt: Date.now() + (Number(tokens.expires_in) * 1000)
    }
  } catch (error) {
    // Could not refresh tokens, return error
    console.error("Error refreshing access and id tokens: ", error);
    return { ...token, error: "RefreshTokensError" as const }
  }
}

Here's what this function is doing before returning the jwt tokens when a token request is made:

1. If it's the first request (i.e initial sign-in action), add accessToken, idToken, refreshToken, and expiresAt to the token object. The expiresAt attribute lets us know when the access and id token will expire. The object returned here will be represented by the token param in all subsequent token requests.

2. If this is not the first request, check if the access and id tokens have expired by comparing the current time to the expiresAt time in the token object that was set in point 1.

3. If the tokens have not expired, return the same token object.

4. If the tokens have expired, fetch new tokens using the refresh token.

5. If the fetch fails, throw the error. In the catch block, return the current token object but append an error property so we can check for it and redirect the user to the sign-in page.

6. If the fetch is successful, replace the accessToken, idToken and expiresAt properties in the token object and return the new token object.

Next, we need to update the session callback. This callback allows us to set session properties that will be accessible in the front end via the useSession hook.

All we need to do here is add the error property of the token to the session object. That way we can detect the error in the front end and redirect the user to the sign-in page.

async session({ session, token }) {
  /* 
    Forward tokens to client in case you need to make authorized API      
    calls to an AWS service directly from the front end.
  */
  session.accessToken = token.accessToken;
  session.idToken = token.idToken;
  /* 
    If there is an error when refreshing tokens, include it so it can 
    be forwarded to the front end.
  */
  session.error = token.error;
  return session;
}

Configuring shared session state

To configure the sessions state, update the _app.tsx or _app.jsx file with the following code:

import '../styles/globals.css';
import { SessionProvider } from "next-auth/react";

function MyApp({ 
  Component,
  pageProps: { session, ...pageProps }
}) {
  return (
    <SessionProvider session={session}>
      <Component {...pageProps} />
    </SessionProvider>
  )
}

export default MyApp

This change allows us to use the session throughout the application.

Frontend login with providers

In order to allow the user to sign in with their preferred provider, we have to configure the sign-in buttons for each provider in our SignIn page component.

import { useEffect, useState } from "react";

/* The getProviders function returns a promise that resolves
   to an object containing all of the configured providers. */
// The signIn functions initiates the signIn process.
import { getProviders, signIn } from "next-auth/react";

export default function SignIn() {
  const [providers, setProviders] = useState(null);

  // Fetch providers on mount.
  useEffect(() => {
    getProviders().then((providers) => setProviders(providers));
  }, []);

  return (
    <div>
      {providers && (
        <>
          <div>
            <button onClick={
              () => signIn(providers["cognito_amazon"].id)
            }>
              Login with Amazon
            </button>
          </div>
          <div>
            <button onClick={
              () => signIn(providers["cognito_apple"].id)
            }>
              Login with Apple
            </button>
          </div>
          <div>
            <button onClick={
              () => signIn(providers["cognito_facebook"].id)
            }>
              Login with Facebook
            </button>
          </div>
          <div>
            <button onClick={
              () => signIn(providers["cognito_google"].id)
            }>
              Login with Google
            </button>
          </div>
        </>
      )}
    </div>
  );
}

For more context, the provider object will take the following shape:

{
  <prodiver_id>: {
    "id": <provider_id>
    "name": <provider_name>
    "type": <provider_type>
    "signinUrl": <provider signin url>
    "callbackUrl": <provider callback url>
  }
}

In this configuration, the provider object will look like this:

{
  "cognito_amazon":
  {
    "id":"cognito_amazon",
    "name":"CognitoAmazon",
    "type":"oauth",
    "signinUrl":
      "http://localhost:3000/api/auth/signin/cognito_amazon",
    "callbackUrl":
      "http://localhost:3000/api/auth/callback/cognito_amazon"
  },
  "cognito_apple":{
    "id": "cognito_apple",
    "name":"CognitoApple",
    "type":"oauth",
    "signinUrl":"http://localhost:3000/api/auth/signin/cognito_apple",
    "callbackUrl":
      "http://localhost:3000/api/auth/callback/cognito_apple"
  },
  "cognito_facebook":{
    "id": "cognito_facebook",
    "name":"CognitoFacebook",
    "type":"oauth",
    "signinUrl":
      "http://localhost:3000/api/auth/signin/cognito_facebook",
    "callbackUrl":
      "http://localhost:3000/api/auth/callback/cognito_facebook"
  },
  "cognito_google":{
    "id":"cognito_google",
    "name":"CognitoGoogle",
    "type":"oauth",
    "signinUrl":
      "http://localhost:3000/api/auth/signin/cognito_google",
    "callbackUrl":
      "http://localhost:3000/api/auth/callback/cognito_google"
  }
}

Frontend useSession hook

In order to access the session data in the front end, we can make use of the useSession hook from next-auth/react.

This hook allows us to access the session data and the user's auth status.

First import the hook.

import { useSession } from 'next-auth/react'

Then inside your component, use it as follows:

const { data, status } = useSession();

The data property will contain all the data that we set in the session callback that we configured [...nextauth.ts] . It will also contain an additional user property with information about the current user such as email address.

The status property is a string containing information about the user's auth status. Its possible values are "loading", "authenticated" or "unauthenticated". You can use this to implement protected routes or redirect to the sign-in page when a session is expired.

Setup

This is article is based on a serverless framework project. You can still implement the concepts discussed here without serverless framework.

You can find the code used in this article in this GitHub repository.

Serverless provider config

The serverless provider configuration will look exactly the same as a regular API project with the exception of one additional property: websocketsApiRouteSelectionExpression.

This property tells API gateway which request body property to use in order to determine where to route the request. In this instance, the action property of the payload will determine which handler gets called.

We'll see this in action when we define the handlers.

provider:
  name: aws
  runtime: nodejs14.x
  profile: ${env:PROFILE, "localstack"}
  stage: ${opt:stage, "local"}
  region: ${opt:region, "us-east-1"}
  websocketsApiRouteSelectionExpression: $request.body.action

Connection table

For our use case, we'll need to maintain a record of active connections in order to send targeted messages.

I've chosen DynamoDB for this demonstration but you might get better mileage using a relational database like PostgreSQL. The concept will be the same, but with a slightly different implementation.

A table row will contain Username and ConnectionId fields for each active connection.

We must create a DynamoDB table resource in the serverless.yaml file. The table will use the Username as the primary key so make sure it's unique.

Since we'll need to query the table by ConnectionId at some point, let's create a global secondary index that uses ConnectionId as the primary key.

You can read more about global secondary indexes here.

resources:  
  Resources:
    ConnectionsTable:
      Type: AWS::DynamoDB::Table
      Properties:
        AttributeDefinitions:
          - AttributeName: Username
            AttributeType: S
          - AttributeName: ConnectionId
            AttributeType: S
        KeySchema:
          - AttributeName: Username
            KeyType: HASH
        ProvisionedThroughput:
          ReadCapacityUnits: 1
          WriteCapacityUnits: 1
        GlobalSecondaryIndexes:
          - IndexName: ConnectionIdIndex
            KeySchema:
              - AttributeName: ConnectionId
                KeyType: HASH
            Projection:
              ProjectionType: ALL
            ProvisionedThroughput:
                ReadCapacityUnits: 1
                WriteCapacityUnits: 1

Connecting

Now let's create the connection handler. This lambda will be called when a client tries to establish a connection.

The main aim of this handler is to save the connection details in the DynamoDB table so this connection can be targeted when sending a message.

First, let's define the function in serverless.yaml in the function section:

wsConnectHandler:
    handler: ./src/connect.handler
    events:
      - websocket:
          route: $connect
    iamRoleStatements:
      - Effect: "Allow"
        Action: "dynamodb:PutItem"
        Resource: !GetAtt [ConnectionsTable, Arn]

This lambda is triggered by the pre-defined $connect websocket route. We also give this function permission to put an item into the DynamoDB table we defined earlier.

The handler logic looks as follows:

import {
  DynamoDBClient,
  PutItemCommand
} from "@aws-sdk/client-dynamodb";

const {
  STAGE,
  REGION,
  CONNECTION_TABLE,
  LOCALSTACK_ENDPOINT
} = process.env;

const getDynamoDbConfig = (): { region: string, endpoint?: string } => {
  if (STAGE === "local") return { region: REGION, endpoint: LOCALSTACK_ENDPOINT };
  return { region: REGION };
}

module.exports.handler = async (event) => {
  const dynamoDBClient = new DynamoDBClient(getDynamoDbConfig());

  const putItemCommand = new PutItemCommand({
    TableName: CONNECTION_TABLE,
    Item: {
      Username: { S: event.queryStringParameters.username },
      ConnectionId: { S: event.requestContext.connectionId }
    }
  });
  await dynamoDBClient.send(putItemCommand);

  return { statusCode: 200 };
}

When the connection route is triggered, we create a record in the connection table that contains the username and connection id.

We retrieve the username from the queryStringParameters of the request. To establish a connection, the client will have to send a connection request to wss://<api-id>.execute-api.<region>.amazonaws.com/<stage>/?username=<username>.

If the username isn't provided, an error is thrown and a connection isn't established. We must return a 200 status code for a successful connection.

You can implement custom error handling here if you'd like but AWS will usually display the error in the cloudwatch logs when encountered.

Disconnecting

Now let's handle disconnection. The disconnection handler is triggered whenever a connection is ended. We can use this to do some cleanup. In our case, the cleanup involves deleting the connection row in our DynamoDB table.

First, let's define the function in serverless.yaml

wsDisconnectHandler:
    handler: ./src/disconnect.handler
    events:
      - websocket:
          route: $disconnect
    iamRoleStatements:
      - Effect: "Allow"
        Action: 
          - "dynamodb:DeleteItem"
          - "dynamodb:Query"
        Resource: 
          - !GetAtt [ConnectionsTable, Arn]
          - !Join ["/", [!GetAtt [ConnectionsTable, Arn], "index", "ConnectionIdIndex"]]

Once again, this is triggered by the disconnect event through the pre-defined $disconnect route.

We need to give this function permission to delete from the connection table and query the global secondary index we created earlier.

Here's the handler implementation:

import {
  DynamoDBClient,
  DeleteItemCommand,
  QueryCommand,
} from "@aws-sdk/client-dynamodb";

const {
  STAGE,
  REGION,
  CONNECTION_TABLE,
  LOCALSTACK_ENDPOINT
} = process.env;

const getDynamoDbConfig = (): { region: string, endpoint?: string } => {
  if (STAGE === "local") return { region: REGION, endpoint: LOCALSTACK_ENDPOINT };
  return { region: REGION };
}

module.exports.handler = async (event, context, callback) => {
  console.log(event);

  const dynamoDBClient = new DynamoDBClient(getDynamoDbConfig());

  // Retrieve the connection using the connectionId
  const queryCommand = new QueryCommand({
    TableName: CONNECTION_TABLE,
    IndexName: "ConnectionIdIndex",
    KeyConditionExpression: "ConnectionId = :a",
    ExpressionAttributeValues: {
      ":a": { S: event.requestContext.connectionId }
    }
  })

  const queryResult = await dynamoDBClient.send(queryCommand);
  const connectionRecord = queryResult?.Items ? queryResult.Items[0] : undefined;

  const deleteItemCommand = new DeleteItemCommand({
    TableName: CONNECTION_TABLE,
    Key: {
      Username: { S : connectionRecord?.Username?.S ? connectionRecord?.Username?.S : "" }
    }
  });
  await dynamoDBClient.send(deleteItemCommand);

  return {
    statusCode: 200,
    body: JSON.stringify({ message: "Connection ended!" })
  };
}

We retrieve the connection id from the request context in the event object and query the index to find the connection record with this connection id.

Once we've retrieved the record, we delete it from the connection table using the username as the key.

Send a message

Now that we've handled connecting and disconnecting, it's time to handle sending messages directly to another connected user.

This handler will work by receiving a payload that contains the recipient's username, finding the recipient's connection record, extracting the connection id, and sending the payload message to the target connection.

First, let's define the function in serverless.yaml:

wsSendMessage:
    handler: ./src/sendMessage.handler
    events:
      - websocket:
          route: "sendMessage"
    iamRoleStatements:
      - Effect: "Allow"
        Action: "dynamodb:Query"
        Resource:
          - !GetAtt [ConnectionsTable, Arn]
          - !Join ["/", [!GetAtt [ConnectionsTable, Arn], "index", "ConnectionIdIndex"]]
      - Effect: "Allow"
        Action: "execute-api:ManageConnections"
        Resource:
          - Fn::Join:
            - "/"
            -
              - Fn::Join:
                - ":"
                -
                  - "arn:aws:execute-api"
                  - ${opt:region, "us-east-1"}
                  - !Ref AWS::AccountId
                  - !Ref WebsocketsApi
              - ${opt:stage, 'dev'}
              - "POST"
              - "@connections"
              - "{connectionId}"

Notice we have a slightly different event. The route is set as sendMessage. This is a custom route defined by us.

If you recall, the provider section has this property defined: websocketsApiRouteSelectionExpression: $request.body.action. This tells API Gateway to look for the route in the action property of the request body.

So in order to trigger the wsSendMessage lambda, the message we send once connected needs to be an object that contains the action property with the value of "sendMessage" like so:

{
  "action": "sendMessage",
  ...
}

Next, we need to give this lambda permission to query both the connection table and the global secondary index. We also must give it permission to send a message to an existing web socket connection.

This is the handler implementation:

import { DynamoDBClient, QueryCommand } from "@aws-sdk/client-dynamodb";

import {
  ApiGatewayManagementApiClient,
  PostToConnectionCommand
} from "@aws-sdk/client-apigatewaymanagementapi";

const {
  STAGE,
  REGION,
  CONNECTION_TABLE,
  LOCALSTACK_ENDPOINT,
  PORT
} = process.env;

const getDynamoDbConfig = (): { region: string, endpoint?: string } => {
  if (STAGE === "local") return { region: REGION, endpoint: LOCALSTACK_ENDPOINT };
  return { region: REGION };
}

const getConnectionEndpoint = (event) => {
  return STAGE === "local" ? 
  `http://${event.requestContext.domainName}:${PORT}` :
  `https://${event.requestContext.domainName}/${event.requestContext.stage}/`
}

module.exports.handler = async (event, context, callback) => {
  console.log(event);

  const body = JSON.parse(event.body);

  const dynamoDBClient = new DynamoDBClient(getDynamoDbConfig());

  /**
   * Query the connection table for the TARGET connection record.
   * The recepient's username used in the query will be included in the event body.
   */
  let queryResult = await dynamoDBClient.send(new QueryCommand({
    TableName: CONNECTION_TABLE,
    KeyConditionExpression: "Username = :a",
    ExpressionAttributeValues: {
      ":a": { S: body.recepient}
    }
  }));

  // Return early if the recepient is not found in the connections table.
  if (!(queryResult?.Items?.length && queryResult?.Items?.length > 0)) return callback(JSON.stringify({
    message: "Recepient not found",
    body
  }));

  const recepientConnection =  queryResult.Items[0];

  /**
   * Query ConnectionIdIndex using the connection id of the sender.
   * The senders connection id can be found in event.requestContext.connectionId.
   * We retrieve this connection record in order to retrieve the sender's username.
   */
  queryResult = await dynamoDBClient.send(new QueryCommand({
    TableName: CONNECTION_TABLE,
    IndexName: "ConnectionIdIndex",
    KeyConditionExpression: "ConnectionId = :b",
    ExpressionAttributeValues: {
      ":b": { S: event.requestContext.connectionId }
    },
  }));

  // Return early if no actively connected sender is found.
  if (!(queryResult?.Items?.length && queryResult?.Items?.length > 0)) return callback(JSON.stringify({
    message: "Sender not found",
    context: event.requestContext
  }));

  const senderConnection = queryResult?.Items ? queryResult.Items[0]: undefined;

  const apiGatewayManagementApiClient = new ApiGatewayManagementApiClient({ 
    region: REGION,
    endpoint: getConnectionEndpoint(event)
  });

  // Post a message to the recipient's connection.
  const postToConnectionCommand = new PostToConnectionCommand({
    ConnectionId: recepientConnection?.ConnectionId.S,
    Data: Buffer.from(JSON.stringify({
      from: senderConnection?.Username.S,
      message: body.message
    }), "utf-8")
  });

try {
    await apiGatewayManagementApiClient.send(postToConnectionCommand);
  } catch (error) {
    console.log(error);
  }

  return { 
    statusCode: 200,
    body: JSON.stringify({ message: "Message sent." })
  };
}

In order to test this, send a message to the connection with the following signature:

{
  "action": "sendMessage",
  "recepient": <username>,
  "message": <string>
}

Make sure the recipient is connected as well.

For more in-depth reading on WebSocket APIs, visit the official AWS docs here.

Cognito offers this functionality built into the hosted-ui. However, if you find the hosted UI to be limited in terms of design or functionality, you might want to implement this authentication method using the AWS SDK in your own custom UI.

Unfortunately, there is no straightforward way of doing this, so this is a possible workaround to that particular limitation.

If you haven't read the other articles in this series, I'm using a react frontend with a nodejs backend for this tutorial. My framework of choice is NextJS. You don't have to use this exact framework. As long as you use a JS frontend and a NodeJS backend, you should be able to adapt this tutorial to your own stack.

Setup

First, we need to install a couple of packages to help us out in our implementation. The 2 packages we need are react-google-login to facilitate Google login in the frontend, and google-auth-library for verifying Google tokens in the backend.

Install the packages with the following command in the terminal: npm install react-google-login google-auth-library.

Registration

On the registration page, import GoogleLogin from react-google-login and add the button to the layout.

You can find more information about this package and how to customise the button here.

<GoogleLogin
    clientId={process.env.NEXT_PUBLIC_GOOGLE_CLIENT_ID}
    responseType={'id_token'}
    buttonText="Register with Google"
    onSuccess={googleRegisterSuccess}
    onFailure={googleRegisterFailure}
/>

We pass the googleRegisterSuccess function to the onSuccess prop. We have a useRegister hook that contains the googleRegisterSuccess function.

Now, define the googleRegisterSuccess function in the useRegister hook as follows:

const googleRegisterSuccess = (googleResponse) => {
        fetch('/api/register/google', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json'
            },
            body: JSON.stringify({ id_token: googleResponse?.tokenId })
        }).then(res => {
            if (!res.ok) throw res
            router.push({
                pathname: '/login'
            })
        }).catch(err => {
            console.error(err)
        })
    }

const googleRegisterFailure = (googleResponse) => {
    console.error(googleResponse)
}

All we're interested in from the response object is the tokenId, which we send to the backend for verification.

In the handler for the Google register route, we have the following code:

import { CognitoIdentityProviderClient, SignUpCommand } from '@aws-sdk/client-cognito-identity-provider'
import { OAuth2Client } from 'google-auth-library'

const {
  COGNITO_REGION,
  COGNITO_APP_CLIENT_ID,
  GOOGLE_TOKEN_ISSUER,
  NEXT_PUBLIC_GOOGLE_CLIENT_ID,
} = process.env

export default async function handler(req, res) {
  if (req.method !== 'POST') return res.status(405).send()

  let googlePayload

  try {
    // Verify the id token from google
    const oauthClient = new OAuth2Client(NEXT_PUBLIC_GOOGLE_CLIENT_ID)
    const ticket = await oauthClient.verifyIdToken({
      idToken: req.body.id_token,
      audience: NEXT_PUBLIC_GOOGLE_CLIENT_ID
    })
    googlePayload = ticket.getPayload()
    if (
      !googlePayload?.iss === GOOGLE_TOKEN_ISSUER ||
      !googlePayload?.aud === NEXT_PUBLIC_GOOGLE_CLIENT_ID
    ) {
      throw new Error("Token issuer or audience invalid.")
    }
  } catch (err) {
    return res.status(422).json({ message: err.toString() })
  }

  // Register the user
  try {
    const params = {
      ClientId: COGNITO_APP_CLIENT_ID,
      Username: googlePayload.email.split("@")[0], // Username extracted from email address
      Password: googlePayload.sub,
      UserAttributes: [
        {
          Name: 'email',
          Value: googlePayload.email
        },
        {
          Name: 'custom:RegistrationMethod',
          Value: 'google'
        }
      ],
      ClientMetadata: {
        'EmailVerified': googlePayload.email_verified.toString()
      }
    }
    const cognitoClient = new CognitoIdentityProviderClient({
      region: COGNITO_REGION
    })
    const signUpCommand = new SignUpCommand(params)
    const response = await cognitoClient.send(signUpCommand)
    return res.status(response['$metadata'].httpStatusCode).send()
  } catch (err) {
    console.log(err)
    return res.status(err['$metadata'].httpStatusCode).json({ message: err.toString() })
  }
}

Let's go through what's happening here:

1. We need to verify the idToken by using the google-auth-library package. After verifying the token, we get a "payload" that contains some useful information about the user and about the token.

2. We need to check that the token issuer is accounts.google.com or https://accounts.google.com and that the audience matches our Google client id.

3. Next, we register the user with the SignUpCommand. We use a substring of the email as the username and the sub as the password. You'll want to be careful, you may need to find a more creative way to set the password.

Just make sure you can reproduce the string that you pass here otherwise the user will not be able to sign in later. Here, I've used sub because it does not change. You can hash this, add a prefix/suffix, or both if you want to go the extra mile.

In ClientMetadata, we specify the verification status for this email address. We will be using this in the pre-signup trigger later on.

Optionally, you can set a custom attribute custom:RegistrationMethod in case you have several registration methods for your user pool.

PreSignup Trigger

In a previous article on cognito pre-signup triggers, we added some custom logic upon signup that checks if the provided email address is already in use. Feel free to check that article out for more details on that.

We are going to add some logic to that pre-signup trigger to cater to users who register using Google.

module.exports.handler = async (event, context, callback) => {
  // Check if the provided email address is already in use

  // Verify user's email address if it's already verified with Google.
  if (event.request.userAttributes['custom:RegistrationMethod'] === "google") {
    let userEmailVerified = event.request.clientMetadata['EmailVerified'] === 'true'
    event.response.autoVerifyEmail = userEmailVerified
    event.response.autoConfirmUser = userEmailVerified
  }

  callback(null, event);
};

If the user's email address is already verified by Google, we don't need to verify it ourselves. So we set autoVerifyEmail to whatever value we set in the EmailVerified attribute of the ClientMetadata.

When you set autoVerifyEmail to true, you also have to do the same for autoConfirmUser as you cannot verify an email address without confirming the user in cognito.

You can find the code for the cognito triggers I use in this series in this Github repo.

Sign in

Now, that we've handled registration, let's handle signing in. On your login page, add the GoogleLogin button just like we did in the registration page.

<GoogleLogin 
    clientId={process.env.NEXT_PUBLIC_GOOGLE_CLIENT_ID}
    buttonText="Login with Google"
    responseType={'id_token'}
    onSuccess={googleSignInSuccess}
    onFailure={googleSignInFailure}
/>

The useAuth hook contains a googleSignInSuccess function that sends the tokenId from Google to the Google login API endpoint on the backend.

const googleSignInSuccess = (googleResponse) => {
    fetch('/api/login/google', {
      method: 'POST',
      headers: { 
        'Content-Type': 'application/json'
      },
      body: JSON.stringify({ id_token: googleResponse?.tokenId })
    }).then(res => {
      if (!res.ok) throw res
      return res.json()
    }).then(data => {
      console.log(data)
    }).catch(err => {
      console.error(err)
    })
  }

  const googleSignInFailure = (googleResponse) => {
    console.error(googleResponse)
  }

In the backend, we handle the login as follows:

import { CognitoIdentityProviderClient, AdminInitiateAuthCommand } from "@aws-sdk/client-cognito-identity-provider";
import { OAuth2Client } from "google-auth-library";

const {
  COGNITO_REGION,
  COGNITO_APP_CLIENT_ID,
  COGNITO_USER_POOL_ID,
  GOOGLE_TOKEN_ISSUER,
  NEXT_PUBLIC_GOOGLE_CLIENT_ID
} = process.env

export default async function handler(req, res){
  if (!req.method === 'POST') return res.status(405).send()

  let googlePayload

  try {
    const oauthClient = new OAuth2Client(NEXT_PUBLIC_GOOGLE_CLIENT_ID)
    const ticket = await oauthClient.verifyIdToken({
      idToken: req.body.id_token,
      audience: NEXT_PUBLIC_GOOGLE_CLIENT_ID
    })
    googlePayload = ticket.getPayload()
    if (
      !googlePayload?.iss === GOOGLE_TOKEN_ISSUER ||
      !googlePayload?.aud === NEXT_PUBLIC_GOOGLE_CLIENT_ID
    ) {
      throw new Error("Token issuer or audience invalid.")
    }
  } catch (err) {
    return res.status(422).json({ message: err.toString() })
  }

  // Sign the user in
  try {
    const params = {
      AuthFlow: 'ADMIN_USER_PASSWORD_AUTH',
      ClientId: COGNITO_APP_CLIENT_ID,
      UserPoolId: COGNITO_USER_POOL_ID,
      AuthParameters: {
        USERNAME: googlePayload?.email,
        PASSWORD: googlePayload?.sub
      }
    }
    const cognitoClient = new CognitoIdentityProviderClient({
      region: COGNITO_REGION
    })
    const adminInitiateAuthCommand = new AdminInitiateAuthCommand(params)
    const response = await cognitoClient.send(adminInitiateAuthCommand)
    return res.status(response['$metadata'].httpStatusCode).json({
      ...response.AuthenticationResult
    })
  } catch (err) {
    console.log(err)
    return res.status(err['$metadata'].httpStatusCode).json({ message: err.toString() })
  }
}

The first part of the handler is quite similar to the registration: we verify the token and grab the user's profile details along with more information about the token.

We check the token's issuer and audience to make sure they are correct, and then we proceed to the sign in.

For signing in, we use AdminInitiateAuthCommand along with the ADMIN_USER_PASSWORD_AUTH authentication flow in the params.

AdminInitiateAuthCommand is the recommended way to handle auth in a secure backend environment as it requires developer credentials, adding an extra layer of security.

Pass the username in the params and then generate the password the same way you did in the registration handler. This is why it's important to be able to reproduce the password you created in the registration.

Although we're logging in with Google on the surface, we're still using username/password "under the hood".

Conclusion

That's it on this workaround on Google login with AWS Cognito without going through the hosted UI. This is not an ideal solution, but it serves the purpose if you want to quickly implement simple Google signup/sign-in in your app.

You can check out this repository for the code I reference in this series.

In this article, I'm going to demonstrate how to achieve the same goal, but using the serverless framework instead of the AWS console to develop and deploy the lambdas and policies.

Setup

First install the serverless framework using the following command: npm install -g serverless.

Once serverless is installed, create a project with the following command; serverless create --template aws-nodejs -n cognito-triggers.

With the command above, we're creating a nodejs serverless project intended to be hosted on the AWS cloud platform. We then pass the name of cognito-triggers for the project.

Feel free to use whatever language you want in this project, just be sure to follow your language's package installation and build steps where necessary.

Implementation

First, let's create a .env file at the root of the project with the following content:

COGNITO_USER_POOL_NAME=<user_pool_name>
COGNITO_USER_POOL_ID=<user_pool_id>
COGNITO_USER_POOL_ARN=<user_pool_arn>
REGION=us-west-2

If you're deploying to multiple environments (e.g. testing, staging, production) then you should have multiple env files in your project named in the format .env.{stage}. So the production env file will be .env.production.

For this project, we will stick to one env file for the sake of simplicity.

Let's install a few packages we're going to need for this project with the following command: npm install @aws-sdk/client-cognito-identity-provider dotenv serverless-dotenv-plugin

We will need the dotenv and serverless-dotenv-plugin for loading environment variables. We also need the serverless-offline package if we want to invoke our lambda functions locally. We won't be going into that in this article but you can install it with npm install -D serverless-offline.

You'll notice that there's a serveless.yml file at the root of the project. Open the file and add the following code:

service: cognito-triggers

useDotenv: true

plugins:
  - serverless-dotenv-plugin
  - serverless-offline

frameworkVersion: '3'

In the code above, we're setting the service name, setting useDotenv: true to allow us to load environment variables.

The plugins section has 2 plugins:

1. serverless-dotenv-plugin for loading environment variables.
2. serverless-offline for running the project on the local machine (in case you want to invoke the functions locally).

Now let's define the provider:

provider:
  name: aws
  runtime: nodejs14.x
  stage: ${opt:stage, 'dev'}
  region: ${env:REGION}
  profile: default
  iam:
    role:
      statements:
        - Effect: 'Allow'
          Action: 'cognito-idp:ListUsers'
          Resource: '${env:COGNITO_USER_POOL_ARN}'

package:
  patterns:
    - '!.gitignore'

We set the provider name to aws because we're using the AWS platform. Our runtime of choice is nodejs14.x. Be sure to use a runtime that's available for the platform you're deploying to.

When setting the stage, we prefer the provided --stage in the command options. If one is not provided, default to dev. This will dictate which environment file to use when running the service offline or deploying. For example sls offline --stage staging will run the service on your local machine using .env.staging while sls deploy --stage production will deploy the service with the .env.production file.

Now for the interesting part, the actual lambda functions! At the top level of the serverless.yml file right below the provider section, create a function section with the following code:

functions:
  pre_signup:
    handler: ./src/pre_signup.handler
    events:
      - cognitoUserPool:
          pool: ${env:COGNITO_USER_POOL_NAME}
          trigger: PreSignUp
          existing: true

The functions section is where we declare the lambda functions in our service. Here we have a pre_signup function.

The handler property of the function points to the handler function exported by a js file. Make sure the path matches the location of your file. Here we have the file in an src folder that's located at the root of the project.

The events property determines what kind of events can trigger this lambda function. This can be an HTTP request through an API gateway, or in our case, a Cognito signup to a user pool.

If you already have an existing user pool, you have to set the existing property to true for the cognitoUserPool while specifying the user pool's name in the pool property.

Let's create the js function to handle all the pre signup logic.

Create an src folder at the root of the project and then create a file called pre_signup.js within that folder. The file will have the following content:

'use strict';
require("dotenv").config({});

const { COGNITO_USER_POOL_ID } = process.env;

const {
  CognitoIdentityProviderClient,
  ListUsersCommand
} = require("@aws-sdk/client-cognito-identity-provider");

module.exports.handler = async (event, context, callback) => {
  const client = new CognitoIdentityProviderClient();

  const listUsersCommand = new ListUsersCommand({
    UserPoolId: COGNITO_USER_POOL_ID,
    Filter: `email = "${event.request.userAttributes.email}"`
  });

  const result = await client.send(listUsersCommand);

  if (result.Users.length > 0) return callback(new Error("Email is already in use."), event);

  callback(null, event);
};

This code is very familiar if you read my previous article on Pre Signup Validation on AWS Cognito. Basically, we're listing the users in our user pool that have the same email address as the one provided in this signup attempt. If we have more than 0, then throw an error stating that the email address is already in use.

Notice, we're exporting a handler function. This is the function that we reference in the serverless.yml file.

While we're here, let's create another function to edit the messages sent to the user's email address. Cognito user pools have a Custom message trigger that allows us to intercept a message before it's sent and edit its content.

In the functions section of serverless.yml, create a function called custom_message with the following properties:

  custom_message:
    handler: ./src/custom_message.handler
    events:
      - cognitoUserPool:
          pool: ${env:COGNITO_USER_POOL_NAME}
          trigger: CustomMessage
          existing: true

It is identical to the pre_signup functions except it's referencing a different handler and hooking into the CustomMessage trigger.

In the src folder create a custom_message.js file with the following content:

'use strict';
require("dotenv").config({});

module.exports.handler = async (event, context, callback) => {
  switch(event.triggerSource) {
    case "CustomMessage_SignUp":
      event.response.smsMessage = `Hi ${event.userName}, your signup code is ${event.request.codeParameter}`;
      event.response.emailSubject = `Your registration code`;
      event.response.emailMessage = `Hi ${event.userName}, your signup code is ${event.request.codeParameter}`;
      break;
    case "CustomMessage_ForgotPassword":
      event.response.smsMessage = `Hi ${event.userName}, your password reset code is ${event.request.codeParameter}. If you did not request this code, ignore this message. Please DO NOT share this code with anyone.`;
      event.response.emailSubject = `Your password reset code`;
      event.response.emailMessage = `Hi ${event.userName}, your password reset code is ${event.request.codeParameter}. If you did not request this code, ignore this email. Please DO NOT share this code with anyone.`;
      break;
    case "CustomMessage_ResendCode":
      event.response.smsMessage = `Hi ${event.userName}, your requested code is ${event.request.codeParameter}`;
      event.response.emailSubject = `Your requested code`;
      event.response.emailMessage = `Hi ${event.userName}, your requested verification code is ${event.request.codeParameter}`;
      break;
    default:
      event.response.smsMessage = `Hi ${event.userName}, your requested code is ${event.request.codeParameter}`;
      event.response.emailSubject = `Your requested code`;
      event.response.emailMessage = `Hi ${event.userName}, your requested code is ${event.request.codeParameter}`;
  }
  callback(null, event);
}

The handler captures different message events and displays a relevant message depending on the message event. CustomMessage_SignUp is the trigger source when the signup verification email is triggered, CustomMessage_ForgotPassword for the password reset email and CustomMessage_ResendCode when a manual code request is triggered (e.g attempting to log in when unconfirmed).

You can find more information about the different trigger sources here.

The event object for this trigger looks like this:

{
  version: '1',
  region: 'us-west-2',
  userPoolId: '<user_pool_id>',
  userName: '<username>',
  callerContext: {
    awsSdkVersion: 'aws-sdk-js-3.58.0',
    clientId: '<client_id>'
  },
  triggerSource: 'CustomMessage_SignUp',
  request: {
    userAttributes: {
      sub: 'd98dad2a-c2f3-4f97-bc49-a3ed3c81f27a',
      email_verified: 'false',
      'cognito:user_status': 'UNCONFIRMED',
      email: '<user_email_address>'
    },
    codeParameter: '{####}',
    linkParameter: '{##Click Here##}',
    usernameParameter: null
  },
  response: { smsMessage: null, emailMessage: null, emailSubject: null }
}

Make sure you include the codeParameter in your custom message.

Deployment

To deploy the app, run: sls deploy. If you are deploying to a specific environment, specify it with the --stage option (e.g. sls deploy --stage staging or sls deploy --stage production).

If you'd like to remove the deployed service, run sls remove or sls remove --stage <stage> and the service along with any resources you might have created in this service will be destroyed.

We want to make sure that we only have one user per email. We could build this logic into the sign-up handler in our app's backend but there are a couple of reasons why it's not the best practice.

1. The validation will only live in this particular backend. If we create another app that interacts with the same user pool, we will have to repeat this logic there too.
2. We're adding bloat to the request handler, we could keep this handler clean by avoiding unnecessary code if we can.

Enter Cognito user pool triggers. We'll specifically look at the pre-signup trigger. This trigger is executed before Cognito's own signup validation with a couple of notable advantages:

1. We can add custom validation when registering a new user to the user pool. In our case, it's to check if the submitted email is already in use.
2. The logic is baked into the user pool. Any signup event for this user pool will be validated regardless of where it's coming from.

Implementation

Lambda

Make sure that the lambda is in the same region as the user pool. The following is the lambda's code:

const { CognitoIdentityProviderClient, ListUsersCommand } = require("@aws-sdk/client-cognito-identity-provider");

exports.handler = async (event, context, callback) => {
    // TODO implement
    const params = {
        UserPoolId: 'us-west-2_rxOJKcUKc',
        Filter: `email = "${event.request.userAttributes.email}"`
    };

    const client = new CognitoIdentityProviderClient();
    const listUsersCommand = new ListUsersCommand(params);

    const data = await client.send(listUsersCommand);

    if (data?.Users?.length > 0) {
        callback(new Error("Email is already taken"), event);
    } else {
        callback(null, event);
    }

};

Before we link this lambda to the user pool, we need to make sure it runs properly. In this example, I'm using the JavaScript v3 SDK. If we try to run this, the @aws-sdk/client-cognito-identity-provider module will not be found.

Layer

This section is only applicable if you're using the v3 SDK at the time of writing this article. If you're not using the v3 SDK, you can skip ahead to the permissions section.

We need to create a lambda layer containing this package so we can use it in the lambda. Create a new project in your development environment. Navigate to the folder and run npm init.

Complete the setup process and then run npm install @aws-sdk/client-cognito-identity-provider to install the package.

Open the package.json file and add the following line to the scripts section:

{
  ...
  "scripts": {
    ...
    "build": "rm -rf nodejs && rm cognito-base-layer.zip && npm install && mkdir nodejs && mv node_modules nodejs && zip -r cognito-base-layer.zip . && cp cognito-base-layer.zip ~/Downloads"
  },
  ...
}

This is the build script for the layer. It does the following:

1. Delete the previous nodejs directory and cognito-base-layer.zip file from the previous build.
2. Install the packages.
3. Create a directory called nodejs.
4. Move the node_modules folder into nodejs.
5. Zip the current directory into a zip file called cognito-base-layer.zip.
6. Copy the zip file to the desired location (Optional).

If you're wondering why we move the node_modules to a subfolder, this is where the lambda will look for the installed packages. If you only have node_modules at the root, your packages will not be found.

When you're ready, run npm run build. A new zip file will be created. Now that we have the zip file, let's create a layer:

Go to the lambda service and click on Layers

Click on "Create layer"

Fill in the layer details:

Name the layer whatever you like. In the upload section, upload the zip file that was generated in the previous build.

In the "compatible architectures" section, make sure to select the same architecture that your lambda is based on. In my case, it's x86_64.

When you're done click "Create".

Let's add this layer to the lambda to allow it to make use of the package(s) in the layer.

Navigate to the lambda function and click on Layers:

Click on Add a layer.

Fill in the layer details:

For the layer source, select Custom layers. In the custom layers dropdown menu, select the layer you just created. Once selected, you'll have the option to select a layer version.

Every time you make another upload on a layer, a new version is created and the old version is kept. This is to prevent breaking lambda functions that depend on the current version as each lambda has to specify which layer version they depend on.

Once you're done, click "Add".

That's it on layers, now our lambda can make use of the AWS JavaScript v3 SDK.

Permissions

We're not done yet. This lambda currently doesn't have the correct permissions to list users from a Cognito user pool. Let's grant the correct permissions to the lambda function. First, take note of the lambda function's role.

You can find this in the Configuration tab of the lambda function:

Navigate to the IAM Management Console and click on the Policies tab:

Click on "Create Policy":

Click on the JSON tab and you should see a template like this:

{
    "Version": "2012-10-17",
    "Statement": []
}

In the statement array add the following code:

{
    "Effect": "Allow",
    "Action": "cognito-idp:ListUsers",
    "Resource": "arn:aws:cognito-idp:<region>:<account_id>:userpool/<userpool_id>"
}

The above policy statement allows us to list users from the specified Cognito user pool. Replace the Resource value with your user pool's ARN.

Now let's attach this policy to the lambda's role in order to allow the lambda to make use of it.

Navigate to "Roles" in the IAM Management Console:

Search for the lambda's role and click on it:

Click on the "Add permissions" dropdown and then click on "Attach policies":

Search for the policy you just created, click on the check box and then click on "Attach policies" at the bottom of the page.

Now we have the right permissions and layers to execute this lambda. All we need to do now is set the lambda as a pre signup trigger for the user pool.

Triggers

Navigate to the user pool's console and click on the "Triggers" tab:

Click on the Pre sign-up dropdown and select the lambda function we created:

Click on "Save changes" at the bottom of the page.

Result

That's all there is to the setup for our use case. Now we can go ahead and try to sign up with an email address that is already used in the user pool. Doing so should return an error that looks like this:

UserLambdaValidationException: PreSignUp failed with error Email is already taken.
...
{
    '$fault': 'client',
    '$metadata': {
        httpStatusCode: 400,
        requestId: '3bc8f968-cbf5-4960-857f-e48daa312870',
        extendedRequestId: undefined,
        cfId: undefined,
        attempts: 1,
        totalRetryDelay: 0
  },
  __type: 'UserLambdaValidationException'
}

Bonus

You can do more with a pre sign-up trigger. The event object passed to the lambda function has a response property with the following structure:

"response": { "autoConfirmUser": "boolean", "autoVerifyPhone": "boolean" "autoVerifyEmail": "boolean" }

These all default to false. They can be set to true if you'd like to skip some parts of the default sign up flow. Before invoking the callback, you can add the following statements:

// Automatically set user's account status to CONFIRMED
event.response.autoConfirmUser = true

// Automatically set phone number as verified
event.response.autoVerifyPhone = true

// Automatically set email as verified
event.response.autoVerifyEmail = true

However, this time it’s a little different. Previous articles have been about managing user authentication yourself. In this article, we will be leveraging AWS Cognito and its user pools for the same functionality.

You can check out some of my previous articles on handling user auth manually here:

1. How to Create Registration & Authentication with Express & PassportJS.
2. How to Handle Password Reset in ExpressJS.
3. How to Verify Users in ExpressJS

Setup

This article assumes that you have an AWS Cognito user pool set up and that you have some NextJS boilerplate code set up as well. I will write another article explaining how to set up a Cognito user pool using the AWS console.

First, let’s understand the project structure.

.
├── components
│   ├── layouts
│   │   └── InputLayout.js
│   ├── AuthLinkText.js
│   ├── InputField.js
│   ├── InputHelperText.js
│   ├── Label.js
│   └── SubmitButton.js
├── hooks
│   ├── useAuth.js
│   ├── useRegister.js
│   └── useValidationSchema.js
├── pages
│   ├── api
│   │   ├── confirm
│   │   │   ├── index.js
│   │   │   └── send.js
│   │   ├── password
│   │   │   ├── reset.js
│   │   │   └── reset\_code.js
│   │   ├── login.js
│   │   └── register.js
│   ├── password
│   │   ├── reset.js
│   │   └── reset\_code.js
│   ├── \_app.js
│   ├── confirm.js
│   ├── index.js
│   ├── login.js
│   └── register.js
├── public
│   ├── favicon.ico
│   └── vercel.svg
├── styles
│   ├── Home.module.css
│   └── globals.css
├── README.md
├── next.config.js
├── package-lock.json
└── package.json

Most of these are nothing special as they’re created automatically when you set up a NextJS project.

The “components” folder contains all the re-usable custom components like form input fields and layouts.

For this article, I will focus on the “hooks” and “pages” folders. If you’d like to look at the full code, you can find it on Github.

The “hooks” folder has 3 files:

1. useAuth.js contains a hook that will handle user sign in and password reset.
2. useRegister.js contains a hook that will handle user registration and verification.
3. useValidationSchema.js contains the form validation schemas. We won’t be covering this in the article. Feel free to check the repo out for more details on this.

I prefer to use hooks because they allow me to separate the business logic from the UI logic in the component.

The “pages” folder has an “API” sub-directory. This is where all the backend code lives.

NextJS uses file-system based routing so the file structure is what determines the API endpoints.

All the other files and sub-directories outside the API sub-directory will be treated as frontend pages.

Make sure to install the @aws-sdk/client-cognito-identity-provider package.

You can refer to the full CognitoIdentityServiceProvider SDK for more in-depth explanations of what is discussed in this article.

Sign Up

First, let’s handle user registration. Navigate to the “pages/api/register.js” and add the following code:

import { CognitoIdentityProviderClient, SignUpCommand } from '@aws-sdk/client-cognito-identity-provider'

const { COGNITO_REGION, COGNITO_APP_CLIENT_ID } = process.env

export default async function handler(req, res) {
    if (req.method !== 'POST') return res.status(405).send()

    const params = {
        ClientId: COGNITO_APP_CLIENT_ID,
        Password: req.body.password,
        Username: req.body.username,
        UserAttributes: [
            {
                Name: 'email',
                Value: req.body.email
            }
        ]
    }

    const cognitoClient = new CognitoIdentityProviderClient({
        region: COGNITO\_REGION
    })
    const signUpCommand = new SignUpCommand(params)

    try {
        const response = await cognitoClient.send(signUpCommand)
        return res.status(response['$metadata'].httpStatusCode).send()
    } catch (err) {
        console.log(err)
        return res.status(err['$metadata'].httpStatusCode).json({ message: err.toString() })
    }
}

This is a handler for the “/api/register” endpoint. Include a guard clause to make sure only POST requests are allowed, return a 405 error for any other type of request.

In the params, we have the following:

1. ClientId – App client Id that you created for this user pool in the AWS console.
2. Password – The user’s chosen password.
3. Username -The user’s chosen username.
4. UserAttributes – Any additional attributes provided by the user upon signing up. The default list of attributes includes address, nickname, birthdate, phone number, email, family name, preferred username, gender, profile, given name, zoneinfo, locale, updated at, middle name, website and name. You can add custom attributes as well (for example, “Department” if you’re creating an auth system for an organisation).

Keep in mind that if you’re setting a custom user attribute you need to follow the following format:

{
    Name: 'custom:<AttributeName>',                
    Value: '<AttributeValue>'
}

In the case of adding a department attribute, it would look like this:

{
    Name: 'custom:Department',                
    Value: 'Engineering'
}

Then we send the SignUpCommand to trigger a user sign up. Return a status 200 response if all goes well, otherwise, return the error code from Cognito and the stringified version of the error.

Cognito will usually return an error that looks like this:

UsernameExistsException: User already exists
...
{
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 400,
    requestId: '9442223e-ef29-40c1-880e-6689638d8042',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  __type: 'UsernameExistsException'
}

Grab the status code and message and then forward them to the front end.

When the request is successful, Cognito will return a response that looks like this:

{
  '$metadata': {
    httpStatusCode: 200,
    requestId: '67f6d99c-d13c-4677-ab46-957d88f62bb9',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  AuthenticationResult: {
    AccessToken: "...",
    ExpiresIn: 3600,
    NewDeviceMetadata: undefined,
    RefreshToken: "...",
    TokenType: "Bearer"
  },
  ChallengeName: undefined,
  ChallengeParameters: {},
  Session: undefined
}

For this application, we’re only interested in the Authentication result as it contains the access and refresh tokens.

Flesh out the useRegister hook in order to make use of the API endpoint we created above.

Create a method called “register” with the following logic:

import { useRouter } from 'next/router'

export default function useRegister() {

    const router = useRouter()

    const register = (values, { setSubmitting }) => {
        fetch('/api/register', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json'
            },
            body: JSON.stringify(values)
        }).then(res => {
            if (!res.ok) throw res
            router.push({
                pathname: '/confirm',
                query: { username: values?.username }
            },
                "/confirm")
        }).catch(err => {
            console.error(err)
        }).finally(() => {
            setSubmitting(false)
        })
    }

    const confirm = (values, { setSubmitting }) => {
        // Confirm the user
    }

    return {
        register,
        confirm
    }
}

The register method is the submit method for our registration form. The “values” object contains the form data and “setSubmitting” allows us to update the loading state once we’re done processing the request.

This will be a common theme with most of the hooks in this project.

Set the Content-Type header to “application/json” to help NextJS parse the request body.

In this method, we call the register endpoint and pass all the form values (which include username, email, and password). If the request is successful, we redirect to the confirm page and prompt the user to verify their email address. More on this in the verification section.

You might notice we have an empty confirm method here. This method will handle the request to verify the user. We will cover this in the verification stage as well.

Here is what the registration form will look like, notice we import the useRegister hook and pass the register method as the form submit handler.

import { Formik } from "formik";
import InputLayout from "../components/layouts/InputLayout";
import Label from "../components/Label";
import InputField from "../components/InputField";
import InputHelperText from "../components/InputHelperText";
import AuthLinkText from "../components/AuthLinkText";
import SubmitButton from "../components/SubmitButton";
import useValidationSchema from "../hooks/useValidationSchema";
import useRegister from '../hooks/useRegister'
import Link from "next/link";

export default function Register() {

    const { registerSchema } = useValidationSchema()
    const { register } = useRegister()

    return (
        <div style={{
            padding: "10px"
        }}>
            <Formik
                initialValues={{
                    username: "",
                    email: "",
                    password: "",
                    confirm_password: ""
                }}
                validationSchema={registerSchema}
                onSubmit={register}
                validateOnMount={false}
                validateOnChange={false}
                validateOnBlur={false}>
                {({
                    isSubmitting,
                    errors,
                    values,
                    handleSubmit,
                    handleChange,
                    handleBlur
                }) => (
                    <form onSubmit={handleSubmit}>
                        <InputLayout>
                            <Label>Username</Label>
                            <InputField
                                type="text"
                                name="username"
                                placeholder="Username"
                                onChange={handleChange}
                                onBlur={handleBlur}
                                value={values?.username}
                            />
                            <InputHelperText isError>{errors?.username}</InputHelperText>
                        </InputLayout>
                        <InputLayout>
                            <Label>Email</Label>
                            <InputField
                                type="email"
                                name="email"
                                placeholder="Email"
                                onChange={handleChange}
                                onBlur={handleBlur}
                                value={values?.email}
                            />
                            <InputHelperText isError>{errors?.email}</InputHelperText>
                        </InputLayout>
                        <InputLayout>
                            <Label>Password</Label>
                            <InputField
                                type="password"
                                name="password"
                                placeholder="Password"
                                onChange={handleChange}
                                onBlur={handleBlur}
                                value={values?.password}
                            />
                            <InputHelperText isError>{errors?.password}</InputHelperText>
                        </InputLayout>
                        <InputLayout>
                            <Label>Confirm password</Label>
                            <InputField
                                type="password"
                                name="confirm_password"
                                placeholder="Confirm password"
                                onChange={handleChange}
                                onBlur={handleBlur}
                                value={values?.confirm_password}
                            />
                            <InputHelperText isError>{errors?.confirm_password}</InputHelperText>
                        </InputLayout>
                        <InputLayout>
                            <AuthLinkText href="/login">Already have an account? Log in</AuthLinkText>
                        </InputLayout>
                        <SubmitButton isSubmitting={isSubmitting} />
                    </form>
                )}
            </Formik>
        </div>
    )
}

Sign In

Now that we’ve registered our users, it’s time to allow them to log into our application by authenticating them against our Cognito user pool.

Add the following code to the “pages/api/login.js” file.

import { CognitoIdentityProviderClient, AdminInitiateAuthCommand } from "@aws-sdk/client-cognito-identity-provider"

const { COGNITO_REGION, COGNITO_APP_CLIENT_ID, COGNITO_USER_POOL_ID } = process.env

export default async function handler(req, res) {
    if (req.method !== 'POST') return res.status(405).send()

    const params = {
        AuthFlow: 'ADMIN_USER_PASSWORD_AUTH',
        ClientId: COGNITO_APP_CLIENT_ID,
        UserPoolId: COGNITO_USER_POOL_ID,
        AuthParameters: {
            USERNAME: req.body.username,
            PASSWORD: req.body.password
        }
    }

    const cognitoClient = new CognitoIdentityProviderClient({
        region: COGNITO_REGION
    })
    const adminInitiateAuthCommand = new AdminInitiateAuthCommand(params)

    try {
        const response = await cognitoClient.send(adminInitiateAuthCommand)
        console.log(response)
        return res.status(response['$metadata'].httpStatusCode).json({
            ...response.AuthenticationResult
        })
    } catch(err) {
        console.log(err)
        return res.status(err['$metadata'].httpStatusCode).json({ message: err.toString() })
    }
}

At the moment, we’re handling only username/password auth. To do this we need to use the ADMIN_USER_PASSWORD_AUTH flow. This authentication flow requires developer credentials. If you're authenticating users in a secure server, this is the recommended method over InitiateAuthCommand with USER_PASSWORD_AUTH.

The final parameter is AuthParameters. This simply contains the username and password passed to the endpoint upon form submission. Be careful here and make sure the auth parameters are in all uppercase as you see here, otherwise, they will not be recognised.

Now let’s divert our attention to the “useAuth” hook. Add this code to the “hooks/useAuth.js” file:

import { useRouter } from "next/router";

export default function useAuth(){

  const router = useRouter()

  const login = (values, { setSubmitting }) => {
    fetch('/api/login', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json'
      },
      body: JSON.stringify(values)
    }).then(res => {
      if (!res.ok) throw res
    }).then(data => {
      console.log(data)
    }).catch(async err => {
      const responseData = await err.json()
      if (responseData?.message?.includes("UserNotConfirmedException:")) {
        // Trigger confirmation code email
        await fetch('/api/confirm/send', {
          method: 'POST',
          headers: {
            'Content-Type': 'application/json'
          },
          body: JSON.stringify({ username: values.username })
        })
        await router.push(
      {
            pathname: "/confirm",
            query: {username: values.username},
          },
          "/confirm")
      }
    }).finally(() => {
      setSubmitting(false)
    })
  }

  const resetPasswordRequest = (values, { setSubmitting }) => {
    // Send the password reset code
  }

  const resetPassword = (values, { setSubmitting }) => {
    // Send request to reset password
  }

  return {
    login,
    resetPasswordRequest,
    resetPassword
  }
}

Here we have a login method that is similar to the register method we created above. The “values” object represents our form data (username and password).

It’s important to note that Cognito will not allow a user to be authenticated if they have not yet been verified/confirmed. So attempting this login with an unconfirmed user will return an error.

To handle this, trigger an endpoint to send a confirmation email to the user and then redirect them to the confirm page so they can be verified. More on this endpoint in the verification section.

Our login page has the following code:

import { Formik } from "formik";
import InputLayout from "../components/layouts/InputLayout";
import Label from "../components/Label";
import InputField from "../components/InputField";
import InputHelperText from "../components/InputHelperText";
import AuthLinkText from "../components/AuthLinkText";
import SubmitButton from "../components/SubmitButton";
import useAuth from "../hooks/useAuth";
import useValidationSchema from "../hooks/useValidationSchema";
import { useRouter } from "next/router";

export default function Login() {

  const router = useRouter();
  const { success } = router.query;

  const { loginSchema } = useValidationSchema();
  const { login } = useAuth();

  return (
    <div style={{
      padding: "10px"
    }}>
      {
        success === "true" &&
        <div style={{
          paddingTop: "10px",
          paddingBottom: "10px",
          color: "green"
        }}>
          {'You\\'re signed up!'}
        </div>
      }
      <Formik
        initialValues={{
          username: "",
          password: ""
        }}
        validationSchema={loginSchema}
        onSubmit={login}
        validateOnMount={false}
        validateOnChange={false}
        validateOnBlur={false}>
        {({
          isSubmitting,
          errors,
          values,
          handleChange,
          handleBlur,
          handleSubmit
        }) => (
          <form onSubmit={handleSubmit}>
            <InputLayout>
              <Label>Username</Label>
              <InputField
                type="text"
                name="username"
                placeholder="Username or email"
                onChange={handleChange}
                onBlur={handleBlur}
                value={values?.username}
              />
              <InputHelperText isError>{errors?.username}</InputHelperText>
            </InputLayout>
            <InputLayout>
              <Label>Password</Label>
              <InputField
                type="password"
                name="password"
                placeholder="Password"
                onChange={handleChange}
                onBlur={handleBlur}
                value={values?.password}
              />
              <InputHelperText isError>{errors?.password}</InputHelperText>
            </InputLayout>
            <InputLayout>
              <AuthLinkText href="/password/reset_code">{'Forgot password?'}</AuthLinkText>
            </InputLayout>
            <InputLayout>
              <AuthLinkText href="/register">{'Don\\'t have an account? Register.'}</AuthLinkText>
            </InputLayout>
            <SubmitButton isSubmitting={isSubmitting} />
          </form>
        )}
      </Formik>
    </div>
  );
}

Just like before, we import the useAuth hook and make use of its login method as the form submission handler.

We will discuss the 2 password reset methods in the useAuth hook in the password reset section.

Verification

So we’ve registered our users, but we cannot authenticate them because they are not verified. Let’s fix that.

A few things to note before we continue:

1. I have set up my user pool to use an email code for verification. This is where AWS will send an email with a code that the user has to enter manually on your app. Other authentication methods include: a clickable verification link and a verification code sent to their phone number. If you are using any of the other verification methods (except verification code to a phone number), then you can go ahead and skip this section.
2. Cognito will automatically trigger the chosen verification method upon successful registration. Verification can also be triggered via the SDK.

In the “pages/api/confirm/index.js” file, add the following code:

import {
    CognitoIdentityProviderClient,
    ConfirmSignUpCommand
} from "@aws-sdk/client-cognito-identity-provider"

const { COGNITO_REGION, COGNITO_APP_CLIENT_ID } = process.env

export default async function handler (req, res) {
    if (req.method !== 'POST') return res.status(405).send()

    const params = {
        ClientId: COGNITO_APP_CLIENT_ID,
        ConfirmationCode: req.body.code,
        Username: req.body.username
    }

    const cognitoClient = new CognitoIdentityProviderClient({
        region: COGNITO_REGION
    })
    const confirmSignUpCommand = new ConfirmSignUpCommand(params)

    try {
        const response = await cognitoClient.send(confirmSignUpCommand)
        console.log(response)
        return res.status(response['$metadata'].httpStatusCode).send()
    } catch (err) {
        console.log(err)
        return res.status(err['$metadata'].httpStatusCode).json({ message: err.toString() })
    }
}

Here create and send confirmSignUpCommand with the following parameters:

1. ClientId.
2. ConfirmationCode – The code sent to the user’s email/phone by Cognito.
3. Username – The username of the user you’d like to verify.

Note that because this is the index file within the confirm directory, we do not need to specify the filename when calling this endpoint. The endpoint would just be “/api/confirm”

In the same confirm directory, we have a file called “send.js”. This file is responsible for manually triggering the confirmation code email.

Add the following code to this file:

import {
    CognitoIdentityProviderClient,
    ResendConfirmationCodeCommand
} from "@aws-sdk/client-cognito-identity-provider"

const { COGNITO_REGION, COGNITO_APP_CLIENT_ID } = process.env

export default async function handler (req, res) {
    if (req.method !== 'POST') return res.status(405).send()

    const params = {
        ClientId: COGNITO_APP_CLIENT_ID,
        Username: req.body.username
    }

    const cognitoClient = new CognitoIdentityProviderClient({
        region: COGNITO_REGION
    })
    const resendConfirmationCodeCommand = new ResendConfirmationCodeCommand(params)

    try {
        const response = await cognitoClient.send(resendConfirmationCodeCommand)
        console.log(response)
        return res.status(response['$metadata'].httpStatusCode).send()
    } catch (err) {
        console.log(err)
        return res.stat(err['$metadata'].httpStatusCode).json({ message: err.toString() })
    }
}

Here, simply create and send ResendConfirmationCodeCommand with a params object containing the ClientId and the username of the user you’d like to verify.

Cognito will search for the user with the specified username, then send a verification code to their email/phone number depending on your settings and/or which one is provided.

You might be wondering why this command is called ResendConfirmationCodeCommand.

Cognito will automatically send a verification code to the user upon signing up.

If you are triggering the registration manually, then you are always “re-sending” the code.

Remember that “confirm” method in the useRegister hook? It’s time to flesh it out.

Add the following logic in that method:

const confirm = (values, { setSubmitting }) => {
        fetch('/api/confirm', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json'
            },
            body: JSON.stringify(values)
        }).then(res => {
            if (!res.ok) throw res
            router.push({
                pathname: '/login',
                query: { confirmed: true }
            },
                "/login")
        }).catch(err => {
            console.error(err)
        }).finally(() => {
            setSubmitting(false)
        })
    }

From this method, call the “/api/confirm” endpoint with the form values. If the confirmation is successful, redirect the user to the login page so they can sign in.

Here’s a look at the confirm page in the “pages/confirm.js” file:

import { Formik } from "formik";
import InputLayout from "../components/layouts/InputLayout";
import Label from "../components/Label";
import InputField from "../components/InputField";
import InputHelperText from "../components/InputHelperText";
import SubmitButton from "../components/SubmitButton";
import useValidationSchema from "../hooks/useValidationSchema";
import useRegister from "../hooks/useRegister";
import { useRouter } from "next/router";

export default function Confirm(){

    const router = useRouter();
    const { username } = router.query;

    const { confirm } = useRegister();
    const { confirmSchema } = useValidationSchema();

    return (
        <div style={{
            padding: "10px"
        }}>
            <Formik
                initialValues={{
                    username: username,
                    code: ""
                }}
                onSubmit={confirm}
                validationSchema={confirmSchema}
                validateOnMount={false}
                validateOnChange={false}
                validateOnBlur={false}>
                {
                    ({
                        isSubmitting,
                        errors,
                        values,
                        handleSubmit,
                        handleChange,
                        handleBlur
                     }) => (
                        <form onSubmit={handleSubmit}>
                            <InputLayout>
                                <Label>Confirmation Code</Label>
                                <InputField
                                    type={"text"}
                                    name={"code"}
                                    placeholder={"Code"}
                                    onChange={handleChange}
                                    onBlur={handleBlur}
                                    value={values?.code}
                                />
                                <InputHelperText isError>{errors?.code}</InputHelperText>
                            </InputLayout>
                            <SubmitButton isSubmitting={isSubmitting} />
                        </form>
                    )
                }
            </Formik>
        </div>
    )
}

Password reset

One of the most important features in any auth system is the ability to reset passwords.

Fortunately, password reset is made dead simple by Cognito.

The password reset flow is similar to the verification flow but with some extra steps:

1. The user clicks the “Forgot password” link and is redirected to a page where they are prompted to enter their username.
2. Send a confirmation code to the user’s email/number.
3. If the code is sent successfully, redirect the user to a reset page where they enter the code, along with their new password.

First, prepare the endpoint to trigger the verification email containing the code.

Add the following code to “pages/api/password/reset_code.js”:

import { CognitoIdentityProviderClient, ForgotPasswordCommand } from '@aws-sdk/client-cognito-identity-provider'

const { COGNITO_REGION, COGNITO_APP_CLIENT_ID } = process.env

export default async function handler (req, res) {
    if (req.method !== 'POST') return res.status(405).send()

    const params = {
        ClientId: COGNITO_APP_CLIENT_ID,
        Username: req.body.username
    }

    const cognitoClient = new CognitoIdentityProviderClient({
        region: COGNITO_REGION
    })
    const forgotPasswordCommand = new ForgotPasswordCommand(params)

    try {
        const response = await cognitoClient.send(forgotPasswordCommand)
        console.log(response)
        return res.status(response['$metadata'].httpStatusCode).send()
    } catch (err) {
        console.log(err)
        return res.status(err['$metadata'].httpStatusCode).json({ message: toString() })
    }
}

Here we accept the ClientId and username params for the “ForgotPasswordCommand” command. Cognito will find a user with the matching username and send them a verification code using the configured method.

In the “pages/api/password/reset.js” file, add the code below:

import {
    CognitoIdentityProviderClient,
    ConfirmForgotPasswordCommand
} from "@aws-sdk/client-cognito-identity-provider"

const { COGNITO_REGION, COGNITO_APP_CLIENT_ID } = process.env

export default async function handler(req, res) {
    if (req.method !== 'POST') return res.status(405).send()

    const params = {
        ClientId: COGNITO_APP_CLIENT_ID,
        ConfirmationCode: req.body.code,
        Password: req.body.password,
        Username: req.body.username
    }

    const cognitoClient = new CognitoIdentityProviderClient({
        region: COGNITO_REGION
    })
    const confirmForgotPasswordCommand = new ConfirmForgotPasswordCommand(params)

    try {
        const response = await cognitoClient.send(confirmForgotPasswordCommand)
        console.log(response)
        return res.status(response['$metadata'].httpStatusCode).send()
    } catch (err) {
        console.log(err)
        return res.status(err['$metadata'].httpStatusCode).json({ message: err.toString() })
    }
}

Here we accept the ConfirmationCode that was sent to the user, their new password and their username.

Cognito will take care of all the logic of making sure that the code provided is valid and is the one that was sent to the user with the specified username.

You’ll remember that in the useAuth hook, we had 2 empty methods “resetPasswordRequest” and “resetPassword”. We are going to flesh those out now.

Add the following logic to those methods:

const resetPasswordRequest = (values, { setSubmitting }) => {
    fetch('/api/password/reset\_code', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json'
      },
      body: JSON.stringify(values)
    }).then(res => {
      if (!res.ok) throw res
      router.push({
        pathname: '/password/reset',
        query: { username: values.username }
      },
        "/password/reset")
    }).catch(err => {
      console.error(err)
    }).finally(() => {
      setSubmitting(false)
    })
  }

  const resetPassword = (values, { setSubmitting }) => {
    fetch('/api/password/reset', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json'
      },
      body: JSON.stringify(values)
    }).then(res => {
      if (!res.ok) throw res
      router.push({
        pathname: '/login',
        query: { reset: true }
      },
        "/login")
    }).catch(err => {
      console.error(err)
    }).finally(() => {
      setSubmitting(false)
    })
  }

The resetPasswordRequest method triggers the “/api/password/reset_code” endpoint.

From the user’s perspective, they are redirected to a page where they enter their username. Once they submit the form, they receive an email and are redirected to the password reset page (if the email was successfully sent).

The resetPassword method triggers the “/api/password/reset” endpoint to actually reset the password.

Here’s how the forgot password form looks:

import { Formik } from "formik";
import InputLayout from "../../components/layouts/InputLayout";
import Label from "../../components/Label";
import InputField from "../../components/InputField";
import InputHelperText from "../../components/InputHelperText";
import SubmitButton from "../../components/SubmitButton";
import useValidationSchema from "../../hooks/useValidationSchema";
import useAuth from '../../hooks/useAuth';

export default function ResetCode(){

    const { resetPasswordRequestSchema } = useValidationSchema();
    const { resetPasswordRequest } = useAuth();

    return (
        <div style={{
            padding: "10px"
        }}>
            <Formik
                initialValues={{
                    username: ""
                }}
                onSubmit={resetPasswordRequest}
                validationSchema={resetPasswordRequestSchema}
                validateOnMount={false}
                validateOnChange={false}
                validateOnBlur={false}
                >
                {
                    ({
                        isSubmitting,
                        errors,
                        values,
                        handleSubmit,
                        handleChange,
                        handleBlur
                    }) => (
                        <form onSubmit={handleSubmit}>
                            <InputLayout>
                                <Label>Username</Label>
                                <InputField
                                    type={"text"}
                                    name={"username"}
                                    placeholder={"Username"}
                                    onChange={handleChange}
                                    onBlur={handleBlur}
                                    value={values?.username}
                                />
                                <InputHelperText isError>{errors?.username}</InputHelperText>
                            </InputLayout>
                            <SubmitButton isSubmitting={isSubmitting} />
                        </form>
                    )
                }
            </Formik>
        </div>
    )
}

Here’s how the password reset form looks:

import { Formik } from "formik";
import InputLayout from "../../components/layouts/InputLayout";
import Label from "../../components/Label";
import InputField from "../../components/InputField";
import InputHelperText from "../../components/InputHelperText";
import SubmitButton from "../../components/SubmitButton";
import useValidationSchema from "../../hooks/useValidationSchema";
import useAuth from '../../hooks/useAuth';
import { useRouter } from "next/router";

export default function Reset(){

    const router = useRouter()
    const { username } = router.query

    const { resetPasswordSchema } = useValidationSchema();
    const { resetPassword } = useAuth()

    return (
        <div style={{
            padding: "10px"
        }}>
            <Formik
                initialValues={{
                    username: username,
                    code: "",
                    password: "",
                    confirm\_password: ""
                }}
                validationSchema={resetPasswordSchema}
                onSubmit={resetPassword}
                validateOnMount={false}
                validateOnChange={false}
                validateOnBlur={false}
            >
                {
                    ({
                        isSubmitting,
                        errors,
                        values,
                        handleSubmit,
                        handleBlur,
                        handleChange
                    }) => (
                        <form onSubmit={handleSubmit}>
                            <InputLayout>
                                <Label>Reset code</Label>
                                <InputField
                                    type={"text"}
                                    name={"code"}
                                    placeholder={"Reset code"}
                                    onChange={handleChange}
                                    onBlur={handleBlur}
                                    value={values?.code}
                                />
                                <InputHelperText isError>{errors?.code}</InputHelperText>
                            </InputLayout>
                            <InputLayout>
                                <Label>New password</Label>
                                <InputField
                                    type={"password"}
                                    name={"password"}
                                    placeholder={"New password"}
                                    onChange={handleChange}
                                    onBlur={handleBlur}
                                    value={values?.password}
                                />
                                <InputHelperText isError>{errors?.password}</InputHelperText>
                            </InputLayout>
                            <InputLayout>
                                <Label>Confirm password</Label>
                                <InputField
                                    type={"password"}
                                    name={"confirm_password"}
                                    placeholder={"Confirm password"}
                                    onChange={handleChange}
                                    onBlur={handleBlur}
                                    value={values?.confirm_password}
                                />
                                <InputHelperText isError>{errors?.confirm_password}</InputHelperText>
                            </InputLayout>
                            <SubmitButton isSubmitting={isSubmitting} />
                        </form>
                    )
                }
            </Formik>
        </div>
    )
}

Caveat

A major caveat with the implementation we’ve just created is that 2 or more users can actually register with the same email.

We don’t want any user to receive a code for another user’s password reset.

For this and many other reasons, you may want to force unique emails for each user.

We can easily achieve this using Cognito triggers which are event triggers that run custom lambda functions. These allow us to customise our workflows.

We can trigger lambdas during any of the following events (and more):

1. Pre-signup – Right before Cognito signup is triggered.
2. Pre-authentication – Right before the user is authenticated by Cognito.
3. Custom message – Right before a verification/confirmation message is sent. We can dynamically edit the message here.
4. Post-authentication – After the user is successfully authenticated. If the authentication fails, this will not be triggered.
5. Post-confirmation – After a user is verified/confirmed. You can use this to send a welcome email or enable certain privileges that are only available to confirmed users.

These are only a few of the triggers available to us. You can find more information on these triggers here.

In our case, the trigger we’re most interested in is the Pre-signup trigger. We can check if there are any current users who have the same email address as the one we’ve just received for signup. If so, throw an error and fail early before the Cognito signup.

I will publish an article that demonstrates how to achieve this.

It’s common for bots or users with fake email addresses and no intention of seriously using your application to register. One way to deal with this at the start is by making sure to verify users.

This article is a tutorial on user verification in ExpressJS and a continuation of my Express web development series. I will be building on top of the concepts discussed in my previous article on handling password resets.

The setup and required packages are specified in that article but you will be able to see what packages are used in the code examples.

I’d suggest taking a look at the other articles in the series first, although you should be able to follow along with this one regardless. Check out the project on GitHub if you’d like to track it as the series progresses.

Models

Let’s first create the model that holds the verification tokens. Navigate to the models folder and create a file called ‘UserVerification.js’. The file should have the following content:

const { Schema, model } = require('mongoose')

const schema = new Schema({
  user : {
    type: Schema.Types.ObjectId,
    ref: 'User',
    required: true
  },
  token: {
    type: Schema.Types.String,
    required: true
  }
}, {
  timestamps: true
})

schema.index({ 'updatedAt': 1 }, { expireAfterSeconds: 300 })

const UserVerification = model('UserVerification', schema)

module.exports = UserVerification

The model schema contains a token that will be included in the verification link, and the user it’s associated with.

Create an index on the ‘updatedAt’ field that instructs MongoDB to delete the record after 5 minutes from the time the record is updated. 5 minutes is reasonable for testing but you’ll want to increase this to something more reasonable in production.

In the user model, add a boolean ‘verified’ property to the schema. Set the default to false as the user would not be verified upon registration.

const { Schema, model } = require('mongoose')

const saltRounds = 10

var userSchema = new Schema({
  name: {
    type: Schema.Types.String,
    required: [true, 'You must provide a name']
  },
  email: {
    type: Schema.Types.String,
    required: [true, 'Email address is required']
  },
  username: {
    type: Schema.Types.String,
    required: [true, 'Username is required']
  },
  password: {
    type: Schema.Types.String,
    required: [true, 'You must provide a password']
  },
  verified: {
    type: Schema.Types.Boolean,
    required: true,
    default: false
  }
})

...

const User = model('User', userSchema)

module.exports = User

Routes

Profile routes

The first route we have to create is the profile route. This route will simply render a template with the user’s profile details. Create a file in the routes folder named ‘profile.js’ and add a route that renders the ‘profile.html’ template.

const router = require('express').Router()

router.get('/profile', (req, res) => {
  if (!req.isAuthenticated()) return res.redirect('/login')
  return res.render('profile.html')
})

module.exports = router

User verification routes

Now let’s create the routes that will handle user verification. In the routes folder, create a file named ‘user-verification.js’. To start with, the file will have the following contents:

const router = require('express').Router()
const { v4 } = require('uuid')
const { User, UserVerification } = require('../models')
const { sendEmail } = require('../helpers')

/* Create routes here */

module.exports = router

Import the User and UserVerification models. Import the ‘sendMail’ helper function that we created in the previous article. This is simply a function that uses NodeMailer to send an email using the arguments passed to it.

Now let’s create the routes.

Create verification url

The first route is a get route ‘/verify’. This route is responsible for creating the verification URL and has the following contents:

router.get('/verify', async (req, res) => {
  if (!req.isAuthenticated()) return res.redirect('/login')
  if (req.user.verified) return res.redirect('back')

  const token = v4().toString().replace(/-/g, '')
  const verificationUrl = `${process.env.DOMAIN}/verify-confirm/${token}`

  await UserVerification.updateOne({ 
    user: req.user._id 
  }, {
    user: req.user._id,
    token: token
  }, {
    upsert: true
  })

  sendEmail({
    to: req.user.email,
    subject: 'Verify your email address',
    text: `Here's your email verification link: ${verificationUrl}`
  })

  req.flash('verify_success', 'Check your email address for your verification link. It may take a few minutes')
  res.redirect('/profile')
})

First, check if the user is authenticated. The user should only be able to request a verification link when they are logged in. If they aren’t, redirect them to the login page.

Check if the user is already verified. We don’t want to send a verification link if the user is already verified. If they are, redirect to the previous page.

Create the token and then the verification URL that contains the token.

Update the UserVerification record associated with the current user. Make sure to set the upsert option to ‘true’. We want to replace the current verification link so that only one can be active at a time, but we also want to create a new one if there isn’t one in the collection.

Send the email containing the user verification link, flash a success message urging the user to check their email address and then redirect to the user’s profile.

Verify user

The second route handles the link sent to the user:

router.get('/verify-confirm/:token', async (req, res) => {
  if (!req.isAuthenticated()) return res.redirect('/login')

  const token = req.params.token

  const userVerification = await UserVerification.findOne({
    user: req.user._id,
    token: token
  })

  if (userVerification) {
    await User.updateOne({ _id: req.user._id }, { verified: true })
    await UserVerification.deleteOne({ 
      user: req.user._id,
      token: token
    })
    sendEmail({
      to: req.user.email,
      subject: 'Verified',
      text: `Congratulations ${req.user.name}, your account is now verified!`
    })
    req.flash('verify_success', 'Congrats, you are now verified!')
  } else {
    req.flash('verify_error', 'Verification link is invalid or has expired.')
  }

  return res.redirect('/profile')
})

This route expects a token which we will validate later. First check whether the user is logged in, if not, redirect to the login page.

Extract the token from the url and query the UserVerification collection for a document with the current token and the current user.

If the document doesn’t exist, flash an error message stating that the link is invalid or expired.

If the document exists, update the user’s verified status to ‘true’ and delete the current UserVerification document in order to prevent the link from being clicked again (this would be pointless anyway but it’s good practice).

Send the user an email confirming their verification status and then flash a success message stating that the user has now been verified. Redirect to the user’s profile page afterwards.

Import routes

Go into the app’s entry folder and include the profile and user-verification routes with the following code:

app.use('/', require('./routes/profile'))
app.use('/', require('./routes/user-verification'))

Templates

There’s one new template that we need to create for this feature: the profile template.

{% extends 'base.html' %}

{% set title = 'Profile' %}

{% block content %}
  {% if messages.verify_success %}
    <div class="alert alert-success" role="alert">
      {{ messages.verify_success }}
    </div>
  {% endif %}
  {% if messages.verify_error %}
    <div class="alert alert-danger" role="alert">
      {{ messages.verify_error }}
    </div>
  {% endif %}
  <div>
    <h5>Hi, {{ user.name }}</h5>
    {% if not user.verified %}
      Your email is not verified, 
      <a class="btn btn-sm btn-warning" href="/verify">Verify Email</a>
    {% endif %}
  </div>
{% endblock %}

{% block scripts %}
{% endblock %}

This template renders the error or success message flashed in the previous request. We have a div that displays the user’s name and a button to generate the verification URL conditionally based on the user’s verified status.

Conclusion

In this article, I demonstrated how to verify users in your Express application. There are many reasons you may want to verify users: You may want to make sure you have active, human users on your app or maybe you want to restrict features that require users to be verified.

Whatever the reason, I hope that this article has provided sufficient guidance on the flow and execution of the verification process.

The next article will be about creating user following and follower relationships using many-to-many relationships in MongoDB.

It is necessary to provide a way for users to recover access to their accounts/data in case of a lost or forgotten password. In this article, I will be demonstrating how to handle password resets in ExpressJS.

In the last 2 articles, I wrote about how to connect ExpressJS application to MongoDB database and building a user registration and authentication system.

Both of these articles tie into today’s article. We’re going to use mongoose and our saved user data to enable password resets.

If you’ve read those articles, or already have your own authentication system, read on. Even if you’re using a different tech stack, you may still gain some valuable ideas from this approach.

As always, this project is hosted on Github. Feel free to clone the project to get access to the source code I use in this article.

The password reset flow

Before we dive into the code, let’s first establish what the password reset flow will look like from the user’s perspective and then design the implementation of this flow.

User’s perspective

From the user’s perspective, the process should go as follows:

1. Click on the ‘Forgot password’ link on the login page.
2. Redirected to a page that requires an email address.
3. Receive the password reset link in an email.
4. Link redirects to a page that requires a new password and password confirmation.
5. After submission, redirected to the login page with a success message.

Reset system characteristics

We also need to understand some characteristics of a good password reset system:

1. Unique password reset link should be generated for the user such that when the user visits the link, they are instantly identified. This means including a unique token in the link.
2. Password reset link should have an expiry time (e.g. 2 hours) after which it is no longer valid and cannot be used to reset the password.
3. The reset link should expire once the password has been reset to prevent the same link from being used to reset the password several times.
4. If the user requests to change password multiple times without following through on the whole process, each generated link should invalidate the previous one. This prevents having multiple active links from which the password can be reset.
5. If the user chooses to ignore the password reset link sent to their email, their current credentials should be left intact and valid for future authentication.

Implementation steps

We now have a clear picture of the reset flow from the user’s perspective and the characteristics of a password reset system. Here are the steps we will take in the implementation of this system:

1. Create a mongoose model called ‘PasswordReset’ to manage active password reset requests/tokens. The records set here should expire after a specified time period.
2. Include the ‘Forgot password’ link in the login form that leads to a route that contains an email form.
3. Once the email is submitted to a post route, check whether a user with the provided email address exists.
4. If the user does not exist, redirect back to the email input form and notify the user that no user with provided email was found.
5. If the user exists, generate a password reset token and save it to PasswordReset collection in a document that references the user. If there already is a document in this collection associated with this user, update/replace the current document (there can only be one per user).
6. Generate a link that includes the password reset token within it, email the link to the user.
7. Redirect to the login page with success message prompting the user to check their email address for the reset link.
8. Once the user clicks the link, it should lead to a GET route that expects the token as one of the route params.
9. Within this route, extract the token and query the PasswordReset collection for this token. If the document is not found, alert the user that the link is invalid/expired.
10. If the document is found, load a form to reset the password. The form should have 2 fields (new password & confirm password fields).
11. When the form is submitted, its post route will update the user’s password to the new password.
12. Delete the password reset document associated with this user in the PasswordReset collection.
13. Redirect the user to the login page with a success message.

Implementation

The setup

Firstly, we’ll have to set up the project. Install the uuid package for generating a unique token, and the nodemailer package for sending emails.

npm install uuid nodemailer

Add the full domain to the environment variables. We’ll need this in order to generate a fill link string to email to the user.

DOMAIN=http://localhost:8000

Make some changes to the app entry file in the following areas:

1. Set useCreateIndex to ‘true’ in the mongoose connection options. This makes mongoose’s default index build use createIndex instead of ensureIndex and prevents MongoDB deprecation warnings.
2. Import a new route file that will contain all the reset routes called ‘password-reset’. We will create these routes later.

const connection = mongoose.connect(process.env.MONGO\_URI, {
  useNewUrlParser: true,
  useUnifiedTopology: true,
  useCreateIndex: true
})

...

app.use('/', require('./routes/password-reset'))

Models

We need to have a dedicated model to handle the password reset records. In the models folder, create a model called ‘PasswordReset’ with the following code:

const { Schema, model } = require('mongoose')

const schema = new Schema({
  user: {
    type: Schema.Types.ObjectId,
    ref: 'User',
    required: true
  },
  token: {
    type: Schema.Types.String,
    required: true
  }
}, {
  timestamps: true
})

schema.index({ 'updatedAt': 1 }, { expireAfterSeconds: 300 })

const PasswordReset = model('PasswordReset', schema)

module.exports = PasswordReset

We have two properties in this model, the user that’s requested the password reset, and the unique token assigned to the particular request.

Make sure to set the timestamps option to true in order to include createdAt and updatedAt fields in the document.

After defining the schema, create an index on the updatedAt field with an expiry time of 300 seconds (5 minutes). I’ve set it this low for testing purposes. In production, you can increase this to something more practical like 2 hours.

In the User model we created in this article (or the user model you currently have), update the pre save hook to the following:

userSchema.pre('save', async function(next){
  if (this.isNew || this.isModified('password')) this.password = await bcrypt.hash(this.password, saltRounds)
  next()
})

Do this to make sure the password field is hashed whether the document is new or the password field has been changed in an existing document.

Routes

Create a new file in the route’s folder called ‘password-reset.js’. This is the file we import in the app entry file.

In this file, import the User and PasswordReset models. Import the v4 function from the uuid package for token generation.

const router  = require('express').Router()
const { User, PasswordReset } = require('../models')
const { v4 } = require('uuid')

/* Create routes here */

module.exports = router

Create the first 2 routes. These routes are associated with the form which accepts the user’s email address.

router.get('/reset', (req, res) => res.render('reset.html'))

router.post('/reset', async (req, res) => {
  /* Flash email address for pre-population in case we redirect back to reset page. */
  req.flash('email', req.body.email)

  /* Check if user with provided email exists. */
  const user = await User.findOne({ email: req.body.email })
  if (!user) {
    req.flash('error', 'User not found')
    return res.redirect('/reset')
  }

  /* Create password reset token and save in collection along with user. 
     If there already is a record with current user, replace it. */
  const token = v4().toString().replace(/-/g, '')
  PasswordReset.updateOne({ 
    user: user._id 
  }, {
    user: user._id,
    token: token
  }, {
    upsert: true
  })
  .then( updateResponse => {
    /* Send email to user containing password reset link. */
    const resetLink = `${process.env.DOMAIN}/reset-confirm/${token}`
    console.log(resetLink)

    req.flash('success', 'Check your email address for the password reset link!')
    return res.redirect('/login')
  })
  .catch( error => {
    req.flash('error', 'Failed to generate reset link, please try again')
    return res.redirect('/reset')
  })
})

The first is a GET route to ‘/reset’. In this route, render the ‘reset.html’ template. We will create this template later.

The second route is a POST route for ‘/reset’. This route expects the user’s email in the request body. In this route:

1. Flash email back for pre-population in case we redirect back to the email form.
2. Check if the user with the email provided exists. If not, flash an error and redirect back to ‘/reset’.
3. Create a token using v4.
4. Update PasswordReset document associated with the current user. Set upsert to true in options to create a new document if there isn’t one already.
5. If update is successful, mail the link to the user, flash a success message and redirect to the login page.
6. If update is unsuccessful, flash an error message and redirect back to the email page.

At the moment, we’re only logging the link to the console. We will implement the email logic later.

Create the 2 routes that come into play when the user visits the link generated link above.

router.get('/reset-confirm/:token', async (req, res) => {
  const token = req.params.token
  const passwordReset = await PasswordReset.findOne({ token })
  res.render('reset-confirm.html', { 
    token: token,
    valid: passwordReset ? true : false
  })
})

router.post('/reset-confirm/:token', async (req, res) => {
  const token = req.params.token
  const passwordReset = await PasswordReset.findOne({ token })

  /* Update user */
  let user = await User.findOne({ _id: passwordReset.user })
  user.password = req.body.password

  user.save().then( async savedUser =>  {
    /* Delete password reset document in collection */
    await PasswordReset.deleteOne({ \_id: passwordReset.\_id })
    /* Redirect to login page with success message */
    req.flash('success', 'Password reset successful')
    res.redirect('/login')
  }).catch( error => {
    /* Redirect back to reset-confirm page */
    req.flash('error', 'Failed to reset password please try again')
    return res.redirect(`/reset-confirm/${token}`)
  })
})

The first route is a get route that expects the token in the url. The token is extracted and then validated. Validate the token by searching the PasswordReset collection for a document with the provided token.

If the document is found, set the ‘valid’ template variable to true, otherwise, set it to false. Be sure to pass the token itself to the template. We will use this in the password reset form.

Check the validity of the token by searching the PasswordReset collection by token.

The second route is a POST route that accepts the password reset form submission. Extract the token from the url and then retrieve the password reset document associated with it.

Update the user associated with this particular password reset document. Set the new password and save the updated user.

Once the user is updated, delete the password reset document to prevent it from being reused to reset the password.

Flash a success message and redirect the user to the login page where they can log in with their new password.

If the update is unsuccessful, flash an error message and redirect back to the same form.

Templates

Once we’ve created the routes, we need to create the templates

In the views folder, create a ‘reset.html’ template file with the following content:

{% extends 'base.html' %}

{% set title = 'Reset' %}

{% block styles %}
{% endblock %}

{% block content %}
  <form action='/reset' method="POST">
    {% if messages.error %}
      <div class="alert alert-danger" role="alert">{{ messages.error }}</div>
    {% endif %}
    <div class="mb-3">
      <label for="name" class="form-label">Enter your email address</label>
      <input 
        type="text" 
        class="form-control {% if messages.error %}is-invalid{% endif %}" 
        id="email" 
        name="email"
        value="{{ messages.email or '' }}"
        required>
    </div>
    <div>
      <button type="submit" class="btn btn-primary">Send reset link</button>
    </div>
  </form>
{% endblock %}

Here we have one email field that is pre-populated with an email value if one was flashed in the previous request.

Include an alert that displays an error message if one has been flashed from the previous request.

Create another template in the same folder named ‘reset-confirm.html’ with the following content:

{% extends 'base.html' %}

{% set title = 'Confirm Reset' %}

{% block content %}
  {% if not valid %}
    <h1>Oops, looks like this link is expired, try to <a href="/reset">generate another reset link</a></h1>
  {% else %}
    <form action='/reset-confirm/{{ token }}' method="POST">
      {% if messages.error %}
        <div class="alert alert-danger" role="alert">{{ messages.error }}</div>
      {% endif %}
      <div class="mb-3">
        <label for="name" class="form-label">Password</label>
        <input 
          type="password" 
          class="form-control {% if messages.password_error %}is-invalid{% endif %}" 
          id="password" 
          name="password">
        <div class="invalid-feedback">{{ messages.password_error }}</div>
      </div>
      <div class="mb-3">
        <label for="name" class="form-label">Confirm password</label>
        <input 
          type="password" 
          class="form-control {% if messages.confirm_error %}is-invalid{% endif %}" 
          id="confirmPassword" 
          name="confirmPassword">
        <div class="invalid-feedback">{{ messages.confirm_error }}</div>
      </div>
      <div>
        <button type="submit" class="btn btn-primary">Confirm reset</button>
      </div>
    </form>
  {% endif %}
{% endblock %}

In this form, check for the value of the ‘valid’ variable that we set in the GET route, if false, render the expired token message. Otherwise, render the password reset form.

Include an alert that displays an error message if one was flashed in the previous request.

Go to the login form that we created in the registration & authentication article and add the following code to the top of the form:

{% if messages.success %}
    <div class="alert alert-success" role="alert">{{ messages.success }}</div>
{% endif %}

This renders the success messages that we flash when we create/send the reset link and when we update the user’s password before redirecting to the login page.

Mail

In the previous routes section, we logged the reset link in the console. Ideally, we should send an email to the user when they’ve requested a password reset link.

For this example, I’ve used ethereal.email to generate a test email account for development purposes. Head over there and create one (it’s a one-click process).

Once you’ve created the test account, add the following variables to your environment variables:

EMAIL_HOST=smtp.ethereal.email
EMAIL_NAME=Leanne Zulauf
EMAIL_ADDRESS=leanne.zulauf@ethereal.email
EMAIL_PASSWORD=aDhwfMry1h3bbbR9Av
EMAIL_PORT=587
EMAIL_SECURITY=STARTTLS

These are my values at the time of writing, plug in your own values here.

Create a ‘helpers.js’ file in the root of the project. This file will have a bunch of useful functions that are likely to be reused across the entire project.

Define these functions here so that we can import them when they’re needed rather than repeating similar logic all over our application.

const nodemailer = require('nodemailer')

module.exports = {
  sendEmail: async ({ to, subject, text }) => {
    /* Create nodemailer transporter using environment variables. */
    const transporter = nodemailer.createTransport({
      host: process.env.EMAIL_HOST,
      port: Number(process.env.EMAIL_PORT),
      auth: {
        user: process.env.EMAIL_ADDRESS,
        pass: process.env.EMAIL_PASSWORD
      }
    })
    /* Send the email */
    let info = await transporter.sendMail({
      from: `"${process.env.EMAIL_NAME}" <${process.env.EMAIL_ADDRESS}>`,
      to,
      subject,
      text
    })
    /* Preview only available when sending through an Ethereal account */
    console.log(`Message preview URL: ${nodemailer.getTestMessageUrl(info)}`)
  }
}

Export an object with various functions. The first being the sendEmail function.

This function takes the recipient’s address, email subject and email text. Create the NodeMailer transporter, using the environment variables defined previously in the options. Send the email using the arguments passed to the function.

The last line of the function logs the message url in the console so you can view the message on Ethereal mail. The test account does not actually send the email.

Go back to the ‘password-reset.js’ routes and add the email functionality. First, import the function:

const { sendEmail } = require('../helpers')

In the ‘/reset’ POST route, instead of logging the reset link on the console, add the following code:

sendEmail({
      to: user.email, 
      subject: 'Password Reset',
      text: `Hi ${user.name}, here's your password reset link: ${resetLink}. 
      If you did not request this link, ignore it.`
    })

Send an additional email to notify the user of a successful password change in the ‘/reset-confirm’ POST route once the user is successfully updated:

user.save().then( async savedUser =>  {
    /* Delete password reset document in collection */
    await PasswordReset.deleteOne({ _id: passwordReset._id })
    /* Send successful password reset email */
    sendEmail({
      to: user.email, 
      subject: 'Password Reset Successful',
      text: `Congratulations ${user.name}! Your password reset was successful.`
    })
    /* Redirect to login page with success message */
    req.flash('success', 'Password reset successful')
    res.redirect('/login')
  }).catch( error => {
    /* Redirect back to reset-confirm page */
    req.flash('error', 'Failed to reset password please try again')
    return res.redirect(`/reset-confirm/${token}`)
  })

Conclusion

In this article, I demonstrated how to implement a password reset feature in ExpressJS using NodeMailer.

In the next article, I will write about implementing a user email verification system in your Express application. I will use a similar approach to the one used in this article, with NodeMailer being the email package of choice.

This project is available on Github. Feel free to clone it if you’d like to follow along.

Setup

Let’s start by setting up the necessary packages and libraries for this portion of the project.

Run the following command to install the necessary package:

npm install passport passport-local express-session bcrypt connect-mongo express-flash joi

Here’s a breakdown of the packages we’ve just installed:

1. passport and passport-local – User authentication.
2. express-session – Sessions in ExpressJS.
3. bcrypt – Password encryption and comparison on authentication.
4. connect-mongo – Mongo store for express sessions.
5. express-flash – Flashing messages for display in the front-end.
6. joi – User input validation.

Include bootstrap (optional, as long as the form can send post data to the server, it will work).

In the base.html file, add the link and script tags for the bootstrap imports. They are imported once and then included in every template that extends the base template.

At this stage, base.html file should look like this:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <title>{{ title }}</title>
    <meta name="description" content="">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <!-- Bootstrap CSS -->
    <link 
      href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta1/dist/css/bootstrap.min.css" 
      rel="stylesheet" 
      integrity="sha384-giJF6kkoqNQ00vy+HMDP7azOuL0xtbfIcaT9wjKHr8RbDVddVHyTfAAsrekwKmP1" 
      crossorigin="anonymous">
    {% block styles %}
      {# This block will be replaced by child templates when importing styles #}
    {% endblock %}
  </head>
  <body>
    {% block content %}
      {# This block will be replaced by child templates when adding content to the  #}
    {% endblock %}

    <!-- Bootstrap JavaScript Bundle with Popper -->
    <script 
      src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta1/dist/js/bootstrap.bundle.min.js" 
      integrity="sha384-ygbV9kiqUc6oa4msXn9868pTtWMgiQaeYH7/t7LECLbyPA2x65Kgf80OJFdroafW" 
      crossorigin="anonymous">
    </script>
    {% block scripts %}
      {# This block will be replaced by child templates when importing scripts #}
    {% endblock %}
  </body>
</html>

Implementation

Go into the entry point file and require the following packages:

const session = require('express-session')
const MongoStore = require('connect-mongo')(session)
const passport = require('passport')

Right after the app declaration, add built-in express middleware to parse incoming requests with url-encoded data to process the data that will be received from the forms.

var app = express()
app.use(express.urlencoded({extended: true}))

Next, set up the session middleware. Make sure to place this code after the mongoose connection as we will use the existing mongoose connection to store the session data. Otherwise, you’ll have to create a new connection for this.

app.use(session({
  secret: process.env.SESSION\_SECRET,
  resave: false,
  saveUninitialized: true,
  store: new MongoStore({
    mongooseConnection: mongoose.connection,
    collection: 'sessions'
  }),
  cookie: {
    secure: false
  }
}))

Let’s walk through the code above:

1. We’re adding the session middleware to the app.
2. secret – The string used to encrypt the session. Declare this in the .env file or system environment variables.
3. resave – Determines whether the session object is saved back into the session store even if it was not modified by the request.
4. saveUninitialized – Determines whether a new session should be saved into the store even before it’s modified.
5. store – The store is used to save session data.

Update models

In this section, I’m referring to the user model we created in the previous article. Take a look here.

Now we need to update the user model in order to enable authentication and password hashing upon saving. We’re doing this in the model in order to avoid writing the authentication login in multiple places shall we need it.

This logic is unique to this model so it makes sense to have it here. Navigate to the User.js model file we created previously and add the following code right after the first require statement:

const bcrypt = require('bcrypt')

const saltRounds = 10

After the schema definition, add the following code:

userSchema.pre('save', async function(next){
  if (this.isNew) this.password = await bcrypt.hash(this.password, saltRounds)
  next()
})

userSchema.static('userExists', async function({username, email}){
  let user = await this.findOne({ username })
  if (user) return { username: 'This username is already in use' }
  user = await this.findOne({ email })
  if (user) return { email: 'This email address is already in use' }
  return false
})

userSchema.static('authenticate', async function(username, plainTextPassword){
  const user = await this.findOne({ $or: [ {email: username}, {username} ] })
  if (user && await bcrypt.compare(plainTextPassword, user.password)) return user
  return false
})

There are a few things happening here:

1. The first is a pre-save hook. This runs before every document save. We use it to determine if the current document is new (not an update call). If the document is new, hash the password. Always save a hashed password rather than plain text.
2. The second block is a static method that checks if the user exists. We will query the database by username and then email. If a user is found, return an object specifying which one is already in use. Otherwise, return false.
3. The third method is a static method added to the schema. We’re using this to authenticate the user. If the user exists and the password comparison between plainTextPassword and the hashed user password passes, return the user object. Otherwise, return false for. failed authentication.

Registration

Create the registration form; a simple form that collects the user’s name, username, email address and password.

Place this code in ‘register.html’ in the views folder.

{% extends 'base.html' %}

{% set title = 'Register' %}

{% block styles %}
  <style>
    form {
      margin-top: 20px;
      margin-left: 20px;
      margin-right: 20px;
    }
  </style>
{% endblock %}

{% block content %}
  <form action="/register" method="POST">
    <div class="mb-3">
      <label for="name" class="form-label">Name</label>
      <input 
        type="text" 
        class="form-control {% if messages.name_error %}is-invalid{% endif %}" 
        id="name" 
        name="name"
        value="{{ messages.name or '' }}"
        placeholder="Full Name">
      <div class="invalid-feedback">{{ messages.name_error }}</div>
    </div>
    <div class="mb-3">
      <label for="username" class="form-label">Username</label>
      <input 
        type="text" 
        class="form-control {% if messages.username_error %}is-invalid{% endif %}" 
        id="username" 
        name="username"
        value="{{ messages.username or '' }}"
        placeholder="Username">
      <div class="invalid-feedback">{{ messages.username_error }}</div>
    </div>
    <div class="mb-3">
      <label for="email" class="form-label">Email address</label>
      <input 
        type="email" 
        class="form-control {% if messages.email_error %}is-invalid{% endif %}" 
        id="email"
        name="email"
        value="{{ messages.email or '' }}"
        placeholder="Email Address">
      <div class="invalid-feedback">{{ messages.email_error }}</div>
    </div>
    <div class="mb-3">
      <label for="password" class="form-label">Password</label>
      <input 
        type="password" 
        class="form-control {% if messages.password_error %}is-invalid{% endif %}" 
        id="password" 
        name="password" 
        value="{{ messages.password or '' }}"
        placeholder="Password">
      <div class="invalid-feedback">{{ messages.password_error }}</div>
    </div>
    <div>
      <button type="submit" class="btn btn-primary">Sign me up!</button>
    </div>
  </form>
{% endblock %}

{% block scripts %}
{% endblock %}

We’re using nunjucks to implement some dynamic behaviour.

The first is adding the is-invalid class to the form controls using flashed messages from the server. This adds an error message attached to the form control.

The second is setting the previous value entered by the user (an optional UX feature for the purposes of this tutorial).

After creating the register template, create the routes associated with the template.

Create a folder named ‘routes’ in the root of the project. This folder will hold all of our routes. Create a file ‘register.js’ in this folder. The contents of this file should be as follows:

var router = require('express').Router()
const Joi = require('joi')
const { User } = require('../models')

const validateRegistrationInfo = async (req, res, next) => {
  for(let [key, value] of Object.entries(req.body)) {
    req.flash(`${key}`, value)
  }
  /* Validate the request parameters.
  If they are valid, continue with the request.
  Otherwise, flash the error and redirect to registration form. */
  const schema = Joi.object({
    name: Joi.string().required(),
    username: Joi.string().alphanum().min(6).max(12).required(),
    email: Joi.string()
        .email({ minDomainSegments: 2, tlds: { allow: ['com', 'net'] } }).required(),
    password: Joi.string().min(8).required()
  })

  const error = schema.validate(req.body, { abortEarly: false }).error
  if (error) {
    error.details.forEach(currentError => {
      req.flash(\`${currentError.context.label}\_error\`, currentError.message)
    })
    return res.redirect('/register')
  }

  /** Check if user exists */
  const userExists = await User.userExists(req.body)
  if (userExists) {
    for(let [key, message\] of Object.entries(userExists)) {
      req.flash(`${key}`, message)
    }
    return res.redirect('/register')
  }

  next()  
}

router.get('/register', (req, res) => res.render('register.html'))

router.post('/register', validateRegistrationInfo, async (req, res) => {
  let savedUser = await (new User(req.body)).save()
  res.redirect('/')
})

module.exports = router

The first significant block of code is a function called validateRegistrationInfo. This is middleware that will be used to validate the user’s registration information.

In the first phase of the validation, we immediately flash the current information for pre-population in case we redirect back to the registration page.

Phase 2 is validating each entry against a validation schema. The Joi package makes this process easy.

If there are any errors on validation, flash each error message for that particular entry before redirecting to the register page. Display this error message in the template.

The final phase of validation is checking whether the username/email provided are already in use. If they are, flash the error message before redirecting to the register route.

Create a GET route that simply renders ‘register.html’. This is the route we redirect to when validation fails.

Create a post route that receives the data entered by the user in the request body passing the validation middleware to it.

In the route handler itself, we don’t have to worry about invalid data as it would have passed all the validation checks if the handler is being executed.

Create a new user using the provided data, save it, and redirect to the home page.

Export this router object and import it in the entry file as follows:

// Import rotues
app.use('/', require('./routes/register'))

Authentication

Now that we’ve taken care of the registration, it’s time to implement the authentication logic of our application.

Start by creating a login form. This form has a username/email field and a password field. We’ll also include a condition that checks for an error message to display in an alert. This will be displayed when we redirect to the login page after flashing a message.

Place this form in a ‘login.html’ template file in the views folder alongside the register template.

{% extends 'base.html' %}

{% set title = 'Login' %}

{% block styles %}
  <style>
    form {
      margin-top: 20px;
      margin-left: 20px;
      margin-right: 20px;
    }
  </style>
{% endblock %}

{% block content %}
  <form action="/login" method="POST">
    {% if messages.error %}
      <div class="alert alert-danger" role="alert">{{ messages.error }}</div>
    {% endif %}
    <div class="mb-3">
      <label for="name" class="form-label">Username or Email</label>
      <input 
        type="text" 
        class="form-control {% if messages.name_error %}is-invalid{% endif %}" 
        id="username" 
        name="username"
        value="{{ messages.name or '' }}">
      <div class="invalid-feedback">{{ messages.name_error }}</div>
    </div>
    <div class="mb-3">
      <label for="name" class="form-label">Password</label>
      <input 
        type="password" 
        class="form-control {% if messages.name_error %}is-invalid{% endif %}" 
        id="password" 
        name="password"
        value="{{ messages.name or '' }}">
      <div class="invalid-feedback">{{ messages.name_error }}</div>
    </div>
    <div>
      <button type="submit" class="btn btn-primary">Login</button>
    </div>
  </form>
{% endblock %}

{% block scripts %}
{% endblock %}

The next task is to define the passport strategy used to authenticate the user. We’re using the strategy from passport-local because we’re authenticating against our own stored user credentials.

Create a new file in the root of the project called ‘passport-helper.js’ with the following contents:

const LocalStrategy = require('passport-local').Strategy
const { User } = require('./models')

module.exports = (app, passport) => {

  passport.use(new LocalStrategy((username, password, done) => {
    User.authenticate(username, password)
    .then( user => {
      done(null, user)
    })
    .catch( error => {
      done(error)
    })
  }))

  passport.serializeUser((user, done) => {
    done(null, user.\_id)
  })

  passport.deserializeUser((id, done) => {
    User.findById(id, (error, user) => {
      if (error) return done(error)
      done(null, user)
    })
  })

  app.use(passport.initialize())
  app.use(passport.session())
}

The first step is importing the Strategy and the User model.

The second step is configuring the strategy. We create a new instance of the strategy passing it a function that takes username, password and a verify callback (done) function that is executed after the authentication process is complete.

The authentication logic is placed inside this function. In order to keep this clean, we will simply use the ‘authenticate’ static method we created in the user model.

When authenticating in passport, a user object is passed to the verify callback upon successful authentication, otherwise false is returned (provided there’s no error thrown in which case, pass the error).

Our authenticate method returns a user object if the user is found and false otherwise so its output is perfect for this scenario.

Once we’ve configured the strategy, we have to specify the user serialization and deserialization logic.

This step is optional if you’re not using sessions, but we’re trying to create a login system with sessions so in our case, it’s necessary.

The serializeUser method takes a function with a user object and a callback as parameters which determines the data that will be stored in the session itself.

To keep the data stored in the session small, we store only the user ID in the session. This serialization process happens on initial login.

The deserializeUser method takes a function that receives the user ID and a callback. This method runs on all subsequent requests after login/serialization.

The user ID is grabbed from the session and the user is retrieved from the database. Once the user is retrieved, they are stored in req.user.

After serialization/deserialization, make sure to add passport initialize and session middleware to the app. We’ll wrap all of this up in a function that takes our app and passport objects as parameters.

Our passport configuration is now complete. The next step is to initialize passport.

In the application entry file, import the function we created in the previous step and then execute it, passing the app and passport objects.

Make sure to have the require statement after the passport require statement. The initialization function must be called after session middleware is defined because the passport session middleware utilises it.

const initializePassport = require('./passport-helper')
...
initializePassport(app, passport)

Now let’s create the login routes. Inside the routes folder, create a file called ‘login.js’ and add the following code:

const createLoginRoutes = passport => {
  const router = require('express').Router()

  router.get('/login', (req, res) => {
    if (req.isAuthenticated()) return res.redirect('/')
    res.render('login.html')
  })

  router.post(
    '/login',
    passport.authenticate('local', {
      failureRedirect: '/login', 
      successRedirect: '/',
      failureFlash: 'User not found', 
    }),
    (error, req, res, next) => {
      if (error) next(error)
    }
  )

  router.get('/logout', (req, res) => {
    req.logout()
    res.redirect('/login')
  })

  return router
}

module.exports = createLoginRoutes

Instead of creating routes in the same way we did in the register route file, we’re doing it a bit differently here.

Since we’re going to need the passport object, we will instead export a function that accepts a passport object as a parameter, defines the routes and returns the router object.

The first route is a GET route for ‘/login’. This renders the form when there is no active session. Use the ‘isAuthenticated’ method provided by passport in the request object in order to determine whether there is currently an active session.

The second route is a POST route from ‘/login’. This route accepts the form input from the user.

Pass the passport.authenticate middleware to this route to handle the authentication. This middleware accepts the strategy type and an options object.

In the options object, specify the redirect path in case of failure and in case of success. The failureFlash property specifies the message to flash in case of authentication failure. This is the message you should check for and display on the login page.

Finally, create a logout route that calls req.logout to end the current user’s session. This logout method is also provided by passport.

Now import the login route creator in the entry file and pass the passport object to it:

app.use('/', require('./routes/auth')(passport))

Update the home page route to the following:

app.get('/', async (req, res) => {
  if (!req.isAuthenticated()) return res.redirect('/login')
  res.render('home.html')
})

The home page route is now a protected route. This means it should only be accessed by an authenticated user.

We achieve this by using the req.isAuthenticated method to make sure the user is authenticated. If not, redirect to the login page.

Go back to the register route file and update the GET route. to the following:

router.get('/register', (req, res) => {
  if (req.isAuthenticated()) return res.redirect('/')
  res.render('register.html')
})

We only need to render the registration form if the user is not authenticated, otherwise, redirect to the home page.

Conclusion

In this article, I demonstrated how to create a simple registration/authentication system in ExpressJS using PassportJS. However, an authentication system is not complete without a password reset feature.

The next article will be a tutorial on creating a password reset feature using mongoose and NodeMailer.

Mongoose is an ODM (Object Document Mapper) that allows interaction with MongoDB databases using JavaScript objects.

It provides extra functionality (such as static methods on the schema) that allows us to enhance database interactions and write cleaner code.

At the time of writing this article, the latest stable version of Mongoose is v5.11.8. This will most likely be different at the time of reading although most of the information here should still be relevant.

Makes sure you have a MongoDB server installed and running on your system before following along. If not, you can sign up for a free cluster at MongoDB Atlas and connect to that instead.

Mongoose setup

Install Mongoose and dotenv using the following command:

npm install mongoose dotenv

Dotenv allows us to load environment variables into our application. We are going to place the MongoDB URI in an environment variable file instead of hard-coding it.

Doing this allows us to connect to different MongoDB instances in different environments by only changing this URI in the environment variable without changing the code itself.

Create a file called .env in the root of your project. The contents of the files should be as follows:

PORT=8000
MONGO_URI=mongodb://localhost:27017/app

We’ve defined the port here along with the MongoDB URI. make sure to change the values according to your setup.

Now go back to your index.js file (or the file in which your app instance is initialised) and add the following line at the beginning of the file:

if (process.env.ENV === 'dev') require('dotenv').config()

This loads the .env file in our project if we’re in the development environment. We can access each environment variable using process.env.<variable\_name>.

The dotenv package will look for the .env file in our project when the config method is invoked.

Placing this at the top of the entry point file ensures the environment variables are available to the entire application when we decide to go with a modular approach with our route organisation.

Now import mongoose:

const mongoose = require('mongoose')

Create a mongoose connection by inserting the following code before the route definitions:

const connection = mongoose.connect(process.env.MONGO_URI, {
  useNewUrlParser: true,
  useUnifiedTopology: true
})

/* Display message in the console if the connection is successful. */
mongoose.connection.once('open', () => {
  console.log('connected!')
})

Models

Our mongoose connection has been established. The next step is to define our models. Models are object representations of the documents that will reside in our database.

Models in mongoose require a schema. A schema specifies the structure of the document.

If you’re familiar with NoSQL databases, particularly MongoDB, you might be aware that one of the benefits is that the schema is dynamic. Meaning you can add new fields to a document on the fly upon creation/update.

This may be a good idea depending on your use case but mongoose requires schemas in order to define the shape of the documents in the collection. This ensures we have consistency in a collection and a reference point for what properties are contained in each document.

Let’s begin setting up our models by creating a folder in the root of our project named ‘model’. Next, create a file inside this folder called ‘User.js’. It’s a good idea to separate models into their own files.

Inside User.js, add the following code:

const { Schema, model } = require('mongoose')

var userSchema = new Schema({
  name: {
    type: Schema.Types.String,
    required: [true, 'You must provide a name']
  },
  email: {
    type: Schema.Types.String,
    required: [true, 'Email address is required']
  },
  username: {
    type: Schema.Types.String,
    required: [true, 'Username is required']
  },
  password: {
    type: Schema.Types.String,
    required: [true, 'You must provide a password']
  }
})

const User = model('User', userSchema)

module.exports = User

Let’s walk through the contents of this file:

• Import Schema and model from mongoose.
• Create a schema instance that defines the structure of the user document in the User collection.
• Create a model instance and pass it the collection name and schema.
• Export the user model for use in routes.

Now create an index file within the models directory. This file will import all of the models from its sibling files and export them in an object. We’re doing this to reduce the number of require statements in other files when importing models.

You can certainly import models directly from their respective files, but this is definitely a cleaner way of doing it.

The contents of this index.js file should look like this for now:

const User = require('./User')

module.exports = {
  User
}

Using the models

It’s time to test if this works. We’re going to insert a user into the collection if the collection is empty and retrieve the users in the collection otherwise.

In the app entry file, import the User model from the models index file as follows:

// Import models
const { User } = require('./models')

Update the home route to the following:

app.get('/', async (req, res) => {
  const users = await User.find({})
  if (users.length) {
    /* Log users if users exists. */
    console.log(users)
  } else {
    /* If no users exist, save new user and log saved user on the console. */
    let newUser = new User({
      name: 'Kelvin Mwinuka',
      email: 'email@kelvinmwinuka.com',
      username: 'kelvin',
      password: 'password'
    })
    let savedUser = await newUser.save()
    console.log(savedUser)
  }
  res.render('home.html')
})

Navigate to this route in the browser and you should notice that for the first time, a single object is printed to the console:

{
  _id: 5fdab492561efb3e9a2c56c7,
  name: 'Kelvin Mwinuka',
  email: 'email@kelvinmwinuka.com',
  username: 'kelvin',
  password: 'password',
  __v: 0
}

When you refresh the page, the results should now be as follows:

[
  {
    _id: 5fdab492561efb3e9a2c56c7,
    name: 'Kelvin Mwinuka',
    email: 'email@kelvinmwinuka.com',
    username: 'kelvin',
    password: 'password',
    __v: 0
  }
]

Notice this is an array of current documents and no new user is created/saved.

That’s it. We’ve successfully set up mongoose and we’re ready to start persisting data in our MongoDB database!

Conclusion

In this article, we’ve gone over connecting our express application to a MongoDB database, creating mongoose models and using those models to save data into our database.

In the next article, I will be going over user registration and authentication using Passport JS.

You can track the progress of this project on Github.

Though they aren’t entirely necessary, they definitely help create more dynamic pages with cleaner HTML code.

Before getting into JavaScript, Flask and Django were my preferred choices for building web applications. Coming from that background, I got very used to the Jinja2 engine.

When I switched to using Express for web app development, I wanted a solution that would have similar syntax and features to Jinja2. Fortunately, Nunjucks is exactly that. This is a templating engine that is heavily inspired by Jinja2.

In this article, I will walk you through the Nunjucks setup for ExpressJS. This is a fairly simple and straightforward process.

Setup

First, create an empty folder. Navigate into the folder and run the following command to initialise npm:

npm init

Follow all the prompts to setup your project.

Once the setup process has completed, you should have a package.json file in the root of your project.

Installing packages

Next, run the following command to install express and nunjucks.

npm install express nunjucks

Defining the entry point

Now create the file that will be the entry point of our application. I name this file index.js and place it at the root of my project.

The contents of index.js are as follows:

const express = require('express')
const nunjucks = require('nunjucks')

var app = express()

nunjucks.configure('views', {
  autoescape: true,
  express: app
})

app.set('view engine', 'html')

const PORT = '8000'

app.get('/', (req, res) => {
  res.render('home.html')
})

app.listen(PORT, () => {
  console.log(`Listening on port ${PORT}...`)
})

Let’s take a moment to go through what’s happening in this file:

• Require the express and nunjucks packages.
• Create an instance of express and assign it to a variable called ‘app’.
• Configuring nunjucks: The first parameter is the path to the template folder. The second parameter is an object containing configuration options. Autoescape makes sure all output to templates are escaped before display (recommended). You can manually mark the output as safe. The express property requires an instance of an express app.
• Set the port number. We’ll use this when starting the application later.
• Create a ‘get’ route at ‘/’ and return a response that simply renders the template home.html.
• Listen for connections on the specified port.

Creating the templates

At the moment if you try to load the application, you’ll get an error because we haven’t created the template that we’re trying to render yet.

Remember that we set the path to templates as ‘views’. Nunjucks will look for the ‘views’ folder in the same directory as the script that runs the configuration.

If you have your template folder in another directory or have named it something different like ‘templates’, make sure you configure the path correctly.

In this case, we have to create a folder called ‘views’ in the root of our project (because this is where index.js resides).

Inside the views folder, create a base.html template. This is the template that will be inherited by other templates. It allows us to define certain UI elements, styles and scripts only once and have them displayed/imported on every template.

This is where we’d define navigation bars and page footers.

The contents of base.html are as follows:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <title>{{ title }}</title>
    <meta name="description" content="">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="stylesheet" href="">
  </head>
  <body>
    {% block content %}{% endblock %}
    <script src="" async defer></script>
  </body>
</html>

We’ve defined a content block that will be overridden by the child templates. This is where the child templates will place their respective content.

Next, create a home.html template that extends base.html and displays a simple header in the content block.

{% extends 'base.html' %}

{% set title = 'Home' %}

{% block content %}
  <h1>Welcome home!</h1>
{% endblock %}

Now that we’ve created the templates, we’re able to load the home page. Visit localhost at the specified port and you should see that your app is up and running!

Conclusion

In this article, we’ve covered setting up the nunjucks templating engine with an express web application. In the next article, I will write about connecting an express application to MongoDB using mongoose so we can persist data.

You can track the progress of this project on Github.

Keep in mind that this demonstration is by no means exhaustive. However, the core concepts conveyed will be applicable in other use cases.

First, here’s an overview of the python script that will be running periodically on my machine.

I will explain each section of this script in more depth. If you understand this script sufficiently, feel free to skip ahead to the scheduling section of this article.

import os
import json
import yagmail
import datetime
import yfinance as yf
from dotenv import load_dotenv

load_dotenv()

FILE_PATH = os.getenv('FILE_PATH')
DEV_EMAIL = os.getenv('DEV_EMAIL')
DEV_PASSWORD = os.getenv('DEV_PASSWORD')
NOTIFICATION_EMAIL = os.getenv('NOTIFICATION_EMAIL')

yesterday = datetime.date.today() - datetime.timedelta(days=1)

data = yf.download(
    tickers="AAPL MSFT",
    group_by="ticker",
    start=yesterday,
    end=yesterday
)[:1]

records = {}

try:
    with open(FILE_PATH, 'r') as file:
        records = json.load(file)
except (FileNotFoundError, json.JSONDecodeError) as e:
    pass

parsed = {}

for ticker, column in data:
    parsed[ticker] = {"Date": str(yesterday)} if ticker not in parsed else parsed[ticker]
    parsed[ticker][column] = int(data[ticker][column]) if ticker == 'Volume' else float(data[ticker][column])

for ticker in parsed:
    if ticker in records:
        records[ticker].append(parsed[ticker])
    else:
        records[ticker] = [parsed[ticker]]

with open(FILE_PATH, 'w') as file:
    json.dump(records, file, indent=2)

yagmail.SMTP(
    DEV_EMAIL,
    DEV_PASSWORD
).send(
    to=NOTIFICATION\_EMAIL,
    subject='Daily Financial Data',
    contents='Your daily financial record update is complete!'
)

Script breakdown

Let’s talk about the standout packages that are employed in this script:

1. yagmail is a Gmail/SMTP client package that I use to send emails. Not a vital feature but I will explain my reasons for using it later in the article. There are other ways to send emails using python, this is my preferred way of doing it when I simply need to send an email with code that can fit in one line
2. dotenv (python-dotenv) allows us to load environment variables from a .env file in our project. This means we don’t have to fiddle with our system’s environment variables directly.
3. yfinance is the library of choice in accessing Yahoo financial data.

load_dotenv()

 # Path to JSON file used for storage
FILE_PATH = os.getenv('FILE_PATH') 

DEV_EMAIL = os.getenv('DEV_EMAIL')
DEV_PASSWORD = os.getenv('DEV_PASSWORD')

# Email to notify when the task is completed
NOTIFICATION_EMAIL = os.getenv('NOTIFICATION_EMAIL')

Our first action is loading the environment variables from the .env file using the load_dotenv() function. This function looks for a .env file in the current and parent directories.

Once the file is found, the variables are loaded and can be accessed like any system environment variable using the os package.

yesterday = datetime.date.today() - datetime.timedelta(days=1)

data = yf.download(
    tickers="AAPL MSFT",
    group_by="ticker",
    start=yesterday,
    end=yesterday
)[:1]

Now for the exciting part. Here, I’m using yfinance to download Microsoft and Apple data from the previous day. You can read more about this package here.

The object returned by the download method is a pandas DataFrame. I make sure to truncate it to only retrieve the first row of the results. This is only a safety measure, as Only one row is expected anyway.

records = {}

try:
    with open(FILE_PATH, 'r') as file:
        records = json.load(file)
except (FileNotFoundError, json.JSONDecodeError) as e:
    pass

This section of the code loads the records that we already have. The records are currently stored in a JSON file specified by FILE_PATH.

If you’re downloading a lot of data, I would advise you to use a database such as MongoDB or MySQL.

parsed = {}

for ticker, column in data:
    parsed[ticker] = {"Date": str(yesterday)} if ticker not in parsed else parsed[ticker]
    parsed[ticker][column] = int(data[ticker][column]) if ticker == 'Volume' else float(data[ticker][column])

for ticker in parsed:
    if ticker in records:
        records[ticker].append(parsed[ticker])
    else:
        records[ticker] = [parsed[ticker]]

with open(FILE_PATH, 'w') as file:
    json.dump(records, file, indent=2)

In this section, I parse the data manually into a Python dictionary. I chose to do this because I’m more comfortable with basic Python dictionaries than pandas DataFrames.

If you’re more comfortable dealing with DataFrames, it’s possible to export the data to a JSON file directly.

After parsing the data, we append it to the records dictionary and then overwrite the JSON file with the new records data.

yagmail.SMTP(
    DEV_EMAIL,
    DEV_PASSWORD
).send(
    to=NOTIFICATION_EMAIL,
    subject='Daily Financial Data',
    contents='Your daily financial record update is complete!'
)

Finally, we use yagmail to send an email notification at the end of the script to notify us that the script has been executed.

## Scheduling execution

### Using loops

The next challenge is scheduling the execution of this script. One option would be to wrap everything in an endless while loop and sleep at the end of the loop. This would look like this:

```python
import time
....
while True:
    # Execute script here
    time.sleep(60 * 60 * 24)  # Seconds to sleep

The issue with this approach is that we have to make sure the script is always running. This means we have to run it manually and leave it running in the background. This defeats the purpose of automation as the script will not run if we ever forget to execute it.

Another problem with this script is that it’s not optimal. We’re sleeping for 24 hours! Unless you have a very specific scenario where this is necessary, if you ever find yourself sleeping for 24 hours, it’s time to consider cronjobs (or see a doctor!).

Using cronjobs

Cronjobs are the more effective solution for scheduling the execution of this script. There are 2 ways to achieve this:

1. Create a bash script that executes the python script
2. Execute the python script directly (recommended)

Before implementing either of these approaches. We first have to find out the location of the python interpreter on our system. On macOS or Linux, you can find this out using the “which” command followed by the python command.

$ which python3
/Library/Frameworks/Python.framework/Versions/3.7/bin/python3

In this case, I’m specifically looking for the python3 interpreter as I want to execute this script with python 3. On my machine, the “python” command maps to python 2.7.

Execute through a bash script

#!/bin/bash

/Library/Frameworks/Python.framework/Versions/3.7/bin/python3 ${HOME}/<path-to-python script>

The code above is contained in a bash script that will execute the python script. The bash script basically contains the command we would input on the terminal but with some key differences.

On the terminal we would run this script from the project root as follows:

$ python3 script.py

On the bash script above we do something similar except instead of the python3 command, we specify the full path of the python3 interpreter. The second part of the command is the full path of the script we want to run.

Keep in mind that the paths to the interpreter and the script are relative from the system root and NOT the project root or the user’s home.

Make bash script executable with the following command:

chmod +x <path-to-bash-script>

Open crontab in edit mode and add a command to execute the bash script we created. I’ve used the @daily schedule here which executes the given command everyday at midnight.

If you’d like to fine-tune the exact hour of the day that you’d like to execute this command, you can read this article to learn how to achieve that.

crontab -e

@daily .<path-to-bash-script>

Execute python script directly

The second alternative is directly executing the python script from cron. In order to do this, we have to go through the same steps to prepare the python script as we did with the bash script.

We have to make the python script executable first. Navigate to the script’s directory in the terminal and then run the following command:

chmod +x script.py

Once you’ve done that, open the script itself and add a similar line to this at the very top of the file:

_#!/Library/Frameworks/Python.framework/Versions/3.7/bin/python3_

This is called a hashbang or shebang. It is made up of 2 parts: “#!” followed by the location of the interpreter you want to use. This is the result you get when you run the “which python3” command.

A hashbang tells the system how to execute the file. You will notice we have a hashbang at the beginning of the bash script as well. This points to the bash interpreter.

The benefit of using a hashbang is that we don’t have to preface the script name with the interpreter command when executing it in the terminal.

So instead of:

$ python3 script.py

We can do:

$ ./script.py

Once we’ve done that, we can edit the crontab entry to execute the file directly using the following code:

@daily /Library/Frameworks/Python.framework/Versions/3.7/bin/python3 ${HOME}/<path-to-python script>

Before you save this, remember that we have the hashbang at the top of the script, so we don’t have to specify the interpreter in the crontab line. Let’s update this entry to the following:

@daily .${HOME}/<path-to-python script>

Hashbangs help us keep our crontab entries much cleaner.

Now the script will run at the scheduled times and specified intervals.

Rendering

Before we dive into the use of these methods, let’s first establish a basic understanding of how react components are re-rendered.

Components in react will re-render when there is a change in their state and/or their props.

Child components will also re-render whenever their parent component is re-rendered. Even when the child’s state/props haven’t changed.

Memoization

The second concept we need to understand is memoization as it is central to how React.memo and useMemo work.

Memoization is the practice of caching the results/outputs of expensive functions or operations and returning these cached results the next time identical input is provided.

This optimises our program by allowing us to skip costly computations entirely if the provided inputs have already been used before.

React.memo and useMemo make use of this concept to determine whether components should be re-rendered or values should be re-computed respectively.

useMemo

Let’s start with useMemo. This is a react hook that we use within functional components in order to memoize values (especially from expensive functions).

useMemo takes 2 parameters: a function that returns a value to be memoized, and an array of dependencies. Dependencies are the variables that determine whether the memoized value should be recomputed.

In other words, as long as the dependencies haven’t changed, do not re-run the function to update the memoized value. Since the dependencies are contained in an array, you can have multiple dependencies for useMemo.

Do note only ONE of the dependencies in the dependency array needs to change in order to trigger the execution of the function/operation.

Now let’s look at an example of useMemo in action.

First, let’s write some simple application code that does not make use of useMemo.

const User = ({ greeting }) => {
  console.log(greeting)
  return (
    <div>
      <p>{greeting}</p>
    </div>
  )
}

Here we have a User component that simply renders a string contained in the greeting prop. This string is also logged to the console. You will see in just a moment why this is important.

Next, let’s define the App component:

const App = () => {

  const [name, setName] = useState('Michael')

  const greet = () => {
    return `Hello, ${name}`
  }

  const greeting = greet()

  return (
    <div className="App">
      <div>
        <form onSubmit={(event) => {
          event.preventDefault()
          const data = new FormData(event.target)
          setName(data.get('name'))
        }}>
          <input type='text' name='name'/>
          <input type='submit' value='Change name'/>
        </form>
      </div>
      <User greeting={greeting} />
    </div>
  )
}

The app component contains a function called greet that performs the unfathomably slow operation of returning a greeting based on the current name in the state (which is defaulted to ‘Michael’).

We have a greeting constant that is computed by calling the greet function. This is the string that is passed to the User component.

We also have a form that when submitted, updates the name in the App component’s state.

When we run this application, nothing out of the ordinary happens. Submitting the form updates the name, which causes App components to re-render. This causes the greeting to be updated and finally the User component re-renders with the updated prop.

For the sake of this example, let’s imagine that the greet function is a very expensive function that eventually returns our greeting. How can we make use of useMemo to prevent it from being executed on every re-render?

We can memoize the greeting by updating it to the following:

const greeting = useMemo( () => {
    return greet()
  }, [])

Now we only compute the value of greeting when the dependencies update.

But hold on a minute, the dependency array is empty. What happens in this case?

If you’re familiar with the useEffect hook, you’ll know that to mimic the functionality of componentDidMount, we pass an empty dependency array so that it executes once upon the first render.

This is exactly what happens here. This value will be computed once on the first render and will be the same for all subsequent renders. No matter how many times the name changes, the value of greeting will not change.

Now let’s use it a little more practically. We want to re-compute the greeting every time name changes. But because doing this basically renders useMemo useless, let’s add a condition to the name update:

We will only update the name in the state if the submitted name contains the string ‘Kelvin’ in it. So let’s update the form’s onSubmit function to the following:

<form onSubmit={(event) => {
          event.preventDefault()
          const data = new FormData(event.target)

          let name = data.get('name')
          if (name.toLowerCase().includes('kelvin')) setName(name)

          setCount(count + 1)
        }}>
          <input type='text' name='name'/>
          <input type='submit' value='Change name'/>
</form>

Now we’re conditionally updating the name so it makes sense to memoize the greeting depending on the name, since it’s not updating on every submit. I’ve also added a count variable in state that increments every time the form is submitted just to force the App component to re-render regardless of wether name is updated.

Now we can update the useMemo hook to the following:

const greeting = useMemo( () => {
    return greet()
  }, [name])

The only difference here, is that we’ve added the dependency of name to it. Every time name changes, only then will the greeting be recomputed.

When we run this app, we can see that on the User component, the greeting doesn’t change when the input doesn’t contain ‘Kelvin’ in it. On those instances, the memoized greeting is still being used.

Remember that console.log statement we had in our User component? If you look at your console, you’ll notice that the greeting gets printed whether the memoized value is being used, or a new value is calculated.

It appears we are preventing the greeting from being re-computed on certain instances, but the component is always being re-rendered. Why is this?

The answer is simple: Even though the prop doesn’t change in those instances, the component still gets re-rendered simply because the parent has been re-rendered thanks to the count increment.

So what if the rendering of a child component is in itself expensive and we want to make sure we prevent re-rendering when props haven’t changed even if the parent has re-rendered?

This is where React.memo comes in!

React.memo

As mentioned before, React.memo prevents a component from re-rendering unless the props passed to it have changed.

To put this into practice, let’s update the User component to the following:

const User = React.memo(({ greeting }) => {
  console.log('User component rendered')
  return (
    <div>
      <p>{greeting}</p>
    </div>
  )
})

We have wrapped the component with React.memo. We’ve also updated the log statement to let us know when the User component has rendered, just for extra clarity.

Add the following statement in the App components body before the return statement in order to indicate whenever the App component has been re-rendered:

console.log('App component rendered')

Run the application and you will notice that ‘Hello, Michael’ is displayed on the page. When you enter any name besides Kelvin, the name is not updated in state. Count is always updated in state just as before.

The difference this time is that the User component will not be re-rendered as you can see from the console logs.

Why is this? Well, when the name is updated to any value other than ‘Kelvin’ the value of greeting is not updated. The App component still re-renders because the value of count is updated.

This re-rendering of the App component does not affect the child component User as React.memo prevents it from re-rendering due to the fact that the value of the props (in this case, greeting) hasn’t changed.

Change the name to ‘Kelvin’ and you’ll notice that this time, the name is updated in App state, which causes the value of greeting to be updated, which in turn allows the User component to be re-rendered.

Manual render

As we’ve seen, React.memo prevents a component from re-rendering when the props haven’t changed.

React.memo uses shallow comparison to compare the previous set of props to the next incoming set of props in order to determine whether the component should be re-rendered.

If shallow comparison is not sufficient for your needs, as props tend to contain very complex objects in larger applications, you can pass a second optional argument to React.memo: A function that takes previous props and next props as parameters which allows you to manually determine whether the component should be re-rendered.

To implement this, let’s update the User component:

const User = React.memo(({ greeting }) => {
  console.log('User component rendered')
  return (
    <div>
      <p>{greeting}</p>
    </div>
  )
}, (prevProps, nextProps) => {
  if (prevProps === nextProps) return true
  return false
})

Note that this function should return false if you DO want the component to re-render and true if you want to skip the re-render.

The problem with redux

If you haven’t read my article about setting up redux in react, I urge you to read it in order to get some context about what will be discussed in this article.

One of the main complaints against redux is that it requires a lot of boilerplate code to set up some fairly simple functionality. Having to include redux and react-redux increases the project’s bundle size. While the setup increases the complexity of the code.

This is through no fault of the redux developers. Redux is designed to be a general state management tool not exclusive to react. As a result, adapting it to any particular framework will always take a little more setup than something designed specifically for that framework.

Redux also has quite a steep learning curve for some beginners as it introduces paradigms that are hard to grasp. I’m not ashamed to say that it took me at least a couple of weeks of tinkering with redux before I felt comfortable with it.

The complexity of redux is justified for large projects. As the state becomes large and complex enough, the elaborate redux setup eventually pays for itself in such scenarios.

However, there are some projects that are not quite large enough to justify using redux but contain state too complex to manage using the much simpler useState hook. This is where useReducer comes in.

How useReducer solves this problem

useReducer is a react hook that offers the basic functionality of state management that comes with redux, without all the boilerplate code in the setup.

For projects that have a need for a more sophisticated state management system but do not need the extra bells and whistles that come with redux, this is the (almost) perfect alternative.

Because useReducer is designed specifically for react, it is extremely easy to integrate into react components.

There are more issues that are addressed by the useReducer hook. I will discuss these later on in the advantages section of this article.

Using useReducer

Alright, enough talk, time to code! Here’s a demonstration of useReducer in action. For the sake of simplifying this tutorial, I have all the code written inside the App component.

Feel free to break the code down into separate components wherever you see fit. This will work regardless.

We are going to be using a functional component as react does not allow us to use hooks in class components. Make sure to import the useReducer hook:

import React, { useReducer } from 'react';

Now, let’s make use of the hook:

const reducer = (state, action) => {
    switch (action.type) {
      case 'ADD_LANGUAGE':
        return { ...state, languages: [...state.languages, action.payload] }
      case 'ADD_FRAMEWORK':
        return { ...state, frameworks: [...state.frameworks, action.payload] }
      case 'REMOVE_LANGUAGE':
        return { ...state, languages: state.languages.filter( (language, index) => index !== action.payload ) }
      case 'REMOVE_FRAMEWORK':
        return { ...state, frameworks: state.frameworks.filter( (framework, index) => index !== action.payload ) }
      default:
        return state
    }
  }

  const initialState = {
    name: 'Kelvin Mwinuka',
    occupation: 'Software Developer',
    languages: ['JavaScript', 'Python'],
    frameworks: ['React', 'Flask', 'Express']
  }

  const [state, dispatch] = useReducer(reducer, initialState)

If you’ve used redux before, a lot of this looks very familiar. In fact, the useReducer hook is basically redux lite.

First, we set up our reducer. This takes the current state and the dispatched action as parameters. Depending on the action type, we return the current state with the relevant data (payload) added to it.

Next, we set up our initial state. This can be an empty object. I’ve put some data in the initial state here because I’d like something to be displayed on the first render. If you do not need this behaviour, feel free to leave this empty.

Finally, we initialise state and dispatch using the useReducer hook. The 2 primary arguments are the reducer and initial state.

We will access state when displaying information while rendering but use dispatch in order to update the state.

Now let’s render the visual elements that will allow us to interact with our state:

return (
    <div className="App">
      <div>
        <p><b>{state.name} </b>({state.occupation})</p>

        <h3>Languages</h3>
        <ul>
          {state.languages.map((language, index) => {
            return (
              <li key={index}>
                <b>{language}</b>
                <button onClick={() => { dispatch({type: 'REMOVE_LANGUAGE', payload: index})} }>
                  Remove
                </button>
              </li>
            )
          })}
        </ul>
        <form onSubmit={handleSubmit}>
          <input type='text' name='language' />
          <input type='submit' value='Add Language' />
        </form>

        <h3>Frameworks</h3>
        <ul>
          {state.frameworks.map((framework, index) => {
            return (
              <li key={index}>
                <b>{framework}</b>
                <button onClick={() => { dispatch({type: 'REMOVE_FRAMEWORK', payload: index})} }>
                  Remove
                </button>
              </li>
            )
          })}
        </ul>
        <form onSubmit={handleSubmit}>
          <input type='text' name='framework' />
          <input type='submit' value='Add Framework' />
        </form>
      </div>
    </div>
  )

Here we create 2 lists that will display our languages and frameworks respectively. Each list has a corresponding form that allows us to add to it. Additionally, each list entry has a delete button that allows us to remove that particular item from its list.

Let’s start with the delete buttons as they have the simplest logic. Each delete button rendered is aware of its index in the list. When clicked, the button dispatches an action that has a type and payload (just like redux).

The payload is the index of the button/item. So how does the reducer know which list to remove from?

Well, the delete buttons in the languages list dispatch an action with type REMOVE_LANGUAGE. As you can see, the reducer listens for this specific action and then deletes the given index in the payload from the languages list.

The delete buttons in the frameworks list dispatch a similar action except they pass a type of REMOVE_FRAMEWORK. The reducer also listens for this type of action and responds by filtering out the item at the index passed in the payload.

Now let’s handle adding to the lists.

Both forms have the same submit handler. Let’s define this within our app component:

const handleSubmit = (event) => {
    event.preventDefault()
    const formData = new FormData(event.target)

    const language = formData.get('language')  // Returns null if 'language' is not defined
    const framework = formData.get('framework')  // Returns null if 'framework' is not defined

    const action = language ? {type: 'ADD_LANGUAGE', payload: language} : 
                  framework ? {type: 'ADD_FRAMEWORK', payload: framework} : null

    dispatch(action)
    event.target.reset()
  }

Here we capture the form submit event (for both forms). We then create a FormData object from the form. Next, we capture the language and framework value from the FormData.

The language key will return null for the framework form and vice versa.

We then use nested ternary operators to determine what the action object should look like. The payload is the same for both forms, a string.

However, in order for the reducer to know which list to append the string to, we need a type of ADD_LANGUAGE in the case language is not null, and a type of ADD_FRAMEWORK when framework is not null.

Finally we dispatch the action we’ve just created and reset the target form.

Working with child components

So the next question is: how do we work with child components?

In redux, we can pass in the relevant portion of the state down to child components along with actions. We can also directly connect each component to a relevant section of the state using mapStateToProps. Action creators can be mapped to props using mapDispatchToProps.

With useReducer, we need not pass anything other than the relevant portion of state and the dispatch function itself for action dispatching.

Let’s look at an example of this.

First, we will separate the languages and frameworks sections into their own components:

const Languages = ({ languages, handleSubmit, dispatch }) => {
  return (
    <div>
      <h3>Languages</h3>
      <ul>
        {languages.map((language, index) => {
          return (
            <li key={index}>
              <b>{language}</b>
              <button onClick={() => { dispatch({ type: 'REMOVE_LANGUAGE', payload: index }) }}>
                Remove
                </button>
            </li>
          )
        })}
      </ul>
      <form onSubmit={handleSubmit}>
        <input type='text' name='language' />
        <input type='submit' value='Add Language' />
      </form>
    </div>
  )
}

const Frameworks = ({ frameworks, handleSubmit, dispatch }) => {
  return (
    <div>
      <h3>Frameworks</h3>
        <ul>
          {frameworks.map((framework, index) => {
            return (
              <li key={index}>
                <b>{framework}</b>
                <button onClick={() => { dispatch({ type: 'REMOVE_FRAMEWORK', payload: index }) }}>
                  Remove
                </button>
              </li>
            )
          })}
        </ul>
        <form onSubmit={handleSubmit}>
          <input type='text' name='framework' />
          <input type='submit' value='Add Framework' />
        </form>
    </div>
  )
}

Now that we’ve extracted this code into separate components, we can update the App component’s JSX:

return (
    <div className="App">
      <div>
        <p><b>{state.name} </b>({state.occupation})</p>

        <Languages languages={state.languages} handleSubmit={handleSubmit} dispatch />

        <Frameworks frameworks={state.frameworks} handleSubmit={handleSubmit} dispatch/>
      </div>
    </div>
  )

If we want to update the state from our child components, all we need to pass down is the dispatch function. The chid component will be responsible for dispatching the appropriate action in its logic.

This prevents having to pass multiple functions and callbacks, which can quickly become overwhelming.

Advantages of useReducer

Now that we’ve seen how to implement useReducer, let’s discuss why you should use this hook:

1. Simplicity

The first reason is one we’ve already discussed before, it is simple. This hook removes all the boilerplate associated with redux. This is invaluable for projects that aren’t large enough to justify the use of redux.

2. Handle more complex state than useState

If your application’s state has multiple levels, using the useState hook can become very tedious. To combat this and achieve a clean state management solution, the useReducer hook is more suitable for the task.

3. Reduces obnoxious prop-drilling

One of the ways in which we update state from child components is using a technique called prop drilling.

This is a technique whereby a callback function is passed down multiple levels until it reaches the relevant component that uses it.

Technically, we’re still prop-drilling the dispatch function through all our components.

However, the dispatch function is potentially relevant to all the components it passes through as it is component agnostic.

4. Removes external library dependencies

Redux is an external library, and therefore adds to the external dependencies of your react project.

If you are conscious about this due to concerns about bundle size or any other reason, then useReducer is a perfect way to manage some fairly complex state without having to rely on an external package.

This article assumes basic knowledge of create-react-app, workbox and the react framework, as we will be using some terminology associated with these technologies. There’s no need to be a pro, beginner-level knowledge should suffice.

1. Update serviceWorker.js

The first step is updating the serviceWorker.js file within the src folder. We will be updating the register function. At the moment, it looks like this:

export function register(config) {
  if (process.env.NODE_ENV === 'production' && 'serviceWorker' in navigator) {
    // The URL constructor is available in all browsers that support SW.
    const publicUrl = new URL(process.env.PUBLIC_URL, window.location.href);
    if (publicUrl.origin !== window.location.origin) {
      // Our service worker won't work if PUBLIC_URL is on a different origin
      // from what our page is served on. This might happen if a CDN is used to
      // serve assets; see https://github.com/facebook/create-react-app/issues/2374
      return;
    }

    window.addEventListener('load', () => {
      const swUrl = `${process.env.PUBLIC\_URL}/service-worker.js`;

      if (isLocalhost) {
        // This is running on localhost. Let's check if a service worker still exists or not.
        checkValidServiceWorker(swUrl, config);

        // Add some additional logging to localhost, pointing developers to the
        // service worker/PWA documentation.
        navigator.serviceWorker.ready.then(() => {
          console.log(
            'This web app is being served cache-first by a service ' +
              'worker. To learn more, visit https://bit.ly/CRA-PWA'
          );
        });
      } else {
        // Is not localhost. Just register service worker
        registerValidSW(swUrl, config);
      }
    });
  }
}

We’re going to update the following line:

const swUrl = `${process.env.PUBLIC_URL}/service-worker.js`;

To:

 const swUrl = `${process.env.PUBLIC_URL}/custom-service-worker.js`;

This will allow us to create and register our own custom service worker file named custom-service-worker.js. You can name this whatever you like.

If you’d like to be able to test this in development, disable the production environment check in the following line:

if (process.env.NODE\_ENV === 'production' && 'serviceWorker' in navigator)

2. Create a custom service worker file

The next step is creating the custom service worker file in the public folder of our application. This should match the name we used in step 1. In our case, ‘custom-service-worker.js’

We can import and use workbox from here:

importScripts('https://storage.googleapis.com/workbox-cdn/releases/5.1.2/workbox-sw.js')

if (workbox) {
  console.log(`Yay! Workbox is loaded ![🎉](https://cdn.hashnode.com/res/hashnode/image/upload/v1649114039486/kFdciRh2-.png)`);
} else {
  console.log(`Boo! Workbox didn't load ![😬](https://cdn.hashnode.com/res/hashnode/image/upload/v1649114040812/CRViTXRw3.png)`);
}

Once Workbox is loaded from the CDN, a workbox object is created allowing us to make use of the workbox API.

We can self-host workbox by downloading the files into a folder in our public folder in one of 2 ways:

1. Using workbox cli’s copyLibraries command
2. Downloading the files from a Github release

Note that workbox will cache all modules referenced when loaded through the CDN. These modules will then be available for offline use after the first time they are referenced.

Therefore, you do not need to host workbox yourself if this is your concern.

In order to load workbox if we’re self hosting, we will need to do the following:

importScripts('/third_party/workbox/workbox-sw.js');

workbox.setConfig({
  modulePathPrefix: '/third_party/workbox/'
});

In this instance, the workbox folder is contained within the third_party folder inside our application’s public folder.

Now we can finally use workbox. We can use destructuring in order to access the different workbox modules:

const { backgroundSync, routing, strategies } = workbox

3. Register service worker

The final step is registering the service worker in our application. Navigate to the index.js file in the src folder and change the following line:

serviceWorker.unregister();

To:

serviceWorker.register();

1. Creating the application

In this example, we will create a simple application that displays a user’s profile containing a name, bio and 2 lists: one for programming languages and one for frameworks.

Let’s create the app using create-react-app: npx create-react-app <app_name>

Let’s add an image in the public folder to be used as the profile picture(optional) and create a component file in /src named TechList.js.

Our folder structure should look like this:

.
├── README.md
├── package-lock.json
├── package.json
├── public
│   ├── favicon.ico
│   ├── index.html
│   ├── logo192.png
│   ├── logo512.png
│   ├── manifest.json
│   ├── pro.jpg
│   └── robots.txt
└── src
    ├── App.css
    ├── App.js
    ├── App.test.js
    ├── TechList.js
    ├── index.js
    ├── logo.svg
    ├── serviceWorker.js
    └── setupTests.js

Let’s define the App component:

import React from 'react';
import TechList from './TechList'
import './App.css';

const App = () => {
  return (
    <div className="App">
      <div className="media">
        <img className="align-self-start mr-3 profile-pic" src="pro.jpg" alt="Profile" />
        <div className="media-body">
          <h5 className="mt-0">{/** Bio will go here */}</h5>
          <p>{/** Bio will go here */}</p>
          <div className="container tech-container">
            <div className="row">
              <div className="col-sm">
                {/** Programming lanugages list */}
                <TechList 
                  items={[]}
                />
              </div>
              <div className="col-sm">
                {/** Programming lanugages list */}
                <TechList 
                  items={[]}
                />
              </div>
            </div>
          </div>
        </div>
      </div>
    </div>
  );
}
export default App;

Now let’s define the TechList component, a reusable component that will display both the languages and frameworks lists:

import React from 'react'

const TechList = ({
    items,
}) => {

    const handleFormubmit = (event) => {
        event.preventDefault()
        event.target.reset()
    }
    return (
        <ul className="list-group">
            {
                items.map( (item, index) => {
                    return <li key={index} className="list-group-item">{item}</li>
                })
            }
            <li className="list-group-item">
                <form onSubmit={handleFormubmit}>
                    <div className="form-row">
                        <div className="col">
                            <input type="text" className="form-control add-tech-text" placeholder="Type new" name="entry" required/>
                        </div>
                        <div className="col">
                            <button type="submit" className="btn btn-primary">Add to list</button>
                        </div>
                    </div>
                </form>
            </li>
        </ul>
    )
}

export default TechList

This component receives an items prop which is an array containing the languages/frameworks that will be displayed in the list. At the moment, we’re passing an empty array from the App component so this will not display anything.

It also contains a form appended at the end of the list that allows us to enter some text to be appended to the list dynamically. We will add functionality to this later.

So far, the app should look something like this:

No information is displayed but this will change once we set up and connect redux.

Next, let’s set up a redux folder inside /src that will contain our action creators and reducer. Inside the folder, we’ll have actions.js and reducer.js. The folder structure should now look like this:

.
├── README.md
├── package-lock.json
├── package.json
├── public
│   ├── favicon.ico
│   ├── index.html
│   ├── logo192.png
│   ├── logo512.png
│   ├── manifest.json
│   ├── pro.jpg
│   └── robots.txt
└── src
    ├── App.css
    ├── App.js
    ├── App.test.js
    ├── TechList.js
    ├── index.js
    ├── logo.svg
    ├── redux
    │   ├── actions.js
    │   └── reducer.js
    ├── serviceWorker.js
    └── setupTests.js

2. Installing the packages

We will need to install the necessary packages with the following command: npm install redux react-redux redux-thunk axios

3. Action creators

Our action creators will be located inside the actions.js file. We will have 2 action creators for now: one that creates an action that sends data to add a programming language to the store, and one that sends data to add a framework.

Our code in actions.js will look like this:

export const addLanguange = (language) => {
    return {
        type: 'ADD_LANGUAGE',
        payload: language
    }
}

export const addFramework = (framework) => {
    return {
        type: 'ADD_FRAMEWORK',
        payload: framework
    }
}

4. Reducer

Our reducer.js file will contain our reducer:

const initial_state = {
    profile: {
        name: 'Kelvin Clement Mwinuka',
        bio: 'I am a software developer with a BS in Computer Science from The University of Nottingham. I’m passionate about web technologies. In my free time, I like blogging and challenging myself physically.',
        languages: [
            'JavaScript', 'Python', 'HTML', 'CSS'
        ],
        frameworks: [
            'React', 'Express', 'Flask', 'Django'
        ]
    },
}

const rootReducer = (state = initial_state, action) => {
    switch (action.type) {
        case 'ADD_LANGUAGE':
            return {
                ...state, 
                profile: {
                    ...state.profile,
                    languages: [...state.profile.languages, action.payload]
                }
            }
        case 'ADD_FRAMEWORK':
            return {
                ...state, 
                profile: {
                    ...state.profile,
                    frameworks: [...state.profile.frameworks, action.payload]
                }
            }
        default:
            return state
    }
}

export default rootReducer

In this example, I’ve set up an initial state with some pre-loaded values. When an action is dispatched, the reducer will figure out which part of the state to append data.

Keep the reducer pure by not having any other logic besides returning the new state. We also should not directly mutate the state.

5. Connecting the app

Now that we have our action creators and reducer, it’s time to connect our application to redux so we can actually use them.

Let’s open the index.js file and make the following changes:

import React from 'react';
import ReactDOM from 'react-dom';
import App from './App';
import \* as serviceWorker from './serviceWorker';
import { createStore, applyMiddleware } from 'redux'
import { Provider } from 'react-redux'
import thunk from 'redux-thunk'
import rootReducer from './redux/reducer'

const store = createStore(
  rootReducer,
  applyMiddleware(thunk)
)

ReactDOM.render(
  <React.StrictMode>
    <Provider store={store}>
      <App />
    </Provider>
  </React.StrictMode>,
  document.getElementById('root')
);

// If you want your app to work offline and load faster, you can change
// unregister() to register() below. Note this comes with some pitfalls.
// Learn more about service workers: https://bit.ly/CRA-PWA
serviceWorker.unregister();

First, we import createStore and applyMiddleware. createStore is exactly what it sounds like: it allows us to create the store that will hold our data. applyMiddleware allows us to extend the functionality of redux by adding packages called middleware.

Next, we import the Provider component from react-redux that will wrap our App component.

Our third import is a middleware package called redux-thunk, I will get into more detail about this in section 7 (Enhancement with redux-thunk).

The final import is our reducer. We only have one to import here. However, if you have multiple reducers, you could merge them into one giant reducer using combineReducer from the redux package.

Now we can create our store using createStore and pass in our reducer, and then apply the middleware.

If you wish to stop here or if this simple setup is sufficient, you do not have to use applyMiddleware at all. You could just pass the reducer and call it a day. I’ve added the middleware here in order to set up for the redux-thunk section.

Now lets go into our App component in App.js and make the following changes:

import React from 'react';
import TechList from './TechList'
import { bindActionCreators } from 'redux'
import { addLanguange, addFramework } from './redux/actions'
import { connect } from 'react-redux'
import './App.css';

const App = ({
  profile,
  action
}) => {
  return (
    <div className="App">
      <div className="media">
        <img className="align-self-start mr-3 profile-pic" src="pro.jpg" alt="Profile" />
        <div className="media-body">
          <h5 className="mt-0">{profile.name}</h5>
          <p>{profile.bio}</p>
          <div className="container tech-container">
            <div className="row">
              <div className="col-sm">
                {/** Programming lanugages list */}
                <TechList 
                  items={profile.languages}
                  action={actions.addLanguange}
                />
              </div>
              <div className="col-sm">
                {/** Programming lanugages list */}
                <TechList 
                 items={profile.languages}
                  action={actions.addFrameworks}
                />
              </div>
            </div>
          </div>
        </div>
      </div>
    </div>
  );
}

function mapStateToProps (state) {
  return {
    profile: state.profile
  }
}

function mapDispatchToProps (dispatch) {
  return {
    actions: bindActionCreators({ 
      addLanguange,
      addFramework 
    }, dispatch)
  }
}

export default connect(mapStateToProps, mapDispatchToProps)(App);

First, we import bindActionCreators from redux which allows us to combine all our action creators into one object with corresponding keys.

This is not necessary but I find this to be a clean way of dealing with action creators especially as the project grows and necessitates the usage of more of action creators.

Next we import our action creators themselves from actions.js.

Finally, we import connect from react-redux. This allows us to connect a particular component to our store. We will have this only on the App component and pass down any action creators or data as props.

If you have a large project, you could use this method on multiple components especially if you want to make sure you’re only subscribing to certain parts of the state rather than the entire store.

We’ve added a mapStateToProps function:

function mapStateToProps (state) {
  return {
    profile: state.profile
  }
}

This takes the state contained in our redux store as a parameter and returns an object that can be considered a subset of the state. The object in question will be passed to the current component via its props.

Right now, we are subscribing to the ‘profile’ object in the state. Meaning the component will only re-render if this section of the state changes.

This is one of the strengths of redux. The component does not have to re-render if the portion of the state it subscribes to has not changed. Even if the state has changed elsewhere.

If we end up expanding our state in reducer.js and add another section other than ‘profile’, the App component and subsequently, its children, will not re-render if the new portion of the state changes.

We’ve also added another function:

function mapDispatchToProps (dispatch) {
  return {
    actions: bindActionCreators({ 
      addLanguange,
      addFramework 
    }, dispatch)
  }
}

This function enables us to fire our action creators within the app component and its children provided they are passed down.

We make the following update on both instances of the TechList component:

              ...
               {/** Programming lanugages list */}
               <TechList 
                  items={profile.languages}
                  action={actions.addLanguange}
                />
              ...
                {/** Programming lanugages list */}
                <TechList 
                 items={profile.languages}
                  action={actions.addFrameworks}
                />

We pass the relevant items list and action creator to each of the instances.

6. Dispatching actions

Now that we’ve connected the application to the redux store, let’s dispatch the actions.

The actions in question add a programming language and a framework to the state’s languages and framework lists respectively. In order to make this possible, we’ll update the TechList component to the following:

import React from 'react'

const TechList = ({
    items,
    action
}) => {

    const handleFormubmit = (event) => {
        event.preventDefault()
        action(event.target.entry.value)
        event.target.reset()
    }
    return (
        <ul className="list-group">
            {
                items.map( (item, index) => {
                    return <li key={index} className="list-group-item">{item}</li>
                })
            }
            <li className="list-group-item">
                <form onSubmit={handleFormubmit}>
                    <div className="form-row">
                        <div className="col">
                            <input type="text" className="form-control add-tech-text" placeholder="Type new" name="entry" required/>
                        </div>
                        <div className="col">
                            <button type="submit" className="btn btn-primary">Add to list</button>
                        </div>
                    </div>
                </form>
            </li>
        </ul>
    )
}

export default TechList

This component takes an items props which it loops through and displays in a list as described before. The second prop is an actions prop. This will contain an action creator that will be invoked and passed the data grabbed from the form submission.

This component is action creator agnostic, even though it is the one invoking the action creator. So it’s important to pass the correct action creator down from the parent component.

Congratulations! you’ve connected your app to redux. Now you can add new items to each of the lists.

Next, we’ll take a look at how to enhance this app. At the moment, action creators can only return an action object. This is great if we already have the data we want to return.

What about a situation where we need to retrieve data from a server through an API call? We can’t do this in the reducer as it needs to be pure. The action creator is the place to do this. We need a way to add this logic here. This is where redux-thunk comes in.

7. Enhancement with redux-thunk

To understand redux-thunk, we first need to understand what a thunk is. A thunk is a function that delays the execution of some code until the exact moment the result of that execution is needed. In our case, that code is dispatching an action.

Why is this important? At the moment, we have to dispatch an action that consists of the type and the payload. This requires that we already have the payload data beforehand.

What if we don’t have that data? What if we need to retrieve that data from a server before we display it? This is what a thunk is useful for. In this case, instead of dispatching an action directly, we want to make a request to the server and then dispatch an action with the data from the response.

Our action creators need to return a function that has this logic and then returns an action at the end of its execution. This is the thunk.

In order to enable thunks in redux, we need to apply the redux-thunk middleware, which we have already done.

First, let’s write. a simple Node server that listens on port 8000 for requests. This server has a ‘/profile’ GET endpoint that returns the user’s profile details, a ‘/languages’ POST endpoint that adds to the user’s list of languages, and a ‘/frameworks’ POST endpoint that adds to the user’s list of frameworks.

Each endpoint returns the latest user object as a JSON response.

var bodyParser = require('body-parser')
var cors = require('cors')
var app = require('express')()

const port = 8000

var profile = {
    name: 'Kelvin Mwinuka',
    bio: 'I am a software developer with a BS in Computer Science from The University of Nottingham. I’m passionate about web technologies. In my free time, I like blogging and challenging myself physically.',
    languages: [],
    frameworks: []
}

app.use(cors())
app.use(bodyParser.json())

app.post('/languages', (req, res) => {
    let language = req.body.language
    if (!profile.languages.map( l => l.toLowerCase()).includes(language.toLowerCase())) {
        profile.languages.push(language)
    }
    res.json(profile)
});

app.post('/frameworks', (req, res) => {
    let framework = req.body.framework
    if (!profile.frameworks.map( f => f.toLowerCase()).includes(framework.toLowerCase())) {
        profile.frameworks.push(framework)
    }
    res.json(profile)
});

app.get('/profile', (req, res) => {
    res.json(profile)
});

http.listen(port, () => {
    console.log(`Server started at port ${port}`)
});

Let’s make the necessary changes in actions.js to enable the desired behaviour:

import axios from 'axios'

export const setProfileData = (profile) => {
    return {
        type: 'SET_PROFILE_DATA',
        payload: profile
    }
}

export const loadProfile = () => {
    return async (dispatch) => {
        let res = await axios.get('http://localhost:8000/profile')
        let profile = res.data
        dispatch(setProfileData(profile))
    }
}

export const addLanguange = (language) => {
    return async (dispatch) => {
        let res = await axios.post('http://localhost:8000/languages', { 
            language: language 
        })
        let profile = res.data
        dispatch(setProfileData(profile))
    }
}

export const addFramework = (framework) => {
    return async (dispatch) => {
        let res = await axios.post('http://localhost:8000/frameworks', { 
            framework: framework 
        })
        let profile = res.data
        dispatch(setProfileData(profile))
    }
}

The first change we’ve made is the addition of a ‘setProfileData’ action creator that behaves like a regular action creator (no thunk) to set the profile data if we already have it.

Notice what we’ve done with the addLanguage and addFramework action creators? Instead of returning a raw action object, we instead return an async function that takes dispatch as a parameter.

This function executes whatever logic is needed first, and only then will it dispatch an action. This is what a thunk is. A thunk can also be used for conditional dispatches but that is outside the scope of this article.

We’ve also added another action creator called loadProfile that is explicitly responsible for retrieving the user profile from the server. It behaves similar to the ‘addLanguage’ and addFramework action creators.

Another important thing to note is that these 3 action creators now pass the ‘setProfileData’ action creator to the dispatch function. We can do this because that action creator returns a raw action. Therefore, it’s equivalent to passing the action object directly to dispatch. I take this approach in order to avoid typing the same action object multiple times.

In the reducer, let’s add one more case for setting the user profile. The data is no longer hardcoded in the initial state and will instead be set by dispatching an action after retrieving it from the server.

const initial_state = {
    profile: {
        name: '',
        bio: '',
        languages: [],
        frameworks: []
    },
}

const rootReducer = (state = initial_state, action) => {
    switch (action.type) {

        case 'SET_PROFILE_DATA':
            return {...state, profile: action.payload}

        case 'ADD_LANGUAGE':
            return {
                ...state, 
                profile: {
                    ...state.profile,
                    languages: [...state.profile.languages, action.payload]
                }
            }
        case 'ADD_FRAMEWORK':
            return {
                ...state, 
                profile: {
                    ...state.profile,
                    frameworks: [...state.profile.frameworks, action.payload]
                }
            }
        default:
            return state
    }
}

export default rootReducer

In the app section, let’s import our new loadProfile action creator and then invoke it right at the top of our app component in order to trigger the user profile retrieval from the server.

import React from 'react';
import TechList from './TechList'
import { bindActionCreators } from 'redux'
import { addLanguange, addFramework, loadProfile } from './redux/actions'
import { connect } from 'react-redux'
import './App.css';

const App = ({
  profile,
  actions
}) => {

  actions.loadProfile()

  return (
    <div className="App">
      <div className="media">
        <img className="align-self-start mr-3 profile-pic" src="pro.jpg" alt="Profile" />
        <div className="media-body">
          <h5 className="mt-0">{profile.name}</h5>
          <p>{profile.bio}</p>
          <div className="container tech-container">
            <div className="row">
              <div className="col-sm">
                {/** Programming lanugages list */}
                <TechList 
                  items={profile.languages}
                  action={actions.addLanguange}
                />
              </div>
              <div className="col-sm">
                {/** Programming lanugages list */}
                <TechList 
                  items={profile.frameworks}
                  action={actions.addFramework}
                />
              </div>
            </div>
          </div>
        </div>
      </div>
    </div>
  );
}

function mapStateToProps (state) {
  return {
    profile: state.profile
  }
}

function mapDispatchToProps (dispatch) {
  return {
    actions: bindActionCreators({ 
      loadProfile,
      addLanguange,
      addFramework
    }, dispatch)
  }
}

export default connect(mapStateToProps, mapDispatchToProps)(App);

That’s it! run the app and you’ll notice that we’ve retained all the functionality we had before from the user’s perspective, but we can now create smarter action creators that allow us to accomplish more with redux.

So you’ve created your react application and now you have to write end-to-end tests in order to make sure your application works as expected from the end user’s perspective. In this article, I will provide a simple step-by-step guide on how to achieve this using the Jest testing framework and the Puppeteer library.

Before we continue, it’s important to note that a react application created using create-react-app will come pre-packaged with Testing Library, which more or less allows us to achieve the same thing this article addresses. Here is a brilliant article that explains how to run tests using Jest and Testing Library.

So why puppeteer? Well, there are many reasons you might choose to go with puppeteer instead. Maybe there is some functionality that is unique to puppeteer that satisfies a very specific use case. For me, it’s a matter of personal preference, I prefer the way Puppeteer handles interaction with the DOM.

1. Creating the app

First, let’s create an application that will be our test subject. I’ve created a simple single-page application that contains a form and a table. The form will allow us to input some data that immediately gets displayed in the table upon submission. Here’s how it looks:

This application is created using create-react-app. The application folder will be structured as shown below.

├── e2e
│   ├── custom-environment.js
│   ├── jest.config.js
│   └── tests
│       └── App.test.js
├── package-lock.json
├── package.json
├── public
│   ├── favicon.ico
│   ├── index.html
│   ├── logo192.png
│   ├── logo512.png
│   ├── manifest.json
│   └── robots.txt
└── src
    ├── App.css
    ├── App.js
    ├── DevsTable.js
    ├── Form.js
    ├── index.js
    ├── serviceWorker.js
    └── setupTests.js

I’ve placed the e2e folder in the project root. This is where our puppeteer tests and Jest configurations will live. Other additional files are the DevsTable, and Form components.

In the form component, we accept the user input and pass it to the App component to update its state.

import React from 'react'

const Form = ({ add }) => {

    const handleSubmit = (event) => {
        event.preventDefault();
        let dev = new FormData(event.target)
        add({name: dev.get('name'), role: dev.get('role')})
        event.target.reset()
    }

    return (
        <form onSubmit={handleSubmit} id="devForm">
            <span>Name: </span>
            <input type="text" name="name" required/>
            <span>Role: </span>
            <input type="text" name="role"/><br/>
            <input class="submit" type="submit" value="Add Developer" required/>
        </form>
    )
}

export default Form

The DevsTable component displays a table and maps each object in the devs prop to a row in the table.

import React from 'react';

const DevsTable = (props) => {
    return (
        <table>
            <thead>
                <tr>
                    <th>Name</th>
                    <th>Role</th>
                </tr>
            </thead>
            <tbody>
                {
                    props.devs.map((dev, index) => {
                        return (
                            <tr key={index}>
                                <td id={`name${index}`}>{dev.name}</td>
                                <td id={`role${index}`}>{dev.role}</td>
                            </tr>
                        )
                    })
                }
            </tbody>
        </table>
    )
}

export default DevsTable

Here’s the App component’s code:

import React, { useState, useCallback } from 'react';
import './App.css';
import Form from './Form';
import DevsTable from './DevsTable'

const App = () => {

  const [devs, setDevs] = useState([]); 

  const addDeveloper = useCallback((dev) => {
    setDevs(devs => [...devs, dev])
  }, [setDevs])

  return (
    <div className="App">
      <DevsTable devs={devs} />
      <Form add={addDeveloper} />
    </div>
  );
}

export default App;

Finally, the styling in App.css:

.App {
  text-align: left;
  font-family: Arial, Helvetica, sans-serif;
}

table {
  width: 50%;
  margin: 50px;
  border: 1px solid black;
  border-collapse: collapse;
}

th, td {
  padding: 10px;
  border: 1px solid black;
}

th {
  color: white;
  background-color: teal;
}

form {
  margin: 50px;
}

input {
  margin-left: 5px;
  margin-right: 50px;
  margin-bottom: 20px;
}

.submit {
  padding: 10px;
  color: white;
  background-color: teal;
  border-width: 0px;
  border-radius: 5px;
  margin-left: 0px;
}

I usually delete the index.css file and its import statement in index.js. I prefer to have the top-level CSS in App.css.

2. Installing the packages

Before we can start writing the tests, we need to make sure we have the right packages. The first will be the Jest testing framework. When using create-react-app, Jest comes pre-packaged in order to run the default test script. However, we will not be able to use this in the terminal when running our own custom test scripts.

To solve this, we need to add Jest to our devDependencies. If you don’t already have devDependencies in your package.json, add it at the top level. Next, navigate to your node_modules folder and look for the Jest module. Open that module’s package.json and check the version of Jest that’s been pre-packaged by create-react-app.

We want to use the same version of jest in our devDependencies to avoid clashes. Inside devDependencies, add Jest and set the version to the same version installed with create-react-app.

Your devDependencies section should look something like this:

"devDependencies": {
    "jest": "24.9.0"
  }

Now run npm install.

We are going to need 2 more packages for our testing: jest-environment-node and Puppeteer.

jest-environment-node allows us to create a custom environment in which to run our tests(I will explain this in more detail later in the article). Puppeteer is the testing library that provides a high-level API for controlling chromium. At the time of writing, puppeteer only offers chromium support.

We only need these in our dev environment so we will install them with the following command:

npm install jest-environment-node puppeteer --save-dev

Once the installation is done, our devDependencies section should look like this:

"devDependencies": {
    "jest": "24.9.0",
    "jest-environment-node": "^26.0.1",
    "puppeteer": "^4.0.0"
  }

3. Creating the custom test environment

Jest runs tests in environments. The default environment used by Jest is a browser-like environment through jsdom. The environment is created for each test suite. We want to create a custom environment so we can control the test suites’ setup and teardown.

Why is this important? We could just create the browser and page objects in beforeAll and then close them in afterAll, right?

While this would work, it’s not the most efficient solution if we have multiple test suites that use puppeteer. This would lead to typing the same setup and teardown logic multiple times.

Instead, we will create one custom environment, set up and tear down the tests here and then use that custom environment for all our tests.

Here’s how we set up the custom environment:

var NodeEnvironemnt = require('jest-environment-node')
var puppeteer = require('puppeteer')

class CustomEnvironment extends NodeEnvironemnt {
    constructor(config, context){
        super(config, context)
    }

    async setup(){
        await super.setup()
        this.global.browser = await puppeteer.launch({
            headless: false,
            slowMo: 100
        })
        this.global.page = await this.global.browser.newPage()
        await this.global.page.goto('http://localhost:3000/', {waitUntil: 'load'})
    }

    async teardown(){
        await this.global.browser.close()
        await super.teardown()
    }
}

module.exports = CustomEnvironment

In the custom environment, we have access to this.global. This is where you put data that needs to be accessible in the tests.

We set up our browser and page objects in the setup method. This method runs before the test suite is executed. The teardown method runs after all the tests in a suite are complete so its where we close the browser.

4. Configuring Jest

Next, we have to configure Jest. We can do this directly in the project’s package.json with a “jest” object, but I prefer using a config file. Let’s navigate to the jest.config.js inside the e2e folder and add the following code:

module.exports = {
    testEnvironment: './custom-environment.js',
    testTimeout: 60000
}

The testEnvironment option allows us to set a custom environment to be used instead of the default environment. Let’s point it to the custom environment we defined earlier.

The testTimeout option allows us to set the amount of time a test has to run before Jest times out and aborts it. The default is 5 seconds. This is quite short for end-to-end tests as launching the browser and loading pages alone can take a few seconds. I’ve set it to 1 minute here but feel free to adjust this according to your needs.

However, it’s important to set a timeout that’s not too long either. One of the criteria to test for is performance. It’s not beneficial to simply adjust your tests to tolerate extremely slow loading times as that’s a sign that your application can be optimised.

Next, we have to create a command to run our tests. In the scripts section inside the package.json, let’s add the following line:

"test:e2e": "jest -c e2e/jest.config.js"

This sets the Jest terminal command that will be run. We also set the configuration file here. Now, all we have to do to run the tests is use the npm run test:e2e command in the terminal.

5. Writing the tests

Finally, we can write the tests! Let’s navigate to e2e/tests/App.test.js and define the test suite. Inside the test suite, we’re exposed to the global object which contains the browser and page objects we created in the custom environment.

Here, I’ve written a simple test that fills the form 3 times with 3 sets of data, and then iterates through the table rows to check if the data is being displayed as expected.

describe('App tests',() => {

  let page = global.page

  test('Loads all elements', async () => {

    const testData = [
      { name: 'Kelvin Mwinuka', role: 'Front-end developer' },
      { name: 'James Mitchel', role: 'Back-end developer' },
      { name: 'Michael Scott', role: 'DevOps' }
    ]

    await page.waitForFunction('document.getElementById("devForm")')

    // Input all the data
    for(let i = 0; i < testData.length; i++){
      await page.type("input[name='name']", testData[i].name)
      await page.type("input[name='role']", testData[i],role)
      await page.click("input[type='submit']")
    }

    // Check if all the data is represented in the table
    for(let i = 0; i < testData.length; i++){
      expect(await page.$eval(`#name${i}`, element => element.innerText))
      .toEqual(testData[i].name)
      expect(await page.$eval(`#role${i}`, element => element.innerText))
      .toEqual(testData[i].role)
    }
  });
})

6. Considerations

There are several improvements we can make for a better testing experience.

1. Dev server

At the moment, we need the application to already be running before we can run our tests. We can improve this by employing the help of jest-dev-server. First let’s install it in our devDependencies npm install jest-dev-server --save-dev

Now let’s create a global-setup.js file in our e2e folder with the following contents:

const { setup: setupDevServer } = require("jest-dev-server")

module.exports = async () => {
    await setupDevServer({
        command: 'npm run start --port 3000',
        launchTimeout: 30000,
        debug: true,
        port: 3000
    })
}

This file starts the server upon setting up our test. Next, let’s create a global-teardown.js file in the same directory with the following content:

const { teardown: teardownDevServer } = require("jest-dev-server")

module.exports = async () => {
    await teardownDevServer()
}

This will shutdown the dev server once the tests have finished executing. In order for jest to run these setup and teardown functions, we must update the jest.config.js file to this:

module.exports = {
    testEnvironment: './custom-environment.js',
    testTimeout: 60000,
    globalSetup: './global-setup.js',
    globalTeardown: './global-teardown.js'
}

It’s important to note that the setup and teardown methods in custom environment run once before and after EACH test suite respectively. The globalSetup and globalTeardown function run before and after ALL tests suites respectively.

Also, note that the global variables set in globalSetup can only be accessed in globalTeardown and cannot be accessed inside the test suites unlike the global variables set in the custom environment.

2. Tests with login

If you have multiple test suites which create user sessions, you’ll want to make sure they are queued to run consecutively. By default, Jest will run test suites concurrently.

The issue arises when one suite logs in and essentially kicks another suite out of its session. To prevent this, limit the max workers to 1 in the jest command by updating the script in package.json to the following:

"test:e2e": "jest -c e2e/jest.config.js --maxWorkers=1"

This will make sure only one test suite is run at a time. Do note that this will increase the total execution time.

3. Mocking requests

If you want to avoid the above problem entirely, it’s advisable to mock requests in your tests. This has multiple advantages including but not limited to:

1. Reducing test execution time as no real network calls are made
2. Having full control over responses when testing for various response scenarios
3. Not being at the mercy of the back-end server when running tests

Here’s a library that provides puppeteer request mocking functionality along with concise documentation.

Python allows us to define our own custom higher-order functions. In fact, I demonstrate this feature in my previous article on lambda functions in Python. I won’t be covering that particular feature in this article. However, I will be covering 3 of the most useful higher-order functions that are built into Python:

1. Map
2. Reduce
3. Filter

These functions make dealing with iterable objects much simpler than using first-class functions or regular loops. For each function, I will explain what the function does, demonstrate how it works with a code example, and suggest possible applications for the function.

Map

The map function in Python allows us to produce a new iterable object from an original iterable object. It takes 2 arguments: a function and an iterable object. The map function applies the function passed to every member of the iterable and then produces a map object. This object can be cast to an iterable type of our choosing. The function passed to the map object does not have to return a value (I will demonstrate this later).

Implementation

For our example, let’s generate a new list (L2) from an initial list (L1) such that each element of L2 is a double of its corresponding element in L1. Here is how we can achieve this with the map function:

def double(x):
    return x * 2

L1 = [3, 1, 4, 22, 45, 1, 245, 6, 34, 4, 8, 14]

L2 = list(map(double, L1))

print("Doubled Numbers: {}".format(L2)) 
# [6, 2, 8, 44, 90, 2, 490, 12, 68, 8, 16, 28]

The code above passes the double function(notice we leave out the parenthesis) and the original list (L1). A map object is returned and then cast to a list. Note, we can cast this map object to any type of iterable we want (list, tuple, set, …).

The map function also works with lambda functions. To make the code above a little more compact, we can rewrite it as follows:

L1 = [3, 1, 4, 22, 45, 1, 245, 6, 34, 4, 8, 14]
L2 = list(map( (lambda x : x \* 2) , L1))
print(L2)

This would be a more desirable solution if your map function is rather simple and will not be needed elsewhere in the program. However, if you’re running a complex operation on an iterable and/or plan to reuse the function outside of this context, it’s best to stick to regular functions.

Remember I mentioned that the function does not actually have to return anything. If you want to map certain functionality to every member of an iterable but don’t care for the resulting map object, the map function will simply return None for every corresponding member of the original iterable.

In fact, we don’t even have to store the returned map object at all. If we have a list of people and we simply want to print out a greeting for each person on the list, we can use a map function:

def greet(person):
    print("Hello, {}!".format(person['name']))
people = [
    { 'name' : 'John Doe', 'age' : 20},
    { 'name' : 'Mike Will', 'age' : 10},
    { 'name' : 'Jane Hill', 'age' : 18},
    { 'name' : 'Alan Smith', 'age' : 16},
    { 'name' : 'Chimuka Chinene', 'age' : 18},
]
list(map(greet, people ))
""" Prints :
Hello, John Doe!
...
Hello, Chimuka Chinene!
"""

We have to cast the map object returned in order to actually get the function to execute on all the members. The list produced is just a list of None type objects as the mapper function did not return anything. If you assign that list to a variable and print it, the output would be: [None, None, None, None, None]

Uses

The most obvious use of a map function is in creating a hash table. A hash table is a data structure that associates a key with a hash that’s generated using the key. The map function could be used to apply the hash function to a list of keys without the need for iteration.

The map function could also be used to replace loops. Instead of looping through every member of an iterable and executing some logic, we could map a function with that logic to the iterable. Remember, we don’t have to return anything from the mapper function nor do we have to care for the resulting map object.

Reduce

A reduce function works similar to a map function in the sense that it accepts an iterable and a function to apply to all the members of an iterable. However, it differs from map as it returns ONE value instead of an iterable mapping to the original. It reduces an iterable down to one value. The reduce function also takes a third argument, the initial value that it starts the iteration with. If this value is not provided, the first element of the iterable will be used as the initial value. With every iteration, the function takes its current value, and the next member in the iterable and returns a value to be used for the next element.

Implementation

Let’s calculate the sum of the doubles of all numbers in an array of integers L1. We could use a loop for this. However, a cleaner solution would be to use a reduce function as follows: Note in Python 3, reduce has been moved to functools so you will have to import it.

from functools import reduce
def addDoubles(a, b):
    return a + (b * 2)
L1 = [3, 1, 4, 22, 45, 1, 245, 6, 34, 4, 8, 14]
doubleSum = reduce(addDoubles, L1, 0)
print(doubleSum) # 774

In the code above, the add function takes 2 parameters: the current sum(a) and the current element in the iterable(b) which is doubled. The total of the double and current sum is returned and carried forward into the next iteration until the end of the iterable. The reduce function also accepts lambda functions. We can simplify the code above as follows:

from functools import reduce
L1 = [3, 1, 4, 22, 45, 1, 245, 6, 34, 4, 8, 14]
doubleSum = reduce( (lambda a,b : a + b * 2) , L1, 0)
print(doubleSum) # 774

This is a simple example, we can define a much more complex reducer function. We can also pick another initial value. For example, if I wanted my starting point to be 1000, I would write the code differently:

from functools import reduce
L1 = [3, 1, 4, 22, 45, 1, 245, 6, 34, 4, 8, 14]
sum = reduce( (lambda a,b : a + b * 2) , L1, 1000) # 1000 as initial value
print(sum) # 1774

Uses

The reduce function can be used anywhere that an iterable needs to be aggregated to one value. We can define the reducer function any way we like.

Filter

The filter function is also similar to the map function. It accepts an iterable, and a function to execute on all the members of the iterable. However, it does come with some different rules:

1. A filter does not have to produce a 1 to 1 mapping between the new iterable and the original one like a map function does.
2. A filter creates a new list based on the elements that return True when the filter function is executed on them

Implementation

Given a list of 5 people, let’s filter out all the people that are younger than 18.

def legal(person):
    return person['age'] >= 18
people = [
    { 'name' : 'John Doe', 'age' : 20},
    { 'name' : 'Mike Will', 'age' : 10},
    { 'name' : 'Jane Hill', 'age' : 18},
    { 'name' : 'Alan Smith', 'age' : 16},
    { 'name' : 'Chimuka Chinene', 'age' : 18},
]
legal_people = list(filter(legal, people))
print(legal_people)
""" Prints [{'name': 'John Doe', 'age': 20}, {'name': 'Jane Hill', 'age': 18}, #{'name': 'Chimuka Chinene', 'age': 18}] """

In the code above, we run the legal function on each person. The function returns either True or False depending on the person’s age. Only the people whose age returns True from the legal function make it into the new list. A filter object is returned. We can cast this to any iterable type we desire.

The filter function can also take a lambda function as an argument. We can write the code above as follows:

people = [
    { 'name' : 'John Doe', 'age' : 20},
    { 'name' : 'Mike Will', 'age' : 10},
    { 'name' : 'Jane Hill', 'age' : 18},
    { 'name' : 'Alan Smith', 'age' : 16},
    { 'name' : 'Chimuka Chinene', 'age' : 18},
]
legal_people = list(filter( (lambda person : person['age'] >= 18) , people))
print(legal_people)
""" Prints [{'name': 'John Doe', 'age': 20}, {'name': 'Jane Hill', 'age': 18}, #{'name': 'Chimuka Chinene', 'age': 18}] """

Uses

This one is pretty straightforward. We use the filter function in all instances where we want to filter out undesirable members of the original list. We can run this on very complex elements in an iterable and define complex filter functions to achieve whatever desired outcome we want. Just remember, the filter function has to return True or False.

Conclusion

Higher-order functions are an invaluable tool when writing programs, especially ones working on iterable objects. They are not necessary as their functionality can be achieved by regular for/while loops. However, they help a great deal in keeping our code neat and concise compared to regular loops. I hope I’ve provided a sufficient explanation of the functions above that will allow you to implement them in your projects. Happy coding!

This isn’t always the case, however. If you’re not aware of the tools python provided in order to achieve this, it is very easy to write verbose code that defeats the point of picking Python for its “compactness”.

One such tool is the lambda function. These allow us to create anonymous functions in python. If you’ve read good python programmers’ code before, you might have come across something that looks like this:

add = lambda a,b : a + b

The code above might look a bit confusing, let me clarify the syntax of a lambda function by breaking it down into its components. Lambda functions have 3 components: the ‘lambda’ keyword (of course), the set of arguments, and an expression.

lambda [(optional) arguments] : [expression]

The Rules

Before we go into detail about the implementation of lambda functions, let’s first discuss their rules:

1. Lambda functions are nameless (unless assigned to a variable). You cannot assign a function name after the ‘lambda’ keyword’ as you would with the ‘def’ keyword.

2. Arguments are optional. Lambda functions can be defined without passing any arguments, just like a regular function.

3. Lambda functions can only have one expression.

4. Just like a regular function, you can return anything from a lambda function. It is also possible to return nothing.

5. Lambda functions can be assigned to variables for repeated use. This is the only way to name a lambda function.

6. Lambda functions can be returned from regular functions.

7. Lambda functions can return other lambda functions.

Why Use Lambda Functions?

If you have never used lambda functions before, you probably think that you would do just fine without them, and you’d be right. However, as I’ve stated before, they play an important role in making sure we maintain clean, compact and pythonic code.

Lambda functions are most useful in situations where an anonymous function is required for single-use or repeated use.

Implementation

As stated before, we can use lambda functions alongside regular functions to generate functions. Let’s create an example where we want to create 3 functions. One that doubles a given number, one that triples a given number, and one that quadruples a given number. We could achieve this by creating 3 different functions:

def doubler(n):
    return n * 2

def tripler(n):
    return n * 3

def quadroupler(n):
    return n * 4

Instead of writing the code above, we could create a function that returns a lambda function with the functionality that we want:

def function_generator(n):
    return lambda a : a * n

Now we create our desired functions:

doubler = function_generator(2)
tripler = function_generator(3)
quadroupler = function_generator(4)

The function_generator function returns a function that will multiply its argument to whatever argument was passed to function_generator. We can test this with the following statement:

print(doubler(4), tripler(4), quadroupler(4)) # Output 8 12 16

The function generator can be simplified even further by nesting 2 lambda functions in one line:

function_generator = lambda n : lambda a : a * n

The previous test line should produce the same output as before.

Remember that a lambda function assigned to a variable makes the variable callable as a function. When introducing the topic, I gave the following example:

add = lambda a,b : a + b

This makes ‘add’ callable. So to add 2 numbers, we would use the following statement:

add(4, 6) # Returns 10

This also applies to functions returning lambdas. As long as a lambda function is returned, it is callable, no matter how deep the nesting goes. The function returned also has no name (hence ‘anonymous’) unless it is assigned to a variable.

We can also call a lambda function immediately after its definition:

(lambda x : x * 2)(4) # Returns 8

Make sure to store/print the value returned as it will not be accessible again later on in your module code. This format is very useful for list comprehensions.

Suppose we have a list of numbers (nums) and we want to generate a new list of numbers such that for each number n in nums, n * 2 > 50 should return True. How can we achieve this?

Using a regular function and list comprehension:

def validate(n):
    return n * 2 > 50

nums = [3, 1, 4, 22, 45, 1, 245, 6, 34, 4, 8, 14]
nums2 = [n for n in nums if validate(n)] # Returns [45, 245, 34]

However, we could eliminate the function definition altogether and include the validation expression within the list comprehension:

``python nums = [3, 1, 4, 22, 45, 1, 245, 6, 34, 4, 8, 14] nums2 = [n for n in nums if (lambda x : x * 2 > 50)(n)] # Returns [45, 245, 34] ```

Using Python lambda functions makes our code much more compact than in the previous example.