James Ingold

One Git, Many Hats = How I Manage Multiple Git Identities

Thu, 05 Dec 2024 00:00:00 +0000

Manually managing Git identities for multiple projects can be tedious. I would know, over the past couple of years, I’ve been using bits and pieces of this guide, but I recently automated my workflow entirely. My goal is to configure Git identities automatically based on a repository’s location while keeping project repositories organized in separate directories.

For instance, if I have ~/Repos/work and ~/Repos/personal, each with its own Git identity, I want commits within those directories to automatically use the appropriate Git identity. This can be achieved using Git’s gitconfig and SSH configuration features.

Here’s a step-by-step guide to managing multiple Git identities effectively.

Global Git Configuration

Start by creating a base Git configuration file as your default. This will be saved as $HOME/.gitconfig.

[user]
  name = Your Name
  email = your.email@example.com
  signingKey = ~/.ssh/mykey.pub

This establishes your default identity for all repositories.

Project-Specific Git Config

To override the global identity for specific projects, create a project-specific Git configuration file at the root of your project folder. For example, if all your work projects are under $HOME/Repos/work, create a .gitconfig file at $HOME/Repos/work/.gitconfig.

[user]
    name = Work Name
    email = work@example.com
    signingKey = ~/.ssh/work.pub

Automate Git Identity Selection

To automate identity selection based on directory paths, use Git’s conditional includes. Edit your global Git configuration file ($HOME/.gitconfig) and add the following:

[includeIf "gitdir:~/Repos/work/"]
    path = ~/Repos/work/.gitconfig

With this setup, any repository within the $HOME/Repos/work/ directory will automatically use your work git identity.

SSH Configuration

There are a couple different ways to handle SSH configuration. One way is to use the sshComand in gitconfig to tell Git use a specific ssh command for all Git operations in the repository. We can tell Git to always use our work ssh private key when interacting with repositories in the ~/Repos/work directory by adding an sshCommand to our work .gitconfig file.

[core]
    sshCommand = "ssh -i ~/.ssh/work"

The option I prefer is to manage all SSH configuration in my $HOME/.ssh/config file. If both work and personal projects use github, we will have a problem that the right SSH key is not being used for work projects. This is because both SSH entries use the same hostname. Here’s an example of how an SSH config file would be set up:

Host github
 AddKeysToAgent yes
 Hostname github.com
 User git
 IdentityFile ~/.ssh/github

Host github-work
 AddKeysToAgent yes
 Hostname github.com
 User git
 IdentityFile ~/.ssh/work

To fix this, we will use the url mapping feature in our gitconfig. This rewrites specific repository URLs automatically. When we are in a work repository, git will automatically rewrite the URL to use github-work and use the correct SSH config.

Update Project Specific with URL Overwrite

[url "git@github-work:"]
    insteadOf = git@github.com:

Full Project-Specific Root .gitconfig

[user]
    name = Work Name
    email = work@example.com
    signingKey = ~/.ssh/work.pub

[url "git@github-work:"]
    insteadOf = git@github.com:

With this setup, you can manage Git identities for different projects. Git will default to your global configuration, but will automatically switch to your project specific configuration when you are in a repository that matches the conditional include.

Split Large Files into Lines with NodeJS

Thu, 08 Dec 2022 00:00:00 +0000

Splitting a file into lines is pretty straight forward when working with smaller files, however larger files can run into memory issues. The convenient way to split a file into lines with NodeJS is to use fs.readFile and then split it by the newline character. You can even make it a one liner:

const lines = fs.readFileSync(filePath).toString("UTF8").split("\n");

We are using fs.readFileSync to read the contents of the file and then converting the buffer into a string before splitting it with the newline character. This method runs into issues when dealing with a large file though. Since we’re reading the file into memory as a string before splitting lines this can cause a memory exception. NodeJS also has a character limit for the String type of 512 MB. Trying to save a string into memory greater than that limit will result in an invalid string length error.

Handling Large Files

A better option is to read the file into a buffer and then split the buffer into an array of buffers by line delimiter. This approach is more efficient for large files because it allows you to process the file in chunks without reading the entire file into memory at once.

Unfortunately, there is not a built-in way in NodeJS to split a buffer by a character. Here are a couple different options:

Option 1: buffer-split package

const bsplit = require("buffer-split");
const buf = fs.readFileSync(fileName);
const delim = Buffer.from("\n");
const result = bsplit(buf, delim);
const lines = _.map(result, (r) => {
  return r.toString("utf8");
});

The downside with this package is that it was published in 2016 and hasn’t received any updates since. There are no vulnerabilities currently.

Option 2: inter-ops package

import { pipe, split } from "iter-ops";

const delim = Buffer.from("\n");
const i = pipe(
  buffer,
  split((value) => value === delim)
);

Option 3: Roll Your Own

function splitBufferToArray(buf, splitStr) {
  const bufArr = [];
  let start = 0;
  let idx = buf.indexOf(splitStr, start);
  while (idx !== -1) {
    const chunk = buf.slice(start, idx);
    bufArr.push(chunk);
    start = idx + 1;
    idx = buf.indexOf(splitStr, start);
  }
  if (bufArr.length > 0) bufArr.push(buf.slice(idx));
  return bufArr;
}

The correct approach to dealing with large files is to read in the file and then split that buffer into chunks. This will allow you to process the file efficiently and avoid running into memory issues.

References:

Maximum String Length NodeJS

Using Binary Files in AWS Lambda

Sun, 06 Mar 2022 00:00:00 +0000

I struggled for a few hours trying to run a binary file in an AWS Lambda function. I kept getting “cannot execute binary file”. I tried all sorts of fixes such as giving the file execute permissions and moving the file into the writeable tmp directory in Lambda to reapply the right permissions. The main problem turned out to be using a binary for macOS instead of the Linux version. However, I did learn some things that might help other developers save some time when running a binary file in a Lambda function.

What is a binary file? A file that consists of compiled and executable code.

The Lambda runtime environment contains many pre-compiled binaries that can be used but sometimes you can find yourself needing to execute a binary not provide in the runtime. For example, using the aws-iam-authenticator to access a kubernetes cluster or a pdf binary to generate pdfs.

Once we have our binary file, we’ll upload a zip file containing the package to be used as an Lambda Layer.

If you need to compile a binary from source, it is recommended to use an EC2 instance or the Cloud9 AWS editor. The other alternative is to use a docker container.

The format I use for my binary was a folder called package with a bin folder underneath it. package => bin with the binary placed in the bin folder.

chmod 755 /package/bin

You’ll probably want to make sure the binary file has executable permissions.

chmod +x /package/bin/binary

Now zip the package

zip -r9 package.zip *

If you’re using an EC2 or Linux server, you can pull the zip file down using scp

scp -i /key username@server:/file .

Upload your zip file as an AWS Lambda Layer.

A Lambda layer is a .zip file archive that can contain libraries, a custom runtime, data, or configuration files. They are used to reduce deployment package size and to promote code sharing and separation of responsibilities so that you can iterate faster on writing business logic.

In your Lambda, use the new layer and update your PATH environment variable to include the executable binary file. $PATH:/opt/package/bin

Now you should be able to execute the binary file in your Lambda function!

Deconstructing an Object Relationship Mapper (ORM) in Typescript

Fri, 04 Feb 2022 00:00:00 +0000

Deconstructing a Typescript ORM - Mapping Objects - Part 1

Have you ever wondered how an ORM works? After working through rolling my own lightweight Typescript ORM, I have some answers. We’re not going to talk through building a full ORM in this article but we will set up a basic object mapper which can be later extended to generate SQL and perform queries. Let’s dive into it!

What is an ORM?

An ORM stands for Object Relational Mapping and these tools map programming languages to databases. ORMs allow you to query and manipulate data from a database generally in an object-oriented paradigm. They bridge your objects in code to your database tables.

Pros

ORMs are inherently DRY making it easier to re-use code.
They take care of some things automatically such as sanitization and transactions.
Relationships are handled elegantly which can be a pain to deal with manually.
Can use your programming language of choice instead of SQL.

Cons

The big issue comes down to performance on ORMs, they generate SQL which can be less optimal than handcrafting your own statements.
Learning curve as each ORM has a different implementation.

The Situation

The main pain point I was trying to solve was connecting Typescript classes to a database. In the codebase I was working in, the following pattern existed: there was a domain model, a repo model (matched database tables), and a dto (data transfer object). The domain model and repo model were manually mapped back and forth, to and from the database. The dto was also manually mapped but I’m going to skip this model for now. This required a fair amount of code to be created whenever someone wanted to create a new model to work with. It also made handling relationships difficult. Parameterized constructors can also become a burden, especially early in a project where requirements are bound to change often. There was an established database access pattern - repository classes using a shared library. Since multiple different services were using the database access shared library, I decided to roll my own light weight object mapper to map objects to the database without using an existing fully fledged ORM library.

Psuedo example of the current code

export class RepoModel {
  static propertyValueMap: IPropertyValueMap<DomainModel> = {
   const mapType = (type: TypeEnum) => {
      return RepoModel.propertyValueMap?.type?.[type] ?? handleError();
    };
  }

  constructor (prop1, prop2, prop3, ...) {}

  toDomain() : DomainModel {
      const mapType = (type: CustomEnum) => {
      const map = Translator.invert(RepoModel.propertyValueMap?.type);
      return map?.[type] ?? handleError();
    };
    ...
    return new DomainModel(
      mapType(this.type_id) as TypeEnum,
      this.id,
      this.createdAt)
  }

  static fromDomain(domain: DomainModel) : RepoModel {
    // code that maps each enum
      const mapType = (type: TypeEnum) => {
	  return RepoModel.propertyValueMap?.type?.[type] ?? handleError();
	};
    ...
    return new RepoModel(
      mapType(domain.type),
      domain.id,
      domain.createdAt)
  }
}

export class DomainModel {
  constructor(
    public type: TypeEnum,
    public id?: string,
    public createdAt?: Date
  ) {}
}

export class ModelRepo {
  public async get(id: string): Promise<DomainModel> {
    const result = await this.db.query(this.db.getSql('getById'), [id]);
    const resultObject = this.db.get(result);
    return new RepoModel(
       resultObject.type_id,
        resultObject.id,
        resultObject.created_at
    ).toDomain();
  }
}

The Vision

I wanted to refactor the domain model to handle the conversion to database tables without requiring manual mapping of a repo model. The repo model would be removed. The domain model would inherit a base model which would handle the ORM logic. Since there was an established database access pattern, I didn’t go the full way to make queries through our makeshift ORM but I will point out the spots that can be extended to achieve this functionality. The goal is to simplify the creation of domain models, transformations to/from the database and reduce the amount of code/complexity to ship features.

General Principles - Connecting your Typescript Classes to the Database

Database columns will be mapped to domain object properties using decorators. This will include relationships and enum types. reflect-metadata stores metadata about the classes and properties. Most of the work is a simple map for each class, renaming db column properties to domain model properties and vice versa. Reflect.defineProperty holds a list of field metadata on the target class. This is where more database ORM logic could live in the future such as column type, length, etc. A base domain model entity will use this metadata to map the models appropriately.

Domain entities use Model, Column, Enum, and HasMany (relationships) decorators to map to the database. A Domain entity extends BaseDomainModel which has toRepo() and fromRepo() functions. These functions do the heavy lifting of using metadata to transform objects.

Here’s what our end state will look like:

@Model("DomainModel")
export class DomainModel extends BaseDomainModel implements IDomainModel {
  @Column("id")
  id?: string;
  @Enum("type_id", () => TypeEnum)
  type: TypeEnum;
  @HasMany("UserDomainModels", "domain_model_id")
  users: UserDomainModel[];
  @Column("created_at")
  createdAt?: Date;
  constructor(obj?: IDomainModel) {
    super();
    Object.assign(this, obj);
  }
}

export interface IDomainModel {
  id?: string;
  type: TypeEnum;
  users: UserDomainModel[];
  createdAt?: Date;
}

export class ModelRepo {
  public async get(id: string): Promise<DomainModel> {
    const result = await this.db.query(this.db.getSql("getById"), [id]);
    return DomainModel.fromRepo(this.db.get(result));
  }
}

Decorators

Decorators provide a way to add both annotations and a meta-programming syntax for class declarations and members. Even though it’s an experimental feature, decorators provide great functionality. We’ll leverage decorators to handle our mapping metadata. We’ll briefly walk through each decorator in our ORM.

Model(identifier: string, alias?: string)

Adds the model and identifier to a class map. An alias can be set to avoid name collision with joins in raw sql for example if alias = model then in sql, select model.id as model_id will allow model_id to be set on child models as id which would be overwritten without an alias id column in the join.

export const classMap = new Map();
export function Model(identifier?: string, alias?: string): ClassDecorator {
  return (target: any) => {
    identifier = identifier || target.name;
    if (!target.prototype.modelName) {
      Reflect.defineProperty(target.prototype, "modelName", {
        value: identifier,
        writable: true,
        configurable: true,
        enumerable: true,
      });
      Reflect.defineProperty(target.prototype, "alias", {
        value: alias || "",
        writable: true,
        configurable: true,
        enumerable: true,
      });
    }
    classMap.set(identifier, target);
  };
}

Column(name: string)

Adds the database column name to a map for the class to be used for transforming. This could be extended to support more options and database support such as column type, size, etc. This is also where further options would live as well such as making a field required.

import "reflect-metadata";
export const METADATA_KEY = "design:type"; // reflect-metadata Type information design type
export type relationType = "HASONE" | "HASMANY";

export function setTransform(
  object: object,
  propertyName: string | symbol,
  name: string | symbol
) {
  const metadataMap = getMetadata(PARAM_TYPE_KEY, object);
  metadataMap.set(propertyName, name); // would need to actually implement a map with db types
}

export function Column(name?: string): PropertyDecorator {
  return (target: any, propertyKey?: string | symbol) => {
    if (!target.fields) {
      Reflect.defineProperty(target, "fields", {
        value: {},
        writable: true,
        configurable: true,
        enumerable: true,
      });
    }
    const designType = Reflect.getMetadata(
      METADATA_KEY,
      target,
      propertyKey as string
    );
    const values: any = { type: designType.name, name }; // This is where we could do more DB ORM mapping if we wanted - column type, size, etc
    Reflect.defineProperty(target.fields, propertyKey as string, {
      value: values,
      writable: true,
      configurable: true,
      enumerable: true,
    });
    setTransform(target, propertyKey as string, name as string);
  };
}

Enum(name: string, () => Dictionary)

Supports mapping to and from an enum type. Parameters are the database column name and a function which points to the enum options to use

export function Enum(name: string, options: () => Dictionary) {
  return (target: any, propertyKey?: string | symbol) => {
    const opts = {
      value: { name: propertyKey as string, enum: true, options: options() },
      writable: true,
      configurable: true,
      enumerable: true,
    };
    Reflect.defineProperty(target.fields, propertyKey as string, opts);
    setTransform(target, propertyKey as string, name as string);
  };
}

export type Dictionary<T = any> = { [k: string]: T };

HasMany(modelName: string, relationKey?: string)

Adds a HasMany relationship to the object map supporting transformation when going fromRepo. relationKey is optional but could be used in the future for more database mapping.

export const PARAM_TYPE_KEY = "PARAM_TYPE_KEY";
import { getMetadata } from "./utils"; // wraps Reflect.getMetadata to return class or property info
export function HasMany(
  modelName: string,
  relationKey?: string
): PropertyDecorator {
  return (target: any, propertyKey?: string | symbol) => {
    if (!target.relationship) {
      Reflect.defineProperty(target, "relationship", {
        value: {},
        writable: true,
        configurable: true,
        enumerable: true,
      });
    }
    const values: any = {
      as: propertyKey as string,
      relationshipType: "HASMANY",
      from: modelName,
      on: { [propertyKey as string]: relationKey },
      type: "left", // could use this for joins in the future
    };
    if (!target.relationship.HASMANY) {
      Reflect.defineProperty(target.relationship, "HASMANY", {
        value: [values],
        writable: true,
        configurable: true,
        enumerable: true,
      });
    } else {
      target.relationship.HASMANY.push(values);
    }
    const originMap = getMetadata(PARAM_TYPE_KEY, target);
    originMap.set("relationship", target.relationship.HASMANY);
  };
}

BaseDomainModel

Each domain model which wants to support object mapping will need to extend BaseDomainModel.

Static functions:

fromRepo(obj): DomainModel
toRepo(): obj

import "reflect-metadata";
import { classMap, PARAM_TYPE_KEY, getMetadata } from "../../decorators/utils";

export class BaseDomainModel {
  static toRepo(data: any): any {
    const retVal = {};
    let cls: any;
    if (data instanceof this) {
      cls = data;
    } else {
      cls = Reflect.construct(this, []);
    }
    const originMap = getMetadata(PARAM_TYPE_KEY, this);
    originMap.forEach((value: string, key: string) => {
      if (cls.fields[key] && cls.fields[key].enum) {
        if (typeof data[key as string] === "number")
          retVal[value] = data[key as string];
        else {
          const options = Object.values(cls.fields[key].options);
          retVal[value] = options.findIndex(
            (x: any) => x === data[key as string]
          );
          if (retVal[value] < 0) retVal[value] = 0;
        }
      } else if (key && Object.prototype.hasOwnProperty.call(data, key)) {
        retVal[value] = data[key];
      }
    });
    return retVal;
  }

  static fromRepo(data: any) {
    const objData = Array.isArray(data) ? data[0] : data;
    let cls: any;
    if (data instanceof this) {
      cls = objData;
    } else {
      if (!isObject(objData)) {
        data = {};
      }
      cls = Reflect.construct(this, []);
    }

    const originMap = getMetadata(PARAM_TYPE_KEY, this);
    originMap.forEach((value: any, key: string) => {
      // set the values
      if (
        value &&
        Object.prototype.hasOwnProperty.call(objData, value as string)
      ) {
        if (cls.fields[key] && cls.fields[key].enum) {
          cls[key] = Object.values(cls.fields[key].options)[
            objData[value as string]
          ];
        } else {
          cls[key] = objData[value as string];
        }
      } else if (key === "relationship" && data.length >= 1) {
        // handle relationships mapping
        value.forEach((r: any) => {
          const model = classMap.get(r.from);
          const om = getMetadata(PARAM_TYPE_KEY, model);
          cls[r.as] = [];
          data.forEach((childData: any, index: number) => {
            cls[r.as].push(new model());
            om.forEach((value: string, key: string) => {
              // set value here
              cls[r.as][index][key] =
                childData[`${model.prototype.alias}_${value}`] ||
                childData[value];
            });
          });
        });
      }
    });
  }
}

Conclusion

That’s it! We now have a basic ORM in place to handle mapping our objects back and forth between database and domain models. In the future we can extend our ORM to generate SQL and provide further database support. Happy Codings!

Let me know what you think at hey[@]jamesingold.com

References:

Reflect Metadata

Great Article on Decorators and Metadata

Sequelize Typescript Decorators

Why I want to build an AutoMapper in Typescript

Typestack Class Transformer

Amazon Cognito MFA with Email Using Lambda Triggers

Sat, 14 Aug 2021 00:00:00 +0000

Amazon Cognito is a great service for easily getting started with authentication. It also has multi-factor authentication (MFA) right out of the box using a cell phone for SMS or a TOTP (Time-based One Time Password) device such as Authy or Google Authenticator. The requirement that I had was to only use MFA via email. I would have preferred to use the built-in AWS Cognito MFA workflows but this was a hard requirement for this project. This was in the healthcare industry and their email was usually restricted to being on the hospital network. Therefore, I had to allow MFA by email only.

This post will describe how to implement MFA with email on Amazon Cognito by using a Custom Authentication Flow and Amazon SES (Simple Email Service).

First, go to the AWS console and set up SES. Make sure that you have an email verified in Amazon SES. If your account is in Sandbox mode for Amazon SES, you will want to make sure to verify both the sender and the receiver email address. When the account is in production mode, you will not need to verify email addresses that you are sending to. I point this out because I spent 2 hours trying to figure out why sending an email was failing due to this issue.

To get started, we have to use a Custom Authentication Flow via Lambdas to handle this deviation from the built-in AWS Cognito MFA workflows. Amazon Cognito works via Lambda Functions and they allow different hooks to customize the authentication flow:

Amazon Cognito Lifecyle Triggers:

Define Auth Challenge: Determines the next challenge in a custom authentication flow (tell Cognito about our custom challenge)
Create Auth Challenge: Creates a challenge in a custom authentication flow (send the MFA code)
Verify Auth Challenge: Determines if a response is correct in a custom authentication flow (verify that the code entered matches the code we created)

There are a couple built in challenges that we can utilize, including FORCE_NEW_PASSWORD and PASSWORD_VERIFIER.

In order to create a custom auth flow that allows us to use MFA code via email we’re going to implement it using these Lambda lifecycle hooks. The authentication flow will be as follows:

The user enters their email and password on the app.
On our secure server side (NodeJS/Express), we’ll use the UpdateUserAttributes Cognito method to generate a new MFA code for the user and save it into their attributes as a custom property.
Our define auth challenge lambda function will be hit. If the user has a temporary password, we’ll return the FORCE_NEW_PASSWORD challenge to the client.
Due to a bug in the custom authentication workflow, after the user changes their password, we’ll force them to log in again. The bug makes it so we won’t get to our custom challenges if the user is changing their password
Alright, now they’re back to the login screen where they’ll enter their email and new password
The PASSWORD_VERIFIER challenge will run verifying the users password. After that, we’ll return a CUSTOM_CHALLENGE - our MFA by email challenge!
Our create auth challenge lambda function will run. In this function, we will send the MFA code. We will have the users attributes and we’ll use SES to email them the code generated on the secure server side.
Once the challenge is created and the email sent, we’ll return the current authentication flow state (CUSTOM_CHALLENGE) which should update the UI to ask for the MFA code.
The user inputs the MFA code that they received from their email. Now we send that answer to the verify auth challenge. This will verify the answer to what we have in the user’s custom attributes for their current MFA code. We’ll also make sure that the time frame for a valid code has not passed.

Part 1: Create Serverless Project for Cognito Lifecycle Hooks

I’m going to use the Serverless framework because it’s easy to manage and I know it well. However, you could just use Lambda functions if you wanted to keep it simple.

First, create a new pool through Amazon Cognito console. You can also use a cloud formation template, such as this example.

Next, add a custom attribute called “authChallenge”. This will hold our MFA code and a timestamp to make sure it’s not expired.

Keep Multi-Factor Authentication (MFA) set to “OFF” since we’re going to be implementing this ourselves via a Custom Auth Flow and the Cognito Lifecycle Triggers.

Once the pool is created, create a new serverless project

npm install -g serverless
serverless create -n cognito-mfa-email-example

Define Lifecycle Function

The Define Cognito Lifecyle Event will determine the next challenge in a custom auth flow. This is a flow, we’re going from one event to another starting with the PASSWORD_VERIFIER. There are a few important properties to return: issueTokens, failAuthentication, and the next challengeName to run. Once the password is confirmed (by Cognito), then we run our custom challenge. The PASSWORD_VERIFIER is a built in Cognito challenge which checks the user’s password.

module.exports.handler = async event => {
  // Kicks off with Secure Remote Password
  if (
    event.request.session.length === 1 &&
    event.request.session[0].challengeName === "SRP_A" &&
    event.request.session[0].challengeResult === true
  ) {
    event.response.issueTokens = false;
    event.response.failAuthentication = false;
    event.response.challengeName = "PASSWORD_VERIFIER";
    return event;
  }
  if (event.request.userNotFound) {
    event.response.failAuthentication = true;
    event.response.issueTokens = false;
    return event;
  }

  // Check result of last challenge
  if (
    event.request.session &&
    event.request.session.length > 2 &&
    event.request.session.slice(-1)[0].challengeResult === true
  ) {
    // The user provided the right answer - issue their tokens
    event.response.failAuthentication = false;
    event.response.issueTokens = true;
    return event;
  }

  event.response.issueTokens = false;
  event.response.failAuthentication = false;
  event.response.challengeName = "CUSTOM_CHALLENGE";
  return event;
};

Create Lifecyle Function

The Create function will run, creating our CUSTOM_CHALLENGE. We’re going to use AWS SES to send an email with the MFA code in this function. The request will contain the user attributes from Cognito, we’ll use the authChallenge custom attribute to get the code created on our server side and email that code to the user which they will use to complete MFA. We also set the publicChallengeParameters and privateChallengeParameters which we’ll pass through the flow.

const mailer = require("./mailer");
module.exports.handler = async event => {
  const challenge = event.request.userAttributes["custom:authChallenge"];
  const [authChallenge, timestamp] = (
    event.request.userAttributes["custom:authChallenge"] || ""
  ).split(",");
  // This is sent back to the client app
  event.response.publicChallengeParameters = {
    email: event.request.userAttributes.email
  };

  // add acceptable answer
  // so it can be verified by the "Verify Auth Challenge Response" trigger
  event.response.privateChallengeParameters = {
    challenge: challenge
  };

  // we want to check and make sure we haven't sent the code before in this login session before sending the code
  if (
    event.request.session.length < 3 &&
    !event.request.session.find(s => s.challengeName === "CUSTOM_CHALLENGE")
  )
    await mailer.send(
      "Your Access Code",
      event.request.userAttributes.email,
      "Please use the code below to login: <br /><br /> <b>" +
        authChallenge +
        "</b>"
    );

  return event;
};

Verify Lifecycle Function

In the verify lifecycle function, we take the code that the user inputted and the one we have in privateChallengeParameters to compare if they have the right code. Amazon Cognito invokes this trigger to verify if the response from the end user for a custom Auth Challenge is valid or not. We also check the timestamp to make sure it’s not expired. We set the challenge response property of answerCorrect based on if they answered correctly. You could have even more challenges and the custom challenge flow loop would repeat until all challenges are answered.

const LINK_TIMEOUT = 30 * 60;

module.exports.handler = async event => {
  // Get challenge and timestamp
  const [authChallenge, timestamp] = (
    event.request.privateChallengeParameters.challenge || ""
  ).split(",");

  // Check if code is equal to what we expect...
  if (event.request.challengeAnswer === authChallenge) {
    // Check if the link hasn't timed out
    if (Number(timestamp) > new Date().valueOf() / 1000 - LINK_TIMEOUT) {
      event.response.answerCorrect = true;
      return event;
    }
  }

  event.response.answerCorrect = false;
  return event;
};

Deploy the Project

I’ve included a full sample serverless.yaml here. You’ll want to have the following IAM role capabilities and the following function declarations. I’m using the Amazon Systems Parameter Store for setting the EMAIL_ADDRESS variable which was used in the Create lifecycle function

serverless.yaml snippet

iamRoleStatements:
    - Effect: "Allow"
      Action:
        - cognito-idp:AdminGetUser
        - cognito-idp:AdminUpdateUserAttributes
      Resource:
        - {"Fn::GetAtt": [UserPool, Arn]}
    - Effect: "Allow"
      Action:
        - ses:SendEmail
      Resource:
        - "*"

functions:
  define:
    handler: defineAuth.handler
    events:
      - cognitoUserPool:
          pool: ${self:custom.userPoolName}
          trigger: DefineAuthChallenge
          existing: true
  create:
    handler: createAuth.handler
    environment:
      EMAIL_ADDRESS: ${ssm:/${self:provider.stage}_EMAIL_ADDRESS}
    events:
      - cognitoUserPool:
          pool: ${self:custom.userPoolName}
          trigger: CreateAuthChallenge
          existing: true
  verify:
    handler: verify.handler
    events:
      - cognitoUserPool:
          pool: ${self:custom.userPoolName}
          trigger: VerifyAuthChallengeResponse
          existing: true

Part 2: Pseudo Backend Code

Now that we’ve got our Cognito handlers set up and a user pool, we can run through some example server side code that handles the auth sequence. We start with the SRP_A (Secure Remote Password) challenge which kicks off the authentication flow.

router.post("/login", async (req, res, next) => {
  try {
    const SRP_A = CognitoHelper.calculateA();
    const userHash = generateHash(
      req.body.username,
      this.secretHash,
      this.clientId
    );
    const params = {
      UserPoolId: this.poolId,
      AuthFlow: "CUSTOM_AUTH",
      ClientId: this.clientId,
      AuthParameters: {
        USERNAME: req.body.username,
        PASSWORD: req.body.password,
        SECRET_HASH: userHash,
        SRP_A: SRP_A,
        CHALLENGE_NAME: "SRP_A"
      }
    };

    try {
      const data = await this.cognitoIdentity
        .adminInitiateAuth(params)
        .promise();
      // SRP_A response
      const { ChallengeName, ChallengeParameters, Session } = data;
      // Set up User for MFA
      const authChallenge = _.map([...Array(8).keys()], n =>
        Math.floor(Math.random() * 10)
      ).join("");
      await this.cognitoIdentity
        .adminUpdateUserAttributes({
          UserAttributes: [
            {
              Name: "custom:authChallenge",
              Value: `${authChallenge},${Math.round(
                new Date().valueOf() / 1000
              )}`
            }
          ],
          UserPoolId: this.poolId,
          Username: req.body.username
        })
        .promise();
      const hkdf = CognitoHelper.getPasswordAuthenticationKey(
        ChallengeParameters.USER_ID_FOR_SRP,
        req.body.password,
        ChallengeParameters.SRP_B,
        ChallengeParameters.SALT,
        this.poolId
      );
      const dateNow = CognitoHelper.getNowString();
      const signatureString = CognitoHelper.calculateSignature(
        hkdf,
        this.poolIdOnly,
        ChallengeParameters.USER_ID_FOR_SRP,
        ChallengeParameters.SECRET_BLOCK,
        dateNow
      );
      const responseParams = {
        ChallengeName: ChallengeName,
        ClientId: this.clientId,
        ChallengeResponses: {
          PASSWORD_CLAIM_SIGNATURE: signatureString,
          PASSWORD_CLAIM_SECRET_BLOCK: ChallengeParameters.SECRET_BLOCK,
          TIMESTAMP: dateNow,
          USERNAME: ChallengeParameters.USER_ID_FOR_SRP,
          SECRET_HASH: generateHash(
            ChallengeParameters.USERNAME,
            this.secretHash,
            this.clientId
          )
        },
        Session: Session
      };
      // Password verifier response, should be set up for custom mfa challenge now
      const respData = await this.cognitoIdentity
        .respondToAuthChallenge(responseParams)
        .promise();
      res.status(200).json(respData).end();
    }
  } catch (err) {
    console.log(err);
    if (err.code === "UserNotFoundException")
      return res.status(400).send("Invalid Credentials");
    res.sendStatus(500);
  }
});

Verify Route: This handles responding to our auth challenge with Cognito which will call our verify lifecycle handler.

router.post("/verify", async (req, res, next) => {
  try {
    const { email, password, confirmPassword, user, code } = req.body;
    const params = {
      ChallengeName: user.ChallengeName,
      ClientId: this.clientId,
      ChallengeResponses: {
        USERNAME: email,
        ANSWER: code,
        SECRET_HASH: generateHash(email, this.secretHash, this.clientId)
      },
      Session: user.Session
    };
    const data = await this.cognitoIdentity
      .respondToAuthChallenge(params)
      .promise();

    // if we failed the custom challenge
    if (
      !data.AuthenticationResult &&
      user.ChallengeName === "CUSTOM_CHALLENGE" &&
      code
    ) {
      data.error = "Invalid Code";
      return res.json({ success: false, data: data });
    } else {
      // add any user claims here
      data.AuthenticationResult.UserClaims = {};
      return res.json({ success: true, data: data });
    }
  } catch (err) {
    console.log(err);
    res.sendStatus(500);
  }
});

Conclusion

That’s it! We’ve now built a custom authentication workflow for handling MFA by email using Amazon Cognito. Our server code initiates the auth flow and then provides challenge responses until we have verified that the user has a valid MFA code. Happy Codings!

References:

Source Code Repository

Cognito Custom Auth Lambdas

Automating Route53 DNS Updates for Blue/Green Deployments (the hard way)

Thu, 29 Jul 2021 00:00:00 +0000

Blue/Green Architecture

A blue/green architecture in some form or fashion is a necessity for highly available systems. This means that you have at least two seperate production environments. The blue one is in use, handling live traffic while the green one is a clone of the production system and ready to go in case of emergency or routine maintenance. Ideally, these systems are in different regions and possibly even different cloud providers depending on your need to reduce risk. There are several ways to achieve blue/green deployments and it could be as simple as having two different servers which you can switch between. Regardless of your setup, at some point you will have to switch the DNS to point from the blue environment to the green environment. For example, AWS CodePipeline manages this for Elastic Beanstalk. Most environments won’t manage the DNS updates for you though, so in this post I’ll show you how to script out updates to AWS Route53 to manage blue/green DNS the hard way. For my use case, I have domains pointed to an elastic load balancer in front of a kubernetes cluster. AWS EKS clusters in different regions make up the blue/green architecture, which means we have to switch the DNS to move between them.

As a side note, it’s good to practice switching environments so that the first time isn’t under the intense pressure of something going wrong. Switching which environment is live should be as seamingless as possible, with no downtime or stress. In this example, I’ll be using Route53 to switch DNS urls from one AWS elastic load balancer to another with a python script.

Boto, the python library for AWS, isn’t the easiest to use for Route53. I’ll break down each step of the script but if you want to skip ahead, here’s the full gist. We’re going to build a python utility script to handle updating DNS entries and then a shell script for handling bulk updates. Let’s write some code!

Python Script for Updating Route53 Entries

Create a new python file and add the following code:

#!/usr/bin/env python

"""
Update DNS for a RecordSet on AWS Route53
"""
import argparse
import sys

import boto3

parser = argparse.ArgumentParser()

parser.add_argument("--zoneId", type=str, default="", help="Route53 Zone Id, recommend setting this as an environment variable")
parser.add_argument("--hostname", type=str, default="", help="The hostname to update example.domain.com")
parser.add_argument("--dns", type=str, default="", help="The new dns value")

args = parser.parse_args()

route53 = boto3.client('route53')
zoneid = args.zoneId
hostname = args.hostname
CNAME = args.dns


if not zoneid or not hostname or not CNAME:
   print("Please provide a zone id, hostname and new dns")
   return

We’re using argparse to pass in our zoneId, hostname, and the new dns. I recommend setting your hosted zone id as an environment variable so you don’t have to remember it. If you only have one Route53 hosted zone, you can use Boto to get all the hosted zones (route53.get_all_hosted_zones()) and then use the first one.

Now that we’ve got our AWS Route53 Hosted Zone ID, we can write the update DNS function. I’m going to use a shell script to pass in an array of domain names because in production, there are some domains that I don’t want to update. If you have a simpler set up, you could iterate through the record sets and update each DNS without the need for the shell script. An environment variable could also hold the domains to update without using the shell script as well.

def updatedns(hostname, newdns):
 sets = route53.list_resource_record_sets(HostedZoneId=zoneid)

 for rset in sets['ResourceRecordSets']:
    if rset['Name'] == hostname and rset['Type'] == 'CNAME':
        curdnsrecord = rset['ResourceRecords']
        print(curdnsrecord)
        if type(curdnsrecord) in [list, tuple, set]:
            for record in curdnsrecord:
                curdns = record
        # print('Current DNS CNAME: %s' % curdns)
        curttl = rset['TTL']
        # print('Current DNS TTL: %s' % curttl)

        if curdns != newdns:
            # UPSERT the record
            print('Updating %s' % hostname)
            route53.change_resource_record_sets(
              HostedZoneId=zoneid,
              ChangeBatch={
                'Changes': [
                  {
                    'Action': 'UPSERT',
                    'ResourceRecordSet': {
                      'Name': hostname,
                      'Type': 'CNAME',
                      'TTL': curttl,
                      'ResourceRecords': [
                        {
                          'Value': newdns
                        }
                      ]
                    }
                  }
                ]
              }
            )

We’re passing in the hostname to update and the new DNS entry to the updatedns function. From there, we check out each RecordSet and see if it matches the hostname that we’re trying to update. If it does, we copy over the time to live property and then change the dns if it doesn’t match what was passed in. The Route53 client’s change_resource_record_set is handy because we can use the ‘UPSERT’ action to create or update by hostname.

That’s it for the python file, check out the full gist here.

Bonus: Shell script to Pass Domains to Update

#!bin/bash
# Shell script to update AWS Route53 DNS when switching Blue/Green Deployments

if [ $# -lt 1 ]
then
  echo "Please supply $1 destination DNS address (likely an ELB address)"
  return 1
fi

# array of domains to update
domains=("example.mydomain.com" "example2.mydomain.com")

for i in "${domains[@]}"
do
  echo "updating $i";
  python updatedns.py --zoneId="$MY_ROUTE53_HOSTED_ZONE" --hostname="$i" --dns="$1"

done

echo "Finished updating Route53 records"

The End

That’s it! We’ve now automated changing DNS for blue/green environments using AWS Route53, the hard way. If you know any easier way, let me know! Happy Codings

References:

Boto Route53 Documentation

updatedns.py gist

Migrating from Sequel Pro to Sequel Ace

Wed, 28 Jul 2021 00:00:00 +0000

Sequel Pro is a MySQL/MariaDB management GUI for macOS and it has been around for a long time. Sequel Pro’s first release was in 2008 and at the time of this writing, the last one was in 2016. I’ve been using Sequel Pro for the past seven years but with the update to Big Sur, it’s time to move on. Sequel Pro has become slow to use but I’ve continued on using it as a habit. A co-worker mentioned that the project forked into one called Sequel Ace. It’s still open source and actively maintained. Here’s how to install Sequel Ace and pull over your favorite database connections from Sequel Pro with the least amount of headache.

Install Sequel Ace

brew install --cask sequel-ace

and if you have a Brewfile, add the cask “sequel-ace”

Copy Preferences and Favorites

cp ~/Library/Preferences/com.sequelpro.SequelPro.plist ~/Library/Containers/com.sequel-ace.sequel-ace/Data/Library/Preferences/com.sequel-ace.sequel-ace.plist

cp ~/Library/Application\ Support/Sequel\ Pro/Data/Favorites.plist ~/Library/Containers/com.sequel-ace.sequel-ace/Data/Library/Application\ Support/Sequel\ Ace/Data/Favorites.plist

Copy Passwords (optional)

The database connections that we copied over will not have the passwords because those are stored in the Keychain Access app. You can re-enter those passwords and save them as you connect to the different databases or you can open KeyChain Access and copy them over ahead of time. Open up KeyChain Access and search for “Sequel Pro”. From there, you can click on the different connections and then check “Show Password” to view the plain text password. Next, copy that password over to Sequel Ace.

If you have many favorite connections, Sequel Ace offers the following bash script:

#!/bin/bash

results=$(security dump-keychain -r | grep "Sequel Pro" | sed -E -e 's/^.*"([^"]+)"$/\1/' | sort | uniq)
IFS=$'\n' read -r -d '' -a items < <(printf '%s\0' "$results")

for key in "${items[@]}"
    do
        newkey=$(echo "$key" | sed -E -e 's/Sequel Pro/Sequel Ace/')
        echo "Migrating: $key --> $newkey"
        account=$(sudo security find-generic-password -l "$key" | grep "acct\"<blob>" | sed -E -e 's/^.*"acct"<blob>="(.+)"$/\1/')
        pwd=$(sudo security find-generic-password -l "$key" -w)
        security add-generic-password -a "$account" -s "$newkey" -w "$pwd" -T "/Applications/Sequel Ace.app" -U
        echo "Done: $key --> $newkey"
done

Uninstall Sequel Pro

brew uninstall --cask sequel-pro

Wrapping Up

Now we’ve got our Sequel Pro connections moved over to Sequel Ace, an actively maintained database GUI ready for the next decade of coding on a Mac!

References:

Migrating From Sequel Pro

Restoring a MySQL Database from .frm and .ibd Files on MacOS

Mon, 01 Mar 2021 00:00:00 +0000

Ending 2020 by losing all your MySQL data seems about right doesn’t it? I spent a full day recovering my data after updating to the latest Docker version on a NAS. It nuked my MySQL container in the process. Luckily, I had some database files but they were in .frm and .ibd format which are not easy to import, especially once the server is gone. In this article I’ll show you how to restore your MySQL data on MacOS if you only have .frm and .ibd files available. Hopefully, this will save you some time if you are in the same position I found myself in.

A brief overview of the files we have and what we’ll be doing to salvage the data:

.frm: Contains a single table definition, essentially create table statements
.ibd: This file holds the data and indexes for a single table on the file system.

Restore Process High-Level:

Generate create statements from .frm files to get our table structures back
Reset the tablespace: when an InnoDB table is truncated, or dropped and recreated, it gets a new table ID. Any ID mismatch between the table in the database and the backed-up table can prevent it from being restored. Since we’re restoring this on a different MySQL Server instance, we need to drop the tables ids.
Put our .ibd files into the data directory
Sync the tablespace: Import new tables ids
Backup our data so we have full backup which can easily be restored

Let’s Get Started

Install MySQL shell client:

brew install mysql-client

Add it to your path:

export PATH="/usr/local/opt/mysql-client/bin:$PATH"

Reload your shell

source ~/.zshrc

Let’s begin by starting up a fresh docker instance that we’ll use to restore our database.

Create config and data volumes. The config volume will hold additional MySQL configuration and the data volume will be where our data goes on disk.

mkdir ~/docker/mysql/conf.d
mkdir ~/docker/mysql/data

Add the following to a ~/docker/mysql/conf.d/config.cnf

[mysqld]
innodb_force_recovery = 0

Run a MySQL docker instance

docker run \
--detach \
--name=mysql2 \
--env="MYSQL_ROOT_PASSWORD=mypassword" \
--publish 32780:3306 \
--volume=/pathToMysqlConfig/conf.d:/etc/mysql/conf.d \
--volume=/pathToDataDirectory/mysql2:/var/lib/mysql:rw mysql:5.7

Step 1: Create Files to Reconstruct Table Structure

There are two ways to reconstruct the tables. The first option is to use MySQL command line utilities and the mysqlfrm tool. The other option is to create our tables with a dummy structure using some command line magic and then replace the dummy .frm’s in our data directory with the actual .frms. I’ve used both but prefer the former, I’ll show you how to do both.

Step 1.1: Use MySQL’s command line utilities

The Python MySQL connector is required. Download it from here. Download MySQL Utilities source code by selecting “Source Code” from “Select Platform”. I downloaded the Windows distribution and unzipped it (running on a mac) From a terminal:

$ cd path/to/unzipped/mysql-utilities-1.5.6
$ python ./setup.py build
$ sudo python ./setup.py install

Use mysqlfrm to generate table creates

mysqlfrm  --diagnostic ~/folder/*.frm --quiet > ~/folder/recoverme-table-structures.sql

Set the ROW_FORMAT to compact

sed -i '' -e 's/ENGINE=InnoDB/ ENGINE=InnoDB ROW_FORMAT=COMPACT DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci/' recoverme-table-structures.sql

Step 2 - Generate dummy tables creates using .frm files

Even though we’re going to use the output from Step 1, we will use the files generated in this step to reset the tablespace and import so go ahead and follow along. This is useful when you have many tables to restore.

First we’ll get the table names from our .frm files

ls -1 *.frm > tables.sql

Prepend create table in front of each table name

awk '{print "create table " $0}' tables.sql > rtables.sql

Replace .frm and add a dummy int column with engine innodb

sed -i '' -e 's/.frm/ (id int) engine=innodb;/' rtables.sql

At this point, you should have three files: recoverme-table-structures.sql, tables.sql, rtables.sql.

Step 3 - Create Our Tables

First let’s create our database

mysql -uroot -p --host=127.0.0.1 -P32780 -e "create database dbName"

Next, run sql

mysql -uroot -p --host=127.0.0.1 -P32780 dbName < recoverme-table-structures.sql

If you were to run rtables.sql instead of recoverme-table-structures.sql, you would have your tables added with a dummy id column. At that point, you would stop mysql and then copy over your .frm files into the mysql database directory, replacing the existing .frm files. After that you would run flush tables.

Step 4 - Replace Data

Now the spot we’re in is that our internal id of our tables is different than the one in our .ibd files. We’ll manually sync our tablespaces.

Create remove-tablespace.sql to remove the existing internal ids.

cp rtables.sql remove_tablespace.sql
sed -i '' -e 's/create/alter/' remove_tablespace.sql
sed -i '' -e 's/ (id int) engine=innodb;/ discard tablespace;/' remove-tablespace.sql

Create sync-tablespace.sql to import tablespace from existing data.

cp rtables.sql sync_tablespace.sql
sed -i '' -e 's/create/alter/' sync_tablespace.sql
sed -i '' -e 's/ (id int) engine=innodb;/ import tablespace;/' sync-tablespace.sql

Run remove-tablespace.sql

mysql -uroot -p --host=127.0.0.1 -P32780 dbName < remove-tablespace.sql

Remove any existing .ibd files in our MySQL directory

cd ~/MySQL/recoverme
rm -f *.ibd

Copy .ibd files over to MySQL directory

cp ~/directoryMyRestoreFilesAre/*.ibd .

Sync tablespace

mysql -uroot -p --host=127.0.0.1 -P32780 dbName < sync-tablespace.sql

If you get error - Schema mismatch (Table has ROW_TYPE_COMPACT row format, .ibd file has ROW_TYPE_DYNAMIC row format.). Restart the process and sed ROW_FORMAT=DYNAMIC instead of ROW_FORMAT=COMPACT

Dump our database to create a backup

mysqldump recover > recoverme-structure-data.sql

Now we have a legitimate MySQL dump of our data. We can import that in with the correct database name

mysql -e "create database actualdatabase"
mysql actualdatabase < recoverme-structure-data.sql

That’s it! There are certainly more issues that can come up if the ibd files are corrupted. I encourage you to check out the Twindb github for some excellent tools on repairing ibd files and general data recovery practices.

References

Dealing with corrupted InnoDB data

Recover Orphan InnoDB Database from .ibd file

SO Recover Table Structure from ibd files

SO Repair/Recover data from MySQL database

Undrop for InnoDB Twindb CLI Tool

Using Undrop

Twindb - Recovering Corrupt MySQL database

Setting up a Serverless Project with Webpack, Babel, and Knex

Wed, 16 Dec 2020 00:00:00 +0000

Using Webpack with the Serverless Framework is handy if you want to use the latest Javascript features along with Babel. It also helps to optimize the packaging of functions so we can make sure we’re only shipping code that is lean and mean. However, adding the delightful query builder Knex to the mix can cause some issues that I spent a good amount of time on. Hopefully this article will help anyone dealing with similar issues save some time.

In this article we’ll go through setting up a Serverless project with Webpack, Babel, and Knex along with Prettier and Eslint. We’ll focus on specific issues with Knex in this scenario and how to solve them. If you want a TLDR; here’s the final output, a Serverless starter template with Webpack, Babel, and Knex ready to go.

Project Setup

Install serverless globally

npm install serverless -g

First, we’ll set up a new Serverless project using a default aws-nodejs template:

serverless create --template aws-nodejs

This will create a bare handler.js and a serverless yaml file to get us started.

Next add a package.json file to manage our dependencies

npm init -y

Add Dev Dependencies and Webpack:

We’re going to add Babel to get access to the latest Javascript features and then we’ll add Webpack to transform our Javascript code in a way that the Serverless platforms (AWS) can handle. We’ll also add Serverless-Offline which emulates AWS and AWS Gateway, allowing us to run our functions locally.

npm install --save-dev @babel/core @babel/preset-env webpack serverless-webpack serverless-offline babel-loader dotenv

Adding Source Map Support

It’s always nice to get stack traces, let’s set up source map support.

npm install source-map-support --save npm install
babel-plugin-source-map-support --save-dev

The source-map-support module provides source map support for stack traces in node via the V8 stack trace API

Babel-plugin-source-map-support prepends this statement to each file, giving us stack traces with the source-map-support package:
import 'source-map-support/register';

Setting up Babel

Create a .babelrc file in the root of the project to handle our Babel configuration:

.babelrc

{
  "plugins": ["source-map-support"],
  "presets": [
    [
      "@babel/preset-env",
      {
        "targets": {
          "node": true
        }
      }
    ]
  ]
}

Adding Knex

Next, we’ll add Knex and MySQL as the driver of choice for this purpose:

npm install --save mysql2 knex

Setting up Knex

Create a knexfile.js in the project root:

import dotenv from "dotenv";
dotenv.config({ silent: true });

module.exports = {
  development: {
    client: "mysql2",
    connection: {
      host: process.env.DB_HOST,
      user: process.env.DB_USER,
      password: process.env.DB_PW,
      database: process.env.DB_DB
    }
    // migrations: {
    // directory: './database/migrations',
    // },
    // seeds: { directory: './database/seeds' }
  },
  staging: {
    client: "mysql2",
    connection: {
      host: process.env.DB_HOST,
      user: process.env.DB_USER,
      password: process.env.DB_PW,
      database: process.env.DB_DB
    }
  },
  production: {
    client: "mysql2",
    connection: {
      host: process.env.DB_HOST,
      user: process.env.DB_USER,
      password: process.env.DB_PW,
      database: process.env.DB_DB
    }
  }
};

Create a folder called queries in your project root, this will be where the data retrieval functions will go:

mkdir queries

Add a knex file: knex.js

const knex = require("knex");

const knexfile = require("../knexfile");

const env = process.env.NODE_ENV || "development";

const configOptions = knexfile[env];

module.exports = knex(configOptions);

Example query file - games.js:

const knex = require("./knex");

export async function getAll() {
  const res = await knex("matches").select("*");
  return res;
}

Setting up Webpack

In the root of the project create a webpack.config.js file and configure Webpack to use Babel to bundle up our Serverless functions. We’ll also exclude node development dependencies using the node externals package.

npm install webpack-node-externals --save-dev

webpack.config.js:

const slsw = require("serverless-webpack");
const nodeExternals = require("webpack-node-externals");

module.exports = {
  entry: slsw.lib.entries,
  devtool: "source-map",
  // Since 'aws-sdk' is not compatible with webpack,
  // we exclude all node dependencies
  externalsPresets: { node: true },
  externals: [nodeExternals()],
  mode: slsw.lib.webpack.isLocal ? "development" : "production",
  optimization: {
    minimize: false
  },
  performance: {
    // Turn off size warnings for entry points
    hints: false
  },
  // Run babel on all .js files - skip those in node_modules
  module: {
    rules: [
      {
        test: /\.js$/,
        loader: "babel-loader",
        include: __dirname,
        exclude: /node_modules/
      }
    ]
  },
  plugins: []
};

Setting up Serverless

Add our plugins to the serverless.yaml file:

- serverless-webpack
- serverless-offline

Add serverless-webpack configuration to serverless.yaml

custom:
  webpack:
    webpackConfig: ./webpack.config.js
    includeModules: true # enable auto-packing of external modules

We’ll add an http endpoint to the default hello handler, so that we can test our api endpoint out:

events:
  - http:
    path: hello
    method: get
    cors: true

Full Serverless.yaml

service: serverless-webpack-babel-knex-starter
frameworkVersion: "2"

provider:
name: aws
runtime: nodejs12.x
apiGateway:
  shouldStartNameWithService: true

plugins:

- serverless-webpack
- serverless-offline

functions:
  hello:
    handler: handler.hello
      events:
        - http:
          path: hello
          method: get
          cors: true

custom:
  webpack:
  webpackConfig: ./webpack.config.js
  includeModules: true # enable auto-packing of external modules

Running and Knex Issues

Let’s test it out! Add a start npm script to package.json

"start": "serverless offline start --stage dev --noAuth"

Call our API

curl --location --request GET 'http://localhost:3000/dev/hello'

Knex Runtime Issues:

ES Modules may not assign module.exports or exports.*, Use ESM export syntax, instead: ./knexfile.js

It doesn’t like that we’re using module.exports in our knexfile, one potential solution would be to use es6 default export syntax export default {}

This ended up causing more problems then it solved dealing with the internal knex library which doesn’t play well with ES modules.

The solution I went for is to use a Babel plugin to transform ESM to CommonJS Modules which is the standard for Node modules. Client-side JavaScript that runs in the browser use another standard, called ES Modules or ESM. In CommonJS, we export with module.exports and import with require statements. Since we’re using Babel we can use import/export and our code will be transformed into CommonJS modules.

npm install --save-dev @babel/plugin-transform-modules-commonjs

Add to our plugins section in .babelrc

{
  "plugins": ["source-map-support", "@babel/plugin-transform-modules-commonjs"],
  "presets": [
    [
      "@babel/preset-env",
      {
        "targets": {
          "node": true
        }
      }
    ]
  ]
}

Using CommonJS should be enough to get you going but you might run into the next issue:

Can’t resolve runtime dependencies

Module not found: Error: Can't resolve 'oracledb'
Module not found: Error: Can't resolve 'pg-native'
Module not found: Error: Can't resolve 'pg-query-stream'
Module not found: Error: Can't resolve 'sqlite3'

If you receive module not found errors for packages that you are not using, then we can fix this by ignoring those drivers/packages. There are different ways this can be approached with Webpack and with Serverless but the solution that I landed on was to use the NormalModuleReplacementPlugin which is bundled with Webpack. This plugin allows you to replace resources that match a regular expression with another resource. We’ll add the noop2 package to replace the drivers we’re not using with a “no operation module”.

npm install --save-dev noop2

const { NormalModuleReplacementPlugin } = require("webpack");

plugins: [
  // Ignore knex runtime drivers that we don't use
  new NormalModuleReplacementPlugin(
    /mssql?|oracle(db)?|sqlite3|pg-(native|query)/,
    "noop2"
  )
];

Adding Eslint and Prettier

To finish this starter template, we’ll add some niceness to the project with eslint and prettier.

npm install --save-dev @babel/eslint-parser eslint eslint-config-prettier eslint-plugin-lodash eslint-plugin-prettier prettier

prettierrc.json

{
  "trailingComma": "none",
  "tabWidth": 2,
  "semi": false,
  "singleQuote": true,
  "printWidth": 120
}

.eslintrc.js

module.exports = {
  env: {
    node: true
  },
  plugins: ["prettier"],
  parser: "@babel/eslint-parser",
  parserOptions: {
    sourceType: "module",
    ecmaFeatures: {
      classes: true,
      experimentalObjectRestSpread: true
    }
  },
  extends: [
    "eslint:recommended",
    "plugin:prettier/recommended",
    "plugin:lodash/recommended"
  ],
  rules: {
    "prettier/prettier": "error"
  }
};

Starter Project

Now we have a nice starter project to get us off the ground with Serverless, Webpack, Babel, and Knex.

To grab all this goodness or if you have improvements, check out the Github repository

The Ultimate Guide to Learning Neo4j with Netflix's Dark Series

Sat, 30 May 2020 00:00:00 +0000

Netflix’s Dark is the best TV series ever created. Hands down. Dark is a science fiction mind-bender and time twister where children in the town of Winden go missing. Folks in the town start asking questions and before you know it your head is spinning through multiple different universes as time unravels. Graph databases were built to handle highly-connected and complicated data. I can’t think of better adverbs for Dark. Time and everything else is woven together, highly related and complex. Let’s learn about graph databases and explore Netflix’s Dark TV Series as we pick up some Neo4j knowledge.

This post is written just before Season 3 comes out, so we’ll be making some predictions in this post.

SPOILER ALERT - THERE WILL BE SPOILERS

Goals

To learn about Graph Databases using Neo4JS
To gain insights into the great Dark series

Neo4j (Better Data, Betters Decisions) Crash Course

Neo4j is a flavor of graph database. These types of databases are relational but they’re not a traditional relational database. In fact, traditional relational databases are not great at handling highly related data. We’ll get more into that later in this post.

Neo4j was originally built on top of MySQL, but it was redesigned from the ground up to focus specifically on optimizing graphs. It features ACID
(Atomicity, Consistency, Isolation, Durability) transactions:

Atomicity => All or nothing, if the transaction fails, it’s rolled back.
Consistency => this is debatable as Neo4j is schemaless but we’ll give it to them as the schema is in a consistent state.
Isolation => This requires that multiple transactions which are executed in parallel on the same database instance do not impact each other.
Durability => Persisted, data in Neo4j is written to disk.

Nodes, Labels, Properties, Relationships

A graph database is a collection of nodes which can be labeled, connected by relationships, both of which can have properties. A node is an object or an entity. In our Dark data, we’ll have nodes that have labels of Character, Family, Year and Sic Mundus. For example, Jonas Kahnwald is a character in Dark and a node in our graph. He is related to another node Hannah Kahnwald. This relationship is that Hannah is his parent. We have a PARENT_OF relationship. Jonas could also have a CHILD_OF relationship pointing back to Hannah. There can be multiple relationships between nodes. These relationships are what connect nodes in the graph. The id of a node is an auto incrementing integer but it can also be set manually. Properties are key value pairs where key is a string and value is a primitive. Properties can be indexed, if the property for the index is not there, the node simply is not indexed. Properties can also have constraints.

Graph databases are schemaless, meaning they are highly flexible. You can easily add properties to new nodes. Relational databases have nullable columns, but for graph databases, you just don’t add the properties. This gives us some nice document database (MongoDB) features with highly relational data.

Why not just use a traditional relational database?

Relational databases are actually quite anti-relational. There is a limit to the number of join operations that a system can effectively perform, before the “join bombs” go off and the system becomes very unresponsive. Anyone who has spent time with an RDMS knows this intuitively but it’s not clear until something better comes along to offer you the red pill. (Neo4j really brings out the Matrix references). Instead of tables like in a relational database, we have nodes. Instead of using joins, we have pattern matching. Joins decrease performance exponentially. Joins are expensive. Relations are cheap. In short, relational databases weren’t actually built for deeply related data.

In Neo4j in Action they compare Neo4j against MySQL for a social network. They show that when you have 110,000 records and need to go 3 levels deep, Neo4j takes less than a second while MySQL takes 30,000 seconds. At a certain point, say five levels deep, the relational database is not even usable.

When to Use

Complex queries: when the types of questions that you want to ask of your data are composed of a number of complex join-style operations. Basically, when you have more than 3 joins, prefer pattern matching queries.
Pathfinding queries: where you will be looking to find out how different data elements are related to each other.

Performance: as soon as you grab a starting node, the database will only explore the things that relate to it and will be completely oblivious to anything that is not connected to the starting node. This makes querying by patterns efficient.

When to not use a Graph Database:

When the complexity is low. Graph databases are meant for highly related and complex data. If you don’t have that, use a traditional relational database or NoSQL solution. You need to envision joining multiple tables together in your project. If you’re going to go 3 or more levels deep, take a look at graph databases.

Query Language

Cypher is the declarative, pattern-matching query language for Neo4j. It makes graph database management systems understandable and workable for any database user. We’ll learn some more about Cypher later in this tutorial.

Getting Started

Now that we’re acquainted, let’s get up and running by starting our own instance of Neo4j. We’ll grab the Docker image, enabled with apoc (Awesome Procedures Of Cypher) - fun fact - Apoc kills Cypher in the Matrix. Apoc isn’t going to rid our use of Cypher but it is the home to lots of cools functions and graph algorithms. Like most databases, Neo4j stores it’s data on the file system (Durability), so we will map some folders to our local system that Neo4j in Docker can use.

docker pull neo4j

docker run \
    --name neo4j \
    -p7474:7474 -p7687:7687 \
    -d \
    -v $HOME/neo4j/data:/data \
    -v $HOME/neo4j/logs:/logs \
    -v $HOME/neo4j/import:/var/lib/neo4j/import \
    -v $HOME/neo4j/plugins:/plugins \
    --env NEO4J_AUTH=neo4j/securepassword \
    --env NEO4JLABS_PLUGINS='["apoc"]' \
    --env apoc.import.file.enabled=true \
    neo4j:latest

Let’s check the logs and make sure we’re up and running

docker container logs testneo4j -f

Ports we exposed:
7474: Neo4J Browser (View our data in a local web browser)
7687: Bolt port (how we communicate with the database)

We mapped some folders: data, logs, plugins to our home/neo4j directory. We created a neo4j user with “securepassword” as the password.

Open up the Neo4j Browser at http://localhost:7474/browser/ and make sure it loads.

Setting up our Dev Environment

Now that we’ve got a Neo4j instance up and running, let’s get a couple things in order for our development environment. Here are a few tools to use when working with Neo4j and the Cypher query language. These are handy for writing Cypher but if you want to stay just in the Neo4j Browser, that will work perfectly fine as well.

VSCode/Codium:

This is an excellent Cypher extension for VSCode editors: agatlin/vscode-cypher-query-language-tools.

Vim: Syntax highlighting

Data

I created my own Dark data set which was heavily influenced by this great reddit post from season one. This Gitub repository will contain characters.json which you will need to move to $HOME/neo4j/import to be able to use the corresponding cypher import script below.

cp characters.json $HOME/neo4j/import

Open up characters.json and we’ll take a look at the data. There are several relations defined in there, this is a series about a small town in Germany so mainly we’re talking about families. People also tend to die of non-nature causes.

siblings
parents
parentOf
married
killedBy
sicMundus

Relations can also contain properties, so killedBy could have date.

The characters.json file defines character objects and the relationships above. Here is a json representation of our main character Jonas Kahnwald.

{
  "name": "Jonas Kahnwald",
  "lastname": "Kahnwald",
  "alias": ["Stranger", "Adam"],
  "years": [1921, 1953, 1986, 2019, 2052, 2085],
  "parents": ["Hannah Kanhwald", "Mikkel Nielsen"],
  "sicMundus": true
}

The Plan => Cypher Language and Importing Data

Now that we’ve had a good look at the data, let’s import it. Take a look at the import cypher file. This is the code we will run to import characters.json into the database. Once you’ve read the below comments, run the file in the Neo4j Browser.

We’ll start out by creating some constraints for our character and year node. We’re indexing families and characters on name. While, we won’t run into performance issues with this small of a data set, it is good practice to get into the habit.

CREATE constraint on (c:Character) assert c.name is unique;
CREATE constraint on (y:Year) assert y.id is unique;
CREATE constraint on (s:SicMundus) assert s.name is unique
CREATE index on :Character(name);
CREATE index on :Family(name);

Next, we’ll manually create some nodes for years and family. We could have done this using the json but I just wanted to get up and running:

CREATE (`1921`:`Year` {id: 1921}), (`1953`:`Year` {id: 1953}),
...

Enter apoc. Now we’ll use an Awesome Procedure on Cypher function to import our character.json file. We reference it by the word value, then we’ll UNWIND the array of characters, basically iterating through each character object in the json. We’ll use another apoc function to clean up any empty or nulls. From there we create a new Character node with a name and then attach the rest of the properties.

CALL apoc.load.json("file:///characters.json") YIELD value
UNWIND value.characters AS character
WITH apoc.map.clean(character, [],['',[''],[],null]) as data
MERGE (c:Character {name:data.name})
SET
c += apoc.map.clean(data, [],['',[''],[],null])

Now onto setting up relationships. Mapping our array relationships is fairly straight forward. We’ll loop through each relationship property and add the appropriate relationship. We’re using MERGE which will create a node or use an existing. Remember, we have a unique constraint on character name. Notice how the relationships can go either way.

FOREACH (name in data.married | MERGE (m:Character {name:name}) MERGE (m)-[:MARRIED]->(c))
FOREACH (name in data.parentOf | MERGE (m:Character {name:name}) MERGE (c)-[:PARENT_OF]<-(m))
FOREACH (name in data.siblings | MERGE (m:Character {name:name}) MERGE (m)-[:SIBLING_OF]->(c))
FOREACH (name in data.killedBy | MERGE (m:Character {name:name}) MERGE (c)-[:KILLED_BY]<-(m))
FOREACH (id in data.years | MERGE (y:Year {id:id}) MERGE (c)-[:WHEN]->(y))

We have a couple properties to relate to nodes, Sic Mundus and Family which are just primitives (boolean and string). We use case to test if the property is null and then empty it out before once again using MERGE to create our nodes and relationships.

FOREACH (name in case data.lastname when null then [] else [data.lastname] end | MERGE (f:Family {name:name}) MERGE (c)-[:FAMILY]->(f))
FOREACH (name in case data.sicMundus when null then [] else [data.sicMundus] end | MERGE (s:SicMundus) MERGE (c)-[:SIC_MUNDUS_MEMBER]->(s))

Perfect, now let’s view entire graph in the Neo4J Browser

MATCH (n) RETURN n

Basic Cypher Quick Hits

Now that we’ve got a populated Neo4j database and an introduction to Cypher, let’s dive a little deeper on this steak loving declarative querying language.

CREATE: This creates nodes and relationships with their properties.

CREATE (`Jonas Kahnwald`:`Character`{ name: 'Jonas Kahnwald' })-[:CHILD_OF]->(`Hannah Kahnwald`:`Character`{ name: 'Hannah Kahnwald' })

We’ve created two nodes Jonas and Hannah and added a relationship of CHILD_OF. Relationships can go both ways or we could reverse it like this:

CREATE (`Jonas Kahnwald`:`Character`{ name: 'Jonas Kahnwald' })<-[:PARENT_OF]-(`Hannah Kahnwald`:`Character` {name: 'Hannah Kahnwald'})

MATCH: basically like a SQL select, it describes a pattern that should match in the database.

WHERE: pretty much like SQL, filters the results.

A difference between SQL and Cypher is that in the latter, you return the data that you want declaratively. As we saw above, this Cypher will return the entire database. Match all nodes and then return them.

MATCH (n) RETURN n

We can query for relationships as well, this will look at all the Years when Magnus Nielsen has been present. We are just returning the nodes here but we could return properties such as c.name, y.id, etc.

MATCH(c:Character)-[r:WHEN]->(y:Year) where c.name = 'Magnus Nielsen' return c, y

We could make a character, not a character by removing a label:

MATCH (c:Character)
WHERE c.name = 'Magnus Nielsen'
REMOVE c:Character
RETURN c ;

Remove a node:

MATCH (c.Character, {name: 'Magnus Nielsen'})
DELETE c

Use detach delete to remove relations and then delete node:

 MATCH (c.Character {name: 'Magnus Nielsen'})
 DETACH DELETE c;

WITH: This is a handy keyword which passes results from one query part to the next. WITH is like RETURN; it allows you to separate the query in parts. You’ll see this in action in the next section.

MERGE: As mentioned above, this is like CREATE but it won’t fail if the node or relationship already exists.
Contrived example of WITH and MERGE

MATCH (h: {name: "Hannah Kahnwald"}) as hannah
WITH MATCH(j:User {name: "Jonas Kahnwald"}) jonas
MERGE (hannah)-[:PARENT_OF]->(jonas)

Questions and Findings

Now we have more than enough Neo4j knowledge to be dangerous. Let’s get to the fun part, finding some patterns in our dataset! When creating a graph database, it’s good to know what kind of questions you will want to ask the data. Here is a list of questions and queries:

Who is the most mysterious (least amount of relations)?

MATCH(c:Character)-[r]-(x) RETURN c.name, COUNT(*) as relations
ORDER BY relations limit 5

Bartosz Tiedeman and Clausen each come in with 2 nodes. Yasin Friese only has one, however Clausen and Bartosz are likely key characters who we will find out more about in Season 3.

Are there any characters not related to a family?

MATCH (c:Character) WHERE NOT (c)-[:FAMILY]->() RETURN c

Nope, every character has a family.
A better question,
Which family has the most members?

MATCH(c:Character)-[r:FAMILY]-(f:Family) RETURN count(c) as cnt, f.name order by cnt desc

Despite Jonas being the main character, the Nielsen’s have the most known family members at 9. While the Kahnwalds sit at 4, not including Mikkel.
The Doppler’s come in at 7.

Which “family” has only one member?

MATCH(c:Character)-[r:FAMILY]-(f:Family)
WITH count(c) as cnt, f
WHERE cnt = 1
RETURN f

What are the most active time segments?

MATCH(c:Character)-[r:WHEN]->(y:Year) RETURN COUNT(c.name) as cname, y.id

2019 is obviously the most when’d timeverse as that’s where Dark starts. As the timeverses edge out, we know less characters who have traveled there.

Who has gone the farthest in time?

MATCH(c:Character)-[r:WHEN]->(y:Year) RETURN c.name, count(r) as whens ORDER BY whens desc

Jonas is the only confirmed character to have traveled to 2085. We will probably learn more about that year in Season 3. He has also seen the most timeverses.

Which characters are in Sic Mundus?

MATCH(c:Character)-[:SIC_MUNDUS_MEMBER]->(s:SicMundus) RETURN c

Now for our most ambitious query of this post, who are the likely other Sic Mundus members?

This type of query can give you a hint to what Neo4j can do in terms of predictions or recommendations out of the box. Notice how we can specify which relationships we’re looking for with -> and -

MATCH(c:Character)-[:SIC_MUNDUS_MEMBER]->(s:SicMundus)
WITH c
MATCH(c)-[r]-(x:Character)
WHERE not (x)-[:SIC_MUNDUS_MEMBER]->() AND not (x)-[:KILLED_BY]->()
RETURN x.name, count(r) as cnt
ORDER BY cnt desc

“Martha Nielsen” 2
“Elisabeth Doppler” 2
“Charlotte Doppler” 2

Martha, Elisabeth, and Charlotte all seem likely Sic Mundus candidates. This query could be much higher quality if we had even more data points to look at.

Which characters have only been to one timeverse?

MATCH(c:Character)-[r:WHEN]->(y:Year) WITH c.name as name, count(r) as whens WHERE whens = 1 RETURN name

“Greta Doppler”
“Doris Tiedeman”
“Daniel Kahnwald”
“Mads Nielsen”
“Torben Woller”
“Erik Obendorf”
“Bartosz Tiedeman”
“Ulla Obendorf”
“Peter Doppler”
“Yasin Friese”
“Jurgen Obendorf”
“Clausen”

Most characters don’t skip time frames, if you’ve been to 1986, you’ve likely been to 1953 or 2019. However, some characters have probably warped around the 33 years time frames, let’s find them. There are a few new functions to explain:
collect will turn a group of nodes or properties into an array.
Size is like a .length or .count - it gets the size of the array.
Reduce lets you do some iteration over a collection.

MATCH(c:Character)-[r:WHEN]->(y:Year)
WITH c as character, y
ORDER by y.id
WITH character, reverse(collect(y.id)) as years
WITH character, years, range(0, size(years)-1) AS is
WITH character, years, is, years + (last(years) - 33) as isa
WITH character, years, reduce(total=0, n in is | (total + toInteger((isa[n] - isa[n+1]) / 33 - 1))) as sum
WHERE sum > 0
RETURN character.name, years

“Franziska Doppler” [2019, 1921]
“Magnus Nielsen” [2052, 2019, 1921]

I think it’s a good bet that we see a lot more of Magnus and Franziska in Season 3 as most characters don’t skip around the time frames.

Which characters have never been to the past?

MATCH(c:Character)-[r:WHEN]->(y:Year)
WITH c as character, MIN(y.id) as my
WHERE my >= 2019
RETURN character.name

Bartosz is questionable - there is a theory that Noah killed him in back in 1921. Bartosz as noted above is one of the more mysterious characters and there are rumors that he is also a Sic Mundus member. Will these characters travel further back in season 3?
“Torben Woller”
“Erik Obendorf”
“Bartosz Tiedeman”
“Ulla Obendorf”
“Peter Doppler”
“Martha Nielsen”
“Elisabeth Doppler”
“Yasin Friese”
“Jurgen Obendorf”
“Clausen”

Which characters are the most deadly?
<> means not equals or does not match.

MATCH(c:Character)-[r:KILLED_BY]->(k:Character)
WHERE c.name <> k.name
RETURN c.name, k.name

Noah has killed Claudia Tiedeman, Mads Nielsen. You could probably add Yasin to this list as well and possibly Bartosz.
Agnes Nielsen killed Noah.
The Nielsen family is the most deadly. What does that mean for Season 3?

What percentage of deaths in Dark have been suicides?

MATCH(c:Character)-[r:KILLED_BY]->(k:Character)
WHERE c.name <> k.name
WITH count(c) as cnt
MATCH(s:Character)-[r:KILLED_BY]->(sk:Character)
WHERE s.name = sk.name
RETURN toFloat(count(s)) / (count(s) + cnt)

40%

Conclusion

Hopefully this post has helped you get up to speed with Neo4j using data from Netflix’s Dark Series. If you noticed while running the queries, all of them take under 10ms. Crazy fast. There are definitely ways to improve the Dark dataset. For instance, we could have a relationship of APPEARS_IN with each season being a node. There could also be an INTERACTS_WITH relationship when characters interact with each other and a count on that relationship. These queries may not be the most efficient to express our questions.

What do you think? Have better ways to create the data model? Got some cool queries to run? Want to add to the Dark dataset? Checkout the Github repo.

James Ingold

One Git, Many Hats = How I Manage Multiple Git Identities

Global Git Configuration

Project-Specific Git Config

Automate Git Identity Selection

SSH Configuration

Update Project Specific with URL Overwrite

Full Project-Specific Root .gitconfig

Split Large Files into Lines with NodeJS

Handling Large Files

Option 1: buffer-split package

Option 2: inter-ops package

Option 3: Roll Your Own

References:

Using Binary Files in AWS Lambda

Deconstructing an Object Relationship Mapper (ORM) in Typescript

Deconstructing a Typescript ORM - Mapping Objects - Part 1

What is an ORM?

Pros

Cons

The Situation

The Vision

General Principles - Connecting your Typescript Classes to the Database

Decorators

Model(identifier: string, alias?: string)

Column(name: string)

Enum(name: string, () => Dictionary)

HasMany(modelName: string, relationKey?: string)

BaseDomainModel

Conclusion

References:

Amazon Cognito MFA with Email Using Lambda Triggers

Part 1: Create Serverless Project for Cognito Lifecycle Hooks

Define Lifecycle Function

Create Lifecyle Function

Verify Lifecycle Function

Deploy the Project

Part 2: Pseudo Backend Code

Login Route: Kick off auth flow and prepare custom challenge

Verify Route: This handles responding to our auth challenge with Cognito which will call our verify lifecycle handler.

Conclusion

References:

Automating Route53 DNS Updates for Blue/Green Deployments (the hard way)

Blue/Green Architecture

Python Script for Updating Route53 Entries

Bonus: Shell script to Pass Domains to Update

The End

References:

Migrating from Sequel Pro to Sequel Ace

Install Sequel Ace

Copy Preferences and Favorites

Copy Passwords (optional)

Uninstall Sequel Pro

Wrapping Up

References:

Restoring a MySQL Database from .frm and .ibd Files on MacOS

Restore Process High-Level:

Let’s Get Started

Step 1: Create Files to Reconstruct Table Structure

Step 1.1: Use MySQL’s command line utilities

Use mysqlfrm to generate table creates

Step 2 - Generate dummy tables creates using .frm files

Step 3 - Create Our Tables

Step 4 - Replace Data

References

Setting up a Serverless Project with Webpack, Babel, and Knex

Project Setup

Add Dev Dependencies and Webpack:

Adding Source Map Support

Setting up Babel

Adding Knex

Setting up Knex

Setting up Webpack

Setting up Serverless

Running and Knex Issues

Knex Runtime Issues:

Adding Eslint and Prettier

Starter Project

The Ultimate Guide to Learning Neo4j with Netflix's Dark Series

Goals