Project Citadel Devlog: #1

February 21st, 2020

I'm a big believer in side projects for those who are in the tech industry and need to learn something new.

Everytime I've needed to learn a new technology or programming language, I always made the most progress when I worked on an actual project.

When I started learning Golang, the sideproject I chose to go with was Chrono. It was a simple time tracking CLI program.

It gave me a lens through which I learned Golang as well as the best practices surrounding it.

While I loved working on Chrono, it was a tool that I built to use at my work. When I was actually using it at work, development on it was pretty active. Sadly, my work switched to a different time tracking system and I no longer was able to use Chrono. Development after that pretty much flat-lined.

After chrono, my next side project became my blog. I chose Gatsby as my site generator which uses the React framework for writing the layout of the site. I'd used React in the past and was really impressed, but using it for my blog made me enjoy it even more.

So for my next side project - I defiently wanted to include at least two technologies: Golang & React.

I also didn't want to make the same mistake I had made in past (with Chrono) by choosing a sideproject that was only usable at work. I wanted something that I'd use in my day-to-day personal life.

So I chose to go with a project management system (because there aren't enough of those as it is!).

Project Citadel (a tempory* name for it) was born.

This is a recap of the first two weeks of development.

*Probably

Design

I knew some of the base features I wanted to include with Citadel.

At the minimal I wanted Kanban boards to be a part of the system. I've used Trello for several years and I really like their product - however there are things I wasn't totally happy with.

Some of things I wanted to include in Project Citadel were:

  • I wanted a better way to organize boards
  • A nicer UI (this is more of a subjective preference, but hey it's my project)
  • Multiple boards for a project
  • A timeline view (Ghantt chart)
  • Better calender [TODO]

There are many other features I'd like to add - however the above list is my current goal.

One issue with sideprojects is that they can drag on forever without being "done." Especially if you're like me and contiuelly expand the scope of the project.

By keeping the list of features for an MVP (Minimum Viable Product) will hopefully prevent the above issue.

Architecture

Citadel is split into two components: a SPA web client written in React & a graphQL backend written in Golang.

Things I learned

Below is a list of technologes or concepts I learned during the first two weeks of development. Please note that while I tried to be thourogh while researching, it's possible I could be mistaken about something or there are better alternatives.

React & React setup

One of the React team's core philosphy on development is UI before API. With this in mind, I chose to start with the React portion of the project first (at least to start with). While I had good idea of what I wanted to build, the actual domain models for the project was something I wasn't sure of.

Because I was going to be creating a Kanban somewhere within this application, I looked into what the best drag & drop (dnd) libraries are for React. The main two seem to be react-dnd and react-beautiful-dnd.

The former seems to be a fairly low-level which wasn't something I needed at this point. So I chose to go with react-beautiful-dnd by Atalassian. I setup a Create-React-App project and installed the library. After several days of fiddling with the library, I had a pretty good idea of how it worked as well as mapped out some of the data models I would need.

Styled Components & CSS-in-JS

Styled components is a library that allows you to write CSS within Javascript as if they were React components. For example, to style a h2 tag:

import styled from 'styled-components';

const Title = styled.h2`
  font-size: 24px;
  color: red;
`

Styled components uses template literals to write the CSS. You can also pass in props to change the styles based on data.

import styled from 'styled-components';

const Title = styled.h2`
  font-size: 24px;
  color: ${props => props.active ? 'red' : 'white'};
`

One of the issues that I came across while using styled components is that when you view the component tree using React Devtools, the display names are mangled & it's hard to tell what a component really is.

Luckily, styled component provides a solution to this through a babel macro.

Since CRA comes with babel macros by default, all the was needed was to change the import path used for styled components from styled-components to styled-components/macro.

Project structure refactor

At this point I had quickly made a mess of the initial Create React App's project structure, so the next step became looking into the best practices surrounding React's project structure. The one I decided to go with is based on a project called jira_clone. TODO

The new structure looks like this:

src/
  App/ # Any code that is considered global the application goes here
  Project/ # An example of a Module
  shared/ # any code that is shared between modules goes here
    components/ # re-usable components go here
    hooks/ # re-usable hooks go here
    utils/ # helper functions go here

The jira_clone also came with it's own webpack config so I chose to use the entire project as the boilerplate for the new structure.

The new project structure also favored using functional components instead of class based components.

In previous versions of React (the ones I began with), components were created the following way:

import React from 'react';

class Title extends React.Component {
  render() {
    return (
      <h1>Hello</h1>
    );
  }
}

export default Title;

There were several life cycle methods you could implement such as componentDidMount() which was called once when the component was created & rendered.

However with later releases of React, functional components & hooks were introduced.

Now the above component can be written this way:

import React from 'react';

const Title = () => (
  <h1>Hello</h1>
);

export default Title;

Instead of the lifecycle methods that were used previosly, you would use hooks instead.

For example, if you wanted to replicate the functionality of componentDidMount, you can use the useEffect hook like so.

import React, {useEffect} from 'react';

const Title = () => {
  useEffect( () => {
    console.log('mounted!');
  }, []):
  return (
    <h1>Hello</h1>
  );
}

The above component would emit "mounted!" to the console when the component was mounted. The second argument to the useEffect method is what accomplishes this. Without the empty array, the console.log would be called every render. TODO: add explanation

With the new structure in place, I began moving everything over the new structure and format. Since I had to rewrite my components as functional components, I cleaned them as I refactored.

Storybook

After I had migrated my first component, I remembered that a library called Storybook existed. It brands itself as an isolated environment to develop components. I had tried adding it to my Gatsby blog but could never get it to work, so I wasn't sure how well adding it to this project was going to go.

Luckily it was super easy. Using the sb init tool provided by storybook, I was up and running in minutes.

Now, as I rewrote the components in the new format, I did so in Storybook rather than the normal React environment.

It made things go much smoother as I could develop each component in isolation. Because I had to develop in isolation I found a fair amount of coupling that was hidden.

Storybook also comes with several extensions such as actions which can be used to connect component actions to the storybook interface.

Storing orderable lists within a database

Along with the hidden coupling, I also ran into the issue of how I was going to store orderable lists within a database. In the react-beautiful-dnd tutorial, the order of the tasks are stored using an array.

While that work well, it breaks down when you introduce the need to store the order outside of the React context.

While researching the different ways to store orderable data, I came across this article: TODO.

It gave three approaches to storing ordable data. Out of the, I chose to go with #2 - using floats to store position data. This is the same approach I believe Trello uses (going by the database returned by its API calls).

One of the main issues with the using floats is that after so many re-orders, the position float turns back into an integer - which breaks the ordering. I haven't come up which a good solution to this issue yet - but that's future Jordan's problem.

GraphQL and Golang

If you've never heard about GraphQL before, you're missing out. Normally, most SPA using an API use a REST based API.

In REST, resources are accessed using URI endpoints. For example, if you wanted to access users you might use the following URL:

http GET http://example.org/users
{
  "users": [
    {
      "username": "jordan",
    }
  ]
}

Which would return a JSON encoded lists of users.

If you wanted to create a new user, you would do a POST request to the same endpoint which the body of the request containing the new user's data.

While this approach works, there are many problems that come with it.

  1. The client has very little control of the data that is returned
  2. Documentation for REST API's typically varies wildly in quality

GraphQL solves the above two problems by exposing a single endpoint that exposes all of the data for an application. Users of the API can write GraphQL queries to specify exactly what information they want.

GraphQL also comes with a built-in documentation for what queries can be performed.

There are three main Golang GraphQL libraries that I saw.

  • graphql-go/graphql
  • graph-gophers/graphql-go
  • gqlgen

The first two seemed fairly popular however gqlgen had the most features and the easiest API to interact with so that's the library I chose to go with.

It worked by taking in a schema file and generating the nessessary resolvers and server code to implement a graphql server.

For the Go router, I chose to go with Chi - as it's lightweight and compatible with standard net/http middleware.

Golang & Database access

Now that I had graphQL setup, I needed to implement the resolvers. It was time to add a database. I'm a big fan of postgres so that's what I chose to go with. Using Docker, I spun up a development database and created some simple tables.

Now that I had a database, I needed to decide how I was going to access it within Golang.

A common solution to this problem is what is called an ORM (object relational mapping). ORMs typically abstract the raw SQL used to communicate with a database and automagically map results returned into objects (or an appropriate datastructure).

I've used ORMs in previous projects and always inevitably fought with it when I tried to do something beyond a basic CRUD application.

So for this project I decided against going with an ORM solution like GORM to handle database access.

Instead I decided to go with sqlx which is an additional helper layer over the standard database/sql library that Golang provides.

For complex queries, it's a much better developer experience than fighting with an ORM however there is a drawback in that the basic queries require a deal of inane boilerplate code.

One solution I'm looking into for handling that is using a library like sqlc or xo to generate Golang code from SQL queries.

I tinkered with sqlc some but I was having some difficulties getting it to work. That will be something I investigate further during the next week's development sprint.

Database Migration

Database migration is the process of dealing with evolving database schemas. Most ORM's typicaly include some sort of migration tool as part of the package, but since I'm not using an ORM - it's a feature I don't have access to out of the box.

Since I was using a helper library where I wrote raw SQL, I wanted my migration tool to allow me to do the same. What I found was go-migrate, a project that ships as both a CLI & library. I could write my migration files as sql files and it would manage applying them to a database.

API Authentication

Since the API I was writing was not meant to be publicly accessible, it meant I needed to find a good way to deal with authentication. After researching the various methods, I decided to go with a refresh + access token system.

The refresh token is a opaque token that is stored in the database with a long expiration date. It's used to request an access token from that api periodically as the access token is a JWT token with a very short expiration date.

The JWT tokens containing all the neccessary data to validate a user for normal authentication without having to make a database call. The downside of not making a database call is that the access tokens can't be revoked once they're issued. That's where the refresh token comes in.

Because the access token has such a short lifespan, the refresh token can be revoked instead. If the token is revoked, then the next time the access token is recycled - the user would be locked out.

While the revoktion is not instant, it's a compromise between keep database calls down & keeping the ability to revoke tokens.

To prevent several types of attacks, the access token is only kept in memory. The refresh token is stored as an httpOnly cookie, which means it's unaccessible through scripting.

React SPA authentiation & Apollo Client

Because the access token will almost defiently be expire during a user's session, a silent refresh flow needed to be added.

The first part of the refresh flow is when a user refresh the page.

Because the access token is stored in memory, when the page is refreshed, the token is lost. So during the React context setup, it tries to get a new access token.

If it succeceds, then the new token is saved back into memory and the application executes normally.

If it doesn't, then the user is redirected to the login page.

The second part of the flow is dealing when the token expires during the course of normal interation with the API.

Since I'm using Apollo client to handle communication with the graphql server, I added some extra checks to the client.

If an 401 error occurs on a request, then it will try to refresh the access token then try again.

If it can't, then the user would be redirect back to the login page.

Password Storage

To finish off the authentication system, I needed to implement user authentication. Previously I was just using a hard coded password but in a production system that would not be viable.

Since I would need to store the passwords for my users, I looked into best practices for storing them within a database.

I knew some of the basic ideas of password storage, they needed to be salted & hashed (like a potato) before being inserted into a table. I wasn't sure of which hashing algorithm to go with however.

Researching this led me to OWASP's password storage recommendations.

OWASP recommends using modern hashing algorithms such as bcrypt or argon2. They include salting built in.

They also recommend adding a pepper to passwords as well.

Typescript & React

At this point, I decided to introduce Typescript to my project. I used it a little in the past and have heard good things about it. Since I was already using the tsserver for javascript code completion in Neovim, switching to it would only make completion even better.

However I ran into some issues with the webpack configuration that came with the jira_clone project, so I decided to migrate back to the CRA template.

So I set up a CRA project with the --typescript flag enabled. I added an eslint, prettier and editorconfig configuration file to the project as well. One thing I really liked about the jira_clone project was the ability to absolute import components instead of having to do relative ones.

So I modified the tsconfig file that came with the CRA project to allow for absolute imports.