Project Citadel Devlog #2

July 18th, 2020

This is the second entry in the Project Citadel devlog. You can find the first one here.

It's been about four months since the last devlog however there has been quite a lot of progress!

Some of the major things I've learned so far:

  • Typescript is both amazing and annoying as hell.
  • Styled components is amazing until you have to work with a library component that doesn't work well with it (I'm looking at you React Datepicker).
  • Project structure is important (obvious, but even more so once some files get over 1k LOC)
  • Polishing features often takes twice the time the base feature takes to implement.
  • Embedding and serving React frontend in Golang binary
  • My preferred React folder structure
  • Breaking up GraphQL schema files in gqlgen
  • Managing apollo cache

Typescript & you

Typescript is great at helping catch bugs and mismatched types. As long as it works as expected. However once you get into non basic TS, things get a little weird...

One of the biggest issues I had with Typescript was getting it to work well with React refs.

Because of how Citadel works, I use refs in a MANY places. I also need to forward them from one component to a child component. What does that look like?

One of the biggest places I used forwarding refs in Citadel is the Card component.

A simple React component in Typescript would look like this

type CardProps = {
  title: string;
}
const Card: React.FC<CardProps> = ({title}) => {
  return (<div>{title}</div>);
}

If you need to forward a ref to a component, you can use the React.forwardRef functions. The issue then is how do I type my props? Also you need to type the ref variable itself (because otherwise Typescript complains).

After a lot of trial and error I eventually had to just do this

type CardProps = {
  title: string;
}
const Card = React.forwardRef(({title}: CardProps, $ref: any) => {
  return (<div>{title}</div>);
})

It's not great but it's the only way I've found to actually get it to semi work.

The other issue I've had with Typescript is dealing with types from other libraries.

Sometimes I need to use their types in my own types.

The issue is that most don't export their types or it's buried so I have to dig around to find it.

Embedding React in single Golang binary

One of the things that I wanted to accomplish with Citadel is for it to be easily deployable - part of that meant that the final binary should have everything it needs embed in it! This includes the built version of the React frontend.

While there are many libraries out there that can handle this, some are nicer than others. The final two that I debated on using either this fork of go-bindata vs vfsgen.

While I've seen go-bindata used in a lot of projects successfully, it had a custom API for interacting with embedded assets.

I ultimately decided on going with vfsgen since it took a different approach by embededing an entire virtual filesystem http.FileSystem (which works well with serving it with Chi).

I added a Magefile target to convert a built version of the frontend into a Go source code file.

Now I needed to implement actually serving the virtual filesystem in a way that made React Router happy.

One of the popular Go HTTP routers Mux has an article on how to serve a React Router SPA. However, I'm using Chi so I had to modify the SPA handler to work with Chi as well as a virtual filesystem. The current final implementation looks something like this:

type FrontendHandler struct {
	indexPath  string
}

// IsDir checks if the given file is a directory. If an error occurs, false is returned.
func IsDir(f http.File) bool {
	fi, err := f.Stat()
	if err != nil {
		return false
	}
	return fi.IsDir()
}

func (h FrontendHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	path, err := filepath.Abs(r.URL.Path) // Get the absolute path of the requested file
	if err != nil {
		http.Error(w, err.Error(), http.StatusBadRequest)
		return
	}

	f, err := frontend.Frontend.Open(path) // Try to access the requested file on the virtual file system
	if os.IsNotExist(err) || IsDir(f) { // Does not exist, serve the index file
		index, err := frontend.Frontend.Open("index.html")
		if err != nil {
			http.Error(w, err.Error(), http.StatusInternalServerError)
			return
		}
		http.ServeContent(w, r, "index.html", time.Now(), index)
		return
	} else if err != nil { // Something went wrong!
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}
	http.ServeContent(w, r, path, time.Now(), f) // The file does exist, serve it instead of the index
}

Now I have an embedded version of my React frontend and I can serve it through Chi!

Golang project folder structure

While I knew how I wanted to structure the React component of the project going in, I was less sure on how to do the same for the Golang component.

At the start of the project I just put everything in a route package and that worked okay while the project stayed small.

Once it started getting larger, I knew I needed to refactor the structure & I'm fairly happy with the layout I chose to go with.

It looks something like this:

ls
cmd/citadel/main.go # Main entry point
frontend/ # React frontend code
internal/ # All Golang code goes
  auth/ # Code related to dealing with tokens & passwords
  route/ # Handler & middleware code
  db/ # generated db code
    queries/ # sqlc query files
  graph/ # GraphQL resolver code
    schema/ # schema files go here
  logger/ # Custom logger implementation for Chi
  commands/ # Cobra command implementation (web, migrate, etc)
migrations/ # golang migrate sql migration files

Breaking up single large GraphQL schema file

One thing I noticed as I expanded my GraphQL API is that the schema file was getting quite large.

Now gqlgen allows you to have separate schema files so I could do something like this

users.graphql # has user related schema declarations
projects.graphql # has project related schema declarations

But there is one major drawback with that approach. You can't shared types between the schema files.

Which didn't work for me. But I still needed to break up the schema somehow. Scripting to the rescue!

I created a new directory internal/graph/schema and broke apart all my schemas into separate files.

I had a _root.gql file that had an empty (I'll explain why soon) Mutation type and as well as my Query type.

Next I had a _models.gql file that contained all my base types. Then for each domain type (users, projects, tasks, etc), I created a file dedicated to storing their mutation queries & input/payload types.

One nice thing you can do with GraphQL is extend a type. The reason the _root.gql file had an empty mutation was so that I could extend the type in each dedicated domain schema file.

# _root.gql
type Mutation

# _example_domain.gql
extend type Mutation {
  createExampleDomain(input: CreateExampleDomain!): CreateExampleDomainPayload!
}

input CreateExampleDomain {
  id: UUID!
}

type CreateExampleDomainPayload {exampleDomain: ExampleDomain!
}

Then I added a Magefile target to combine all my schema files in a specific order into a single file schema.graphql which gqlgen then used to generate the resolvers.

Now I had the best of both worlds, I could split up my schemas and still have gqlgen work.

Managing Apollo Cache

One of the nice features Apollo has is that it will automagically update the data for data based on the ID. For example if I get a Project object through a query and then later update it through a mutation, as long as the __typename & id fields match, the new data will be updated and any component that uses the data will be re-rendered.

The above works just fine if all you're doing is updating objects, but once you start creating or deleting them with Apollo, the automagic can no longer help you.

Now you have to deal directly with Apollo's cache - which while not hard to do, does get very repetitive. So I wrote a custom function that handles reading and write the cache, while also making sure all of the data is correctly typed for Typescript. It looks something like this:

import { DataProxy } from '@apollo/client';
import { DocumentNode } from 'graphql';

type UpdateCacheFn<T> = (cache: T) => T;

export function updateApolloCache<T>(
  client: DataProxy,
  document: DocumentNode,
  update: UpdateCacheFn<T>,
  variables?: object,
) {
  let queryArgs: DataProxy.Query<any>;
  if (variables) {
    queryArgs = {
      query: document,
      variables,
    };
  } else {
    queryArgs = {
      query: document,
    };
  }
  const cache: T | null = client.readQuery(queryArgs);
  if (cache) {
    const newCache = update(cache);
    client.writeQuery({
      ...queryArgs,
      data: newCache,
    });
  }
}

export default updateApolloCache;

I usually use it in conjunction with immer, a JS library for dealing with immutable operations. So now when I do a mutation that adds an object, the update field on the mutation config looks something like this:

const [deleteTeamMember] = useDeleteTeamMemberMutation({
  update: (client, response) => {
    updateApolloCache<GetTeamQuery>(
      client,
      GetTeamDocument,
      cache =>
        produce(cache, draftCache => {
          draftCache.findTeam.members = cache.findTeam.members.filter(
            member => member.id !== response.data.deleteTeamMember.userID,
          );
        }),
      { teamID },
    );
  },
});

It makes dealing with the Apollo cache much easier!

Next steps

While this isn't a comprehensive list of everything I've learned, it is everything I thought was interesting and wanted written down somewhere for the future.

The next steps for Citadel is to continue working on polishing the already implemented features & add the calendar tab.