Migrating and Modernizing the high-traffic website

The Company

Client with US Govt Project, USA

The Project

The client has a US Government Project which is a high-traffic public transportation website. This website receives over 70k sessions per month. The goal of this website is to provide the correct information in as little time as possible. The most used features included Live Map, Schedules, Notices, and Downloadable route files.

Tech Stack
  • ReactJS

  • JavaScript

  • GatsbyJS

  • Material-UI

  • nginx

The Challenges
  • The client struggled to release new changes such as Notices, and News. This was because for every new notice, they had to deploy the entire website.

  • The code, data, and styles for the webpages were mixed into the single HTML file. This coupling increased the repository size into multiple gigabytes. This resulted in increased git checkout times before a developer can work on the project, and also increased hosting costs.

  • The development and release model were working from a single master branch. This caused multiple challenges such as

    • The team cannot work in parallel on multiple features.

    • There was no way to test the current deployment to the new upcoming release.

  • Any regressions in the latest deployment halted the development efforts for the next release.

  • Every webpage on the website was duplicate content copied to match the header, footer, styles, and structure with the other webpages. This presented huge challenges in releasing new features because the code needs to duplicate among other pages as well. There were more than 150 webpages like these.

  • The team was using WYSIWYG tools to develop and maintain the website, and lacked the skills to develop website using modern JavaScript tools and frameworks.

The Approach
  • Reviewed the codebase, software release lifecycle, and prepared the action plan for the team. This involved finding the right technology, release patterns to meet the needs of the client. The choice in this case was to use GatsbyJS (see the entire tech stack below).

  • Ran Lighthouse audits on the website to find out the problems with the latest deployment.

  • Analyze the website traffic and identified the key areas which require care and will also bring the impact once the fixes are deployed. This helped to prioritized the work for the upcoming releases.

  • Trained the team on JavaScript, React and GatsbyJS. This involved presentations, hand-on coding sessions, quizzes and creating apps to acquire necessary skills for the project.

  • Developed the interface inventory with the team. Also created Components using Material-UI, React and Storybook.

  • Developed solutions to unknown problems such as embedding the custom fonts using Gatsby and Material-UI.

  • Developed solution to separate the code from the data. Using GatsbyJS and GraphQL the data was kept in separate files such as JSON, CSV, each suited for a specific use case.

  • Implemented a git branching model to enable parallel development, scheduled and hot-fix releases.

  • Developed a strategy where Notices could be fetched using remote datastore. The idea was to display new Notices without making a new website release. This saved a lot of time for the engineering team because the Project Management Teams can use an internal interface to post new Notices to remote datastore. On the website, these new Notices could be fetch using HTTP API calls and display to the users.

  • Developed migration plan so that existing features could be deployed using newer stack transparently without users knowing about it. This involved deploying 2 versions of the website (existing stack, and new stack) and use load-balancer to serve the traffic based on URL patterns.

  • Addressed the issues reported in the Lighthouse Audit reports.

Open-Source Contribution
  • The open-source starter version for GatsbyJS, Material-UI, and Storybook is available at GitHub under the MIT license.

  • The solution to embedding custom fonts using GatsbyJS and Material-UI is open-sourced and hosted on GitHub under MIT License as a way to give back to the community.

The Impact
  • The team was able to learn, contribute, and deploy using the new stack in less than 3 months.

  • The team started following code review process to review changes for their peers.

  • The team was able to progressively migrate features to newer stack, therefore avoiding the big-bang rewrite. This helped the client to maximize the ROI of the engineer's time. Now, they are in position to push multiple releases a day depending on the nature of change.

  • The team now has a reproducible plan to work and evolve the codebase in an organized manner.

  • The team uses modern development stack based on open-source technologies. This has helped them leverage the work done by the communities and developers around the world and not re-invent the wheel for every single use-case.

  • The team takes a data-driven approach to decide what features should be prioritized, worked upon and measure the impact by comparing it with previous releases.

  • The client has transparent, predictable system and process in place to deliver new features on the website.

Team Structure

The client had less than 3 engineers working on this project