Using sveltekit to build a dataviz platform
Every newsroom needs a way to create datavisualisations for their articles. Most companies prefer to use third party tools like datawrapper , but at NZZ we run our own custom built platform called Q . We just finished a year's long refactor of the entire architecture using svelte/kit. In this article we'll provide you with insights to the why, how and the pro's and con's of working with svelte (v4) & sveltekit (v2) at such a scale.
Why: Inheriting a legacy micro service system
The original platform consists of about 20 micro service apps built using HAPI (NodeJs) that server-side renders the visualisations and returns an html response. A main service, called the q-server, is the entry point that resolves each vizualistion request to the correct micro service and forwards the result. Couchdb was used for storage, and it is a very robust setup, but one with issues under the hood:
- Plain javascript:
The entire codebase was written in plain javascript. You are entitled to your strong opinions on typescript, but with the scale and complexity of this project, it is astounding that the decision was made to not type anything. It made debugging and adding features unnecessarily difficult.
- Extremely outdated dependencies:
Many of the services were using outdated dependencies and NodeJS versions, which we could not update without breaking something, essentially forcing a refactor.
- Server-side rendering:
Server-side rendering was problematic because we wanted to have more interactivity in our visualisations. The old team solved this by dynamically adding javascript on the server to the response. While it is a working solution, it was atrocious to debug and very error prone. We wanted a better and modern way of writing interactive apps.
- Knowledge loss:
The original team who had written the codebase over 7 years ago have all left the company. While there was good documentation left behind, There was a huge knowledge loss.
The result of this situation was that maintaining, adding features and debugging were all long and costly processes. After carefull consideration we decided to rethink the whole system and port it to a new architecture.
How: Architecture of the new system
- Monorepo to the rescue:
This was the best decision we made of the entire project. Since a lot of apps were simply copied templates and shared similar functionalities, a monorepo made it a breeze to share, write and debug code. It slightly complicates releases, but is still more practical than having to manage dozens of repositories and shared packages.
- Client-side rendering:
We went from serverside rendering to pure client side rendering. Each old micro-service became a svelte package in the monorepo that gets compiled to a js file and hosted on S3 behind a cdn. Our new server simply replies with the data and a link to the versioned .js file.
The cost for the client from going server-side render to client rendered have been negligible in our testing. The urls are all versioned, gziped and cached. With modern day devices and the low level of complexity of our visualisations, this does not present an issue. The improvement in developer experience and speed alone are worth the trade-off in our opinion.
- SaaS instead of multi-tenancy:
The microservices for the editor & visualisations were independently deployed for each subsidiary publication of NZZ with their own stylistic preferences. We consolidated this into 1 SaaS service and are using user defined settings for any difference in style. This has made it much easier to manage and deploy new features than managing multi-tenant setups.
The result of this is that we went from 20 microservices to this setup:
- 1 Sveltekit app (visualisation creator/editor)
- 5 Fastify apps (delivery/auth/exporting/large data-processing/quiz)
- 15 svelte packages hosted on a cdn (vizualisation apps)
The Pro’s and Con’s of using SvelteKit/Svelte for large systems:
While generally our conclusion is that sveltekit/svelte is absolutely incredible and simple to work with, there are some real downsides to choosing it as your stack.
Svelte
Huge Pro: DX & productivity are incredible.
Svelte really does not overcomplicate the development process and in our experience the mental model behind is quite easy to grasp. It cannot be stated enough how incredibly fast and easy it was for the whole team to become productive using svelte, even new members who never touched it before. We all have worked professionally with Angular for years, and our minds were just blown by how simple things could actually be.
PRO: Reactivity syntax is so simple
Svelte’s (v4) reactivity syntax is incredibly simple that it’s just mind-blowing that nobody came up with it before. There are a few gotchas along the way that you have to learn while doing, but nothing major that takes away from the experience.
CON: Reactivity hell
Reactivity is amazing, but keeping it simple on large projects is a challenge. When you start using svelte stores, derived stores, multi-line reactive statements,... It’s very easy to lose perspective and create a complicated mess of reactive statements.
I’ve had the same issue with a fully reactive angular codebase that was using rxJS, so I wouldn’t consider this necessarily a ‘svelte’ issue, but it’s more of a ‘reactive programming issue’. I think this was also partly due to the teams general lack of experience with reactive programming on such a scale.
svelte v5 is going to completely change the structure and syntax with it’s new signal system called runes. The old syntax will still be available but will eventually be removed. This new v5 system should improve the stucture and make it easier to keep reactivity simple. We haven’t tried out svelte 5 yet as it’s still not production ready, but we are excited to see what it brings.
Con: Svelte’s ecosystem is still lacklustre
It’s always the same main issue with any emerging technology: a lacking ecosystem. It’s constantly improving but it’s absolutely nowhere near the level of the other frameworks. Expect difficulties finding good svelte libraries for things that are common and established in other frameworks. This has been our number one painpoint.
You could argue that svelte allows very easy integration of pure js libraries, and that’s true. But generally you don't really want to spend time writing custom integrations, you just want to install a package and move on.
CON: Some svelte language issues
There are some minor inconveniences in svelte v4 that Rich Harris has pointed out in this video. We have bumped into most of them, and I highly recommend watching it because it explains the issues much better and how they plan to address them in svelte 5.
Sveltekit
Pro: incredible DX
We were initially sceptical about the whole front-end = the back-end tech stack, but the DX and speed of development is absolutely incredible. It takes a bit getting used to but once you get the hang of it it's really tough going back to a traditional split stack for front & back-end.
Features like granular route settings so you can statically render some pages, while keeping others server-side rendered and still transition into a SPA with incredible ease. Sveltekit adapters for letting you deploy to many different providers. Good interactive tutorials for onboarding. It truly has a lot going for it in the DX department.
Con: Sveltekit is an incomplete server-side solution.
Svelte is such a paradoxical framework. It offers some incredible conveniences and new ways of working, but leaves you hanging in other crucial areas. It’s so frustrating that if you want to whitelist a route for CSRF, you either have to turn it off completely, or you have to re-implement the entire code yourself because they still haven’t implemented such a basic feature already requested in 2022. (I wrote this article in 06/2024)
If you have ever worked with Laravel you will understand how convenient it makes your life with built-in features like auth, cronjobs, rate-limiting, etc... At the moment Sveltekit leaves you hanging in some crucial server-side areas.
PRO & CON: File based routing is very transparent, but comes with organisational overhead
With any sizable app your routes folder will have dozens if not hundreds of route folders and files. When you start using route groups, nested layout, layout overrides, it becomes even more overwhelming to keep track of how it all interacts with each other.
We are conflicted about this point, because it is very transparent at first but can get wildly confusing depending on how much complexity you throw into your route structure.
CONCLUSION
Using Svelte/kit on such a big project required a serious mind-shift coming from a more
traditional setup, but it has also been an incredible breath of fresh air, despite some of the
downsides. In the end, our development speed has sky-rocketed while delivering simpler
software and having a lot more fun doing it!
We are convinced that svelte/kit was worth the risk, although for heavy server-side services we
still think it's best to use more traditional back-end tech.
If the svelte team can iron out all the inconveniences from svelte v4, and then focus on making sveltekit a more complete server-side solution like laravel, that would be a dream framework for our team.