You don’t need Google Analytics

Today it seems like wherever we go we are followed by our Big Brothers recording our every move. And, although we don’t like being followed, we’re often the ones responsible for helping Big Tech record our own users.

Through this article I’ll try to give you an overview of how you can develop your own Analytics Service so that you can better understand how they work, what are the alternatives and hopefully make you think more about your own and the privacy of your users.

First of all, let’s start with the great Google Analytics Open Source alternatives.

The OGs in this space are definitely Countly and Matomo. Those are great heavy weight alternatives with more than a decade of experience. They pretty much have the majority of Google Analytics features.

Newcomers in this space are lightweights like Plausible, Umami, Ackee and Shynet. Those are great alternatives if you want to track basic stuff.

Nevertheless, building the base of such a system would somewhat look like this.

The Collector 🦹‍♂️

The collector is a part of the system that would collect analytical data from the frontend of your application. This might be a typical server-side application connected to a Relational Database that has an /analytics endpoint for data collection.

The most basic data record collected consists of:

• User/Session identifier - The Ghost 👻, we’ll talk more about them below
• Visited URL - the URL where the collector was activated
• Timestamp - Time of the visit
• Additional analytical data - Geo data, Device data, Referred data etc.

A major dilemma you might have with the Collector is whether to use a REST API or a Pixel approach.

REST API approach would consist of you building a server-side application that has an endpoint receiving JSON data and returning JSON HTTP response. This approach is easily scalable and would enable the Collector to only collect data without major processing on a server side.

The other approach is the good ol’ Pixel approach. How pixel works is instead of building a REST API that receives your analytical data sent from the client, you are building a HTTP GET request that returns a HTTP response with Content-type: image/png which consists of a solely 1x1 pixel transparent image. The request is triggered by adding a simple <img/> tag somewhere on the page referencing a pixel URL. With this approach all of the tracking process is done server-side. You track User/Session identifier through randomly-generated server-side cookies, Visited URL from request Origin, Geo Location from the IP, Device data from the User-Agent etc.

The REST approach is better because it enables you more precise tracking as well as it easily scales to mobile and other non-web-based applications. It can also be immune to VPNs, disabled cookies as well as an Incognito Mode 😬. (Ghost is sneaky 👻)

On the other hand Pixel approach is better when you have web-based applications without enabled JavaScript or you want to track even those non-Javascript fanatics. The great examples are Email clients and RSS feed applications, which don’t run JavaScript but load images.

But in real life, most of the analytics applications use a hybrid approach between both of them.

And so.., oh no…, who’s there!?

The Ghost 👻

Mostly client-side, the Ghost is a part of your analytics application that is responsible for sending data to the Collector. I refer to it as the Ghost because it’s silent in the shadows of any app, but tries to be as precise as possible when identifying users and their devices.

When it works with a Pixel Collector 🦹‍♂️, it is pretty simple. Every tracked website has it’s Pixel URL, so we build our Ghost simply by adding <img src=”https://collector.api/some-cool-applictaion-id/pixel.png”/> to our website. That’s it.

But when it comes to REST API Collectors 🦹‍♂️, that’s when the Ghost becomes interesting (and scary 😥).

The Ghost, because it is a client-side code, has access to your device, therefore all the major data that comes with it. So even if you are using for example VPN, the Ghost is still loaded as a client-side application and knows where you are hiding.

The easiest way for identifying repeated users is with a randomly generated User/Session id that can be generated and stored by server or client in a cookie. If the cookie doesn’t exist, just generate a random ID, and if it does exist, just add it for a current request.

But the harder and more sinister way is by what we call device fingerprinting. What this is, is creating a unique hash (fingerprint) of every user based on their available device data therefore tracking them based on their unique hardware identifier without storing any tracking data in cookies or on their local storage. Because this approach is not dependent on storing any data to the user's device it can track users even in incognito mode, on VPN etc. There are a lot of proven techniques for device fingerprinting and you can further learn about it from FingerprintJS Blog or from their GitHub Repository.

So now when you know what to build you only need to decide on techniques you wanna use and start building your client-side libraries. Our next protagonist will love you for this 😄

The analyst 🕵️‍♂️

When the Collector 🦹‍♂️ and the Ghost 👻 have joined forces, your analytical data collection super team is almost assembled, the only one left is the Analyst 🕵️‍♂️.

The Analyst is interested in two things. Optimization and Visualization.

When you have multiple rows of data for every user session and you have thousands of sessions every minute, you really don’t want to pull all that historical data every time you want a yearly report. This kind of historical data is perfect for data normalization. What that means in layman's terms is that your existing analytical data won't be updated through history. Visits that happened in January 2021 will still be legit in March 2022, so if you are tracking data on a monthly basis your data can be normalized for previous months and only consist for example of the number of visits for a particular URL in a particular month. Therefore your database goes through a lot of less rows every time you need a report.

Having visitors' data is great, but unless you are a hard core table fanatic, you would need some kind of data visualization to better understand it. There are a lot of great JavaScript libraries for data visualization, even Google Charts is Open Source. There you can find any chart used in Google Analytics for free 🥳.

Other options are free and paid BI platforms as well as good ol’ CSV database export and data visualization with Google Sheets or Microsoft Excel.

That’s it the Analytics team is assembled. The trio works together to provide you a Google Analytics alternative. But like Batman 🦇 has Inspector Gordon 👮‍♂️ those three have..

The Judge 👩‍⚖️

You can eliminate cookie banners by using Ghost’s Fingerprinting Techniques 👻 but you can not evade the Judge 👩‍⚖️

Basically there are a lot of laws protecting users from being tracked online and their data being collected without their knowledge. So, when building this kind of application you need to take a privacy-first approach.

In general:

• If you are storing anything on your user's device you need their consent
• You should never store their personal data without their consent (including IP address)
• Although this is a gray area, your fingerprinting hashes should at least never be reversible. You should never be able to take a hash of some user and with any kind of algorithm be able to reverse it to a particular IP address or any other personal information.

GDPR fines can be astonishingly big so reading up a bit about these regulations or consulting with an expert is probably a good idea if you plan on building a solution out of this.

Conclusion 😴

With this blog post I hope I’ve helped you understand how tracking works and helped you understand how Big Tech tracks us among different apps with their sneaky ghosts 👻

They are not reading our minds, but maybe, we are enabling them to hear and see everything by using their tracking systems.

I hope you’ve had a fun time reading this. If there are any questions I can help you with, see you in comments 👇