crawl analytics

ok, couple of days in and it has been a whirlwind: from amazement how quickly something you visualize in your head can be turned into an actual and functioning solution... to utter frustration when Cursor decides to eradicate something you have been working on for hours and you had not initiated git yet.

That latter was eventually salvaged by the fact that I keep my local folder in Dropbox, which gave me version control... still lost more than an hour of optimized code, and of course the result was just a bit different.

The fact that AI apologizes made me smirk, but I still had that feeling after my sister kicked down my free form LEGO tower and when I had finally rebuilt it, it was just not as perfect.

Luckily I'm in that phase of life where I accept good enough over perfect.

Current status:

• Flow to check url of any website and getting a rating for how well optimized they are: done

• Getting the specific settings for how they optimized: done

• dashboard that shows real-time crawlers: done (with dummy data)

• knowledge base with information about crawlers and best practices: done

• setup Polar for payment gateway: done

Still a lot to implement and test, but making good progress. And especially the results of different sites work inspiring. This is exciting!

That moment that you realize you have to push pause on something else, because this is (well, might be) bigger than that other thing.

Shift

I was working on a new project that deals with product ingredients, mostly in food supply chains. With the change that is happening in that industry, there is a need to better structure certain data so that it can be easily consumed by LLMs.

I wanted to better understand the how behind this and came to the level of a simple website: If there is SEO there must be something like AICO (is that a thing? AI Crawler Optimization).

I quickly realized (of course with the help of OpenAI, Perplexity, Grok and Manus) that there are not that many solutions that offer both insights (how is a site optimized, yes or no), knowledge (which crawlers are out there, what do they do) and which crawlers visit my site (analytics).

crawl analytics

I decided to draft the outline, have AI of course write the prompt (for Cursor, I skipped the step with Lovable for this project) and off we were.

I was playing around with different fancy names, but decided to keep is super simple for this project and crawlanalytics.com was still available.

So... let's see if we can get to a working full solution that covers it all.

First off: the functionality to check a website, any website, and see if and how they optimize their website for, or block their website from AI crawlers.