Getting hit in the face

That moment that you realize you have to push pause on something else, because this is (well, might be) bigger than that other thing.

Shift

I was working on a new project that deals with product ingredients, mostly in food supply chains. With the change that is happening in that industry, there is a need to better structure certain data so that it can be easily consumed by LLMs.

I wanted to better understand the how behind this and came to the level of a simple website: If there is SEO there must be something like AICO (is that a thing? AI Crawler Optimization).

I quickly realized (of course with the help of OpenAI, Perplexity, Grok and Manus) that there are not that many solutions that offer both insights (how is a site optimized, yes or no), knowledge (which crawlers are out there, what do they do) and which crawlers visit my site (analytics).

crawl analytics

I decided to draft the outline, have AI of course write the prompt (for Cursor, I skipped the step with Lovable for this project) and off we were.

I was playing around with different fancy names, but decided to keep is super simple for this project and crawlanalytics.com was still available.

So... let's see if we can get to a working full solution that covers it all.

First off: the functionality to check a website, any website, and see if and how they optimize their website for, or block their website from AI crawlers.