CocoIndex

🚀 New Feature Launch: Seamless Query + Indexing with CocoIndex & CocoInsight

https://cocoindex.io/blogs/query-support

We’re excited to introduce Query Handlers in CocoIndex, now fully integrated with CocoInsight’s Query Mode.

With this launch, you can:

• Iterate fast on indexing strategies.

• Trace results back to source data step by step.

• Run queries seamlessly inside CocoInsight, without building custom UIs.

This means you can validate transformations, inspect missing chunks, and refine retrieval strategies—all in one place.

👉 Check out the docs, star us on GitHub, and start building smarter indexing flows today.

#CocoIndex #CocoInsight #VectorDatabases #DataEngineering #AIInfrastructure

We’re excited to announce native Kuzu support in CocoIndex https://github.com/cocoindex-io/cocoindex — now you can build real-time knowledge graphs with Kuzu -  a high-performance, lightweight graph database that’s also #OpenSource

🔍 Why it matters:
CocoIndex makes building continuously updated knowledge graphs easy with declaring transformations — and now with Kuzu, exporting your graph is just a config switch away.

📚 In our example, we used LLMs to extract relationships from documents and export the graph to Kuzu in under 200 lines of Python. If you’re using Neo4j, switching to Kuzu is plug-and-play — thanks to our standardized GraphDB interface.

💡 Highlights:
- Real-time indexing with LLM-powered extraction
- Incremental processing
- Open source end-to-end
- Ultra performant end-to-end (data processing + query)

👀 Learn more about this integration and how it works: https://cocoindex.io/blogs/kuzu-integration

🙌 GitHub Repo: https://github.com/cocoindex-io/cocoindex

We're excited to roll out a packed update over ~15 releases focused on real-time, incremental data processing — all with developers in mind at CocoIndex.

https://github.com/cocoindex-io/cocoindex

Here's what's new:

✅ Native Amazon S3 + SQS Support
Plug into your S3 bucket and stream updates in real-time with built-in incremental change detection.

⚙️ CLI Overhaul
A new standalone CLI makes launching and updating indexing pipelines as easy as: cocoindex setup

🔍 Refined Query Philosophy
Define your embedding (or any) transformation with  @transform_flow — use it both in indexing and querying. Simpler, cleaner, consistent.

We’ve deprecated SimpleQueryHandler for flexibility - users directly talk to the database to have maximum control. And we’re doubling down on what we do best: real-time indexing + transformation.

🌐 App Namespaces
Separate your staging from prod using COCOINDEX_APP_NAMESPACE. Organize flows by app, environment, or team — no collisions.

👀 New: Pretty Print Your Flow
Run cocoindex show to get a clean view of your flow’s spec, lineage, and schema.

📚  Tutorials & Examples
- Real-time semantic search from S3
- Image search with vision models
- FastAPI + Docker Compose search stack

We're all in on building the best real-time incremental data framework — and we couldn’t be more excited.

Red the full story here: https://cocoindex.io/blogs/cocoindex-changelog-2025-05-31

CocoIndex https://github.com/cocoindex-io/cocoindex now offers native support for Amazon S3 as a data source—making it easier than ever to integrate with your existing cloud storage. Paired with AWS Simple Queue Service (SQS), CocoIndex delivers real-time, event-driven processing of new or updated data in your S3 buckets.

🌱 Why Incremental Data Processing Is a Game-Changer

Incremental processing means handling only the data that has changed—whether added, modified, or deleted—since the last run. Unlike full reprocessing, which is time-consuming and resource-intensive, this approach is faster, cheaper, and scalable, especially when working with large datasets or latency-sensitive applications.

⚡ When You Need Incremental Processing the Most
1. When Fresh Data Is Mission-Critical
Real-time search, analytics, and AI systems depend on up-to-date information. If a user edits a document, they expect that change to be reflected instantly—especially in search results or AI-generated responses. Outdated data risks misinformed decisions and poor user experiences.

2. When Processing Is Expensive
Tasks like generating embeddings, running ML models, or performing complex transformations can be computationally heavy. Reprocessing unchanged data wastes time and resources. Incremental processing ensures compute efficiency by skipping redundant operations.

3. When You’re Operating at Scale
For datasets in the terabyte or petabyte range, full reprocessing is impractical. Incremental updates are the only way to stay within performance and budget constraints without sacrificing data freshness.

If T is your acceptable staleness threshold, and full reprocessing every T isn’t feasible, incremental workflows become essential.

⚙️ CocoIndex: Purpose-Built for Incremental, Real-Time Processing on S3

🔍 Smart File & Chunk-Level Change Detection
CocoIndex intelligently detects new or modified S3 files, eliminating unnecessary processing. Even within files, it identifies changed chunks—allowing it to recompute only what’s needed, not entire datasets.

⚡ Instant Updates with AWS SQS Integration
Get real-time responsiveness without polling delays. With native SQS support, CocoIndex reacts the moment data is updated in S3, triggering transformation workflows automatically.

🔒 Built-In Data Lineage & Integrity
Track the complete data processing lifecycle. CocoIndex maintains a persistent state of processed data, ensuring output consistency and providing full data lineage visibility across updates.

🧠 Granular Smart Caching for Faster Results
Reduce latency and compute cost through fine-grained caching. CocoIndex caches transformation results at the chunk level, reprocessing only the segments that have changed—perfect for workflows involving vector embeddings, NLP, and more.

👉 Read more: https://cocoindex.io/blogs/s3-incremental-etl
🌟 Repo: https://github.com/cocoindex-io/cocoindex

🎉 We are thrilled to announce the open-source release of https://github.com/cocoindex-io/cocoindex 🥥 🌴 . It is an ETL framework to transform data for AI, with real-time incremental processing - keep index up to date with low latency on source update.

If you like our work, please support us with ⭐ star Cocoindex on Github to support us ❤️. Thank you so much with a warm coconut hug 🥥🤗.

It is the world’s first open source compute engine that supports:
1) incremental processing
2) custom logic
3) heavy transformations beyond SQL. E.g., LLM inference in the ETL, vector embeddings, knowledge graph / structured extraction.

Key features:
- Keep index fresh on source updates with low latency with live updates.
- Support custom logic like lego; with data flow programming model.
- Native built-ins to support use cases for vector embeddings, knowledge graphs, and other data transformations.
- Super performant RUST engine, with Python SDK

Get started: https://cocoindex.io/docs/getting_started/quickstart 🚀

hashtag#OpenSource hashtag#ETL hashtag#DataEngineering hashtag#AI hashtag#LLM hashtag#DataInfrastructure hashtag#RAG hashtag#VectorEmbeddings hashtag#KnowledgeGraph