In this Leading with Data session, we are joined by Bob Van Luijt, CEO of Weaviate. Together, we explore the shift to AI-native applications, the importance of open-source communities, and advancements in AI databases. Discover how Weaviate drives innovation, the role of generative feedback loops, and tips for building impactful AI projects in today’s dynamic landscape.
You can listen to this episode of Leading with Data on popular platforms like Spotify, Google Podcasts, and Apple. Pick your favorite to enjoy the insightful content!
Let’s look into the details of our conversation with Bob Van Luijt!
My journey into AI began in 2015 when I started working with machine learning and word embeddings. It’s been nearly a decade since then, and the field has evolved tremendously. Initially, the focus was on creating relations between words in vector space, which was a relatively new concept in digital technology for machine learning applications.
The significant moments for me were the advent of BERT and sentence transformers. These developments drastically improved the quality of search results and recommendations, marking a shift from a niche to a mainstream focus on AI. The explosion of AI’s prominence was something I hadn’t anticipated, but it has been an incredible wave to ride.
Weaviate started as a classic open-source story, recognizing the need for a database where embeddings are a first-class citizen. Unlike libraries like Faiss from Facebook, which are great but not databases, Weaviate is built from scratch to be a dedicated vector database. Over time, the community’s feedback has shaped our offerings, leading to features like filtering and hybrid search.
Our focus now is on what we call AI-native use cases, which are applications that wouldn’t function without AI at their core. Weaviate’s offerings have evolved to support these use cases, with tools like the workbench tailored for AI-native applications.
AI-native use cases are applications that rely so heavily on AI that removing it would render them non-functional. These are different from AI-enabled applications, which would still work without AI but would lack certain features. Weaviate enables AI-native use cases by focusing on the integration of AI at the heart of the application, providing the necessary infrastructure and tools to support such innovation.
Traditional databases struggle with unstructured data, often requiring complex SQL statements that can fail due to data inconsistencies. Weaviate’s approach is to prompt the database with your data needs, and it autonomously searches, analyzes, and updates the data. This AI-native paradigm simplifies data management and allows for more dynamic and efficient handling of unstructured data.
I’m particularly excited about the concept of generative feedback loops (GFLs), where you prompt the database instead of the model. This allows for more dynamic interactions with the data, such as specifying language preferences for data entries or triggering actions based on content. The future of AI-native databases lies in their ability to become more efficient and multidirectional in their operations, moving beyond the early stages of today’s generative AI.
Weaviate places a strong emphasis on education, providing developers with the knowledge and tools to build AI-native applications. Our community is a crucial part of our growth, helping us understand what works and what’s needed in the market. As we introduce new concepts like GFLs, we rely on the community to experiment, provide feedback, and ultimately drive adoption.
Building an open-source business requires understanding the value you create and how to capture it. Initially, focus on growing the community and observing the value generated. As the company matures, the open-source community becomes a funnel for potential customers. Finally, transparency and trust become paramount as the company scales. It’s also crucial to seek advice from veterans in the field and to be open to learning continuously.
For those starting their career in AI, it’s essential to focus on building something great without being preoccupied with financial success. The industry is full of opportunities, and now is the best time to dive in. Embrace the fun and challenges of creating something new, and don’t be afraid to fail and try again
This conversation highlights the growing importance of AI-native applications and community-driven progress. Bob’s journey shows how focusing on creating value, learning from challenges, and exploring new ideas can lead to success in AI. The future of AI offers endless opportunities for those ready to innovate and experiment.
For more engaging sessions on AI, data science, and GenAI, stay tuned with us on Leading with Data.