From Big Data to ML: Integrating Spark and Kubeflow with Juju

In modern data science, the "Data Lakehouse" is a powerful way to manage large-scale data. However, for many teams, the "last mile" of integration—connecting Machine Learning platforms like Kubeflow to big data processing engines like Apache Spark—remains a major technical challenge. In a shared Kubernetes cluster, managing security, user permissions, and connection strings across different user profiles often leads to manual errors and security risks.

In this session, we will explore an open-source solution to this problem using Juju and Charmed Operators. We will walk through a "Multi-Integrator" architecture where each Kubeflow profile receives its own secure connection to the data platform. Instead of manual configuration, we use automated integrators to inject Spark Service Accounts, configurations, and security secrets directly into user namespaces.

By using Charmed Spark and Charmed Kubeflow, we can create a modular system where data scientists can move from a simple notebook to a distributed Spark job without worrying about the underlying YAML files or credentials. We will discuss the logic of this design, how it handles multi-tenancy, and how it simplifies the path to a production-ready data lakehouse on Ubuntu.

More info: https://events.canonical.com/event/146/contributions/951/