Skip to main content

Kubeflow for Machine Learning

Classification Published technical book / ML infrastructure

Status Published technical book

Period 2019-2022

Role Lead author

Full title Kubeflow for Machine Learning: From Lab to Production

Materials Kubeflow, Kubernetes, machine learning workflows, Apache Spark, Apache Mahout, CT scan denoising, low-dose CT image processing, pipelines, notebooks, model serving, production ML tooling, open-source ML infrastructure

Problem​

Kubeflow for Machine Learning: From Lab to Production was a technical book about using Kubeflow for machine learning workflows.

The book was mainly a "how to use Kubeflow" book. That was harder than it sounds. Kubeflow was not one simple product. It was a collection of tools deployed on Kubernetes for machine learning workflows. Even a simple hello-world example could involve multiple services, followed by the caveat that users did not necessarily need all of them.

Build​

I was the lead author. That meant writing my own section, keeping the broader book moving, and helping coordinate a project that kept changing underneath us. Kubeflow itself shifted repeatedly while the book was being written. Several drafts had to be reworked as the underlying platform changed.

Chapter 9 was my main technical contribution: a comprehensive demo built around medical image processing. The workflow used Apache Spark and Apache Mahout to decompose CT scans and reduce noise in low-dose images. The user could control the amount of denoising, and a doctor could review the output.

That demo connected to related peer-reviewed work on low-dose CT scan denoising. During early COVID, CT scans were being explored as one possible detection method, although that direction eventually lost out to nasal swabs.

  1. Inputlow-dose CT scans
  2. ComputeSpark workflow
  3. MathMahout decomposition
  4. Controldenoising amount
  5. Reviewdoctor review
tool pile != solved problemreview stays in the loop

What worked​

  • The book gave readers a concrete path into Kubeflow.
  • The medical imaging demo showed how multiple tools could be tied into a real ML workflow.
  • The work forced a production-oriented view of ML systems.
  • The project developed into later Kubeflow consulting work.
  • It showed how much coordination a serious technical book requires.

What was hard​

  • Kubeflow kept changing while the book was being written.
  • Simple examples became complicated quickly.
  • The platform offered many choices without always giving users a clear problem-shaped path.
  • The book had to explain a shifting system clearly enough for people to use it.

Lesson​

A pile of powerful tools is not the same as a solved problem.

Kubeflow had powerful parts, but the open-source version often felt like too much freedom around too little specific problem definition. It was a pile of capable open-source components. That did not automatically make it a clear solution for a particular user.

Why it matters now​

This artifact is one of the reasons aboriginal armadillo starts with the workflow, not the tooling.

The open-source promise of the 2010s often sounded like: take the right foundation, deploy the right stack, and your problem gets easier. Kubeflow showed the limit of that idea. A broad platform can be powerful and still fail to give a specific user the simple thing they actually need.

Bespoke agent work is moving in the opposite direction. Start with the itch. Build the small widget. Keep the tool close to the problem. If the workflow is specific, the system can be specific too.

Evidence​