Code-First CI/CD in Microsoft Fabric with fabric-cicd

Continuous Integration and Continuous Delivery/Deployment (CI/CD) has been a standard practice in software development for years. By automating the deployments, CI/CD improves reliability and accelerates deployment process. Data platforms, however, have often lagged behind. With Microsoft Fabric maturing and adding stronger CI/CD support, developers can now bring modern DevOps practices to data projects.

In this post, I’ll show you how to use the fabric-cicd library to implement a code-first deployment workflow in Microsoft Fabric.

What is fabric-cicd?

fabric-cicd is a Python library built by Microsoft to manage Fabric items in your workspace. It is open-source, and you can find it here: https://github.com/microsoft/fabric-cicd.


Official definition:

This library supports code-first Continuous Integration / Continuous Deployment (CI/CD) automations to seamlessly integrate Source Controlled workspaces into a deployment framework. The goal is to assist CI/CD developers who prefer not to interact directly with the Microsoft Fabric APIs.

In practice, the library provides an abstraction layer on top of the Fabric REST APIs. Instead of calling endpoints directly, you can use Python code to deploy items in your workspace. As the library depends on Fabric REST APIs and Git integration, it can only support items with API endpoints or those included in Git integration. It may not cover every Fabric item yet, so check if your item type is supported before using it in production projects.

With fabric-cicd, you can build automated deployment pipelines across different environments, allowing you to deploy items from development to higher environments such as test or production.

Git integration and Deployment Flow

To use fabric-cicd, your development workspace must be connected to Git. Microsoft Learn provides an excellent tutorial on setting this up, so I won’t cover those steps here.

You can find example items in my GitHub repository, created through Git integration. These are the same items used later, giving you a clear view of how everything fits together.

The overall deployment flow works as follows:

  • Only the development workspace is connected to Git.
  • Higher environments (test, prod etc.) are not connected to Git.
  • Deployments to these environments are managed through the fabric-cicd library.

You can see a clear illustration of this process in the documentation:

https://microsoft.github.io/fabric-cicd/latest/how_to/getting_started/#git-flow

Parameterization with parameter.yaml

Deployments often require environment-specific values, such as pointing a notebook to the correct Lakehouse in production.

fabric-cicd handles this through a parameterization file called parameter.yml. This file contains find-and-replace rules that are applied automatically during deployment. It must be placed in the root of the repository folder containing your Fabric items.

Here’s an example:

find_replace:
  # Lakehouse Connection Guid regex
  - find_value: \#\s*META\s+"default_lakehouse":\s*"([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})"
    is_regex: "true"
    replace_value:
      prod: "$items.Lakehouse.lh_anssitehti.id"
      
  # Lakehouse workspace id regex
  - find_value: \#\s*META\s+"default_lakehouse_workspace_id":\s*"([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})"
    is_regex: "true"
    replace_value:
      prod: "$workspace.id"

This example uses regex patterns to find values and replace them with environment-specific values. The correct environment is passed as a parameter to the library.

More parameterization rules and options are described in the official docs.

Getting started with deployments

The core functionality for deployments is provided by the fabric-cicd library through the FabricWorkspace class and utility functions like publish_all_items and unpublish_all_orphan_items. These allow you to programmatically manage and deploy items from your Git-synced development workspace to higher environments.

  • FabricWorkspace
    • represents a Fabric workspace and manages items for deployment.
  • publish_all_items(workspace)
    • publishes all items in scope to the target workspace.
  • unpublish_all_orphan_items(workspace)
    • removes items in the target workspace that no longer exist in the source repository.

By using these components directly, you can fully control deployments from code.

Helper Script: deploy_fabric_items.py

To simplify the deployment process, I created a helper Python script called deploy_fabric_items.py. It wraps the core library functionality, validates input arguments, and provides friendly output messages during deployment. Internally, it uses FabricWorkspace, publish_all_items, and unpublish_all_orphan_items to handle the deployment logic. Authentication is automatically managed by the Azure Identity library, whether you’re using your Azure CLI credentials or a Service Principal. This makes it easy to run fabric-cicd scripts in Azure DevOps Pipelines, or GitHub Actions without any extra setup.

Here’s how you can deploy multiple item types to a production workspace using the script:

python deploy_fabric_items.py \
  --workspace_id ff5eff96-7b42-47bc-961d-07a5f7656032  \
  --environment prod \
  --source_directory anssitehti \
  --items_in_scope "Lakehouse, Notebook, Eventhouse, DataPipeline"

The script requires the following parameters:

  • workspace_id
    • the ID of the target Fabric workspace.
  • environment
    • the deployment environment (e.g., prod) used in parameterization.
  • source_directory
    • the repository folder containing your Git-synced Fabric items.
  • items_in_scope
    • a comma-separated list of Fabric item types to deploy.

The following snippet from deploy_fabric_items.py demonstrates how the FabricWorkspace object works together with the publish_all_items and unpublish_all_orphan_items functions:

def create_workspace_config(config: DeploymentConfig) -> FabricWorkspace:
    """Create and return Fabric workspace configuration."""
    print("🔧 Creating Fabric workspace configuration...")
    try:
        target_workspace = FabricWorkspace(
            workspace_id=config.workspace_id,
            environment=config.environment,
            repository_directory=config.repository_directory,
            item_type_in_scope=config.item_types,
        )
        print("✅ Workspace configuration created successfully!")
        print()
        return target_workspace
    except Exception as e:
        print(f"❌ Error creating workspace configuration: {e}")
        sys.exit(1)

def deploy_items(workspace: FabricWorkspace) -> None:
    """Deploy all items to the workspace."""
    print("📤 Publishing all items to workspace...")
    try:
        publish_all_items(workspace)
        print("✅ Successfully published all items!")
        print()
    except Exception as e:
        print(f"❌ Error publishing items: {e}")
        sys.exit(1)


def cleanup_orphaned_items(workspace: FabricWorkspace) -> None:
    """Remove orphaned items from the workspace."""
    print("🧹 Cleaning up orphaned items...")
    try:
        unpublish_all_orphan_items(workspace)
        print("✅ Successfully removed orphaned items!")
        print()
    except Exception as e:
        print(f"❌ Error cleaning up orphaned items: {e}")
        sys.exit(1)

Summary

fabric-cicd provides a code-first approach to CI/CD in Microsoft Fabric. It enables developers to manage deployments with a GitOps process, where configuration is stored in Git and pipelines handle the deployment of changes across environments. The library is still evolving, but it already supports most common Fabric items and makes the deployment process smoother.

While Fabric also offers Deployment Pipelines (GUI), in my opinion the code-first approach gives developers more flexibility to build pipelines that fit organizational needs.

You can find the full code example in my GitHub repo: http://github.com/matonen/blog-examples.