DevOps & Platform Engineering

Overview

Integration teams in financial services operate under a peculiar kind of friction. Spinning up a new integration service requires a Jira ticket to request a namespace, another to get a Kafka topic created, a third to have the MQ queue configured, a fourth for the Vault secret path, and then a wait — often measured in days — while each team processes the request in sequence. By the time the integration engineer has an environment they can write code against, three of them have switched to other work and the original deadline has moved.

An Internal Developer Platform (IDP) collapses that queue. The developer fills out a form, the scaffolder creates the repo from a template, the Tekton pipeline wires up CI, and Crossplane provisions the Kafka topic and namespace in a single reconcile loop. The platform team’s job is to build and maintain the paved path; the integration team’s job is to write the integration logic, not the scaffolding around it.

The trade-off that most teams underestimate: building an IDP is a significant engineering investment with delayed payback. The first three months feel like you are solving the wrong problem. The payback arrives at scale — when 15 teams are onboarding simultaneously and none of them are filing tickets.

Why self-service fails without a platform

Self-service attempts without a platform follow a predictable arc. A motivated platform engineer writes a bash script or an Ansible playbook that automates the ticket-filing sequence. The script works for one team’s setup, breaks when another team has a slightly different namespace convention, gets forked, and within six months there are seven versions of the onboarding script, none of which are authoritative.

The underlying problem is that “self-service” without abstractions is just automation of the ticket. The developer still needs to know which Kafka cluster to target, which secret store path to use, which Helm chart version is current, which namespace naming convention applies to their service type. An IDP hides those decisions behind a form backed by opinionated defaults; the platform team encodes the conventions once in the scaffolder, and every team who uses it gets the current convention automatically.

Backstage portal

Backstage is Spotify’s open-source developer portal, now a CNCF incubating project. Its three core capabilities for integration teams are the Software Catalog (a registry of all services, APIs, and integrations), the Scaffolder (a form-based service generator backed by git templates), and TechDocs (documentation pulled from the repo and rendered in the portal).

Running Backstage on OpenShift requires a PostgreSQL instance (the catalog backend), a Kubernetes service account with read access to the cluster, and OAuth integration with your IdP (Entra ID or Keycloak). The Red Hat Developer Hub, which is the supported downstream of Backstage, ships as an Operator on OpenShift and is the supported path for production use under a SAMA audit.

app:
  title: SAIB Integration Platform
  baseUrl: https://idp.integration.saib.internal

backend:
  baseUrl: https://idp.integration.saib.internal
  database:
    client: pg
    connection:
      host: ${POSTGRES_HOST}
      port: 5432
      user: ${POSTGRES_USER}
      password: ${POSTGRES_PASSWORD}
      database: backstage

auth:
  providers:
    microsoft:
      development:
        clientId: ${ENTRA_CLIENT_ID}
        clientSecret: ${ENTRA_CLIENT_SECRET}
        tenantId: ${ENTRA_TENANT_ID}

catalog:
  providers:
    github:
      integrationOrg:
        organization: saib-integration
        catalogPath: /catalog-info.yaml
        filters:
          branch: main
          repository: .*-integration   # only integration repos
        schedule:
          frequency: { minutes: 30 }
          timeout: { minutes: 3 }

Scaffolder templates

A scaffolder template is a Backstage entity stored in git that defines a form, a set of parameters, and a sequence of steps that execute when the form is submitted. Steps are Backstage actions: create a git repo, open a PR, create a Crossplane claim, trigger a Tekton pipeline. The template itself is YAML; the form renders in the Backstage UI.

For integration teams, the two most valuable templates are the ACE Flow service template (creates an IBM ACE project skeleton with a Tekton pipeline and an ArgoCD Application) and the Kafka consumer template (creates a Java/Quarkus service skeleton with the correct Kafka consumer configuration, a topic claim, and a Vault secret claim for the SASL credentials).

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: ace-flow-service
  title: IBM ACE Integration Flow
  description: New IBM ACE message flow with CI/CD, ArgoCD, and Vault secrets
  tags: [ace, ibm, integration]
spec:
  owner: platform-engineering
  type: integration-service
  parameters:
    - title: Service details
      required: [name, domain, owner]
      properties:
        name:
          title: Service name
          type: string
          pattern: '^[a-z][a-z0-9-]{2,30}$'
        domain:
          title: Business domain
          type: string
          enum: [payments, lending, treasury, retail, corporate]
        owner:
          title: Team
          type: string
          ui:field: OwnerPicker
          ui:options: { allowedKinds: [Group] }
  steps:
    - id: fetch-template
      name: Fetch skeleton
      action: fetch:template
      input:
        url: ./skeleton
        values:
          name: ${{ parameters.name }}
          domain: ${{ parameters.domain }}
          owner: ${{ parameters.owner }}

    - id: publish
      name: Create repository
      action: publish:github
      input:
        repoUrl: github.com?owner=saib-integration&repo=${{ parameters.name }}-integration
        defaultBranch: main
        repoVisibility: private
        topics: [integration, ${{ parameters.domain }}, ace]

    - id: infra-claim
      name: Provision infrastructure
      action: kubernetes:apply
      input:
        clusterRef: dev-cluster
        namespace: crossplane-system
        manifest: ./claims/integration-env-claim.yaml

    - id: register
      name: Register in catalog
      action: catalog:register
      input:
        repoContentsUrl: ${{ steps['publish'].output.repoContentsUrl }}
        catalogInfoPath: /catalog-info.yaml

The key design choice: keep templates thin. A template should set names, owners, and domain; it should not encode platform implementation details like Kafka cluster hostnames or Vault paths. Those come from the Crossplane composition, which the platform team controls. If a team forks the template to hard-code a hostname, the template has failed at its job.

Golden paths

A golden path is the happy path the platform team has paved, tested, and recommends. It is not the only path — teams can deviate with explicit approval — but it is the path that requires zero ticket-filing and comes with platform support. For integration teams on OpenShift running ACE and Kafka, three golden paths cover 90% of new work:

“Time to first deploy” means from filling the scaffolder form to a running pod in the dev cluster. That metric is the north star for IDP quality — if it drifts above an hour, developers stop using the IDP and file tickets again.

Software catalog

The Backstage Software Catalog is a registry of every service, API, resource, and team in the integration estate. Its value is not discovery — engineers already know what exists — it is operational: every component has a listed owner, a runbook link, an on-call contact, and a dependency graph. When an integration fails at 2 a.m., the on-call engineer needs to know who owns the upstream API, what SLA it carries, and where the runbook is. The catalog answers those questions without a Slack DM.

Every repo created by the scaffolder includes a catalog-info.yaml at root. Backstage ingests it and registers the component automatically. The platform team owns the schema; service teams fill in the values.

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payments-ips-adapter
  description: IPS payment instruction adapter — ISO 20022 to ACH transform
  annotations:
    backstage.io/techdocs-ref: dir:.
    github.com/project-slug: saib-integration/payments-ips-adapter-integration
    argocd/app-name: payments-ips-adapter-prod
  tags: [payments, iso20022, ace, ips]
  links:
    - url: https://wiki.saib.internal/runbooks/payments-ips-adapter
      title: Runbook
      icon: docs
    - url: https://monitoring.saib.internal/d/payments-ips
      title: Dashboard
      icon: dashboard
spec:
  type: integration-service
  lifecycle: production
  owner: group:payments-integration-team
  system: payments-platform
  dependsOn:
    - component:ips-clearing-api
    - resource:mq-payments-cluster
  providesApis:
    - payments-ips-adapter-api

Catalog data quality degrades if teams see it as bureaucracy. The platform team’s job is to make the catalog useful enough that teams fill it in because it helps them, not because they are told to. Wiring the ArgoCD sync status, the Tekton pipeline status, and the last deploy date into the catalog entity page is usually enough to tip the balance — teams start maintaining the catalog to get the operational view for free.

Infrastructure on demand

Crossplane is the Kubernetes-native infrastructure provisioner. Rather than filing a ticket to the Kafka admin team, an integration team submits a KafkaTopic claim; the Crossplane composition resolves it to the correct Strimzi cluster, creates the topic with the right partition count and retention policy, and writes the connection details to a Vault secret that the service can consume.

The key abstraction is the Composite Resource Definition (XRD): the platform team defines what the claim looks like and what the provisioning rules are; the service team only sees the claim API. They do not need to know which Strimzi cluster is the target, what the naming convention for topic names is, or what retention policy SAMA requires for financial event data.

# Composite Resource Definition — what service teams submit
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xintegrationenvironments.platform.saib.internal
spec:
  group: platform.saib.internal
  names:
    kind: XIntegrationEnvironment
    plural: xintegrationenvironments
  claimNames:
    kind: IntegrationEnvironment    # what teams create
    plural: integrationenvironments
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [serviceName, domain, tier]
              properties:
                serviceName: { type: string }
                domain:
                  type: string
                  enum: [payments, lending, treasury, retail, corporate]
                tier:
                  type: string
                  enum: [dev, test, prod]
                kafkaTopics:
                  type: array
                  items:
                    type: object
                    required: [name, partitions]
                    properties:
                      name: { type: string }
                      partitions: { type: integer, minimum: 1, maximum: 48 }
                      retentionMs: { type: integer }

The composition that backs this XRD creates the OpenShift namespace, the Strimzi KafkaTopic objects, the Vault secret path, and an ExternalSecret that copies the Kafka credentials into the namespace. The service team submits one claim; four resources are provisioned in the correct order with the correct naming conventions. That is the value of Crossplane over Terraform: the claim is reconciled continuously — if someone deletes the namespace manually, Crossplane recreates it.

RBAC & multi-tenancy

Integration teams in a bank are not equal in their trust level. The payments team’s namespace has PCI-DSS classification; the retail banking team’s namespace does not. Both use the same IDP, but the Crossplane composition that provisions a payments-domain service must enforce stricter NetworkPolicy (deny all egress except to known payment system CIDRs), a more restrictive ResourceQuota, and a Vault policy that limits which secret paths the service account can read.

The approach that scales: encode the classification in the claim schema (domain: payments), and use the composition to inject domain-specific constraints. The service team never writes NetworkPolicy; the composition injects the correct policy based on domain. Auditors can then verify that every namespace in the payments domain has the required NetworkPolicy by querying the Crossplane composite resources rather than inspecting 40 namespaces individually.

1. Define domain groups in Backstage. Each business domain (payments, lending, treasury, retail, corporate) maps to a Backstage Group entity. Users belong to one or more groups. The group membership drives which scaffolder templates they can access and which domains their claims can target.
2. Enforce claim namespace via OPA / Kyverno. A Kyverno policy validates that every IntegrationEnvironment claim is submitted from a namespace owned by the declaring team. A claim targeting domain: payments submitted from the lending team’s namespace is rejected at admission.
3. Map domain to Vault policy in the composition. The composition reads the domain field and applies the corresponding Vault policy to the generated service account. The payments domain policy allows reads from secret/payments/*; it does not allow reads from secret/treasury/*.
4. Inject NetworkPolicy from the composition. Each domain has a baseline NetworkPolicy stored in a ConfigMap in the crossplane namespace. The composition patches the correct policy into the provisioned namespace. Override policies require a platform team PR and a change record.
5. Surface the policy in the Backstage catalog. The catalog entity for each service shows the effective NetworkPolicy, the Vault policy, and the ResourceQuota. Auditors get a single pane; platform engineers can see immediately if a service is mis-classified.
6. Test multi-tenancy with negative-path assertions. Add a nightly CI job that tries to submit a cross-domain claim and verifies it is rejected. A policy that is never tested is a policy that will fail silently when it matters.

IDP metrics

Three metrics determine whether the IDP is delivering value. Measure them weekly; trend them quarterly.

A secondary metric worth tracking: ticket deflection. Count the infra-related Jira tickets filed to the platform team per month before and after IDP launch. A successful IDP should reduce that count by 60–70% within three months of reaching critical mass. If ticket deflection is low, the templates are not covering the real bottlenecks — go back and interview the teams about where they still need to file tickets.

Common pitfalls

Production checklist

1. Backstage backed by PostgreSQL; config in Vault via External Secrets; no plaintext credentials in app-config.yaml.
2. Auth via Entra ID or Keycloak; group membership synced automatically; no manually maintained user list.
3. Software Catalog populated via automated GitHub discovery; stale component detection alert wired to platform team.
4. At least two golden-path scaffolder templates covering >60% of new integration service types; each tested nightly in CI.
5. Crossplane compositions for namespace, Kafka topics, MQ queues, and Vault secret paths; all idempotent and drift-correcting.
6. Domain-based NetworkPolicy injected by Crossplane composition; no team-written NetworkPolicy in production namespaces.
7. Kyverno admission policy blocking cross-domain claims; negative-path tests in nightly CI.
8. Time-to-first-deploy SLO measured and alerted; target <30 min for all golden paths.
9. Developer NPS collected quarterly; top verbatim items triaged into IDP backlog within 2 weeks.
10. IDP itself operated as a production service: SLO, on-call rotation, incident process, changelog.