/ Context
Model serving had to work across an AWS estate of more than 100 accounts, where account boundaries are real and teams retain autonomy inside them. The platform's job was to make open-source model serving available across that estate while security and lifecycle standards stayed centrally owned.
/ Role & decision scope
- Designed the secure ML and model-serving platform across 100+ AWS accounts.
- Implemented it, including automated model packaging and GPU-backed serving on SageMaker.
- Owned where the central governance boundary sits relative to team-owned accounts.
- Defined how open-source models are packaged and served.
/ Constraints
- An estate of 100+ AWS accounts with their own ownership boundaries.
- Open-source HuggingFace models rather than a single managed vendor model.
- GPU-backed inference has to be supported.
- Central security and lifecycle standards apply across the estate.
/ Architecture & key decisions
- D01
SageMaker as the serving substrate
GPU-backed inference is standardised on Amazon SageMaker, giving one consistent serving surface across the estate rather than a different stack in each account.
- D02
Automated model packaging
Open-source HuggingFace models go through an automated packaging pipeline, so what is served is a packaged artifact produced by the platform.
- D03
Centralised governance controls
Security and lifecycle controls are owned centrally and applied across the estate, rather than being restated account by account.
- D04
Serving designed for a multi-account estate
Design focus: the platform spans 100+ accounts as a first-class constraint, so account structure shapes the architecture instead of being worked around.
/ Trade-offs
- Distributed account autonomy versus centralised security and lifecycle standards
- Teams keep their own accounts; packaging, serving and governance controls stay central — autonomy inside the account, shared standards across the estate.
- Open-source model choice versus supportability
- Open-source HuggingFace models are supported, served through the platform's packaging pipeline and SageMaker rather than as unmanaged deployments.
/ Evidence & outcomes
- Artifact
- Secure ML and model-serving platform spanning 100+ AWS accounts
- Artifact
- Automated model packaging pipelines for open-source HuggingFace models
- Artifact
- GPU-backed inference on Amazon SageMaker
- Decision
- Centralised governance controls over a distributed account estate