Migrating from GOV.UK PaaS

Migrating from GOV.UK PaaS

Last updated

  • 2 November 2022 - first published

Government cloud hosting strategy

The Government Digital Service (GDS) has provided GOV.UK PaaS since 2015, supporting and providing a public cloud platform for departments using a shared hosting and responsibility model.

Following an extensive analysis period, GDS has concluded that, while the platform has been successful in its aims, the underlying technology would now require investment before it could meet its goals in the long term.

Faced with this need for re-investment GDS has decided to decommission the platform, in order to focus its budget and energy on other GDS products for common use by Government.

GDS will not be providing a replacement hosting service.

Tenants should look to their own department, or parent department, and use existing support for public cloud hosting services.

In deciding where to host their service next, tenants should consider the following principles from the Central Digital & Data Office (CDDO):

  • Tenants should continue to use public cloud services, or internal platforms based on public cloud offerings. These continue to be the platforms of choice for government services.
  • The Central Digital & Data Office (CDDO) has secured Director General level agreement to help share services. Larger departments are prepared to share knowledge, patterns and platforms with the rest of government. If tenants have no existing cloud engineering capability, you could approach the CDDO to benefit from the sharing agreements.
  • When selecting a cloud platform, the tenant should assure themselves that:
    • the public sector cloud supplier is regionally appropriate, i.e. that the data will reside in locations suitable for your workload; with GOV.UK PaaS this was predominantly the AWS eu-west-2 (London) region, but some tenancies were hosted in the AWS eu-west-1 (Ireland) region
    • billing processes are clearly defined and understood
    • any ‘service wrap’ provided by the department or supplier is appropriate and proportionate to the needs of your application
    • you have suitable arrangements in place for retention or archiving of data and its safe transit through migration
  • they understand and are prepared to manage and accept any risks of using the new platform and security assurance is carried out before the new service transition date

Introduction

The purpose of this guidance

To help GOV.UK PaaS tenants plan to move their services to an alternative hosting platform by 23 December 2023 and understand how the GOV.UK PaaS team can assist in the migration.

What does this mean for my service?

By 23 December 2023 all services hosted on GOV.UK PaaS will need to have migrated to an alternate hosting platform.

Any services that are not hosted on another platform by that date will go offline.

All platform and tenant data will be permanently deleted by 28 February 2024.

If you miss the deadline, and you need to migrate your applications, contact us via the normal support routes. If you contact us before we have deleted all tenant data, we will be able to provide this data. We will not be able to provide access to the applications or their configuration on GOV.UK PaaS.

Key dates

These are some key dates in the lifetime of the platform until decommission.

Milestone Date What this means for you
Announcement of the decision to decommission 12 July 2022 Public notification of the decision to decommission the platform and the overall timeline to retirement. Potential tenants cannot request a 90-day trial account for evaluation. Trial tenants cannot request an upgrade to a billable account in order to deploy a beta or production service.
Festive season code freeze 16 December 2022 until 9 January 2023 Annual platform freeze over the festive period for non essential engineering. This should not affect you.
Deprecation of PostgreSQL 10 plans 7 February 2023 From this date we will be deprecating all our PostgreSQL 10 plans. You will no longer be able to create new PostgreSQL 10 databases on GOV.UK PaaS. You will need to upgrade any existing plans to ensure continuity of operation of your application. This is happening because our provider is retiring their Postgres 10 offering.
End of the financial year 31 March 2023 After this date you may have additional funds available from your department to help the migration.
Automatic upgrade of PostgreSQL 10 databases to PostgreSQL 14 17 April - 18 July 2023 Between 17 April and 18 July, AWS will forcibly upgrade any PostgreSQL 10 databases to PostgreSQL 14 during a maintenance window. GOV.UK PaaS has no plans to support PostgreSQL 14 through the marketplace using cf create-service. Any databases forcibly upgraded by AWS will still be functional, but you will not be able to manage them through Cloud Foundry, other than to delete them or connect via Conduit. We recommend upgrading your PostgreSQL 10 databases to at least PostgreSQL 11 before 17 April if you expect to be with us by that point. This change may break your application. You are advised to ensure your applications work with the versions of the relevant libraries that are compatible with PostgreSQL14, if you plan to allow AWS to automatically upgrade your database. We will be in touch on this topic regularly.
Deprecation of Redis 3 30 June 2023 From this date we will be deprecating all our Redis 3 plans. You will have to upgrade any existing plans to ensure continuity of operation of your application. This is happening because our provider is retiring their Redis 3 offering.
Forced upgrade of PostgreSQL 10 databases to PostgreSQL 14 18 July 2023 From this date AWS will force the upgrade of PostgreSQL 10 databases to PostgreSQL 14, regardless of maintenance window. This change may break your application.
GOV.UK PaaS decommission 23 December 2023 After this date services hosted on the platform will stop operating or responding to requests from the web. The GOV.UK PaaS API will not respond to requests.
Final bills issued by GDS Business Operations January - March 2024 You will receive your final bill.
Deletion of tenant data and GOV.UK PaaS platform data 28 February 2024 After this date all tenant and platform data will be destroyed.
Deadline to pay final bills 31 March 2024 You must pay any outstanding bills by this date.
End of financial year 2023/2024 31 March 2024 The end of the financial year
Deletion of tenant data and paas platform data April - July 2024 Your data will be finally deleted

Support for GOV.UK PaaS

GDS will continue to provide a team of site reliability engineers to perform critical platform operations and timely support

The GOV.UK PaaS platform will be maintained to the usual standards until the 23 December 2023.

PaaS maintenance schedule
Maintenance Frequency What this means for you
Monthly buildpack upgrades regular upgrades, typically monthly Regular communication from the GOV.UK PaaS team highlighting any major changes to language support, deprecations.
Ad hoc security patches as required within 5 days of reporting The platform will be maintained to our usual standards for security patching until it is retired.
Annual platform pentest January to March of each year The platform will be assured to the usual standards until it is demised. This means annual health checks and remediation of any priority findings.
monthly paas-announce email to tenants regular monthly updates Regular communication from GOV.UK PaaS team highlighting any key milestones or guidance for tenants.
PaaS IT Health Check for 2023/2024 January to March of each year Annual health check for the coming year to ensure we are independently assessed and we remediate any findings

Unless you no longer require your service, it must be migrated before this date.

You will receive regular communications from the GOV.UK PaaS team via email.

We offer help and advice though our usual channels:

Changing responsibility model

GOV.UK PaaS is designed and built with a shared responsibility model. After migration, tenants will assume the responsibilities that the GOV.UK PaaS team once owned for their own infrastructure.

Examples include:

  • Security information and event management (SIEM) threat detection compliance and security incident management
  • Information assurance and IT Health checks
  • Infrastructure scaling
  • Infrastructure cost management
  • Infrastructure-level log management and retention
  • User management
  • Ingress traffic routing
  • Egress traffic routing
  • TLS certificates management
  • Secrets management
  • Platform/Operating system stack upgrades -> Upgrade deployment strategy
  • Platform monitoring and alerting
  • Tenant service monitoring and alerting
  • Back-up management
  • Disaster recovery
  • Securing SSH access to applications

Prepare to migrate

Before tenants migrate their services, there are a number of things to consider.

Audit your services

Understand all the services that you need to migrate. You could consider:

  • The relative importance of your services: which ones are higher priority? Should any be retired instead of migrated?
  • The technical effort required to migrate each service to another platform
  • Who owns the service? Smaller services may not have an obvious owner due to changes in the team or department structure
  • The “7 Rs” are commonly used cloud migration strategies

Think about your timing and budget

  • Do you have the budget to fund the migration?
  • Can you raise a business case for the year 2023/2024?
  • What is the timeframe for your migration?
  • Are there any events or features that may constraint your migration plan? For example if your service has a - higher number of users in April, you may decide to migrate a month after when it will have less of an impact
  • Can you identify a sequence to migrate the services if you have more than one?
  • You’ll need to explain to stakeholders that any data migration will incur some downtime. You can avoid this if absolutely necessary (see the Data migration section below)

Review your skills

  • Do you have the skills to migrate your services available in your department? You will need:
    • people with the skills to work with the infrastructure (defined in the DDAT framework as an infrastructure devops engineer role and also referred to as Site Reliability Engineering in the wider technology marketplace), to enable the migration, change your pipelines
    • people with the skills to refactor your applications to accommodate the new hosting arrangement (i.e. software developers)
  • Do you need to outsource to a delivery partner?
  • Do you need to bring in contractors to supplement your team?
  • Do you need to procure cloud migration expertise from the Digital Marketplace to assist with the migration?

Consider your department’s cloud hosting strategy

  • Understand your department's hosting standards and preferred cloud provider
  • Determine if there is a departmental strategy to follow
  • Determine if there is a departmental hosting platform available in-house. Some departments may have multiple hosting platforms for different types of workload. (for example one platform for general purpose web - hosting for OFFICIAL level services and one for data heavy workloads for data scientists)

Assess your security needs

You will need to review the threat model of your service and re-assess your risks so that they can be accepted by your senior responsible owner (SRO)

Decide where to host your services

There are many possible hosting options, but not all of them will necessarily make sense for your service. You should plan to spend time and effort exploring options and evaluating technical solutions according to your specific needs.

You should prefer known technologies with a proven track record like your public cloud provider’s virtual machines or basic container hosting offerings.

You should consider services with higher levels of abstraction than Public Cloud IaaS, such as:

  • AWS Copilot (to provision ECS)
  • AWS ECS on Fargate
  • AWS Lambda
  • Google App Engine
  • Google Cloud Run
  • Heroku
  • Microsoft Azure App Service

These are just some examples rather than endorsements so ensure that whatever you choose meets the needs of your service.

When making your technology choices, consider not only what is suitable for your team and department, but also what is suitable for each service in turn; not all services will be best placed running on the same technology.

For example, a static website for documentation could easily be hosted on GitHub Pages, or in an AWS S3 bucket with AWS CloudFront in front; but a complex set of microservices with multiple persistent stores would be better on ECS on Fargate, or even Kubernetes if your team/department has the necessary skills to run it.

Review the existing guidance

Here is some of the existing guidance relevant to cloud hosting.

Guidance Description
Government Cloud First Policy The government introduced a ‘Cloud First’ policy in 2013 for all technology decisions.
Technical code of practice

Point 5: Use cloud first
The Technology Code of Practice is a set of criteria to help public sector organisations design, build and buy technology.
Service Standard

9. Create a secure service which protects users’ privacy

11. Choose the right tools and technology

14. Operate a reliable service
The Service Standard helps teams to create and run great public services.
Service Manual

Technology
Helping teams to create and run great public services that meet the Service Standard.
CDDO Creating and implementing a cloud hosting strategy This guidance outlines how to create and implement a cloud strategy, and when to consider a single, hybrid or multi-cloud solution.
NCSC cloud security guidance Cloud Security Principles

Approach

  • Keep it simple
  • Find the largest building blocks you can and consider solutions that take care of the underlying infrastructure, allowing you to focus on the specifics of your service
  • Use mature, well documented technologies and that you can readily get skills for in the marketplace
  • Consider running some technical spikes to evaluate alternatives and to inform your decision

Ongoing support for your service

Once your service is migrated to a new hosting platform you will need to look after it on an ongoing basis for a long time, potentially many years

  • How is your team going to support your service over the longer term? Will there be a long lived team?
  • Do you need and can you afford to offer out of hours support, including cyber security?
  • Consider how you are going to support debugging of your applications (GOV.UK PaaS permits direct SSH access to containers but other hosting providers may not)
  • Consider how you are going to secure an engineer’s access to backing services to allow you to support your service
  • Consider how you are going to patch container vulnerabilities in a timely manner

Data migration

Data migration may involve downtime. If you need zero downtime it will not be easy or cheap

Use documented PaaS data techniques

Our technical documentation covers a number of topics relevant for migrating services.

These documented processes are self-service and can be performed by tenants independent of the GOV.UK PaaS team.

The documentation is mature and tested and allows you to coordinate any system downtime yourselves.

Connecting to GOV.UK PaaS backing services

Some GOV.UK PaaS backing services (PostgreSQL, MySQL, InfluxDB, OpenSearch and Redis) only accept connections from GOV.UK PaaS apps. You’ll need to connect through a GOV.UK PaaS app to access one of these backing services from your local machine.

Conduit lets you connect to your remote backing service instances from your local system. This allows you to use the standard tool for your backing service, for example, psql for PostgreSQL or mysql for MySQL CLI tools to make backups and interrogate your backing services.

InfluxDB times series data

Time series data is often scrapped during migration.

MySQL relational data

OpenSearch non-relational data

OpenSearch is not intended as a persistent data store, and we don’t recommend moving the data you store in it. You should instead rebuild your indices at the destination.

PostgreSQL relational data

Redis cache data

Redis is not intended as a persistent data store, and we don’t recommend moving the data you store in it. You should instead rebuild your indices at the destination. If you have been using Redis for persistent data, you will need to move it.

S3 file storage

Consider your data migration strategy options

  • Can you reduce the volume of data to migrate by archiving legacy data?
  • Can you reduce the volume data to migrate during the production migration by only synchronising changes?
  • Can you migrate your data ahead of your applications?

Do you have more complex data migration needs?

Some tenants may face data migration challenges using the documented approaches above.

Things that may affect the decision include:

  • the complexity of your data schema
  • the need for high availability
  • the difficulty of exporting a database dump owing to a variety of factors like size of data, connection speed, or the reliability of the connection

We recommend you make a small number of attempts to export a dump of your databases using conduit in order to assess how reliably you can do it.

If you think that your service will face issues migrating data using the documented approaches you should get in contact with the team using our support form to discuss your needs further.

PaaS CDN Migration

We implement Cloud Foundry CDN routes as AWS Cloudfront distributions behind the scenes.

If you are migrating from a Cloud Foundry CDN route, you will need to remove the domain from your cdn-route service before you can add it to your new AWS Cloudfront distribution. If you don't do this, you will receive a 'CNAMEAlreadyExists' error. AWS documents this issue.

You can either remove the cdn-route service:

cf delete-service my-cdn-route

or update the service to remove the domain you are migrating (if you have multiple domains):

cf update-service my-cdn-route -c '{"domain": "<remaining domains>"}'

Removing the domain and re-adding it to your new AWS Cloudfront Distribution will cause your website to go down. Unfortunately changes to AWS Cloudfront can take between 5 and 20 minutes to apply.

AWS support can transfer your domain instantly between AWS Cloudfront Distributions, however you will still get a small amount downtime as you cutover the DNS yourself. This requires some coordination and prerequisite work with AWS support and will require you to discover your current Distribution ID.

To avoid downtime you can migrate your domain to something other than AWS Cloudfront for a short time or use the wildcard approach provided by AWS.

Changes to your service when migrating

You will have to make some changes to the source code of your service, its supporting stack and pipelines to allow it to migrate to an alternative hosting provider.

Apps hosted on the GOV.UK PaaS need to follow 12 factor principles which is an established cloud best-practice that will help moving containers to another platform.

If you are using GOV.UK PaaS buildpacks you will have to find an alternative to create your containers. Cloud Native buildpacks or Packer are some examples but other solutions are available.

You will have to provision a container registry for your container images. Our tenants currently use ECR, ACR, GCR, Dockerhub, GHCR, or Quay.io, but other solutions are available.

You will need to update your pipelines to build your containers and target your chosen hosting platform. Our tenants use all of the commonly available cicd tools including Jenkins, Concourse, GitHub actions, CircleCI.

You may want to consider modifying your pipelines to deploy to the new hosting platform in parallel to the existing GOV.UK PaaS platform, so that you can develop and maintain their existing system whilst exploring their new hosting platform.

GOV.UK PaaS apps get credentials for their databases from the VCAP_SERVICES environment variable, you will need to refactor your application to solve this problem without using the Cloud Foundry conventions or you will have to implement the environment variable mapping in your platform outside of the application code.

You will need to find other means to bring your own domains to use with your applications and generate TLS certificates. GOV.UK PaaS makes use of AWS CloudFront and ACM.

PaaS uses log drains to ship logs to a permanent log service for analysis and storage. You will have to find an alternative way to do this. Using a self managed ELK stack, Logit.io and Papertrail are some examples used by our tenants but other solutions are available.

You will need to reconfigure your uptime monitoring to monitor your application. Tenants are currently using tools such as pingdom and cronitor.

You will need to find other means to implement your preferred observability tools to collect metrics and raise alerts to support staff. Tenants are using tools such as prometheus, alertmanager, grafana, sentry pagerduty.

How to contact us

If you have any questions about your migration please contact us via the GOV.UK PaaS Team (support form).


Feedback

Please let us know if this guidance is useful or not useful or raise a GitHub issue to give us feedback