Our Enhancing infrastructure security delved into the complexities of security observability and the proactive measures taken. We now shift our focus to another critical facet of infrastructure security. In our upcoming article, we'll continue our journey by addressing the importance of key management and rotation. As we transition from the AWS ecosystem to Google Cloud, we'll uncover the challenges and solutions related to GCP service account key rotation.
There is no doubt that security is a high-priority aspect of any infrastructure, no matter the scale of your project. One of the most common practices, along with the least privilege concept, is regular key rotation, which prevents your keys from leaking and, thus, untrusted access to your resources. This post will show how we implemented a simple GCP service account key rotation mechanism using Kubernetes and Python.
First, let’s dive into the problem itself. IAM service accounts, which give your workloads programmatic access to your Google Cloud resources, are heavily used everywhere. Although there are some alternatives, such as workload identities, you will likely still rely on service accounts. In our case, we had a few of them used in GitLab CI pipelines.
The solution is pretty straightforward - a Python script can access GCP and GitLab, running in CronJob periodically. The writing itself accomplishes this in three steps - deletes the last service account key, generates a new one, and sends JSON to GitLab destination (is it group or project variable). Here is the script:
import os
from google.oauth2 import service_account
import googleapiclient.discovery # type: ignore
import gitlab
import base64
def create_key(service_account_id: str) -> str:
"""Creates a key for a service account."""
service = googleapiclient.discovery.build("iam", "v1")
key = (
service.projects()
.serviceAccounts()
.keys()
.create(name=f"projects/{os.environ['PROJECT_ID']}/serviceAccounts/" + str(service_account_id), body={})
.execute()
)
json_key = base64.b64decode(key['privateKeyData']).decode('utf-8')
print("Key created")
return json_key
def delete_key() -> None:
"""Deletes a service account key."""
project_id = os.environ["PROJECT_ID"]
sa_id = os.environ["SA_ID"]
sa_full_name = f'projects/{project_id}/serviceAccounts/{sa_id}'
service = googleapiclient.discovery.build("iam", "v1")
key_list = service.projects().serviceAccounts().keys().list(name=sa_full_name).execute()['keys']
print(key_list)
key_to_delete = key_list[-1]['name']
service.projects().serviceAccounts().keys().delete(name=key_to_delete).execute()
print("Deleted key: " + key_to_delete)
def update_gl_var(json_key: str) -> None:
gl = gitlab.Gitlab(url=os.environ["GITLAB_HOST"], private_token=os.environ["GITLAB_TOKE_1"])
group = gl.groups.get(os.environ["GL_GROUP_ID"])
variable = group.variables.get(os.environ["GROUP_VAR_NAME"])
variable.value = json_key
variable.save()
glElixir = gitlab.Gitlab(url=os.environ["GITLAB_HOST"], private_token=os.environ["GITLAB_TOKEN_2"])
projectElixir = glElixir.projects.get(os.environ["GL_PROJECT_ID"])
variableElixir = projectElixir.variables.get(os.environ["VAR_NAME"])
variableElixir.value = json_key
variableElixir.save()
delete_key()
json_key = create_key(os.environ["SA_ID"])
update_gl_var(json_key)As you can see, it's pretty simplistic and straightforward. In this example, we rotate the key for one project variable and a group variable.
Also, let’s look at Dockerfile since the base image and libs needed are essential, although Dockerfile itself is pretty simple.
FROM python:3.8.2-alpine
WORKDIR /app
RUN apk add --no-cache --virtual .build-deps g++ python3-dev
RUN pip install --upgrade pip
RUN pip install --upgrade setuptools wheel
RUN pip install --upgrade google-api-python-client
RUN pip install google-cloud-iam
RUN pip install --upgrade python-gitlab
COPY . .
CMD ["python", "/app/script.py"]Now, let’s take a look at Kubernetes resources. We already mentioned that CronJob was used, but there are two things we should keep in mind - GitLab tokens and GCP access for the script itself. Tokens are managed pretty quickly since they sit in Kubernetes Secret resources and are later mounted to CronJob’s pod as an environment variable. If you are worried about GitLab tokens, you can keep them in GCP Secret Manager and mount values from Secret Manager directly with the secret store CSI provider.
But what should we do with the GCP access your workload requires in order to delete and create keys? Creating a new IAM service account doesn’t look like a good answer since the key for this service account should be rotated as well, and thus, we increase configuration and management overhead. The best solution here is to use the workload identity we mentioned earlier.
In this case, it suits us perfectly well since we use GKE and can easily map Kubernetes service accounts with GCP service accounts without creating and managing any keys. All you need is to create a Kubernetes service account with a specific annotation that references a Google service account with a Service Account Key Admin role, configure principals in GCP, and you are good to go. Generally speaking, it’s always a preferable way of Google Cloud access for your GKE workloads.
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
iam.gke.io/gcp-service-account: 'key-rotation-controller@my-project-id.iam.gserviceaccount.com'
name: rotation-controller-sa
namespace: sa-key-rotation-controllerThe overall manifest structure looks like this. We won’t get into details of Kubernetes resources configuration since they are widespread, and it's not the main topic of this post.
That’s it, I hope you like it. Another primary purpose of this post is to show you how easy it is to overcome some uncommon issues with some scripting and creativity. Don’t be afraid to create new tools and experiment, even if it’s something as simple as our key rotation controller.
Streamline CORS for your APIs on AWS Gateway with Terraform and Lambda secure scale done
Cut your Kubernetes cloud bill with these 5 hacks for smarter scaling and resource tuning
PostgreSQL blends relational and NoSQL for modern app needs