How to start managing developer secrets with 1Password and Pulumi

Together, 1Password Developer and Pulumi offer a code-driven approach to tackling secret management challenges.

In this guide, we’ll show you how to use Pulumi ESC (Environments, Secrets, and Configuration) and 1Password together to have a consistent interface for working with secrets, all while following security best practices.

You’ll learn how to declare your 1Password resources (vaults, secrets, and service accounts) using Pulimi IaC (Infrastructure as Code) and manage the lifecycle via automated and consistent workflows, aligning with DevOps best practices.

What is Pulumi ESC?

Pulumi ESC is a centralized management solution for secrets and configurations. It is offered as part of Pulumi Cloud and consolidates various secret managers under a single umbrella, providing a unified consumption interface. For instance, you can use 1Password as your single source of truth to store all your secrets and make them consistently available across all environments via Pulumi ESC.

In Pulumi ESC, you organize your secrets and configurations into collections called “Environments”. Environments can be defined via a YAML/Table editor in the browser, the SDK, or the CLI. Here’s an example of an ESC Environment:

imports:
  - pulumi-esc-dev
  - oidc-gcp
values:
  1password:
    secrets:
      fn::open::1password-secrets:
        login:
          # prod-vault-read-service-account
          serviceAccountToken:
            fn::secret:
              ciphertext: ZXN...
        get:
          docker_usr:
            ref: "op://prod-vault/docker/username"
          docker_pat:
            ref: "op://prod-vault/docker/credential"
          cloudflare_token:
            ref: "op://prod-vault/cloudflare/credential"
          cloudflare_zone:
            ref: "op://prod-vault/cloudflare/zone"
          cloudflare_domain:
            ref: "op://prod-vault/cloudflare/username"
          # same name overrides inherited ones
          google_oauth_client_id:
            ref: "op://prod-vault/google-oauth/username"
          google_oauth_client_secret:
            ref: "op://prod-vault/google-oauth/credential"
  environmentVariables:
    DOCKER_PAT: ${1password.secrets.docker_pat}
    DOCKER_USR: ${1password.secrets.docker_usr}
    GOOGLE_OAUTH_CLIENT_ID: ${1password.secrets.google_oauth_client_id}
    GOOGLE_OAUTH_CLIENT_SECRET: ${1password.secrets.google_oauth_client_secret}
    CLOUDFLARE_API_TOKEN: ${1password.secrets.cloudflare_token}
    CLOUDFLARE_ZONE: ${1password.secrets.cloudflare_zone}
    CLOUDFLARE_DOMAIN: ${1password.secrets.cloudflare_domain}

With Pulumi ESC, you gain access to:

• Centralization. Manage secret sprawl with a uniform consumption interface for your applications and infrastructure.
• Hierarchy. Reference secrets and configurations defined in other ESC Environments, even if stored across multiple 1Password vaults.
• Short-term credentials. Configure OpenID Connect for AWS, Google Cloud, and Azure.

ESC simplifies secret management across diverse cloud infrastructures, enhances security, and streamlines operations for modern DevOps teams.

How to configure Pulumi ESC with 1Password

Next, we’ll show you how to connect Pulumi ESC with 1Password. You’ll learn to set up and use 1Password as a secure storage for your secrets. We’ll cover the basic steps to link these tools, making your secret management easier and safer. By the end, you’ll be ready to use 1Password and Pulumi ESC in your projects.

Before you start

• A 1Password account. You can sign up or sign in with any account (Individual, Family, Teams, or Business)
• A Pulumi Cloud account. You can sign up for a free individual account.

Configurations

You can complete the 1Password configuration using the 1Password CLI, 1Password desktop app, or the browser. You will need:

• A 1Password vault.
• A Service Account with read access to the vault. Read 1Password’s guide on how to create one.
• An “API Credential” item such as:
  • username: my-api-key-name
  • password: my-api-key-value Pulumi ESC Providers let you dynamically import secrets and configurations into your Environment. To configure the Pulumi ESC 1Password Provider, create a new ESC Environment and add the following definition to your ESC Environment:

values:   
    1password:     
        secrets:       
            fn::open::1password-secrets:         
                login:           
                    serviceAccountToken:             
                    fn::secret: ops_YOUR_SA_TOKEN  
                get:           
                    k:             
                        ref: "op://buzz-dev/API Credential/username"      
                    v:             
                        ref: "op://buzz-dev/API Credential/credential" 
    environmentVariables:     
        KEY: ${1password.secrets.k}     
        VALUE: ${1password.secrets.v}

Make sure to substitute ops_YOUR_SA_TOKEN with the 1Password Service Account token.

Lastly, you’ll need to create a Pulumi Access token to access the newly created ESC Environment programmatically. You can restrict access to the ESC Environment with RBAC if you’re on one of Pulumi’s enterprise or business-critical tiers. Learn more about all the 1Password Provider options.

Using Pulumi ESC to access 1Password secrets and more

The following scenario assumes you have configured 1Password and Pulumi. You’ve followed the configuration steps and are ready to start developing your application named Buzz.

Environment variables in a Golang web app

Buzz is a Golang web application that requires Google OAuth 2.0 and a Gemini key. Once the user is logged in, Buzz takes in a string input and spells out each letter. This is done by sending the input to Gemini. For example, the input “hello” would result in “Hotel, Echo, Lima, Lima, Oscar”. All the code is available in the Buzz GitHub repository. Here is a screenshot of the Buzz application after a user signs in and enters “pulumi” as the input:

Develop the application

The first step towards having a secure development environment is to avoid storing secrets as environment variables. To start, let’s jump into the following scenario, you are now developing the Buzz Golang web application in your local environment. Here are code snippets for the app:

var googleOauthConfig = &oauth2.Config{
    ClientID:     os.Getenv("GOOGLE_OAUTH_CLIENT_ID"),    
    ClientSecret: os.Getenv("GOOGLE_OAUTH_CLIENT_SECRET"),    
    Scopes:       
[]string{"https://www.googleapis.com/auth/userinfo.email", 
"https://www.googleapis.com/auth/userinfo.profile"},    
    Endpoint:     google.Endpoint, 
} 

//... other code 

client, err := genai.NewClient(ctx, 
option.WithAPIKey(os.Getenv("GEMINI_API_KEY")))

Traditionally, you would define local environment variables in a file or load them onto a terminal session to test the code. However, in this instance, you want to follow best practices and thus not have any locally stored environment variables. Instead, 1Password and ESC are configured so you can load the needed variables at runtime.

# sans secrets management, expected to fail as env vars not set 😢 
$ go run main.go 
2024/08/11 14:32:09 Starting HTTP Server. Listening at ":8000" 
Missing required parameter: client_id 
Error 400: invalid_request

Add Google OAuth and a Gemini Key to 1Password

Now to begin configuring 1Password and ESC, you or a platform engineer adds Google OAuth 2.0 credentials and a Gemini API Key to the 1Password development vault.

Configure Pulumi ESC to access the 1Password secrets

Next, you need to create a Pulumi ESC Environment to fetch the 1Password-stored credentials. This is done with the Pulumi ESC 1Password provider, as shown below. Learn more about the secret reference syntax.

values:   
    1password:     
        secrets:       
            fn::open::1password-secrets:         
                login:           
                    # dev-vault-read-service-account           
                    serviceAccountToken:             
                        fn::secret: ops_YOUR_SA_TOKEN         
                get:           
                    google_oauth_client_id:             
                        ref: "op://dev-vault/google-oauth/username"         
                    google_oauth_client_secret:             
                        ref: "op://dev-vault/google-oauth/credential"           
                    gemini:             
                        ref: "op://dev-vault/google-gemini/credential"   
    environmentVariables:     
        GOOGLE_OAUTH_CLIENT_ID: ${1password.secrets.google_oauth_client_id}     
        GOOGLE_OAUTH_CLIENT_SECRET: ${1password.secrets.google_oauth_client_secret}     
        GEMINI_API_KEY: ${1password.secrets.gemini}

Test the application

With the configuration in place, you can now go back to your development environment and retrieve 1Password-stored credentials at runtime using the ESC CLI. The secrets are loaded as environment variables so there is no need to make any code changes to adopt this security best practice.

# with pulumi esc + 1password integration ✨🔐✨ 
$ esc login                                              
$ esc run pulumi-esc-dev go run main.go 
2024/08/11 14:32:09 Starting HTTP Server. Listening at ":8000"

In addition, the Pulimi access token given to the developer can be restricted to only access to the pulumi-esc-dev ESC Environment and audit all its activity:

Next steps

In this blog post, you’ve learned how Pulumi ESC can enhance security across diverse development environments. Pulumi ESC provides a uniform solution regardless of where secrets are stored. This approach simplifies secret management, ensures appropriate permissions, and reduces potential vulnerabilities associated with traditional storage methods.

In particular, we’ve covered the manual creation of 1Password Service Accounts, vaults, and secret items and how to use them in a Golang application.

We encourage you to dive into the above capabilities and discover how to enhance your workflows and security practices!

Diana Esteves

Solutions Architect, Business Partners – Pulumi