| Tested on | Rocky Linux 10.2 (Red Quartz) workstation |
|---|---|
| Package | kubectl 1.36.3 |
| Additional tool | OpenSSL 3.x for generating the TLS lab files |
| Applies to | Any host with kubectl configured; any Kubernetes cluster |
| Cert prep | CKA · CKAD · CKS |
| Lab environment | Multi-node kubeadm cluster with containerd — install Kubernetes with kubeadm |
| Privilege | Normal user (no sudo required on the workstation) |
| Scope | Opaque, TLS, and registry Secret types; creation from literals, files, and stringData; secretKeyRef and envFrom; read-only Secret volumes; decode and inspect; immutable Secrets. Does not cover encryption at rest, external secret stores, production certificate issuance, or complete private-registry pull testing. |
A Secret keeps passwords, tokens, and keys out of container images. This walkthrough creates common Secret types in secret-lab, then consumes application credentials through secretKeyRef, envFrom, and a read-only Secret volume.
Understand Kubernetes Secrets
What a Secret Stores
A Secret holds a small amount of confidential data—passwords, API tokens, SSH keys, or TLS material. An individual Kubernetes Secret is limited to 1 MiB. Secrets are namespace-scoped. Pods read Secret data as environment variables or as read-only files on a volume.
Cluster node access in this lab assumes SSH from your workstation.
Values in the data field are base64-encoded. That is encoding for transport and storage in the API, not encryption. Access is still governed by Kubernetes API authorization and RBAC. Use Secrets for credentials; use Kubernetes ConfigMaps for non-confidential configuration.
Create the lab namespace:
kubectl create namespace secret-labSample output:
namespace/secret-lab createdMain Secret Types
Every Secret has a type that identifies its intended use. Opaque accepts arbitrary key-value data, while several built-in types require or validate particular key names or payload formats.
| Type | Typical use |
|---|---|
Opaque |
Application credentials and arbitrary key-value data |
kubernetes.io/basic-auth |
Username and password |
kubernetes.io/ssh-auth |
SSH private key |
kubernetes.io/tls |
TLS certificate and private key |
kubernetes.io/dockerconfigjson |
Private container registry authentication |
kubernetes.io/service-account-token |
Legacy or explicitly created long-lived ServiceAccount token |
Most application credentials use Opaque (the default for kubectl create secret generic). Modern Pod ServiceAccount credentials normally use projected short-lived tokens rather than long-lived Secret objects—see Kubernetes ServiceAccounts for that flow.
Create Opaque Secrets
Create from Literal Values
Literal flags suit a few key-value pairs such as a database username and password. Use demonstration values in labs—not production credentials.
kubectl create secret generic app-credentials -n secret-lab --from-literal=username=demo-user --from-literal=password=demo-passwordSample output:
secret/app-credentials createdConfirm the object exists without dumping every value:
kubectl get secret app-credentials -n secret-labExample output; AGE varies:
NAME TYPE DATA AGE
app-credentials Opaque 2 0sInspect metadata and key sizes:
kubectl describe secret app-credentials -n secret-labSample output:
Name: app-credentials
Namespace: secret-lab
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
password: 13 bytes
username: 9 bytesdescribe shows key names and byte counts, not the decoded values.
Create from Files
File-based creation loads key material from disk. The file content becomes the Secret value.
Save a demonstration API key:
printf '%s' 'sk-demo-api-key-12345' > api-key.txtThis command produces no output. printf avoids adding a trailing newline to the stored value.
Create the Secret with an explicit key name:
kubectl create secret generic application-keys -n secret-lab --from-file=api-key=api-key.txtSample output:
secret/application-keys createdUsing api-key=path sets the Secret key to api-key regardless of the source filename. You can also point --from-file at a directory so each file becomes a separate key, or use --from-env-file for a simple KEY=value list—the same pattern as ConfigMap env files.
Create a Manifest with stringData
Declarative manifests are easier to review in Git—though you should not commit real Secret values to source control. Use stringData to supply plain strings; the API stores them in encoded data.
apiVersion: v1
kind: Secret
metadata:
name: app-credentials-yaml
namespace: secret-lab
type: Opaque
stringData:
username: demo-user
password: demo-passwordCreate the Secret from the manifest:
kubectl create -f app-credentials-yaml.yamlSample output:
secret/app-credentials-yaml createdstringData is a write-only input field. The API merges these plain strings into the encoded data field and does not return stringData when the Secret is read.
kubectl get secret app-credentials-yaml -n secret-lab -o yamlAbridged sample output; server-generated metadata is omitted:
apiVersion: v1
data:
password: ZGVtby1wYXNzd29yZA==
username: ZGVtby11c2Vy
kind: Secret
metadata:
name: app-credentials-yaml
namespace: secret-lab
type: OpaqueWhen you must supply pre-encoded values directly, use data instead of stringData and base64-encode each value yourself.
Inspect and Decode a Secret
Base64 is reversible. Decoding proves the API stored your lab value—not that the Secret is encrypted.
kubectl get secret app-credentials -n secret-lab -o jsonpath='{.data.password}' | base64 -d
echoSample output:
demo-passwordUse this only with demonstration credentials. In production, limit who can read Secret objects through RBAC rather than relying on encoding for secrecy.
Use Secrets in Pods
Inject One Key with secretKeyRef
Use secretKeyRef when the application expects one credential under a specific environment variable name. The container variable does not have to match the Secret key.
apiVersion: v1
kind: Pod
metadata:
name: env-single
namespace: secret-lab
spec:
containers:
- name: app
image: busybox:1.36
command: ["sh", "-c", "echo APP_DB_USER=$APP_DB_USER; sleep 3600"]
env:
- name: APP_DB_USER
valueFrom:
secretKeyRef:
name: app-credentials
key: username
restartPolicy: NeverThe Secret and key must exist before the Pod starts unless you set optional: true on the reference. Apply the Pod and wait until it is ready:
kubectl apply -f env-single.yamlSample output:
pod/env-single createdWait for the Pod to become ready:
kubectl wait --for=condition=Ready pod/env-single -n secret-lab --timeout=60sSample output:
pod/env-single condition metThe command returns after the Pod reports the Ready condition. The kubelet decodes the Secret value when setting the environment variable:
kubectl exec env-single -n secret-lab -- printenv APP_DB_USERSample output:
demo-userEnvironment variables are fixed at container start. Updating the Secret later does not change values in a running process—see how ConfigMap and Secret updates reach running Pods.
Import Keys with envFrom
When the application expects environment variable names that match Secret keys, envFrom imports every key in one block.
apiVersion: v1
kind: Pod
metadata:
name: env-from
namespace: secret-lab
spec:
containers:
- name: app
image: busybox:1.36
command: ["sh", "-c", "env | grep -E '^(username|password)=' | sort; sleep 3600"]
envFrom:
- secretRef:
name: app-credentials
restartPolicy: NeverApply the Pod:
kubectl apply -f env-from.yamlSample output:
pod/env-from createdWait for Ready, then read the imported variables:
kubectl wait --for=condition=Ready pod/env-from -n secret-lab --timeout=60sSample output:
pod/env-from condition metBoth keys appear as environment variables with matching names:
kubectl logs env-from -n secret-labSample output:
password=demo-password
username=demo-userPrefer explicit secretKeyRef entries when you need to rename keys, import only a subset, or mix Secrets with other env sources.
Mount Secret Keys as Files
A Secret volume projects each key as a filename. Mounts are read-only. Use a dedicated directory such as /etc/app-secret rather than masking files baked into the image.
Mount the app-credentials Secret at /etc/app-secret with selected keys and custom filenames:
apiVersion: v1
kind: Pod
metadata:
name: volume-mount
namespace: secret-lab
spec:
containers:
- name: app
image: busybox:1.36
command: ["sh", "-c", "ls -l /etc/app-secret; cat /etc/app-secret/username; echo; sleep 3600"]
volumeMounts:
- name: app-secret-vol
mountPath: /etc/app-secret
readOnly: true
volumes:
- name: app-secret-vol
secret:
secretName: app-credentials
defaultMode: 0440
items:
- key: username
path: username
- key: password
path: db-password
restartPolicy: NeverThe items list selects keys and renames the target files. defaultMode sets file permissions (here 0440). Apply the Pod:
kubectl apply -f volume-mount.yamlSample output:
pod/volume-mount createdWait for the Pod, then inspect the mount:
kubectl wait --for=condition=Ready pod/volume-mount -n secret-lab --timeout=60sSample output:
pod/volume-mount condition metThe startup log lists the projected files and prints the username value:
kubectl logs volume-mount -n secret-labExample output; timestamps and symlink lengths vary:
total 0
lrwxrwxrwx 1 root root 18 Jul 26 15:55 db-password -> ..data/db-password
lrwxrwxrwx 1 root root 15 Jul 26 15:55 username -> ..data/username
demo-userConfirm the projected file permissions:
kubectl exec volume-mount -n secret-lab -- stat -L -c '%a %n' /etc/app-secret/db-password /etc/app-secret/usernameSample output:
440 /etc/app-secret/db-password
440 /etc/app-secret/usernameNormal Secret volumes are eventually refreshed when the source object changes. Files mounted with subPath do not—see how ConfigMap and Secret updates reach running Pods and subPath volume examples.
Create Typed Secrets
TLS Secret
TLS Secrets store a certificate and private key for Ingress controllers, web servers, or other TLS-terminated workloads. Generate or obtain tls.crt and tls.key first with OpenSSL—this section covers the Kubernetes object, not production certificate issuance.
openssl req \
-x509 \
-noenc \
-newkey rsa:2048 \
-keyout tls.key \
-out tls.crt \
-days 30 \
-subj '/CN=web.example.com' \
-addext 'subjectAltName=DNS:web.example.com'OpenSSL prints key-generation progress to standard error. The sequence of dots and plus signs varies, so no fixed sample output is shown.
Create the TLS Secret from the generated certificate and private key:
kubectl create secret tls web-tls -n secret-lab --cert=tls.crt --key=tls.keySample output:
secret/web-tls createdConfirm the type and expected keys:
kubectl describe secret web-tls -n secret-labExample output; certificate and key byte counts vary:
Name: web-tls
Namespace: secret-lab
Labels: <none>
Annotations: <none>
Type: kubernetes.io/tls
Data
====
tls.crt: 1168 bytes
tls.key: 1704 byteskubectl create secret tls requires a PEM-encoded certificate whose public key matches the supplied private key. How Ingress or an application consumes web-tls belongs in the relevant networking or workload guide.
Private Registry Secret
A registry Secret stores Docker config JSON for pulling images from a private registry.
kubectl create secret docker-registry registry-credentials -n secret-lab --docker-server=registry.example.com --docker-username=demo --docker-password=demo-pass --docker-email=demo@example.comSample output:
secret/registry-credentials createdThe object type is kubernetes.io/dockerconfigjson:
kubectl get secret registry-credentials -n secret-lab -o jsonpath='{.type}{"\n"}'Sample output:
kubernetes.io/dockerconfigjsonReference the Secret in imagePullSecrets on the Pod spec, or attach it to a ServiceAccount so Pods using that account inherit pull credentials. Complete registry authentication, verification, and failure handling are covered in pull images from a private registry and ImagePullBackOff troubleshooting.
Immutable Secrets
Set immutable: true when Secret data should never change in place—similar to immutable ConfigMaps:
apiVersion: v1
kind: Secret
metadata:
name: app-secret-immutable
namespace: secret-lab
type: Opaque
stringData:
token: release-token
immutable: trueCreate the immutable Secret, then try to patch it:
kubectl create -f app-secret-immutable.yamlSample output:
secret/app-secret-immutable createdA patch against stringData is rejected:
kubectl patch secret app-secret-immutable -n secret-lab -p '{"stringData":{"token":"changed"}}'Sample output:
The Secret "app-secret-immutable" is invalid: data: Forbidden: field is immutable when `immutable` is setOnce a Secret is marked immutable, you cannot change its data or set immutable back to false. Create a replacement Secret, or delete and recreate the object. Pods mounting a deleted immutable Secret should also be recreated.
Create a new Secret name and update the workload reference to ship new credentials. Replacement and rollout behaviour are covered in how ConfigMap and Secret updates reach running Pods.
Common Secret Problems
| Error or symptom | Likely cause | Fix |
|---|---|---|
Pod has CreateContainerConfigError or a Secret volume cannot mount |
Secret name or namespace is wrong | Run kubectl describe pod <pod> -n <namespace> and create the Secret in the same namespace; the kubelet retries automatically |
| Required key not found | Typo in secretKeyRef.key or volumes[].secret.items[].key |
Compare the Pod reference with kubectl describe secret <name> -n <namespace>; a missing required key prevents container startup |
| Invalid base64 data | Malformed value in data |
Use stringData in manifests, or encode with base64 -w0 before placing values in data |
TLS Secret missing tls.crt or tls.key |
Wrong keys or wrong type |
Use kubectl create secret tls or ensure keys match the kubernetes.io/tls layout |
| Registry Secret in wrong namespace | imagePullSecrets is namespace-scoped |
Recreate the Secret in the Pod namespace or reference it through a ServiceAccount in that namespace |
| Environment variable unchanged after Secret edit | Env vars are set at container start | Roll out new Pods—see the propagation guide |
| Volume file updated but app unchanged | Application reads config only at startup | Reload through app mechanism or restart containers—see the propagation guide |
What's Next
- Kubernetes Multi-Tenancy and Workload Isolation
- Run Sandboxed Containers with RuntimeClass and gVisor
- Encrypt Kubernetes Pod Traffic with Cilium WireGuard
References
- Secrets — Secret types, limits, updates and immutable Secrets
- Managing Secrets using kubectl — literals and file-based creation
- Managing Secrets using Configuration File —
dataandstringData - Distribute Credentials Securely Using Secrets — environment variables and Secret volumes
- kubectl create secret — imperative Secret commands
- kubectl create secret tls — TLS certificate and key requirements
- kubectl create secret docker-registry — registry Secret creation options
- kubectl wait — readiness wait behavior
- openssl req — self-signed certificate and private-key generation
Summary
You created Opaque application Secrets from literals, files, and stringData manifests, then verified metadata with kubectl get and describe. The same app-credentials Secret fed three consumption paths: one renamed environment variable through secretKeyRef, all keys through envFrom, and selected keys through a read-only volume.
TLS and registry Secrets use typed layouts—tls.crt/tls.key for kubernetes.io/tls and Docker config JSON for kubernetes.io/dockerconfigjson. Base64 in the API is encoding, not encryption; RBAC and operational practices protect access. When credentials change, plan for Pod recreation or application reload—environment variables and subPath mounts do not update in place.

