- Introduction
- Setup kube2iam in your cluster
- Setting up the Role in AWS
- Troubleshooting
- Conclusion
- More Resources:
The open source project kube2iam is Amazon aws specific. It allows your pod to have access to an aws role, without other pods on the same node having the same privledges. For example if your pod needs to use the aws cli command to copy / sync files with an s3 bucket you can use kube2iam in your cluster so that ONLY your pod has those rights.
Amazon has stated at Kubecon 2017 that their EKS solution ( Amazon hosted Kubernetes cluster), that they intend to use kube2iam to handle roles and permissions for pods.
To have a seamless deployment of kube2iam in your cluster there are some steps to follow that aren't exactly in the readme files. The person responsible for deploying the kube2iam software should deploy it as a daemonset in the cluster so that every node has it. Also until the development team or ops team is comfortable with kube2iam you should probably deploy it with the --verbose and --debug flags.
To insure that all the requests for credentials get caught, I personally would recommend using the iptables=true option. Althought from personal experience I have seen where removing this option or setting it to false doesn't remove the iptable entry. You actually have to restart the node. Finally if you are using a network overlay with Kubernetes ( if not you should be ), you need to tell it what host interface to use. Personally we use Calico.
My deployment file for kube2iam looks like this:
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: kube2iam
labels:
app: kube2iam
namespace: kube-system
spec:
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
name: kube2iam
spec:
hostNetwork: true
serviceAccount: kube2iam
containers:
- image: jtblin/kube2iam:latest
name: kube2iam
args:
- "--auto-discover-base-arn"
- "--iptables=true"
- "--host-ip=$(HOST_IP)"
- "--host-interface=cali+"
- "--verbose"
- "--debug"
env:
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
ports:
- containerPort: 8181
hostPort: 8181
name: http
securityContext:
privileged: true
One final note to mention. If you are using RBAC with your Kubernetes cluster you are going to need to setup some Kubernete Roles and Service Accounts for kube2iam. I didn't find a good reference online so I created the following:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: kube2iam
namespace: kube-system
---
apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: kube2iam
rules:
- apiGroups: [""]
resources: ["namespaces","pods"]
verbs: ["get","watch","list"]
- apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: kube2iam
subjects:
- kind: ServiceAccount
name: kube2iam
namespace: kube-system
roleRef:
kind: ClusterRole
name: kube2iam
apiGroup: rbac.authorization.k8s.io
kind: ListWhen you setup a role you are saying "I give permission for my software to have access to do X on Resource Y". A role is made up of a number of Amazon policies. For example here is the only policy attached to the role 'backup-2-s3':
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::backup"
]
},
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::backup/*"
]
}
]I'm giving the user of this role the right to ListBuckets and to do whatever they want inside the s3 bucket named "backup". Let's assume that you've done this and you've deployed kube2iam with the above role and you can't get access to the s3 bucket. Well that's because you actually don't have permission to use this new role. You might have seen an error in the kube2iam logs like this:
time="2018-01-17T20:12:01Z" level=info msg="GET /latest/meta-data/iam/security-credentials/ (200) took 9281 ns" req.method=GET req.path=/latest/meta-data/iam/security-credentials/ req.remote=100.106.175.8 res.duration=9281 res.status=200
time="2018-01-17T20:12:01Z" level=error msg="Error assuming role AccessDenied: User: arn:aws:sts::221044447717:assumed-role/nodes.test.c.fobar.com/i-0d803755a438f8069 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::221044447717:role/backup-2-s3\n\tstatus code: 403, request id: adf9683b-fbc2-11e7-8d79-93262b130f27" ns.name=test pod.iam.role="arn:aws:iam::221044447717:role/backup-2-s3" req.method=GET req.path=/latest/meta-data/iam/security-credentials/backup-2-s3 req.remote=100.106.175.8
Why? keep reading...
This is the topic that confused me for a long time. It wasn't until i sat down and worked through some examples did I really understand what was happening with the trust relationship. The online AWS documentation describes configuring and using a role as:
When you run commands using the role profile, the AWS CLI uses the source profile's credentials to call AWS Security Token Service and assume the specified role. The source profile must have permission to call sts:assume-role against the role, and the role must have a trust relationship with the source profile to allow itself to be assumed.
What does this mean??? We have 2 roles:
- The first role
Role 1is the kubernetes worker node role. This is the role that every worker node has associated to it in your Kubernetes cluster. This role has numerous policies attached to it that give the role the right to DescribeInstances and get certain directories in certain s3 buckets that kops uses to setup your nodes. Role 2is the role that does the special work that you want, in this case to have access to a specific s3 bucket. This is the role that will be assumed.
You must allow Role 2 to be assumed by Role 1. So two additional steps must be done here:
- You have to modify
Role 2so that it can be Assumed byRole 1. You do this by modifying the trust relationship. You trustRole 2to give it the action to assumeRole 1.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
},
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::221044447717:role/nodes.k8s.baz.foobar.io"
]
},
"Action": "sts:AssumeRole"
}
]
}
- You now need to add the policy to allow your Kubernete worker nodes to assume roles that are not in their default role. In kops you can do this as an additional policy. Or you can just attach a new policy to your existing node role:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Action": [
"sts:AssumeRole"
],
"Resource": [
"arn:aws:iam::221044447717:instance-profile/backup-2-s3"
]
}
]
}
One thing that you could do in this additional role, since you don't want to have to add additional fields every time you add a resource you want this node to assume, you could change the arn:aws:iam::221044447717:instance-profile/backup-2-s3 to a *. That way you wouldn't have to edit it every time you want to add a new role for that node to assume.
Using kube2iam in your cluster is actually pretty easy. In your deployment file have an annotation in the metadata that specifies the name of the role that you want to use.
For example to use my backup s3 role I would add the line iam.amazonaws.com/role: backup-2-s3 under annotations:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: remote
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: remote
minReadySeconds: 5
template:
metadata:
labels:
app: remote
annotations:
iam.amazonaws.com/role: backup-2-s3
spec:
containers:
- name: remote
image: "iotapi322/worker:v4"
resources:
limits:
cpu: "200m"
memory: "256Mi"
requests:
cpu: "100m"
memory: "128Mi"To show if you truly hvae access to the s3 bucket you can do two things.
- Check if you get the correct role value back
- Try to use the resource
Well after the above steps, I verified that I was getting the correct role back:
root@remote-79cdc8dd46-mg2g5:/# curl 169.254.169.254/latest/meta-data/iam/security-credentials/
backup-2-S3
and I verfied that I could list the contents of my s3 bucket:
root@remote-79cdc8dd46-mg2g5:/# aws s3 ls s3://backup/EFSshares/
PRE api-adventurebot-01/
PRE api-adventurebot-02/
PRE hello-world-nginx/
PRE lockbot-db2/
PRE lockbot/
PRE test-permissions/
PRE test/
Troubleshooting permissions and Roles in AWS can be frustrating, however I've found that the logs from kube2iam to be really quite good. When in doubt find the node that your pod is running on. Next find the kube2iam pod that is also running on that node and start tailing the log.
You can curl inside your pod to the meta-data ip address to find out what your role is. If you have specified an annotation and you don't get that annotation your deployment is probably setup wrong.
Send your curl request set to:
curl 169.254.169.254/latest/meta-data/iam/security-credentials/ (you have to have the trailing slash), you should get returned the name of the role that you placed in the annotation. If not something has gone awry.
I would suggest going into the kubernetes dashboard and finding which node your container is running on, then find the kube2iam pod that is running on that same node and looking at the kube2iam logs.
If you don't have the trust relationship in place you might see the following log lines from the kube2iam pod
time="2018-01-18T16:12:39Z" level=info msg="GET /latest/meta-data/iam/security-credentials (301) took 802140 ns" req.method=GET req.path=/latest/meta-data/iam/security-credentials req.remote=100.96.7.199 res.duration=802140 res.status=301
time="2018-01-17T20:12:01Z" level=error msg="Error assuming role AccessDenied: User: arn:aws:sts::221044447717:assumed-role/nodes.test.c.tropo.com/i-0d803755a438f8069 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::221044447717:role/backup-2-s3\n\tstatus code: 403, request id: adf9683b-fbc2-11e7-8d79-93262b130f27" ns.name=test pod.iam.role="arn:aws:iam::221044447717:role/backup-2-s3" req.method=GET req.path=/latest/meta-data/iam/security-credentials/backup-2-s3 req.remote=100.106.175.8
The highlights from the above error blog are:
i-0d803755a438f8069 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::221044447717:role/backup-2-s3.
This is showing that the trust relationship has not been set on the worker node.
When you remote into your pod and run the aws cli command aws s3 ls s3://<s3-bucket>/ you will get back an error like this:
Unable to locate credentials. You can configure credentials by running "aws configure".
This is because you don't have a trust relationship setup between the kubernetes node and your role
I hope this helps clear up some of the confusion with Roles and Trust Relationships. kube2iam is a very powerful and necessary tool in your InfoSec toolbox.
Hi,
I followed the steps you mentioned, but unable to get it working. I get the log max up-to;
But I don't get something like 'AccessDenied' - I am not sure where to look the issue, or is just an issue with kube2iam? . Any idea?