CHANGELOG-1.31.md

v1.31.0

Documentation

Downloads for v1.31.0

Source Code

filename	sha512 hash
kubernetes.tar.gz	6343ef4fe96441c9f4e5da359ef90ab10f14d6e51ac41094bea29a624683f9c4527d835b6c3a644afd5b0b0dd60400c1b86a9f05b0cf71ef16a8bb6b6fb72d0f
kubernetes-src.tar.gz	5565c7d99601ff9fd2fae7b37b94d5333201a9745c27dc79c38aa6883204e5e447098c1f04b84dcb1485e42bc6ec9619b8b813f27871709b615f638b42f8ded4

Client Binaries

filename	sha512 hash
kubernetes-client-darwin-amd64.tar.gz	bafae6dba3a6cbe07bee62d49f30f435378e5a4ac7df364ea7a5bc1d4654c8f9a7f4a6b5f37afebaad24ed5d75c9bde5172548be58c303799650503aaad22e6e
kubernetes-client-darwin-arm64.tar.gz	d7b55d624e8faf88a63ea2c3327378de558a45ca6f091caebd761eca4c37d340fcae038b19dd493011da9b67141a0ce51736324df5269261d68ba2de96d8c786
kubernetes-client-linux-386.tar.gz	2f9cca6b0c36d70bb599b784813a1457a2a2fd71925c84eef1cc7a6274e5cd05a78302118d83a8fec46e21162a4130e004b9a57bb5de21b386133278c5695504
kubernetes-client-linux-amd64.tar.gz	36242264b366378ec202ee4117782eb26ff14c0a96526a42caa9e36246452faa7bb2ea0732a0818d56fe074bbaf0674321660d791fce1e9e132d508f2e52c7f9
kubernetes-client-linux-arm.tar.gz	baad6720df7215e2b3e2bb343c371a8e90c25fa5dce81e53e287d29ff10efe9a92a793b234d357fb70a0b0d40ee386f7ab9a893c0e21a798a6f98117d574b0ce
kubernetes-client-linux-arm64.tar.gz	ee3239e13a94bc22f8718755c41662a5ab739e1f929581fd3f8a881201a192cd677e7d342bb78796a04cb770966056938ef1b601b81c649bfa10fe5596e00d98
kubernetes-client-linux-ppc64le.tar.gz	8cf97b790a22d38d1196c9696a8ff83aee03ff28ab976b5f3038d960862dbf0de1c79a3dfdc725c8a9e5d643c17e2c01534208acf8f28b1eee363f17e66e664c
kubernetes-client-linux-s390x.tar.gz	09922e8bb10b44055e87f3359c14984d0c15306f84624657d8b03121f44675560c73d7423c91d32152be85ebcfbdebefcba45ad801e2557b10e3ab4232b94563
kubernetes-client-windows-386.tar.gz	d5e3fbc75e4b46043cc46a8d89b7c93d4cfbe3217a0ad2618794576a99a2709088e1b530ca97b5262baa7ffe2795bcd9b59f0fb5d7d86fa6a41e0613200b6c7b
kubernetes-client-windows-amd64.tar.gz	2a86c2fcdb9be3819e021149394485b3717c6fc3285aa911b04a975417c5ac686e8fc86506d9ec77fc4adf08e5559f44c0c8fd83ae66cb2eccdad025b7476755
kubernetes-client-windows-arm64.tar.gz	7599b4cf90025c747393abf6732fd064ab7b11008d1145dcdbe98b9bf7ec0a98e2fbf6b1d3cf52d6dc48be1378e527735744587c6044811fa98c8695c6b2ed95

Server Binaries

filename	sha512 hash
kubernetes-server-linux-amd64.tar.gz	4d73777e4f139c67c4551c1ca30aefa4782b2d9f3e5c48b8b010ffc329065e90ae9df3fd515cc13534c586f6edd58c3324943ce9ac48e60bb4fa49113a2e09d4
kubernetes-server-linux-arm64.tar.gz	3978a6cd8bed01efbaf6955a741122e3d39173e2a380396214672c96c7dbf5da8e275c4cea716a1789d3034ed9649417ad43d3d73c2dcaae5df4c91ff0f4bdaf
kubernetes-server-linux-ppc64le.tar.gz	ceaa8327e96f17baaa883a47cfdc3d8281658ad9e5fb3f652141570de936e2afa296cf7140db6a8b394bd3346a0fc43bdaf31d302f83ac8ecb5d613348d2ba1f
kubernetes-server-linux-s390x.tar.gz	8f36054506d73f13a7795b74074642613ddc5ebf5b0f6a8b02e674d1e2e668785bbdc74c9bf4c0bf4b7ca0758f2ffa97b57729ecbd3bc89e23243bbdf48825ef

Node Binaries

filename	sha512 hash
kubernetes-node-linux-amd64.tar.gz	8c196abfbf0f6a781fdc0308f83be1bbeecf8e6286e397a208a895638b2a6b4a5e2477eeb902e4989dc60b6b0a1abb6097ca50c31ea39bb0948f028bcaf41095
kubernetes-node-linux-arm64.tar.gz	6c20f283e3297274e185bcd828d2ea38a9a85055828abdc33845367e9d279a1873630a872d1510bc98197e0cfdf7ea85fdd140e9223998103ab8c8cbf89d2fb1
kubernetes-node-linux-ppc64le.tar.gz	3d12f9f96e6d9621578948a94b399ae4090fa082fa62f525e24cb3dc7caa53588e81c2d764e9fb7187903df6559ad74a84027676101b923db7b8ab3c9fa7c19e
kubernetes-node-linux-s390x.tar.gz	f17dbe3438d0cd0d921ee6eaa62ba59c7eb1f15df9957cac2923a07e00d134d4c6ebc9b2c8a5e55242e304c3916e0217f3977d5a8690d943b50be1364db41e28
kubernetes-node-windows-amd64.tar.gz	36f25295089dc706920a7dad80d614f3379f610112d21652ae5356bd56d7f2b6e127cac35ff4d1b0a8f1ddc22fdcff6bf1a2e35b033f20673a5c401247eb23c1

Container Images

All container images are available as manifest lists and support the described architectures. It is also possible to pull a specific architecture directly by adding the "-$ARCH" suffix to the container image name.

name	architectures
registry.k8s.io/conformance:v1.31.0	amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.31.0	amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.31.0	amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.31.0	amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.31.0	amd64, arm64, ppc64le, s390x
registry.k8s.io/kubectl:v1.31.0	amd64, arm64, ppc64le, s390x

Changelog since v1.30.0

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Added support to the scheduler to start using QueueingHint registered for Pod/Updated event to determine whether unschedulable Pods update make them schedulable, when the feature gate SchedulerQueueingHints is enabled. Previously, when unschedulable Pods are updated, the scheduler always put Pods back to activeQ/backoffQ. But, actually not all updates to Pods make Pods schedulable, especially considering many scheduling constraints nowadays are immutable. Now, when unschedulable Pods are updated, the scheduling queue checks with QueueingHint(s) whether the update may make the pods schedulable, and requeues them to activeQ/backoffQ only when at least one QueueingHint(s) return Queue.

Action required for custom scheduler plugin developers: Plugins have to implement a QueueingHint for Pod/Update event if the rejection from them could be resolved by updating unscheduled Pods themselves. Example: suppose you develop a custom plugin that denies Pods that have a schedulable=false label. Given Pods with a schedulable=false label will be schedulable if the schedulable=false label is removed, this plugin would implement QueueingHint for Pod/Update event that returns Queue when such label changes are made in unscheduled Pods. (#122234, @AxeZhan) [SIG Scheduling and Testing] [sig/scheduling,sig/testing]

• Kubelet flag --keep-terminated-pod-volumes was removed. This flag was deprecated in 2017. (#122082, @carlory) [SIG Apps, Node, Storage and Testing] [sig/storage,sig/node,sig/apps,sig/testing]
• Reduced state change noise when volume expansion fails. Also mark certain failures as infeasible.

ACTION REQUIRED: If you are using the RecoverVolumeExpansionFailure alpha feature gate then after upgrading to this release, you need to update some objects. For any existing PersistentVolumeClaimss with status.allocatedResourceStatus set to either "ControllerResizeFailed" or "NodeResizeFailed", clear the status.allocatedResourceStatus. (#126108, @gnufied) [SIG Apps, Auth, Node, Storage and Testing] [sig/storage,sig/node,sig/auth,sig/apps,sig/testing]

Changes by Kind

Deprecation

• 'kubeadm: marked the sub-phase of ''init kubelet-finilize'' called ''experimental-cert-rotation'' as deprecated and print a warning if it is used directly; it will be removed in a future release. Add a replacement sub-phase ''enable-client-cert-rotation''.' (#124419, @neolit123) [SIG Cluster Lifecycle] [sig/cluster-lifecycle]
• Added a warning when creating or updating a PersistentVolume (PV) with the deprecated annotation volume.beta.kubernetes.io/mount-options. (#124819, @carlory) [sig/storage]
• CephFS volume plugin ( kubernetes.io/cephfs) was removed in this release and the cephfs volume type became non-functional. Alternative is to use CephFS CSI driver (https://github.com/ceph/ceph-csi/) in your Kubernetes Cluster. A re-deployment of your application is required to use the new driver if you were using kubernetes.io/cephfs volume plugin before upgrading cluster version to 1.31+. (#124544, @carlory) [SIG Node, Scalability, Storage and Testing] [sig/scalability,sig/storage,sig/node,sig/testing]
• CephRBD volume plugin ( kubernetes.io/rbd) was removed in this release. And its csi migration support was also removed, so the rbd volume type became non-functional. Alternative is to use RBD CSI driver (https://github.com/ceph/ceph-csi/) in your Kubernetes Cluster. A re-deployment of your application is required to use the new driver if you were using kubernetes.io/rbd volume plugin before upgrading cluster version to 1.31+. (#124546, @carlory) [SIG Node, Scalability, Scheduling, Storage and Testing] [sig/scalability,sig/scheduling,sig/storage,sig/node,sig/testing]
• Kube-scheduler deprecated all non-csi volumelimit plugins and removed those from defaults plugins.
• AzureDiskLimits
• CinderLimits
• EBSLimits
• GCEPDLimits

The NodeVolumeLimits plugin can handle the same functionality as the above plugins since the above volume types are migrated to CSI. Please remove those plugins and replace them with the NodeVolumeLimits plugin if you explicitly use those plugins in the scheduler config. Those plugins will be removed in the release 1.32. (#124500, @carlory) [SIG Scheduling and Storage] [sig/scheduling,sig/storage]

• Kubeadm: deprecated the kubeadm RootlessControlPlane feature gate (previously alpha), given that the core K8s UserNamespacesSupport feature gate graduated to beta in 1.30. Once core Kubernetes support for user namespaces is generally available and kubeadm has started to support running the control plane in userns pods, the kubeadm RootlessControlPlane feature gate will be removed entirely. Until kubeadm supports the userns functionality out of the box, users can continue using the deprecated RootlessControlPlane feature gate, or opt-in UserNamespacesSupport by using kubeadm patches on the static pod manifests. (#124997, @neolit123) [SIG Cluster Lifecycle] [sig/cluster-lifecycle]
• Removed k8s.io/legacy-cloud-providers from staging. (#124767, @carlory) [SIG API Machinery, Cloud Provider and Release] [sig/api-machinery,sig/release,sig/cloud-provider]
• Removed legacy cloud provider integration code (undoing a previous reverted commit). (#124886, @carlory) [SIG Cloud Provider and Release] [sig/release,sig/cloud-provider]

API Change

• 'ACTION REQUIRED: The Dynamic Resource Allocation (DRA) driver's DaemonSet must be deployed with a service account that enables writing ResourceSlice and reading ResourceClaim objects.' (#125163, @pohly) [SIG Auth, Node and Testing] [sig/node,sig/auth,sig/testing]
• Add UserNamespaces field to NodeRuntimeHandlerFeatures (#126034, @sohankunkerkar) [SIG API Machinery, Apps and Node] [sig/node,sig/api-machinery,sig/apps]
• Added Coordinated Leader Election as Alpha under the CoordinatedLeaderElection feature gate. With the feature enabled, the control plane can use LeaseCandidate objects (coordination.k8s.io/v1alpha1 API group) to participate in a leader election and let the kube-apiserver select the best instance according to some strategy. (#124012, @Jefftree) [SIG API Machinery, Apps, Auth, Cloud Provider, Etcd, Node, Release, Scheduling and Testing] [sig/scheduling,sig/node,sig/api-machinery,sig/auth,sig/apps,sig/testing,sig/release,sig/cloud-provider,sig/etcd]
• Added a .status.features.supplementalGroupsPolicy field to Nodes. The field is true when the feature is implemented in the CRI implementation (KEP-3619). (#125470, @everpeace) [SIG API Machinery, Apps, Node and Testing] [sig/node,sig/api-machinery,sig/apps,sig/testing]
• Added an allocatedResourcesStatus to each container status to indicate the health status of devices exposed by the device plugin. (#126243, @SergeyKanzhelev) [SIG API Machinery, Apps, Node and Testing] [sig/node,sig/api-machinery,sig/apps,sig/testing]
• Added support to the kube-proxy nodePortAddresses / --nodeport-addresses option to accept the value "primary", meaning to only listen for NodePort connections on the node's primary IPv4 and/or IPv6 address (according to the Node object). This is strongly recommended, if you were not previously using --nodeport-addresses, to avoid surprising behavior. (This behavior is enabled by default with the nftables backend; you would need to explicitly request --nodeport-addresses 0.0.0.0/0,::/0 there to get the traditional "listen on all interfaces" behavior.) (#123105, @danwinship) [SIG API Machinery, Network and Windows] [sig/network,sig/api-machinery,sig/windows]
• Added the feature gates StrictCostEnforcementForVAP and StrictCostEnforcementForWebhooks to enforce the strict cost calculation for CEL extended libraries. It is strongly recommended to turn on the feature gates as early as possible. (#124675, @cici37) [SIG API Machinery, Auth, Node and Testing] [sig/node,sig/api-machinery,sig/auth,sig/testing]
• Changed how the API server handles updates to .spec.defaultBackend of Ingress objects. Server-side apply now considers .spec.defaultBackend to be an atomic struct. This means that any field-owner who sets values in that struct (they are mutually exclusive) owns the whole struct. For almost all users this change has no impact; for controllers that want to change the default backend port from number to name (or vice-versa), this makes it easier. (#126207, @thockin) [SIG API Machinery] [sig/api-machinery]
• Component-base/logs: when compiled with Go >= 1.21, component-base will automatically configure the slog default logger together with initializing klog. (#120696, @pohly) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Storage and Testing] [sig/network,sig/storage,sig/api-machinery,sig/cluster-lifecycle,sig/auth,sig/cli,sig/instrumentation,sig/testing,sig/architecture,sig/cloud-provider]
• CustomResourceDefinition objects created with non-empty caBundle fields which are invalid or do not contain any certificates will not appear in discovery or serve endpoints until a valid caBundle is provided. Updates to CustomResourceDefinition are no longer allowed to transition a valid caBundle field to an invalid caBundle field, because this breaks serving of the existing CustomResourceDefinition. (#124061, @Jefftree) [SIG API Machinery] [sig/api-machinery]
• Dynamic Resource Allocation (DRA): Added a feature so the number of ResourceClaim objects can be limited per namespace and by the number of devices requested through a specific class via the v1.ResourceQuota mechanism. (#120611, @pohly) [SIG API Machinery, Apps, Auth, CLI, Etcd, Node, Release, Scheduling and Testing] [sig/scheduling,sig/node,sig/api-machinery,sig/auth,sig/apps,sig/cli,sig/testing,sig/release,sig/etcd]
• Dynamic Resource Allocation (DRA): client-side validation of a ResourceHandle would have accepted a missing DriverName, whereas server-side validation then would have raised an error. (#124075, @pohly) [sig/apps]
• Dynamic Resource Allocation (DRA): in the pod.spec.recourceClaims array, the source indirection is no longer necessary. Instead of e.g. source: resourceClaimTemplateName: my-template, one can write resourceClaimTemplateName: my-template. (#125116, @pohly) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing] [sig/scheduling,sig/node,sig/api-machinery,sig/auth,sig/apps,sig/testing]
• Enhanced the Dynamic Resource Allocation (DRA) with an updated version of the resource.k8s.io API group. The primary user-facing type remains the ResourceClaim, however significant changes have been made, resulting in the new version, v1alpha3, which is not compatible with the previous version. (#125488, @pohly) [SIG API Machinery, Apps, Auth, CLI, Cluster Lifecycle, Etcd, Node, Release, Scheduling, Storage and Testing] [sig/scheduling,sig/storage,sig/node,sig/api-machinery,sig/cluster-lifecycle,sig/auth,sig/apps,sig/cli,sig/testing,sig/release,sig/etcd]
• Fixed a 1.30.0 regression in OpenAPI descriptions of the imagePullSecrets and hostAliases fields to mark the fields used as keys in those lists as either defaulted or required. (#124553, @pmalek) [sig/api-machinery]
• Fixed a 1.30.0 regression in openapi descriptions of PodIP.IP and HostIP.IP fields to mark the fields used as keys in those lists as required. (#126057, @thockin) [sig/api-machinery]
• Fixed a bug in the API server where empty collections of ValidatingAdmissionPolicies did not have an items field. (#124568, @xyz-li) [SIG API Machinery] [sig/api-machinery]
• Fixed a deep copy issue when retrieving the controller reference. (#124116, @HiranmoyChowdhury) [SIG API Machinery and Release] [sig/api-machinery,sig/release]
• Fixed code-generator client-gen to work with api/v1-like package structure. (#125162, @sttts) [SIG API Machinery and Apps] [sig/api-machinery,sig/apps]
• Fixed incorrect "v1 Binding is deprecated in v1.6+" warning in kube-scheduler log. (#125540, @pohly) [SIG API Machinery] [sig/api-machinery]
• Fixed the comment for the Job's managedBy field. (#124793, @mimowo) [SIG API Machinery and Apps] [sig/api-machinery,sig/apps]
• Fixed the documentation for the default value of the procMount entry in securityContext within a Pod. The documentation was previously using the name of the internal variable DefaultProcMount, rather than the actual value, "Default". (#125782, @aborrero) [SIG Apps and Node] [sig/node,sig/apps]
• Graduate PodDisruptionConditions to GA and lock (#125461, @mimowo) [SIG Apps, Node, Scheduling and Testing] [sig/scheduling,sig/node,sig/apps,sig/testing]
• Graduated MatchLabelKeys/MismatchLabelKeys feature in PodAffinity/PodAntiAffinity to Beta. (#123638, @sanposhiho) [SIG API Machinery, Apps, Scheduling and Testing] [sig/scheduling,sig/api-machinery,sig/apps,sig/testing]
• Graduated JobPodFailurePolicy to GA and locked it to it's default. (#125442, @mimowo) [SIG API Machinery, Apps, Scheduling and Testing] [sig/scheduling,sig/api-machinery,sig/apps,sig/testing]
• Graduated the Job successPolicy field to beta.

The new reason label, "SuccessPolicy" and "CompletionsReached" are added to the "jobs_finished_total" metric. Additionally, if you enable the JobSuccessPolicy feature gate, the Job gets "CompletionsReached" reason for the "SuccessCriteriaMet" and "Complete" condition type when the number of succeeded Job Pods (.status.succeeded) reached the desired completions (.spec.completions). (#126067, @tenzen-y) [SIG API Machinery, Apps and Testing] [sig/api-machinery,sig/apps,sig/testing]

• Graduated the DisableNodeKubeProxyVersion feature gate to beta. By default, the kubelet no longer attempts to set the .status.kubeProxyVersion field for its associated Node. (#123845, @HirazawaUi) [SIG API Machinery, Cloud Provider, Network, Node and Testing] [sig/network,sig/node,sig/api-machinery,sig/testing,sig/cloud-provider]
• Improved scheduling performance when many nodes, and prefilter returned 1-2 nodes (e.g. daemonset)

For developers of out-of-tree PostFilter plugins, note that the semantics of NodeToStatusMap are changing: A node with an absent value in the NodeToStatusMap should be interpreted as having an UnschedulableAndUnresolvable status. (#125197, @gabesaba) [sig/scheduling]

• Introduced a new boolean kubelet flag --fail-cgroupv1. (#126031, @harche) [SIG API Machinery and Node] [sig/node,sig/api-machinery]
• K8s.io/apimachinery/pkg/util/runtime: Added support for new calls to handle panics and errors in the context where they occur. PanicHandlers and ErrorHandlers now must accept a context parameter for that. Log output is structured instead of unstructured. (#121970, @pohly) [SIG API Machinery and Instrumentation] [sig/api-machinery,sig/instrumentation]
• KEP-1880: Users of the new feature to add multiple service CIDR will use by default a dual-write strategy on the new ClusterIP allocators to avoid the problem of possible duplicate IPs allocated to Services when running skewed kube-apiservers using different allocators. They can opt-out of this behavior by enabled the feature gate DisableAllocatorDualWrite. (#122047, @aojea) [SIG API Machinery, Apps, Instrumentation and Testing] [sig/api-machinery,sig/apps,sig/instrumentation,sig/testing]
• Kube-apiserver: Added Alpha features to allow API server authz to check the context of requests:
• The AuthorizeWithSelectors feature gate enables including field and label selector information from requests in webhook authorization calls.
• The AuthorizeNodeWithSelectors feature gate changes node authorizer behavior to limit requests from node API clients, so that each Node can only get / list / watch its own Node API object, and can also only get / list / watch Pod API objects bound to that node. Clients using kubelet credentials to read other nodes or unrelated pods must change their authentication credentials (recommended), adjust their usage, or obtain broader read access independent of the node authorizer. (#125571, @liggitt) [SIG API Machinery, Auth, Node, Scheduling and Testing] [sig/scheduling,sig/node,sig/api-machinery,sig/auth,sig/testing]
• Kube-apiserver: ControllerRevision objects are now verified to contain valid JSON data in the data field. (#125549, @liggitt) [SIG API Machinery and Apps] [sig/api-machinery,sig/apps]
• Kube-apiserver: the --encryption-provider-config file is now loaded with strict deserialization, which fails if the config file contains duplicate or unknown fields. This protects against accidentally running with config files that are malformed, mis-indented, or have typos in field names, and getting unexpected behavior. When --encryption-provider-config-automatic-reload is used, new encryption config files that contain typos after the kube-apiserver is running are treated as invalid and the last valid config is used. (#124912, @enj) [SIG API Machinery and Auth] [sig/api-machinery,sig/auth]
• Kube-controller-manager: the horizontal-pod-autoscaler-upscale-delay and horizontal-pod-autoscaler-downscale-delay flags have been removed (deprecated and non-functional since v1.12). (#124948, @SataQiu) [SIG API Machinery, Apps and Autoscaling] [sig/api-machinery,sig/autoscaling,sig/apps]
• Made kube-proxy Windows service control manager integration (--windows-service) configurable in v1alpha1 component configuration via windowsRunAsService field. (#126072, @aroradaman) [SIG Network and Scalability] [sig/network,sig/scalability]
• PersistentVolumeLastPhaseTransitionTime feature is stable and enabled by default. (#124969, @RomanBednar) [SIG API Machinery, Apps, Storage and Testing] [sig/storage,sig/api-machinery,sig/apps,sig/testing]
• Promoted LocalStorageCapacityIsolation to beta; the behaviour is enabled by default. Within the kubelet, storage capacity isolation is active if the feature gate is enabled and the specific Pod is using a user namespace. (#126014, @PannagaRao) [SIG Apps, Autoscaling, Node, Storage and Testing] [sig/storage,sig/node,sig/autoscaling,sig/apps,sig/testing]
• Promoted StatefulSetStartOrdinal to stable. This means --feature-gates=StatefulSetStartOrdinal=true are not needed on kube-apiserver and kube-controller-manager binaries and they'll be removed soon following policy at https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecation. (#125374, @pwschuurman) [SIG API Machinery, Apps and Testing] [sig/api-machinery,sig/apps,sig/testing]
• Promoted feature-gate VolumeAttributesClass to beta (disabled by default). Users need to enable the feature gate and the storage.k8s.io/v1beta1 API group to use this feature. Promoted the VolumeAttributesClass API to beta. (#126145, @carlory) [SIG API Machinery, Apps, CLI, Etcd, Storage and Testing] [sig/storage,sig/api-machinery,sig/apps,sig/cli,sig/testing,sig/etcd]
• Removed deprecated command flags --volume-host-cidr-denylist and --volume-host-allow-local-loopback from kube-controller-manager. (#124017, @carlory) [SIG API Machinery, Apps, Cloud Provider and Storage] [sig/storage,sig/api-machinery,sig/apps,sig/cloud-provider]
• Removed feature gate CustomResourceValidationExpressions. (#126136, @cici37) [SIG API Machinery, Cloud Provider and Testing] [sig/api-machinery,sig/testing,sig/cloud-provider]
• Reverted a change where ConsistentListFromCache was moved to beta and enabled by default. (#126139, @enj) [sig/api-machinery]
• Revised the Pod API with Alpha support for volumes derived from OCI artifacts. This feature is behind the ImageVolume feature gate. (#125660, @saschagrunert) [SIG API Machinery, Apps and Node] [sig/node,sig/api-machinery,sig/apps]
• Supported fine-grained supplemental groups policy (KEP-3619), which enabled fine-grained control for supplementary groups in the first container processes. This allows you to choose whether to include groups defined in the container image (/etc/groups) for the container's primary UID or not. (#117842, @everpeace) [SIG API Machinery, Apps and Node] [sig/node,sig/api-machinery,sig/apps]
• The (alpha) nftables mode of kube-proxy now requires version 1.0.1 or later of the nft command-line, and kernel 5.13 or later. (For testing/development purposes, you can use older kernels, as far back as 5.4, if you set the nftables.skipKernelVersionCheck option in the kube-proxy config, but this is not recommended in production since it may cause problems with other nftables users on the system.) (#124152, @danwinship) [SIG Network] [sig/network]
• To enhance usability and developer experience, CRD validation rules now support direct use of (CEL) reserved keywords as field names in object validation expressions for existing expressions in storage, will fully support runtime in next release for compatibility concern. (#126188, @cici37) [SIG API Machinery and Testing] [sig/api-machinery,sig/testing]
• Updated the feature MultiCIDRServiceAllocator to beta (disabled by default). Users need to enable the feature gate and the networking v1beta1 group to be able to use this new feature, that allows to dynamically reconfigure Service CIDR ranges. (#125021, @aojea) [SIG API Machinery, Apps, CLI, Etcd, Instrumentation, Network and Testing] [sig/network,sig/api-machinery,sig/apps,sig/cli,sig/instrumentation,sig/testing,sig/etcd]
• Use omitempty for optional Job Pod Failure Policy fields. (#126046, @mimowo) [sig/apps]
• User can choose a different static policy option SpreadPhysicalCPUsPreferredOption to spread cpus across physical cpus for some specific applications (#123733, @Jeffwan) [SIG Node] [sig/node]
• When the featuregate AnonymousAuthConfigurableEndpoints is enabled users can update the AuthenticationConfig file with endpoints for with anonymous requests are alllowed. (#124917, @vinayakankugoyal) [SIG API Machinery, Auth, Cloud Provider, Node and Testing] [sig/node,sig/api-machinery,sig/auth,sig/testing,sig/cloud-provider]

Feature

• 'kubeadm: enhanced the "patches" functionality to be able to patch coredns deployment. The new patch target is called "corednsdeployment" (e.g. patch file "corednsdeployment+json.json"). This makes it possible to apply custom patches to coredns deployment during "init" and "upgrade".' (#124820, @SataQiu) [SIG Cluster Lifecycle] [sig/cluster-lifecycle]
• 'kubeadm: marked the flag "--experimental-output'' as deprecated (it will be removed in a future release) and added a new flag ''--output" that serves the same purpose. Affected commands are - "kubeadm config images list", "kubeadm token list", "kubeadm upgrade plan", "kubeadm certs check-expiration".' (#124393, @carlory) [SIG Cluster Lifecycle] [sig/cluster-lifecycle]
• ACTION REQUIRED for custom scheduler plugin developers: EventsToRegister in the EnqueueExtensions interface gets ctx in the parameters and error in the return values. Please change your plugins' implementation accordingly. (#126113, @googs1025) [SIG Node, Scheduling, Storage and Testing] [sig/scheduling,sig/storage,sig/node,sig/testing]
• Add --for=create option to kubectl wait (#125868, @soltysh) [SIG CLI and Testing] [sig/cli,sig/testing]
• Add a TopologyManager policy option: max-allowable-numa-nodes to configures maxAllowableNUMANodes for kubelet. (#124148, @cyclinder) [SIG Node and Testing] [sig/node,sig/testing]
• Added Custom resource field selectors in beta and enabled them by default. Check out kubernetes/enhancements#4358 for more details. (#124681, @jpbetz) [SIG API Machinery, Auth and Testing] [sig/api-machinery,sig/auth,sig/testing]
• Added Extra.DisableAvailableConditionController for Generic Control Plane setup. (#125650, @mjudeikis) [SIG API Machinery] [sig/api-machinery]
• Added OCI VolumeSource Container Runtime Interface API fields and types. (#125659, @saschagrunert) [SIG Node] [sig/node]
• Added --keep-* flags to kubectl debug, which enables to control the removal of probes, labels, annotations and initContainers from copy pod. (#123149, @mochizuki875) [SIG CLI and Testing] [sig/cli,sig/testing]
• Added cri-client staging repository. (#123797, @saschagrunert) [SIG API Machinery, Node, Release and Testing] [sig/node,sig/api-machinery,sig/testing,sig/release]
• Added storage_class and volume_attributes_class labels to pv_collector_bound_pvc_count and pv_collector_unbound_pvc_count metrics. (#126166, @AndrewSirenko) [SIG Apps, Instrumentation, Storage and Testing] [sig/storage,sig/apps,sig/instrumentation,sig/testing]
• Added a feature to report an event about a Pod if kubelet observes a failed attach operation, even if the kubelet is running with --enable-controller-attach-detach=false. (#124884, @carlory) [sig/storage]
• Added a warning log, an event for cgroup v1 usage and a metric for cgroup version. (#125328, @harche) [sig/node]
• Added apiserver.latency.k8s.io/apf-queue-wait annotation to the audit log to record the time spent waiting in APF queue. (#123919, @hakuna-matatah) [sig/api-machinery]
• Added check for etcd version to warn about deprecated etcd versions if ConsistentListFromCache is enabled. (#124612, @ah8ad3) [SIG API Machinery] [sig/api-machinery]
• Added completion for kubectl set image. (#124592, @ah8ad3) [SIG CLI] [sig/cli]
• Added field management support to the fake client-go typed client. Use fake.NewClientset() instead of fake.NewSimpleClientset() to create a clientset with managed field support. (#125560, @jpbetz) [SIG API Machinery, Auth, Instrumentation and Testing] [sig/api-machinery,sig/auth,sig/instrumentation,sig/testing]
• Added flag to kubectl logs called --all-pods to get all pods from a object that uses a pod selector. (#124732, @cmwylie19) [SIG CLI and Testing] [sig/cli,sig/testing]
• Added namespace autocompletion for kubectl config set-context command. (#124994, @TessaIO) [SIG CLI] [sig/cli]
• Added ports autocompletion for kubectl port-foward command. (#124683, @TessaIO) [SIG CLI] [sig/cli]
• Added support for CEL(Common Expression Language) expressions and additionalProperties to be used under nested quantifiers in CRD schemas. (#124381, @alexzielenski) [SIG API Machinery] [sig/api-machinery]
• Added support for building Windows kube-proxy container image. A container image for kube-proxy on Windows can now be built with the command make release-images KUBE_BUILD_WINDOWS=y. The Windows kube-proxy image can be used with Windows Host Process Containers. (#109939, @claudiubelu) [SIG Windows] [sig/windows]
• Added support for kube-proxy iptables mode to track packets that were wrongfully marked invalid by conntrack and subsequently dropped by introducing kubeproxy_iptables_ct_state_invalid_dropped_packets_total metric. (#122812, @aroradaman) [SIG Instrumentation, Network and Testing] [sig/network,sig/instrumentation,sig/testing]
• Added the WatchList method to the rest client in client-go. When used, it establishes a stream to obtain a consistent snapshot of data from the server. This method is meant to be used by the generated client. (#122657, @p0lyn0mial) [SIG API Machinery] [sig/api-machinery]
• Added the ability to the kubelet server to dynamically load certificate files. (#124574, @zhangweikop) [SIG Auth and Node] [sig/node,sig/auth]
• Allowed creating ServiceAccount tokens bound to Node objects. This allows users to bind a service account token's validity to a named Node object, similar to Pod bound tokens. Use with kubectl create token <serviceaccount-name> --bound-object-kind=Node --bound-object-node=<node-name>. (#125238, @munnerz) [SIG Auth and CLI] [sig/auth,sig/cli]
• Built Kubernetes with Go 1.22.3. (#124828, @cpanato) [SIG Release and Testing] [sig/testing,sig/release]
• Built Kubernetes with Go 1.22.4. (#125363, @cpanato) [SIG Architecture, Cloud Provider, Release, Storage and Testing] [sig/storage,sig/testing,sig/release,sig/architecture,sig/cloud-provider]
• Promoted CRI communication of the cgroup driver mechanism to beta. The KubeletCgroupDriverFromCRI feature gate is now in beta and enabled by default. This allows the kubelet to query the container runtime using CRI to determine the mechanism for cgroup management. If the container runtime doesn't support this, the kubelet falls back to using the configuration file (you can also use the deprecated --cgroup-driver command line argument). (#125828, @haircommander) [SIG Node] [sig/node]
• CEL: added name formats library. (#123572, @alexzielenski) [SIG API Machinery] [sig/api-machinery]
• Changed Linux swap handling to restrict access to swap for containers in high priority Pods. New Pods that have a node- or cluster-critical priority are prohibited from accessing swap on Linux, even if your cluster and node configuration could otherwise allow this. (#125277, @iholder101) [SIG Node and Testing] [sig/node,sig/testing]
• Client-go/reflector: warns when the bookmark event for initial events hasn't been received (#124614, @p0lyn0mial) [SIG API Machinery] [sig/api-machinery]
• Continued streaming kubelet logs when the CRI server of the runtime was unavailable. (#124025, @saschagrunert) [SIG Node] [sig/node]
• Delay setting terminal Job conditions until all pods are terminal.

Additionally, the FailureTarget condition is also added to the Job object in the first Job status update as soon as the failure conditions are met (backoffLimit is exceeded, maxFailedIndexes, or activeDeadlineSeconds is exceeded).

Similarly, the SuccessCriteriaMet condition is added in the first update as soon as the expected number of pod completions is reached.

Also, introduce the following validation rules for Job status when JobManagedBy is enabled:

1. the count of ready pods is less or equal than active
2. when transitioning to terminal phase for Job, the number of terminating pods is 0
3. terminal Job conditions (Failed and Complete) should be preceded by adding the corresponding interim conditions: FailureTarget and SuccessCriteriaMet (#125510, @mimowo) [SIG Apps and Testing] [sig/apps,sig/testing]

• Dependencies: started using registry.k8s.io/pause:3.10. (#125112, @neolit123) [SIG CLI, Cloud Provider, Cluster Lifecycle, Node, Release, Testing and Windows] [sig/node,sig/cluster-lifecycle,sig/windows,sig/cli,sig/testing,sig/release,sig/cloud-provider]
• Enabled feature gates for PortForward (kubectl port-forward) over WebSockets by default (beta).
• Server-side feature gate: PortForwardWebsocket
• Client-side (kubectl) feature gate: PORT_FORWARD_WEBSOCKETS environment variable
• To turn off PortForward over WebSockets for kubectl, the environment variable feature gate must be explicitly set - PORT_FORWARD_WEBSOCKETS=false (#125528, @seans3) [SIG API Machinery and CLI] [sig/api-machinery,sig/cli]
• Enforced kubelet to request serving certificates only once it has at least one IP address in the .status.addresses of its associated Node object. This avoids requesting DNS-only serving certificates before externally set addresses are in place. Until 1.33, the previous behavior can be opted back into by setting the deprecated AllowDNSOnlyNodeCSR feature gate to true in the kubelet. (#125813, @aojea) [SIG Auth, Cloud Provider and Node] [sig/node,sig/auth,sig/cloud-provider]
• Fixed a missing behavior where Windows nodes did not implement memory-pressure eviction. (#122922, @marosset) [SIG Node, Testing and Windows] [sig/node,sig/windows,sig/testing]
• Graduated Kubernetes' support for AppArmor to GA. You now cannot disable the AppArmor feature gate. (#125257, @vinayakankugoyal) [SIG Apps, Node and Testing] [sig/node,sig/apps,sig/testing]
• Graduated support for Container Device Interface (CDI) device IDs to general availability. The DevicePluginCDIDevices feature gate is now enabled unconditionally. (#123315, @bart0sh) [SIG Node] [sig/node]
• Graduated the WatchList feature gate to beta for kube-apiserver and enabled WatchListClient for kube-controller-manager (KCM). (#125591, @p0lyn0mial) [SIG API Machinery and Testing] [sig/api-machinery,sig/testing]
• If the feature-gate VolumeAttributesClass is enabled, when finding a suitable persistent volume for a claim, the kube-controller-manager will be aware of the volumeAttributesClassName field of PVC and PV objects. The volumeAttributesClassName field is a reference to a VolumeAttributesClass object, which contains a set of key-value pairs that present mutable attributes of the volume. It's forbidden to change the volumeAttributesClassName field of a PVC object until the PVC is bound to a PV object. During the binding process, if a PVC has a volumeAttributesClassName field set, the controller will only consider volumes that have the same volumeAttributesClassName as the PVC. If the volumeAttributesClassName field is not set or set to an empty string, only volumes with empty volumeAttributesClassName will be considered. (#121902, @carlory) [SIG Apps, Scheduling, Storage and Testing] [sig/scheduling,sig/storage,sig/apps,sig/testing]
• Implemented event_handling_duration_seconds metric, which is the time the scheduler takes to handle each kind of events. (#125929, @sanposhiho) [sig/scheduling]
• Implemented queueing_hint_execution_duration_seconds metric, which is the time the QueueingHint function takes. (#126227, @sanposhiho) [sig/scheduling]
• Implemented new cluster events UpdatePodScaleDown and UpdatePodLabel for scheduler plugins. (#122628, @sanposhiho) [sig/scheduling]
• Improved memory usage of kube-apiserver by dropping.metadata.managedFields field that self-requested informers of kube-apiserver didn't need. (#124667, @linxiulei) [SIG API Machinery] [sig/api-machinery]
• In the client-side apply on create, defining the null value as "delete the key associated with this value". (#125646, @HirazawaUi) [SIG API Machinery, CLI and Testing] [sig/api-machinery,sig/cli,sig/testing]
• Introduces new functionality to the client-go's List method, allowing users to enable API streaming. To activate this feature, users can set the client-go.WatchListClient feature gate.

It is important to note that the server must support streaming for this feature to function properly. If streaming is not supported by the server, client-go will revert to using the normal LIST method to obtain data. (#124509, @p0lyn0mial) [SIG API Machinery, Auth, Instrumentation and Testing] [sig/api-machinery,sig/auth,sig/instrumentation,sig/testing]

• Introduces new functionality to the dynamic client's List method, allowing users to enable API streaming. To activate this feature, users can set the client-go.WatchListClient feature gate.

It is important to note that the server must support streaming for this feature to function properly. If streaming is not supported by the server, the client will revert to using the normal LIST method to obtain data. (#125305, @p0lyn0mial) [SIG API Machinery and Testing] [sig/api-machinery,sig/testing]

• KEP-3857: promoted RecursiveReadOnlyMounts feature to beta. (#125475, @AkihiroSuda) [SIG Node] [sig/node]
• Kube-apiserver: Added support to disable http/2 serving with a --disable-http2-serving flag. (#122176, @slashpai) [SIG API Machinery] [sig/api-machinery]
• Kube-apiserver: when the Alpha UserNamespacesPodSecurityStandards feature gate is enabled, Pod Security Admission enforcement of the baseline policy now allows procMount: Unmasked for user namespace pods that set hostUsers: false. (#126163, @haircommander) [sig/auth]
• Kube-proxy's nftables mode (--proxy-mode=nftables) is now beta and available by default. (#124383, @danwinship) [SIG Cloud Provider and Network] [sig/network,sig/cloud-provider]
• Kube-scheduler implemented scheduling hints for the CSILimit plugin. The scheduling hints allow the scheduler to retry scheduling a Pod that was previously rejected by the CSILimit plugin if a deleted pod has a PersistantVolumeClaim (PVC) from the same driver. (#121508, @utam0k) [SIG Scheduling and Storage] [sig/scheduling,sig/storage]
• Kube-scheduler implemented scheduling hints for the VolumeRestriction plugin. Scheduling hints allow the scheduler to retry scheduling Pods that were previously rejected by the VolumeRestriction plugin if the Pod is deleted and the deleted Pod conflicts with the existing volumes of the current Pod. (#125279, @HirazawaUi) [SIG Scheduling and Storage] [sig/scheduling,sig/storage]
• Kube-scheduler implements scheduling hints for the VolumeRestriction plugin. Scheduling hints allow the scheduler to retry scheduling Pods that were previously rejected by the VolumeRestriction plugin if a new pvc added, and the pvc belongs to pod. (#125280, @HirazawaUi) [SIG Scheduling and Storage] [sig/scheduling,sig/storage]
• Kube-scheduler implements scheduling hints for the VolumeZone plugin. The scheduling hints allow the scheduler to only retry scheduling a Pod that was previously rejected by the VolemeZone plugin if addition/update of node, addition/update of PV, addition/update of PVC, or addition of SC matches pod's topology settings. (#124996, @Gekko0114) [SIG Scheduling and Storage] [sig/scheduling,sig/storage]
• Kube-scheduler implements scheduling hints for the VolumeZone plugin. The scheduling hints allow the scheduler to only retry scheduling a Pod that was previously rejected by the VolemeZone plugin if addition/update of node, addition/update of PV, addition/update of PVC, or addition of SC matches pod's topology settings. (#125000, @Gekko0114) [SIG Scheduling and Storage] [sig/scheduling,sig/storage]
• Kube-scheduler implements scheduling hints for the VolumeZone plugin. The scheduling hints allow the scheduler to only retry scheduling a Pod that was previously rejected by the VolemeZone plugin if addition/update of node, addition/update of PV, addition/update of PVC, or addition of SC matches pod's topology settings. (#125001, @Gekko0114) [SIG Scheduling and Storage] [sig/scheduling,sig/storage]
• Kubeadm: Ensured that during "upgrade" , if the "etcd.yaml" static pod did not need upgrade, still consider rotating the etcd certificates and restarting the etcd static pod if the "kube-apiserver.yaml" manifest was to be upgraded and if certificate renewal was not disabled. (#124688, @neolit123) [sig/cluster-lifecycle]
• Kubeadm: Switched kubeadm to start using the CRI client library instead of shelling out of the crictl binary for actions against a CRI endpoint. The kubeadm deb/rpm packages will continue to install the cri-tools package for one more release, but in you must adapt your scripts to install crictl manually from https://github.com/kubernetes-sigs/cri-tools/releases or a different location.

The kubeadm package will stop depending on the cri-tools package in Kubernetes 1.32, which means that installing kubeadm will no longer automatically ensure installation of crictl. (#124685, @saschagrunert) [sig/cluster-lifecycle]

• Kubeadm: Switched to using the new etcd endpoints introduced in 3.5.11 - /livez (for liveness probe) and /readyz (for readyness and startup probe). With this change it is no longer possible to deploy a custom etcd version older than 3.5.11 with kubeadm 1.31. If so, please upgrade etcd to a supported version. (#124465, @neolit123) [sig/cluster-lifecycle]
• Kubeadm: Used output/v1alpha3 to print structural output for the commands "kubeadm config images list" and "kubeadm token list". (#124464, @carlory) [sig/cluster-lifecycle]
• Kubeadm: added the ControlPlaneKubeletLocalMode feature gate. It can be used to tell kubeadm to use the local kube-apiserver endpoint for the kubelet when creating a cluster with "kubeadm init" or when joining control plane nodes with "kubeadm join". The "kubeadm join" workflow now includes two new experimental phases called "control-plane-join-etcd" and "kubelet-wait-bootstrap" which will be used when the feature gate is enabled. This phases will be marked as non-experimental when ControlPlaneKubeletLocalMode becomes GA. During "kubeadm upgrade" commands, if the feature gate is enabled, modify the "/etc/kubernetes/kubelet.conf " to use the local kube-apiserver endpoint. This upgrade mechanism will be removed once the feature gate goes GA and is hardcoded to true. (#125582, @chrischdi) [sig/cluster-lifecycle]
• Kubeadm: enabled the v1beta4 API. For a complete changelog since v1beta3 please see https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta4/.

The API does include a few breaking changes:

• The "extraArgs" component construct is now a list of "name"/"value" pairs instead of a string/string map. This has been done to support duplicate args where needed.
• The "JoinConfiguration.discovery.timeout" field has been replaced by "JoinConfiguration.timeouts.discovery".
• The "ClusterConfiguration.timeoutForControlPlane" field has been replaced by "{Init|Join}Configuration.timeouts.controlPlaneComponentHealthCheck". Please use the command "kubeadm config migrate" to migrate your existing v1beta3 configuration to v1beta4.

v1beta3 is now marked as deprecated but will continue to be supported until version 1.34 or later. The storage configuration in the kube-system/kubeadm-config ConfigMap is now a v1beta4 ClusterConfiguration. (#125029, @neolit123) [sig/cluster-lifecycle]

• Kubelet would not restart the container when fields other than image in the Pod spec change. pod spec change. (#124220, @HirazawaUi) [sig/node]
• Kubelet/stats: set INFO log level for stats not found in cadvisor memory cache error. (#125656, @gyuho) [sig/node]
• Kubelet: warn instead of error for the unsupported options on Windows "CgroupsPerQOS" and "EnforceNodeAllocatable". (#123137, @neolit123) [SIG Node and Windows] [sig/node,sig/windows]
• Kubemark: added two flags, --kube-api-qps which indicates the maximum QPS to the apiserver, and --kube-api-burst which indicates maximum burst for throttle to the apiserver. (#124147, @devincd) [sig/scalability]
• Kubernetes is now built with go 1.22.5. (#125894, @cpanato) [SIG Release and Testing] [sig/testing,sig/release]
• LogarithmicScaleDown is now GA. (#125459, @MinpengJin) [SIG Apps and Scheduling] [sig/scheduling,sig/apps]
• Moved ConsistentListFromCache feature flag to beta and enabled it by default. (#123513, @serathius) [SIG API Machinery and Testing] [sig/api-machinery,sig/testing]
• Promote HonorPVReclaimPolicy to beta and enable the feature-gate by default (#124842, @carlory) [SIG Apps, Storage and Testing] [sig/storage,sig/apps,sig/testing]
• Promoted generateName retries to beta, and made the NameGenerationRetries feature gate enabled by default. You can read https://kep.k8s.io/4420 for more details. (#124673, @jpbetz) [sig/api-machinery]
• Promoted the ProcMountType feature gate to beta. (#125259, @sohankunkerkar) [sig/node]
• Promoted the feature gate KubeProxyDrainingTerminatingNodes to stable (#125082, @alexanderConstantinescu) [sig/network]
• Promoted the metrics for both ValidatingAdmissionPolicy (VAP) and CustomResourceDefinition (CRD) validation rules to beta. (#126237, @cici37) [SIG API Machinery and Instrumentation] [sig/api-machinery,sig/instrumentation]
• Scheduler changes its logic of calculating evaluatedNodes from "contains the number of nodes that filtered out by PreFilterResult and Filter plugins" to "the number of nodes filtered out by Filter plugins only". (#124735, @AxeZhan) [sig/scheduling]
• Services implemented a field selector for the ClusterIP and Type fields. The Kubelet uses this field selector to avoid monitoring Headless Services, which helps reduce memory consumption. (#123905, @aojea) [SIG Apps, Node and Testing] [sig/node,sig/apps,sig/testing]
• Starting in 1.31, container_engine_t was added to the list of allowed SELinux types in the baseline Pod Security Standard. (#126165, @haircommander) [sig/auth]
• The PodDisruptionBudget spec.unhealthyPodEvictionPolicy field has graduated to GA. This field may be set to AlwaysAllow to always allow unhealthy pods covered by the PodDisruptionBudget to be evicted. (#123428, @atiratree) [SIG Apps, Auth, Node and Testing] [sig/node,sig/auth,sig/apps,sig/testing]
• The Service trafficDistribution field has graduated to beta and is now available for configuration by default, without the need to enable any feature flag. Services that do not have the field configured will continue to operate with their existing behavior. Refer to the documentation https://kubernetes.io/docs/concepts/services-networking/service/#traffic-distribution for more details. (#125838, @gauravkghildiyal) [SIG Network and Testing] [sig/network,sig/testing]
• The KubeletSeparateDiskGC feature gate is now beta. This split image filesystem feature enables kubelet to perform garbage collection of images (read-only layers) and/or containers (writeable layers) deployed on separate filesystems. gate is now beta. (#126205, @kwilczynski) [sig/node]
• The feature-gate CSIMigrationPortworx was promoted to beta in Kubernetes 1.25, but turned off by default. In 1.31, it was turned on by default. Before upgrading to 1.31, please make sure that the corresponding portworx csi driver is installed if you are using Portworx. (#125016, @carlory) [SIG Storage] [sig/storage]
• The iptables mode of kube-proxy now tracks accepted packets that are destined for node-ports on localhost by introducing kubeproxy_iptables_localhost_nodeports_accepted_packets_total metric. This will help users to identify if they rely on iptables.localhostNodePorts feature and ulitmately help them to migrate from iptables to nftables. (#125015, @aroradaman) [SIG Instrumentation, Network and Testing] [sig/network,sig/instrumentation,sig/testing]
• The kube-proxy command line flag --proxy-port-range, which was previously deprecated and non-functional, has now been removed. (#126293, @aroradaman) [SIG Network] [sig/network]
• The kube-scheduler added scheduling hints for the InterPodAffinity plugin. These hints allow the scheduler to retry scheduling a Pod that was previously rejected by the InterPodAffinity plugin if there are changes (create, delete, or update) to a related Pod or a node that matches the pod affinity criteria. (#122471, @nayihz) [SIG Scheduling and Testing] [sig/scheduling,sig/testing]
• The kube-scheduler added support for scheduling hints for the CSIStorageCapacity resource within the VolumeBinding plugin. The scheduling hints allowed the scheduler to retry scheduling a Pod that was previously rejected by the VolumeBinding plugin only if a new resource referenced by the plugin was created or an existing resource referenced by the plugin was updated. (#124961, @bells17) [SIG Scheduling and Storage] [sig/scheduling,sig/storage]
• The kube-scheduler added support for scheduling hints for the PersistentVolumeClaim resource within the VolumeBinding plugin. The scheduling hints allowed the scheduler to retry scheduling a Pod that was previously rejected by the VolumeBinding plugin only if a new resource referenced by the plugin was created or an existing resource referenced by the plugin was updated. (#124959, @bells17) [SIG Scheduling and Storage] [sig/scheduling,sig/storage]
• The kube-scheduler added support for scheduling hints for the StorageClass resource within the VolumeBinding plugin. The scheduling hints allowed the scheduler to retry scheduling a Pod that was previously rejected by the VolumeBinding plugin only if a new resource referenced by the plugin was created or an existing resource referenced by the plugin was updated. (#124958, @bells17) [SIG Scheduling and Storage] [sig/scheduling,sig/storage]
• The name of CEL(Common Expression Language) optional type has been changed from optional to optional_type. (#124328, @jiahuif) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Network and Node] [sig/network,sig/node,sig/api-machinery,sig/auth,sig/cli,sig/architecture,sig/cloud-provider]
• The scheduler implemented QueueingHint in the TaintToleration plugin, enhancing the throughput of scheduling. (#124287, @sanposhiho) [SIG Scheduling and Testing] [sig/scheduling,sig/testing]
• The scheduler implements QueueingHint in VolumeBinding plugin's CSINode event, which enhances the throughput of scheduling. (#125097, @YamasouA) [SIG Scheduling and Storage] [sig/scheduling,sig/storage]
• The sidecars' finish time will now be accounted for when calculating the job's finish time. (#124942, @AxeZhan) [SIG Apps] [sig/apps]
• This PR added tracing support to the kubelet's read-only endpoint, which currently does not have tracing. It makes use of the WithPublicEndpoint option to prevent callers from influencing sampling decisions. (#121770, @frzifus) [SIG Node] [sig/node]
• Updated kubernetes to build with Go 1.23rc2. (#126047, @cpanato) [SIG Release and Testing] [sig/testing,sig/release]
• Updated the CEL default compatibility environment version to 1.30, ensuring that extended libraries added before version 1.30 are available for use. (#124779, @cici37) [sig/api-machinery]
• Users can traverse all the pods that are in the scheduler and waiting in the permit stage through method IterateOverWaitingPods. In other words, all waitingPods in scheduler can be obtained from any profiles. Before this commit, each profile could only obtain waitingPods within that profile. (#124926, @kerthcet) [SIG Scheduling] [sig/scheduling]
• Windows Kubeproxy will use the update load balancer API for load balancer updates, instead of the previous delete and create APIs.
• Deletion of remote endpoints will be triggered only for terminated endpoints (those present in the old endpoints map but not in the new endpoints map), whereas previously it was also done for terminating endpoints. (#124092, @princepereira) [SIG Network and Windows] [sig/network,sig/windows]
• --custom flag in kubectl debug will be enabled by default and yaml support is added. (#125333, @ardaguclu) [SIG CLI and Testing] [sig/cli,sig/testing]
• ElasticIndexedJob is graduated to GA. (#125751, @ahg-g) [SIG Apps and Testing] [sig/apps,sig/testing]
• pause: Added a -v flag to the Windows variant of the pause binary, which prints the version of pause and exits. The Linux pause binary already has this flag. (#125067, @neolit123) [sig/windows]

Failing Test

• Fixed bug in kubelet if the SplitImageFilesystem feature gate is turned on but the container runtime is not configured. (#126335, @kannon92) [sig/node]
• Fixed issue where following Windows container logs would prevent container log rotation. (#124444, @claudiubelu) [SIG Node, Testing and Windows] [sig/node,sig/windows,sig/testing]
• Introduced Wait(context.Context) error method in pkg k8s.io/apiserver/pkg/storage/cacher to improve watch cache initialization resilience. (#125450, @mauri870) [sig/api-machinery]
• Reverted remove legacycloudproviders from staging. (#124864, @carlory) [sig/release]

Bug or Regression

• "Fixed the ResourceClaim controller forgetting to wait for podSchedulingSynced and templatesSynced." (#124589, @carlory) [SIG Apps and Node] [sig/node,sig/apps]
• 'kubeadm: Stopped storing the ResolverConfig in the global KubeletConfiguration and sets it dynamically for each node instead.' (#124038, @SataQiu) [sig/cluster-lifecycle]
• 'kubeadm: fixed a regression where the KubeletConfiguration is not properly downloaded during "kubeadm upgrade" command from the kube-system/kubelet-config ConfigMap, resulting in the local ''/var/lib/kubelet/config.yaml'' file being written as a defaulted config.' (#124480, @neolit123) [sig/cluster-lifecycle]
• .status.terminating field now gets tracked faster when active Pods are deleted, specifically when Job is failed, gets suspended or has too many active pods. (#125175, @dejanzele) [SIG Apps and Testing] [sig/apps,sig/testing]
• Added /sys/devices/virtual/powercap to default masked paths. It avoids the potential security risk that the ability to read these files may offer a power-based sidechannel attack against any workloads running on the same kernel. (#125970, @carlory) [sig/node]
• Added an extra line between two different key value pairs under data when running kubectl describe configmap. (#123597, @siddhantvirus) [sig/cli]
• Added kubectl support for:
• kubectl create secret docker-registry --from-file=<path/to/.docker/config.json>
• kubectl create secret docker-registry --from-file=.dockerconfigjson=<path/to/.docker/config.json> (#119589, @carlory) [sig/cli]
• Added metrics for the nftables kube-proxy mode rather than it reporting metrics with "iptables" in their names. (#124557, @danwinship) [SIG Network and Windows] [sig/network,sig/windows]
• Allowed calling Stop multiple times on RetryWatcher without panicking. (#126125, @mprahl) [sig/api-machinery]
• Allowed parameter to be set along with proto file path. (#124281, @fulviodenza) [sig/api-machinery]
• Cel: converting a quantity value into a quantity value failed. (#123669, @pohly) [sig/api-machinery]
• Client-go/tools/record.Broadcaster: Fixed automatic shutdown on WithContext cancellation. (#124635, @pohly) [sig/api-machinery]
• Do not remove the "batch.kubernetes.io/job-tracking" finalizer from a Pod, in a corner case scenario, when the Pod is controlled by an API object which is not a batch Job (e.g. when the Pod is controlled by a custom CRD). (#124798, @mimowo) [SIG Apps and Testing] [sig/apps,sig/testing]
• Dropped the additional rule requirement (cronjobs/finalizers) for roles using kubectl create cronjobs to ensure backward compatibility. (#124883, @ardaguclu) [sig/cli]
• Dynamic Resource Allocation (DRA): using structured parameters with a claim that gets reused between pods may have led to a claim with an invalid state (allocated without a finalizer) which then caused scheduling of pods using the claim to stop. (#124931, @pohly) [SIG Node and Scheduling] [sig/scheduling,sig/node]
• Dynamic Resource Allocator (DRA): Enhanced validation for the ResourceClaimParametersReference and ResourceClassParametersReference with the following rules:

1. apiGroup: If set, it must be a valid DNS subdomain (e.g. 'example.com').
2. kind and name: It must be valid path segment name. It may not be '.' or '..' and it may not contain '/' and '%' characters. (#125218, @carlory) [sig/node]

• Enabled kubectl to find kubectl-create-subcommand plugins when positional arguments exists, e.g. kubectl create subcommand arg. (#124123, @sttts) [sig/cli]
• Ensured daemonset controller counts old unhealthy pods towards max unavailable budget. (#123233, @marshallbrekka) [sig/apps]
• Fix a bug that when PodTopologySpread rejects Pods, they may be stuck in Pending state for 5 min in a worst case scenario. The same problem could happen with custom plugins which have Pod/Add or Pod/Update in EventsToRegister, which is also solved with this PR, but only when the feature flag SchedulerQueueingHints is enabled. (#122627, @sanposhiho) [SIG Scheduling and Testing] [sig/scheduling,sig/testing]
• Fix bug where Server Side Apply causing spurious resourceVersion bumps on no-op patches containing empty maps. (#125317, @jpbetz) [SIG API Machinery and Testing] [sig/api-machinery,sig/testing]
• Fix endpoints status out-of-sync when the pod state changes rapidly (#125675, @tnqn) [SIG Apps, Network and Testing] [sig/network,sig/apps,sig/testing]
• Fix the bug where PodIP field is temporarily removed for a terminal pod (#125404, @mimowo) [SIG Node and Testing] [sig/node,sig/testing]
• Fixed "-kube-test-repo-list" e2e flag may not take effect. (#123587, @huww98) [SIG API Machinery, Apps, Autoscaling, CLI, Network, Node, Scheduling, Storage, Testing and Windows] [sig/network,sig/scheduling,sig/storage,sig/node,sig/api-machinery,sig/autoscaling,sig/apps,sig/windows,sig/cli,sig/testing]
• Fixed EDITOR/KUBE_EDITOR with double-quoted paths with spaces when on Windows cmd.exe. (#112104, @oldium) [SIG CLI and Windows] [sig/windows,sig/cli]
• Fixed a bug in storage-version-migrator-controller that would cause migration attempts to fail if resources were deleted when the migration was in progress. (#126107, @enj) [SIG API Machinery, Apps, Auth and Testing] [sig/api-machinery,sig/auth,sig/apps,sig/testing]
• Fixed a bug in the JSON frame reader that could cause it to retain a reference to the underlying array of the byte slice passed to read. (#123620, @benluddy) [sig/api-machinery]
• Fixed a bug in the scheduler where it would crash when prefilter returns a non-existent node. (#124933, @AxeZhan) [SIG Scheduling and Testing] [sig/scheduling,sig/testing]
• Fixed a bug that Pods could stuck in the unschedulable pod pool if they're rejected by PreEnqueue plugins that could change its result by a change in resources apart from Pods.

DRA plugin is the only plugin that meets the criteria of the bug in in-tree, and hence if you have DynamicResourceAllocation feature flag enabled, your DRA Pods could be affected by this bug. (#125527, @sanposhiho) [SIG Scheduling and Testing] [sig/scheduling,sig/testing]

• Fixed a bug that init containers with Always restartPolicy may not terminate gracefully if the pod hasn't initialized yet. (#125935, @gjkim42) [SIG Node and Testing] [sig/node,sig/testing]
• Fixed a bug where kubectl describe incorrectly displayed NetworkPolicy port ranges (showing only the starting port). (#123316, @jcaamano) [sig/cli]
• Fixed a bug where hard evictions due to resource pressure allowed pods to use the full termination grace period instead of shutting down instantly. This bug also affected force deleted pods. Both cases now receive a termination grace period of 1 second. (#124063, @olyazavr) [sig/node]
• Fixed a bug where the Kubelet miscalculated the process usage of pods, causing pods to never get evicted for PID usage. (#124101, @haircommander) [SIG Node and Testing] [sig/node,sig/testing]
• Fixed a missing status prefix in custom resource validation error messages. (#123822, @JoelSpeed) [sig/api-machinery]
• Fixed a race condition in kube-controller-manager and the scheduler, caused by a bug in the transforming informer during the Resync operation, by making the transforming function idempotent. (#124352, @wojtek-t) [SIG API Machinery and Scheduling] [sig/scheduling,sig/api-machinery]
• Fixed a race condition in the transforming informer that occurred when objects were accessed during the Resync operation. (#124344, @wojtek-t) [sig/api-machinery]
• Fixed a regression where kubelet --hostname-override no longer worked correctly with an external cloud provider. (#124516, @danwinship) [sig/node]
• Fixed an issue that prevents the linking of trace spans for requests that are proxied through kube-aggregator. (#124189, @toddtreece) [sig/api-machinery]
• Fixed an issue where kubelet on Windows would fail if a pod had a SecurityContext with RunAsUser. (#125040, @carlory) [SIG Storage, Testing and Windows] [sig/storage,sig/windows,sig/testing]
• Fixed an issue where the Service LoadBalancer controller was not correctly considering the service.Status new IPMode field and excluding the Ports when checking if the status was changed, resulting in the changed field potentially not to update the service.Status correctly. (#125225, @aojea) [SIG Apps, Cloud Provider and Network] [sig/network,sig/apps,sig/cloud-provider]
• Fixed bug where Server Side Apply causes spurious resourceVersion bumps on no-op patches to custom resources. (#125263, @jpbetz) [SIG API Machinery and Testing] [sig/api-machinery,sig/testing]
• Fixed bug where kubectl get with --sort-by flag does not sort strings alphanumerically. (#124514, @brianpursley) [sig/cli]
• Fixed fake clientset ApplyScale subresource from status to scale. (#126073, @a7i) [sig/api-machinery]
• Fixed kubelet so it would no longer crash when a DRA(Dynamic Resource Allocation) driver returns a nil as part of the Node(Un)PrepareResources response instead of an empty struct (Did not affect drivers written in Go, first showed up with a driver written in Rust). returns a nil as part of the Node(Un)PrepareResources response instead of an empty struct (did not affect drivers written in Go, first showed up with a driver written in Rust). (#124091, @bitoku) [sig/node]
• Fixed node reporting "notReady" with the reason 'container runtime status check may not have completed yet' after kubelet restart. (#124430, @AllenXu93) [sig/node]
• Fixed null lastTransitionTime in Pod condition when setting the scheduling gate. (#122636, @lianghao208) [SIG Node and Scheduling] [sig/scheduling,sig/node]
• Fixed recursive LIST from watch cache returning object matching key. (#125584, @serathius) [SIG API Machinery and Testing] [sig/api-machinery,sig/testing]
• Fixed sample-cli-plugin help text to be consistent and always use kubectl ns. (#125641, @nirs) [sig/cli]
• Fixed the bug where if Endpointslices mirrored from Endpoints by the EndpointSliceMirroring controller they would not reconcile if modified. were not reconciled if modified (#124131, @zyjhtangtang) [SIG Apps and Network] [sig/network,sig/apps]
• Fixed the format of the error indicating that a user does not have permission on the object referenced by paramRef in ValidatingAdmissionPolicyBinding. (#124653, @m1kola) [sig/api-machinery]
• Fixed throughput when scheduling DaemonSet pods to reach 300 pods/s, if the configured QPS allows it. (#124714, @sanposhiho) [sig/scheduling]
• Fixed: during the kube-controller-manager restart, when the corresponding Endpoints resource was manually deleted and recreated, causing the endpointslice to fail to be created normally. (#125359, @yangjunmyfm192085) [SIG Apps and Network] [sig/network,sig/apps]
• For statically provisioned PVs, if its volume source is CSI type or it has migrated annotation, when it's deleted, the PersisentVolume controller won't changes its phase to the Failed state.

With this patch, the external provisioner can remove the finalizer in next reconcile loop. Unfortunately if the provious existing pv has the Failed state, this patch won't take effort. It requires users to remove finalizer. (#125767, @carlory) [SIG Apps and Storage] [sig/storage,sig/apps]

• Improved scheduling latency when there are many gated pods and events that trigger requeueing from the unschedulable pool. (#124618, @gabesaba) [SIG Scheduling and Testing] [sig/scheduling,sig/testing]
• Kube-apiserver: fixed a 1.27+ regression watching a single namespace via the deprecated /api/v1/watch/namespaces/$name endpoint where watch events were not delivered after the watch was established. (#125145, @xyz-li) [SIG API Machinery, Node and Testing] [sig/node,sig/api-machinery,sig/testing]
• Kube-apiserver: fixed a 1.28 regression printing pods with invalid initContainer status. (#124906, @liggitt) [sig/node]
• Kube-apiserver: fixed a potential crash serving CustomResourceDefinitions that combine an invalid schema and CEL validation rules. (#126167, @cici37) [SIG API Machinery and Testing] [sig/api-machinery,sig/testing]
• Kube-apiserver: timeouts configured for authorization webhooks in the --authorization-config file are now honored, and webhook timeouts are accurately reflected in webhook metrics with result=timeout (#125552, @liggitt) [SIG API Machinery, Auth and Testing] [sig/api-machinery,sig/auth,sig/testing]
• Kubeadm: Added --yes flag to the list of allowed flags so that it can be mixed with kubeadm upgrade apply --config. (#125566, @xmudrii) [sig/cluster-lifecycle]
• Kubeadm: Added support during the preflight check "CreateJob" of "kubeadm upgrade" to check if there are no nodes where a Pod can be scheduled. If there are none, show a warning and skip this preflight check. This can happen in single node clusters where the only node was drained. (#124503, @neolit123) [sig/cluster-lifecycle]
• Kubeadm: Fixed a bug where the PublicKeysECDSA feature gate was not respected when generating kubeconfig files. (#125388, @neolit123) [sig/cluster-lifecycle]
• Kubeadm: Fixed a regression where the JoinConfiguration.discovery.timeout was no longer respected and the value was always hardcoded to "5m" (5 minutes). (#125480, @neolit123) [sig/cluster-lifecycle]
• Kubeadm: Removed support for mounting /etc/pki as an additional Linux system CA location in kube-apisever and kube-controller-manager pods. Instead, it shifted to supporting the mounting of /etc/pki/ca-trust and /etc/pki/tls/certs. The locations /etc/ca-certificate, /usr/share/ca-certificates, /usr/local/share/ca-certificates, and /etc/ssl/certs continued to be supported. (#124361, @neolit123) [sig/cluster-lifecycle]
• Kubeadm: The healthz address:port configured in the KubeletConfiguration was used during kubelet health checks, instead of hardcoding localhost:10248. (#125265, @neolit123) [sig/cluster-lifecycle]
• Kubeadm: during the validation of existing kubeconfig files on disk, handle cases where the "ca.crt" is a bundle and has intermediate certificates. Find a common trust anchor between the "ca.crt" bundle and the CA in the existing kubeconfig on disk instead of treating "ca.crt" as a file containing a single CA. (#123102, @astundzia) [sig/cluster-lifecycle]
• Kubeadm: fixed a bug on 'kubeadm join' where using patches with a kubeletconfiguration target was not respected when performing the local kubelet healthz check. (#126224, @neolit123) [SIG Cluster Lifecycle] [sig/cluster-lifecycle]
• Kubeadm: fixed a bug where the path of the manifest can not be specified when kubeadm upgrade diff specified a config file, and the --api-server-manifest, --controller-manager-manifest and --scheduler-manifest flags of kubeadm upgrade diff are marked as deprecated and will be removed in a future release. (#125779, @SataQiu) [sig/cluster-lifecycle]
• Kubeadm: the --feature-gates flag is deprecated and no-op for kubeadm upgrade apply/plan, and it will be removed in a future release. The upgrade workflow is not designed to reconfigure the cluster. Please edit the 'featureGates' field of ClusterConfiguration which is defined in the kube-system/kubeadm-config ConfigMap instead. (#125797, @SataQiu) [sig/cluster-lifecycle]
• Kubectl: Show the Pod phase in the STATUS column as 'Failed' or 'Succeeded' when the Pod is terminated (#122038, @lowang-bh) [sig/cli]
• Kubelet now hard rejects pods with AppArmor if the node does not have AppArmor. (#125776, @vinayakankugoyal) [sig/node]
• Mount-utils: treated syscall.ENODEV as corrupted mount. (#126174, @dobsonj) [sig/storage]
• Now the .status.ready field is tracked faster when active Pods are deleted, specifically when Job is failed, gets suspended or has too many active pods. (#125546, @dejanzele) [sig/apps]
• Removed admission plugin PersistentVolumeLabel. Please use https://github.com/kubernetes-sigs/cloud-pv-admission-labeler instead if you need a similar functionality. (#124505, @jsafrane) [SIG API Machinery, Auth and Storage] [sig/storage,sig/api-machinery,sig/auth]
• Reverted "Graduates the WatchList feature gate to beta for kube-apiserver and enables WatchListClient for kube-controller-manager (KCM)". (#126191, @p0lyn0mial) [SIG API Machinery and Testing] [sig/api-machinery,sig/testing]
• Set ProcMountType feature to disabled by default, to follow the lead of UserNamespacesSupport (which it relies on). (#126291, @haircommander) [SIG Node] [sig/node]
• StatefulSet autodelete respected controlling owners on PVC claims as described in kubernetes/enhancements#4375. (#122499, @mattcary) [SIG Apps and Testing] [sig/apps,sig/testing]
• Stopped using wmic on Windows to get uuid in the kubelet. (#126012, @marosset) [SIG Node and Windows] [sig/node,sig/windows]
• The "fake" clients generated by client-gen now have the same semantics on error as the real clients; in particular, a failed Get(), Create(), etc, no longer returns nil. (It now returns a pointer to a zero-valued object, like the real clients do.) This will break some downstream unit tests that were testing result == nil rather than err != nil, and in some cases may expose bugs in the underlying code that were hidden by the incorrect unit tests. (#122892, @danwinship) [SIG API Machinery, Auth, Cloud Provider, Instrumentation and Storage] [sig/storage,sig/api-machinery,sig/auth,sig/instrumentation,sig/cloud-provider]
• The emission of RecreatingFailedPod and RecreatingTerminatedPod events has been removed from the StatefulSet lifecycle. (#123809, @atiratree) [SIG Apps and Testing] [sig/apps,sig/testing]
• The scheduler retries scheduling Pods rejected by PreFilterResult (PreFilter plugins) more appropriately; it now takes events registered in those rejector PreFilter plugins into consideration. (#122251, @olderTaoist) [SIG Scheduling and Testing] [sig/scheduling,sig/testing]
• Updated description of default values for --healthz-bind-address and --metrics-bind-address parameters. (#123545, @yangjunmyfm192085) [sig/network]
• When schedulingQueueHint is enabled, the scheduling queue doesn't update Pods being scheduled immediately. (#125578, @nayihz) [sig/scheduling]
• Job: Fixed a bug where SuccessCriteriaMet could be added to the Job with successPolicy regardless of the featureGate being enabled. (#125429, @tenzen-y) [sig/apps]

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: (#125272, @mauri870) [sig/testing]

• kubeadm: Allowed the kubeadm init phase certs sa command to accept the --config flag. (#125396, @Kavinraja-G) [sig/cluster-lifecycle]
• kubeadm: Improved the IsPrivilegedUser preflight check to not fail on certain Windows setups. (#124665, @neolit123) [sig/cluster-lifecycle]
• lastSuccessfullTime in cronjobs will now be set reliably. (#122025, @lukashankeln) [sig/apps]

Other (Cleanup or Flake)

• "Removed the ability to run kubectl exec [POD] [COMMAND] without a -- separator. The -- separator has been recommended since the Kubernetes v1.18 release, which also deprecated the legacy way of invoking kubectl exec.

This change aligns with the deprecation of legacy kubectl exec command execution and enforces the use of kubectl exec [POD] -- [COMMAND] for improved compatibility and adherence to recommended practices." (#125437, @ardaguclu) [SIG CLI and Testing] [sig/cli,sig/testing]

• "kubectl describe service" and "kubectl describe ingress" will now use endpointslices instead of endpoints. (#124598, @aroradaman) [SIG CLI and Network] [sig/network,sig/cli]
• ACTION-REQUIRED: Dynamic Resource Allocation (DRA) drivers using the v1alpha2 kubelet gRPC API are no longer supported and need to be updated. (#124316, @pohly) [SIG Node and Testing] [sig/node,sig/testing]
• API Priority and Fairness feature was promoted to GA in 1.29, the corresponding feature gate 'APIPriorityAndFairness' has been removed in 1.31. (#125846, @tkashem) [SIG API Machinery] [sig/api-machinery]
• Added a testcase to check hostname and hostNetwork. (#124428, @yashsingh74) [SIG Architecture, Network and Testing] [sig/network,sig/testing,sig/architecture]
• Built etcd image v3.5.13. (#124026, @liangyuanpeng) [SIG API Machinery and Etcd] [sig/api-machinery,sig/etcd]
• Built etcd image v3.5.14. (#125235, @humblec) [sig/api-machinery]
• Cleaned deprecated context.StopCh in favor of ctx. (#125661, @mjudeikis) [sig/api-machinery]
• Container Storage Interface (CSI) spec support has been lifted to v1.9.0 in this release. (#125150, @humblec) [SIG Storage and Testing] [sig/storage,sig/testing]
• Drop support for the deprecated and unsupported kubectl run flags:
• filename
• force
• grace-period
• kustomize
• recursive
• timeout
• wait

Drop support for the deprecated --delete-local-data from kubectl drain, users should use --delete-emptydir-data, instead. (#125842, @soltysh) [SIG CLI] [sig/cli]

• Dynamic Resource Allocation (DRA): fixed some small, unlikely race condition during pod scheduling. (#124595, @pohly) [SIG Node, Scheduling and Testing] [sig/scheduling,sig/node,sig/testing]
• E2e.test and e2e_node.test: tests which depend on alpha or beta feature gates now have Feature:Alpha or Feature:Beta as Ginkgo labels. The inline text is [Alpha] or [Beta], as before. (#124350, @pohly) [sig/testing]
• Ensured that the Node Admission plugin to reject CSR requests created by a node identity for the signers kubernetes.io/kubelet-serving or kubernetes.io/kube-apiserver-client-kubelet with a CN starting with system:node:, but where the CN is not system:node:${node-name}. The feature gate AllowInsecureKubeletCertificateSigningRequests defaults to false, but can be enabled to revert to the previous behavior. This feature gate will be removed in Kubernetes v1.33. (#126441, @micahhausler) [sig/auth]
• Etcd: Updated to v3.5.13. (#124027, @liangyuanpeng) [SIG API Machinery, Cloud Provider, Cluster Lifecycle, Etcd and Testing] [sig/api-machinery,sig/cluster-lifecycle,sig/testing,sig/cloud-provider,sig/etcd]
• Exposed the apiserver_watch_cache_resource_version metric to simplify debugging problems with watchcache. (#125377, @wojtek-t) [SIG API Machinery and Instrumentation] [sig/api-machinery,sig/instrumentation]
• Exposed the kube-scheduler "/livez" and "/readz" endpoints for health checks that are in compliance with https://kubernetes.io/docs/reference/using-api/health-checks/#api-endpoints-for-health. (#118148, @linxiulei) [SIG API Machinery, Scheduling and Testing] [sig/scheduling,sig/api-machinery,sig/testing]
• Finished initial generic controlplane refactor of kube-apiserver, providing a sample binary building a Kubernetes-like control plane, but without container orchestration resources. (#124530, @sttts) [SIG API Machinery, Apps, Cloud Provider, Network, Node and Testing] [sig/network,sig/node,sig/api-machinery,sig/apps,sig/testing,sig/cloud-provider]
• Fixed a typo in the help text for the pod_scheduling_sli_duration_seconds metric in kube-scheduler. (#124221, @arturhoo) [SIG Instrumentation, Scheduling and Testing] [sig/scheduling,sig/instrumentation,sig/testing]
• Improved the documentation clarity for building Kubernetes in Docker Environment, making it more understandable for new users and contributors. (#125536, @this-is-yaash) [sig/release]
• Job-controller: the JobReadyPods feature flag has been removed (deprecated since v1.31). (#125168, @kaisoz) [sig/apps]
• Kube-apiserver: the --enable-logs-handler flag and log-serving functionality which was already deprecated is now switched off by default and scheduled to be removed in v1.33. (#125787, @dims) [SIG API Machinery, Network and Testing] [sig/network,sig/api-machinery,sig/testing]
• Kubeadm: Removed the deprecated UpgradeAddonsBeforeControlPlane feature gate; Ensured that the upgrade of the CoreDNS and kube-proxy addons would not be triggered until all the control plane instances were upgraded. (#124715, @SataQiu) [sig/cluster-lifecycle]
• Kubeadm: Strictly enabled only the supported klog flags, disallowing previously available but unrecommended options. This means that hidden flags about klog (including --alsologtostderr, --log-backtrace-at, --log-dir, --logtostderr, --log-file, --log-file-max-size, --one-output, --skip-log-headers, --stderrthreshold and --vmodule) are no longer allowed to be used. (#125179, @SataQiu) [sig/cluster-lifecycle]
• Kubeadm: The global --rootfs flag considered non-experimental. (#124375, @neolit123) [sig/cluster-lifecycle]
• Kubeadm: improved the warning/error messages of validateSupportedVersion to include the checked resource kind name. (#125758, @SataQiu) [sig/cluster-lifecycle]
• Kubeadm: removed the EXPERIMENTAL tag from the phase "kubeadm join control-plane-prepare download-certs". (#124374, @neolit123) [sig/cluster-lifecycle]
• Kubeadm: removed the deprecated output.kubeadm.k8s.io/v1alpha2 API for structured output. Please use v1alpha3 instead. (#124496, @carlory) [sig/cluster-lifecycle]
• Kubeadm: removed the deprecated and NO-OP "kubeadm join control-plane-join update-status" phase. (#124373, @neolit123) [sig/cluster-lifecycle]
• Kubelet is no longer able to recover from device manager state file older than 1.20. If the proper recommended upgrade flow is followed, there should be no issue. (#123398, @ffromani) [SIG Node and Testing] [sig/node,sig/testing]
• Migrated the pkg/proxy to use contextual logging. (#122979, @fatsheep9146) [SIG Network and Scalability] [sig/network,sig/scalability]
• Moved remote CRI implementation from kubelet to k8s.io/cri-client repository. (#124634, @saschagrunert) [SIG Node, Release and Testing] [sig/node,sig/testing,sig/release]
• Optimized log output to avoid printing out redundant information of the pod. (#124055, @yangjunmyfm192085) [sig/scheduling]
• Removed GA ServiceNodePortStaticSubrange feature gate. (#124738, @xuzhenglun) [sig/network]
• Removed Kubelet flags --iptables-masquerade-bit and --iptables-drop-bit as they were deprecated in v1.28. in v1.28 and have now been removed entirely. (#122363, @carlory) [SIG Network and Node] [sig/network,sig/node]
• Removed ENABLE_CLIENT_GO_WATCH_LIST_ALPHA environment variable from the reflector. To activate the feature set KUBE_FEATURE_WatchListClient environment variable or a corresponding command line option (this works only with binaries that explicitly expose it). (#122791, @p0lyn0mial) [SIG API Machinery and Testing] [sig/api-machinery,sig/testing]
• Removed generally available feature gate CSINodeExpandSecret. (#124462, @carlory) [sig/storage]
• Removed generally available feature gate ConsistentHTTPGetHandlers. (#124463, @carlory) [sig/node]
• Removed generally available feature gate ReadWriteOncePod. (#124329, @chrishenzie) [sig/storage]
• Removed the following feature gates:
• InTreePluginAWSUnregister
• InTreePluginAzureDiskUnregister
• InTreePluginAzureFileUnregister
• InTreePluginGCEUnregister
• InTreePluginOpenStackUnregister
• InTreePluginvSphereUnregister (#124815, @carlory) [SIG Storage] [sig/storage]
• Removed the last remaining in-tree gcp cloud provider and credential provider. Please use the external cloud provider and credential provider from https://github.com/kubernetes/cloud-provider-gcp instead. (#124519, @dims) [SIG API Machinery, Apps, Auth, Autoscaling, Cloud Provider, Instrumentation, Network, Node, Scheduling, Storage and Testing] [sig/network,sig/scheduling,sig/storage,sig/node,sig/api-machinery,sig/autoscaling,sig/auth,sig/apps,sig/instrumentation,sig/testing,sig/cloud-provider]
• Scheduler framework: Allowed PreBind implementations to return Pending and Unschedulable status codes. (#125360, @pohly) [sig/scheduling]
• Set LocalStorageCapacityIsolationFSQuotaMonitoring to false by default, to match UserNamespacesSupport (which the feature relies on). (#126355, @haircommander) [sig/node]
• The ValidatingAdmissionPolicy metrics have been redone to count and time all validations, including failures and admissions. (#126124, @cici37) [SIG API Machinery and Instrumentation] [sig/api-machinery,sig/instrumentation]
• The feature gate "DefaultHostNetworkHostPortsInPodTemplates" has been removed. This behavior was deprecated in v1.28, and has had no reports of issues since. (#124417, @thockin) [sig/apps]
• The feature gate "SkipReadOnlyValidationGCE" has been removed. This gate has been active for 2 releases with no reports of issues (and was such a niche thing, we didn't expect any). (#124210, @thockin) [sig/apps]
• Updated CNI Plugins to v1.5.0. (#125113, @bzsuni) [SIG Cloud Provider, Network, Node and Testing] [sig/network,sig/node,sig/testing,sig/cloud-provider]
• Updated cni-plugins to v1.4.1. (#123894, @saschagrunert) [SIG Cloud Provider, Node and Testing] [sig/node,sig/testing,sig/cloud-provider]
• Updated cri-tools to v1.30.0. (#124364, @saschagrunert) [SIG Cloud Provider, Node and Release] [sig/node,sig/release,sig/cloud-provider]
• Updated kubernetes to build with Go 1.22.5. (#126330, @ArkaSaha30) [SIG Release and Testing] [sig/testing,sig/release]
• kubeadm: The NodeSwap check that kubeadm performs during preflight, has a new warning to verify if swap has been configured correctly. (#125157, @carlory) [sig/cluster-lifecycle]
• kubectl describe service now shows internal traffic policy and ip mode of load balancer IP. (#125117, @tnqn) [SIG CLI and Network] [sig/network,sig/cli]

Dependencies

Added

• cel.dev/expr: v0.15.0
• github.com/antlr4-go/antlr/v4: v4.13.0
• github.com/go-task/slim-sprig/v3: v3.0.0
• gopkg.in/evanphx/json-patch.v4: v4.12.0

Changed

• cloud.google.com/go/compute/metadata: v0.2.3 → v0.3.0
• cloud.google.com/go/firestore: v1.11.0 → v1.12.0
• cloud.google.com/go/storage: v1.10.0 → v1.0.0
• cloud.google.com/go: v0.110.6 → v0.110.7
• github.com/Microsoft/hcsshim: v0.8.25 → v0.8.26
• github.com/alecthomas/kingpin/v2: v2.3.2 → v2.4.0
• github.com/cenkalti/backoff/v4: v4.2.1 → v4.3.0
• github.com/cespare/xxhash/v2: v2.2.0 → v2.3.0
• github.com/chzyer/readline: 2972be2 → v1.5.1
• github.com/cncf/udpa/go: c52dc94 → 269d4d4
• github.com/cncf/xds/go: e9ce688 → 555b57e
• github.com/container-storage-interface/spec: v1.8.0 → v1.9.0
• github.com/cpuguy83/go-md2man/v2: v2.0.2 → v2.0.4
• github.com/davecgh/go-spew: v1.1.1 → d8f796a
• github.com/envoyproxy/go-control-plane: v0.11.1 → v0.12.0
• github.com/envoyproxy/protoc-gen-validate: v1.0.2 → v1.0.4
• github.com/felixge/httpsnoop: v1.0.3 → v1.0.4
• github.com/fxamacker/cbor/v2: v2.6.0 → v2.7.0
• github.com/go-logr/logr: v1.4.1 → v1.4.2
• github.com/go-openapi/swag: v0.22.3 → v0.22.4
• github.com/golang/glog: v1.1.0 → v1.2.1
• github.com/golang/mock: v1.6.0 → v1.3.1
• github.com/google/cel-go: v0.17.8 → v0.20.1
• github.com/google/pprof: 4bb14d4 → 4bfdf5a
• github.com/google/uuid: v1.3.0 → v1.6.0
• github.com/googleapis/gax-go/v2: v2.11.0 → v2.0.5
• github.com/grpc-ecosystem/grpc-gateway/v2: v2.16.0 → v2.20.0
• github.com/ianlancetaylor/demangle: 28f6c0f → bd984b5
• github.com/jstemmer/go-junit-report: v0.9.1 → af01ea7
• github.com/matttproud/golang_protobuf_extensions: v1.0.4 → v1.0.2
• github.com/moby/spdystream: v0.2.0 → v0.4.0
• github.com/moby/sys/mountinfo: v0.6.2 → v0.7.1
• github.com/moby/term: 1aeaba8 → v0.5.0
• github.com/onsi/ginkgo/v2: v2.15.0 → v2.19.0
• github.com/onsi/gomega: v1.31.0 → v1.33.1
• github.com/opencontainers/runc: v1.1.12 → v1.1.13
• github.com/pmezard/go-difflib: v1.0.0 → 5d4384e
• github.com/prometheus/client_golang: v1.16.0 → v1.19.1
• github.com/prometheus/client_model: v0.4.0 → v0.6.1
• github.com/prometheus/common: v0.44.0 → v0.55.0
• github.com/prometheus/procfs: v0.10.1 → v0.15.1
• github.com/rogpeppe/go-internal: v1.10.0 → v1.12.0
• github.com/sergi/go-diff: v1.1.0 → v1.2.0
• github.com/sirupsen/logrus: v1.9.0 → v1.9.3
• github.com/spf13/cobra: v1.7.0 → v1.8.1
• github.com/stretchr/objx: v0.5.0 → v0.5.2
• github.com/stretchr/testify: v1.8.4 → v1.9.0
• go.etcd.io/bbolt: v1.3.8 → v1.3.9
• go.etcd.io/etcd/api/v3: v3.5.10 → v3.5.14
• go.etcd.io/etcd/client/pkg/v3: v3.5.10 → v3.5.14
• go.etcd.io/etcd/client/v2: v2.305.10 → v2.305.13
• go.etcd.io/etcd/client/v3: v3.5.10 → v3.5.14
• go.etcd.io/etcd/pkg/v3: v3.5.10 → v3.5.13
• go.etcd.io/etcd/raft/v3: v3.5.10 → v3.5.13
• go.etcd.io/etcd/server/v3: v3.5.10 → v3.5.13
• go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc: v0.42.0 → v0.53.0
• go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp: v0.44.0 → v0.53.0
• go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc: v1.19.0 → v1.27.0
• go.opentelemetry.io/otel/exporters/otlp/otlptrace: v1.19.0 → v1.28.0
• go.opentelemetry.io/otel/metric: v1.19.0 → v1.28.0
• go.opentelemetry.io/otel/sdk: v1.19.0 → v1.28.0
• go.opentelemetry.io/otel/trace: v1.19.0 → v1.28.0
• go.opentelemetry.io/otel: v1.19.0 → v1.28.0
• go.opentelemetry.io/proto/otlp: v1.0.0 → v1.3.1
• golang.org/x/crypto: v0.21.0 → v0.24.0
• golang.org/x/exp: a9213ee → f3d0a9c
• golang.org/x/lint: 6edffad → 1621716
• golang.org/x/mod: v0.15.0 → v0.17.0
• golang.org/x/net: v0.23.0 → v0.26.0
• golang.org/x/oauth2: v0.10.0 → v0.21.0
• golang.org/x/sync: v0.6.0 → v0.7.0
• golang.org/x/sys: v0.18.0 → v0.21.0
• golang.org/x/telemetry: b75ee88 → f48c80b
• golang.org/x/term: v0.18.0 → v0.21.0
• golang.org/x/text: v0.14.0 → v0.16.0
• golang.org/x/tools: v0.18.0 → e35e4cc
• google.golang.org/api: v0.126.0 → v0.13.0
• google.golang.org/genproto/googleapis/api: 23370e0 → 5315273
• google.golang.org/genproto/googleapis/rpc: b8732ec → f6361c8
• google.golang.org/genproto: f966b18 → b8732ec
• google.golang.org/grpc: v1.58.3 → v1.65.0
• google.golang.org/protobuf: v1.33.0 → v1.34.2
• honnef.co/go/tools: v0.0.1-2020.1.4 → v0.0.1-2019.2.3
• k8s.io/klog/v2: v2.120.1 → v2.130.1
• k8s.io/utils: 3b25d92 → 18e509b
• sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.29.0 → v0.30.3
• sigs.k8s.io/knftables: v0.0.14 → v0.0.17
• sigs.k8s.io/kustomize/api: 6ce0bf3 → v0.17.2
• sigs.k8s.io/kustomize/cmd/config: v0.11.2 → v0.14.1
• sigs.k8s.io/kustomize/kustomize/v5: 6ce0bf3 → v5.4.2
• sigs.k8s.io/kustomize/kyaml: 6ce0bf3 → v0.17.1
• sigs.k8s.io/yaml: v1.3.0 → v1.4.0

Removed

• github.com/GoogleCloudPlatform/k8s-cloud-provider: f118173
• github.com/antlr/antlr4/runtime/Go/antlr/v4: 8188dc5
• github.com/evanphx/json-patch: v4.12.0+incompatible
• github.com/fvbommel/sortorder: v1.1.0
• github.com/go-gl/glfw/v3.3/glfw: 6f7a984
• github.com/go-task/slim-sprig: 52ccab3
• github.com/golang/snappy: v0.0.3
• github.com/google/martian/v3: v3.2.1
• github.com/google/s2a-go: v0.1.7
• github.com/googleapis/enterprise-certificate-proxy: v0.2.3
• google.golang.org/genproto/googleapis/bytestream: e85fd2c
• google.golang.org/grpc/cmd/protoc-gen-go-grpc: v1.1.0
• gopkg.in/gcfg.v1: v1.2.3
• gopkg.in/warnings.v0: v0.1.2
• rsc.io/quote/v3: v3.1.0
• rsc.io/sampler: v1.3.0