Jul 17 17:29:11.730460 ip-10-0-136-43 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 17 17:29:11.730475 ip-10-0-136-43 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 17 17:29:11.730484 ip-10-0-136-43 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 17 17:29:11.730837 ip-10-0-136-43 systemd[1]: Failed to start Kubernetes Kubelet. Jul 17 17:29:23.051823 ip-10-0-136-43 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 17 17:29:23.051839 ip-10-0-136-43 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 1f7699c3e4114ebcaa96b73b1284edf8 -- Jul 17 17:31:18.771502 ip-10-0-136-43 systemd[1]: Starting Kubernetes Kubelet... Jul 17 17:31:19.251807 ip-10-0-136-43 kubenswrapper[2553]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 17:31:19.251807 ip-10-0-136-43 kubenswrapper[2553]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 17 17:31:19.251807 ip-10-0-136-43 kubenswrapper[2553]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 17:31:19.251807 ip-10-0-136-43 kubenswrapper[2553]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 17 17:31:19.251807 ip-10-0-136-43 kubenswrapper[2553]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 17:31:19.251807 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.165047 2553 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 17 17:31:19.251807 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169384 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:31:19.251807 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169395 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:31:19.251807 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169399 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:31:19.251807 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169403 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:31:19.251807 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169406 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:31:19.251807 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169410 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:31:19.251807 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169413 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:31:19.251807 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169416 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:31:19.246196 ip-10-0-136-43 systemd[1]: Started Kubernetes Kubelet. Jul 17 17:31:19.253632 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169419 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:31:19.253632 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169421 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:31:19.253632 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169424 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:31:19.253632 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169427 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:31:19.253632 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169430 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:31:19.253632 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169432 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:31:19.253632 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169435 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:31:19.253632 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169438 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:31:19.253632 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169440 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:31:19.253632 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169443 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:31:19.253632 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169446 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:31:19.253632 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169448 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:31:19.253632 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169451 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:31:19.253632 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169454 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:31:19.253632 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169457 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:31:19.253632 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169459 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:31:19.253632 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169462 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:31:19.253632 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169465 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:31:19.253632 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169468 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:31:19.253632 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169470 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:31:19.254588 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169474 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:31:19.254588 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169485 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:31:19.254588 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169488 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:31:19.254588 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169491 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:31:19.254588 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169493 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:31:19.254588 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169496 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:31:19.254588 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169498 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:31:19.254588 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169501 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:31:19.254588 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169504 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:31:19.254588 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169506 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:31:19.254588 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169509 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:31:19.254588 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169512 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:31:19.254588 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169515 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:31:19.254588 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169517 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:31:19.254588 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169520 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:31:19.254588 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169523 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:31:19.254588 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169526 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:31:19.254588 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.169528 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:31:19.254588 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170149 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:31:19.254588 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170155 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:31:19.256404 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170159 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:31:19.256404 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170164 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:31:19.256404 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170169 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:31:19.256404 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170172 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:31:19.256404 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170175 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:31:19.256404 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170177 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:31:19.256404 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170180 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:31:19.256404 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170183 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:31:19.256404 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170186 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:31:19.256404 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170189 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:31:19.256404 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170191 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:31:19.256404 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170196 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:31:19.256404 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170201 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:31:19.256404 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170204 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:31:19.256404 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170208 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:31:19.256404 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170219 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:31:19.256404 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170222 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:31:19.256404 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170225 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:31:19.257319 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170228 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:31:19.257319 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170231 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:31:19.257319 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170234 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:31:19.257319 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170236 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:31:19.257319 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170239 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:31:19.257319 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170242 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:31:19.257319 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170245 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:31:19.257319 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170247 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:31:19.257319 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170250 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:31:19.257319 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170254 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:31:19.257319 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170257 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:31:19.257319 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170259 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:31:19.257319 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170262 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:31:19.257319 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170265 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:31:19.257319 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170274 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:31:19.257319 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170278 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:31:19.257319 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170281 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:31:19.257319 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170284 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:31:19.257319 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170286 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:31:19.257319 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170289 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:31:19.258665 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170291 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:31:19.258665 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170294 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:31:19.258665 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170298 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:31:19.258665 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170301 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:31:19.258665 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170305 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:31:19.258665 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170309 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:31:19.258665 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170312 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:31:19.258665 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170315 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:31:19.258665 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170318 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:31:19.258665 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170321 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:31:19.258665 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170324 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:31:19.258665 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170326 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:31:19.258665 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.170329 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:31:19.258665 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171070 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:31:19.258665 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171077 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:31:19.258665 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171082 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:31:19.258665 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171086 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:31:19.258665 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171090 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:31:19.258665 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171093 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:31:19.259598 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171096 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:31:19.259598 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171099 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:31:19.259598 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171102 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:31:19.259598 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171105 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:31:19.259598 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171108 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:31:19.259598 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171111 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:31:19.259598 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171114 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:31:19.259598 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171117 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:31:19.259598 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171121 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:31:19.259598 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171123 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:31:19.259598 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171126 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:31:19.259598 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171129 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:31:19.259598 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171131 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:31:19.259598 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171134 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:31:19.259598 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171137 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:31:19.259598 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171140 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:31:19.259598 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171143 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:31:19.259598 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171146 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:31:19.259598 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171149 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:31:19.259598 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171152 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:31:19.261468 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171155 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:31:19.261468 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171158 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:31:19.261468 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171161 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:31:19.261468 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171164 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:31:19.261468 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171166 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:31:19.261468 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171169 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:31:19.261468 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171172 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:31:19.261468 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171175 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:31:19.261468 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171177 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:31:19.261468 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171180 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:31:19.261468 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171183 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:31:19.261468 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171185 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:31:19.261468 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171188 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:31:19.261468 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171191 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:31:19.261468 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171194 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:31:19.261468 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171196 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:31:19.261468 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171199 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:31:19.261468 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171201 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:31:19.261468 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171204 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:31:19.261468 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171207 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:31:19.263743 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171210 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:31:19.263743 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171213 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:31:19.263743 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171224 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:31:19.263743 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171227 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:31:19.263743 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171230 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:31:19.263743 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171233 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:31:19.263743 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171237 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:31:19.263743 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171240 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:31:19.263743 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171242 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:31:19.263743 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171245 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:31:19.263743 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171249 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:31:19.263743 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171252 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:31:19.263743 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171254 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:31:19.263743 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171258 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:31:19.263743 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171261 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:31:19.263743 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171264 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:31:19.263743 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171266 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:31:19.263743 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171269 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:31:19.263743 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171271 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:31:19.263743 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171274 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:31:19.264301 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171277 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:31:19.264301 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171280 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:31:19.264301 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171284 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:31:19.264301 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171288 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:31:19.264301 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171291 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:31:19.264301 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171294 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:31:19.264301 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171296 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:31:19.264301 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171299 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:31:19.264301 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171303 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:31:19.264301 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171307 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:31:19.264301 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171311 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:31:19.264301 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171313 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:31:19.264301 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171316 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:31:19.264301 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171319 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:31:19.264301 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171322 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:31:19.264301 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171325 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:31:19.264301 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171328 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:31:19.264301 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171330 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:31:19.264301 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171334 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:31:19.264772 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171336 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:31:19.264772 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171339 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:31:19.264772 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171342 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:31:19.264772 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171345 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:31:19.264772 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171347 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:31:19.264772 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171350 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:31:19.264772 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171352 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:31:19.264772 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171355 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:31:19.264772 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171358 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:31:19.264772 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171360 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:31:19.264772 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171363 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:31:19.264772 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171365 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:31:19.264772 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171368 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:31:19.264772 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171370 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:31:19.264772 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171754 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:31:19.264772 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171759 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:31:19.264772 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171762 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:31:19.264772 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171765 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:31:19.264772 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171768 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:31:19.265272 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171771 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:31:19.265272 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171774 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:31:19.265272 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171777 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:31:19.265272 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171779 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:31:19.265272 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171783 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:31:19.265272 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171785 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:31:19.265272 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171788 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:31:19.265272 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171791 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:31:19.265272 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171794 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:31:19.265272 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171797 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:31:19.265272 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171799 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:31:19.265272 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171802 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:31:19.265272 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171805 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:31:19.265272 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171811 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:31:19.265272 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171815 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:31:19.265272 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171818 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:31:19.265272 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171821 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:31:19.265272 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171825 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:31:19.265272 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171829 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:31:19.265739 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171832 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:31:19.265739 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171836 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:31:19.265739 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171839 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:31:19.265739 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171843 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:31:19.265739 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171846 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:31:19.265739 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171849 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:31:19.265739 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171852 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:31:19.265739 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171855 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:31:19.265739 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171857 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:31:19.265739 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171860 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:31:19.265739 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171863 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:31:19.265739 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171866 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:31:19.265739 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171869 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:31:19.265739 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171871 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:31:19.265739 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171875 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:31:19.265739 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171878 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:31:19.265739 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171881 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:31:19.265739 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171884 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:31:19.265739 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171886 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:31:19.265739 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171889 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:31:19.266259 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171891 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:31:19.266259 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171894 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:31:19.266259 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171897 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:31:19.266259 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171901 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:31:19.266259 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171903 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:31:19.266259 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171907 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:31:19.266259 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171909 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:31:19.266259 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171912 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:31:19.266259 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171915 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:31:19.266259 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171918 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:31:19.266259 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171936 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:31:19.266259 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171940 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:31:19.266259 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171943 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:31:19.266259 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171946 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:31:19.266259 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171948 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:31:19.266259 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171951 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:31:19.266259 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171954 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:31:19.266259 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171956 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:31:19.266259 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171959 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:31:19.266259 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171962 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:31:19.266785 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171964 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:31:19.266785 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171967 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:31:19.266785 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171972 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:31:19.266785 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171975 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:31:19.266785 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171978 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:31:19.266785 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171981 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:31:19.266785 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171983 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:31:19.266785 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171986 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:31:19.266785 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171989 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:31:19.266785 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171992 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:31:19.266785 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171994 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:31:19.266785 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.171997 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:31:19.266785 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172002 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:31:19.266785 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172005 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:31:19.266785 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172008 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:31:19.266785 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172012 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:31:19.266785 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172014 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:31:19.266785 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172017 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:31:19.266785 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172020 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172022 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172025 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172028 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172030 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172033 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172036 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172039 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172041 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172044 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172046 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172049 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172051 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172054 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172057 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172059 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172063 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172138 2553 flags.go:64] FLAG: --address="0.0.0.0" Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172149 2553 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172156 2553 flags.go:64] FLAG: --anonymous-auth="true" Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172160 2553 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 17 17:31:19.267441 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172164 2553 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172168 2553 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172172 2553 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172176 2553 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172180 2553 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172183 2553 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172186 2553 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172190 2553 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172193 2553 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172197 2553 flags.go:64] FLAG: --cgroup-root="" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172200 2553 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172203 2553 flags.go:64] FLAG: --client-ca-file="" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172206 2553 flags.go:64] FLAG: --cloud-provider="external" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172209 2553 flags.go:64] FLAG: --cluster-dns="[]" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172214 2553 flags.go:64] FLAG: --cluster-domain="" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172217 2553 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172220 2553 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172223 2553 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172227 2553 flags.go:64] FLAG: --container-log-max-files="5" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172230 2553 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172233 2553 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172237 2553 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172240 2553 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172243 2553 flags.go:64] FLAG: --contention-profiling="false" Jul 17 17:31:19.268269 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172246 2553 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172249 2553 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172252 2553 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172257 2553 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172263 2553 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172266 2553 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172269 2553 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172272 2553 flags.go:64] FLAG: --enable-load-reader="false" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172276 2553 flags.go:64] FLAG: --enable-server="true" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172279 2553 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172283 2553 flags.go:64] FLAG: --event-burst="100" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172286 2553 flags.go:64] FLAG: --event-qps="50" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172289 2553 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172292 2553 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172295 2553 flags.go:64] FLAG: --eviction-hard="" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172299 2553 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172302 2553 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172305 2553 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172323 2553 flags.go:64] FLAG: --eviction-soft="" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172326 2553 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172329 2553 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172332 2553 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172335 2553 flags.go:64] FLAG: --experimental-mounter-path="" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172338 2553 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172341 2553 flags.go:64] FLAG: --fail-swap-on="true" Jul 17 17:31:19.269305 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172344 2553 flags.go:64] FLAG: --feature-gates="" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172352 2553 flags.go:64] FLAG: --file-check-frequency="20s" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172355 2553 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172358 2553 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172362 2553 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172365 2553 flags.go:64] FLAG: --healthz-port="10248" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172368 2553 flags.go:64] FLAG: --help="false" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172370 2553 flags.go:64] FLAG: --hostname-override="ip-10-0-136-43.ec2.internal" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172374 2553 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172376 2553 flags.go:64] FLAG: --http-check-frequency="20s" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172381 2553 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172386 2553 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172389 2553 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172392 2553 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172395 2553 flags.go:64] FLAG: --image-service-endpoint="" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172398 2553 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172401 2553 flags.go:64] FLAG: --kube-api-burst="100" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172404 2553 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172407 2553 flags.go:64] FLAG: --kube-api-qps="50" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172410 2553 flags.go:64] FLAG: --kube-reserved="" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172413 2553 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172416 2553 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172420 2553 flags.go:64] FLAG: --kubelet-cgroups="" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172422 2553 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 17 17:31:19.270333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172425 2553 flags.go:64] FLAG: --lock-file="" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172428 2553 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172437 2553 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172440 2553 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172446 2553 flags.go:64] FLAG: --log-json-split-stream="false" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172449 2553 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172453 2553 flags.go:64] FLAG: --log-text-split-stream="false" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172456 2553 flags.go:64] FLAG: --logging-format="text" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172459 2553 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172462 2553 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172465 2553 flags.go:64] FLAG: --manifest-url="" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172468 2553 flags.go:64] FLAG: --manifest-url-header="" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172473 2553 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172476 2553 flags.go:64] FLAG: --max-open-files="1000000" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172480 2553 flags.go:64] FLAG: --max-pods="110" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172483 2553 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172486 2553 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172489 2553 flags.go:64] FLAG: --memory-manager-policy="None" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172493 2553 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172499 2553 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172502 2553 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172505 2553 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172513 2553 flags.go:64] FLAG: --node-status-max-images="50" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172516 2553 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172519 2553 flags.go:64] FLAG: --oom-score-adj="-999" Jul 17 17:31:19.271389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172522 2553 flags.go:64] FLAG: --pod-cidr="" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172525 2553 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172531 2553 flags.go:64] FLAG: --pod-manifest-path="" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172534 2553 flags.go:64] FLAG: --pod-max-pids="-1" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172537 2553 flags.go:64] FLAG: --pods-per-core="0" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172540 2553 flags.go:64] FLAG: --port="10250" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172543 2553 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172546 2553 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04770533fa2e35f93" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172549 2553 flags.go:64] FLAG: --qos-reserved="" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172552 2553 flags.go:64] FLAG: --read-only-port="10255" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172561 2553 flags.go:64] FLAG: --register-node="true" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172565 2553 flags.go:64] FLAG: --register-with-taints="" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172569 2553 flags.go:64] FLAG: --registry-burst="10" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172572 2553 flags.go:64] FLAG: --registry-qps="5" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172574 2553 flags.go:64] FLAG: --reserved-cpus="" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172577 2553 flags.go:64] FLAG: --reserved-memory="" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172581 2553 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172584 2553 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172587 2553 flags.go:64] FLAG: --rotate-certificates="false" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172590 2553 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172593 2553 flags.go:64] FLAG: --runonce="false" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172596 2553 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172599 2553 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172602 2553 flags.go:64] FLAG: --seccomp-default="false" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172605 2553 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 17 17:31:19.272421 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172611 2553 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172614 2553 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172619 2553 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172622 2553 flags.go:64] FLAG: --storage-driver-password="root" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172625 2553 flags.go:64] FLAG: --storage-driver-secure="false" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172628 2553 flags.go:64] FLAG: --storage-driver-table="stats" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172631 2553 flags.go:64] FLAG: --storage-driver-user="root" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172634 2553 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172637 2553 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172640 2553 flags.go:64] FLAG: --system-cgroups="" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172643 2553 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172648 2553 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172651 2553 flags.go:64] FLAG: --tls-cert-file="" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172654 2553 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172661 2553 flags.go:64] FLAG: --tls-min-version="" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172664 2553 flags.go:64] FLAG: --tls-private-key-file="" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172667 2553 flags.go:64] FLAG: --topology-manager-policy="none" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172670 2553 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172674 2553 flags.go:64] FLAG: --topology-manager-scope="container" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172677 2553 flags.go:64] FLAG: --v="2" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172681 2553 flags.go:64] FLAG: --version="false" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172685 2553 flags.go:64] FLAG: --vmodule="" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172689 2553 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.172692 2553 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172807 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:31:19.273567 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172811 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:31:19.274664 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172815 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:31:19.274664 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172818 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:31:19.274664 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172822 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:31:19.274664 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172825 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:31:19.274664 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172828 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:31:19.274664 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172831 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:31:19.274664 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172835 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:31:19.274664 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172838 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:31:19.274664 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172842 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:31:19.274664 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172845 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:31:19.274664 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172848 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:31:19.274664 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172850 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:31:19.274664 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172853 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:31:19.274664 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172856 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:31:19.274664 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172859 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:31:19.274664 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172863 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:31:19.274664 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172866 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:31:19.274664 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172869 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:31:19.274664 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172872 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:31:19.275527 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172874 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:31:19.275527 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172877 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:31:19.275527 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172880 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:31:19.275527 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172883 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:31:19.275527 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172885 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:31:19.275527 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172888 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:31:19.275527 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172892 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:31:19.275527 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172894 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:31:19.275527 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172897 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:31:19.275527 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172900 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:31:19.275527 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172902 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:31:19.275527 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172905 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:31:19.275527 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172908 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:31:19.275527 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172910 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:31:19.275527 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172913 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:31:19.275527 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172916 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:31:19.275527 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172919 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:31:19.275527 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172938 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:31:19.275527 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172941 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:31:19.276368 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172947 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:31:19.276368 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172951 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:31:19.276368 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172956 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:31:19.276368 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172959 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:31:19.276368 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172963 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:31:19.276368 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172966 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:31:19.276368 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172968 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:31:19.276368 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172971 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:31:19.276368 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172974 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:31:19.276368 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172976 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:31:19.276368 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172979 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:31:19.276368 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172982 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:31:19.276368 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172985 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:31:19.276368 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172988 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:31:19.276368 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172991 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:31:19.276368 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172994 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:31:19.276368 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172996 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:31:19.276368 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.172999 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:31:19.276368 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173002 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:31:19.277183 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173005 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:31:19.277183 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173008 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:31:19.277183 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173011 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:31:19.277183 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173014 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:31:19.277183 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173016 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:31:19.277183 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173019 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:31:19.277183 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173021 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:31:19.277183 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173025 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:31:19.277183 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173028 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:31:19.277183 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173031 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:31:19.277183 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173034 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:31:19.277183 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173036 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:31:19.277183 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173039 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:31:19.277183 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173043 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:31:19.277183 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173046 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:31:19.277183 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173050 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:31:19.277183 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173052 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:31:19.277183 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173055 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:31:19.277183 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173058 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:31:19.277183 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173060 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:31:19.278104 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173063 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:31:19.278104 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173066 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:31:19.278104 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173068 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:31:19.278104 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173071 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:31:19.278104 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173074 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:31:19.278104 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173076 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:31:19.278104 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173079 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:31:19.278104 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173082 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:31:19.278104 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173084 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:31:19.278104 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173089 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:31:19.278104 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173092 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:31:19.278104 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173095 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:31:19.278104 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173098 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:31:19.278104 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173101 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:31:19.278104 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173104 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:31:19.278104 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173107 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:31:19.278104 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173110 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:31:19.278104 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173112 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:31:19.278104 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173115 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:31:19.278104 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.173118 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:31:19.279004 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.173124 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 17:31:19.279004 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.180498 2553 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 17 17:31:19.279004 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.180513 2553 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 17 17:31:19.279004 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.180534 2553 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 17 17:31:19.279004 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.180540 2553 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 17 17:31:19.279004 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180597 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:31:19.279004 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180602 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:31:19.279004 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180606 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:31:19.279004 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180609 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:31:19.279004 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180613 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:31:19.279004 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180616 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:31:19.279004 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180619 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:31:19.279004 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180622 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:31:19.279004 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180625 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:31:19.279004 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180628 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:31:19.279690 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180631 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:31:19.279690 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180633 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:31:19.279690 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180636 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:31:19.279690 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180640 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:31:19.279690 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180643 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:31:19.279690 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180646 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:31:19.279690 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180649 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:31:19.279690 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180651 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:31:19.279690 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180654 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:31:19.279690 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180657 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:31:19.279690 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180660 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:31:19.279690 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180663 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:31:19.279690 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180666 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:31:19.279690 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180669 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:31:19.279690 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180671 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:31:19.279690 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180674 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:31:19.279690 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180677 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:31:19.279690 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180679 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:31:19.279690 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180682 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:31:19.279690 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180686 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:31:19.280587 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180689 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:31:19.280587 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180691 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:31:19.280587 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180694 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:31:19.280587 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180697 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:31:19.280587 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180701 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:31:19.280587 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180705 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:31:19.280587 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180709 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:31:19.280587 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180712 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:31:19.280587 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180715 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:31:19.280587 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180718 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:31:19.280587 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180721 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:31:19.280587 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180724 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:31:19.280587 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180726 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:31:19.280587 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180729 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:31:19.280587 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180732 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:31:19.280587 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180735 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:31:19.280587 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180737 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:31:19.280587 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180740 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:31:19.280587 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180742 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:31:19.281406 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180745 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:31:19.281406 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180748 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:31:19.281406 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180750 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:31:19.281406 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180753 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:31:19.281406 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180756 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:31:19.281406 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180759 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:31:19.281406 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180763 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:31:19.281406 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180767 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:31:19.281406 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180771 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:31:19.281406 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180773 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:31:19.281406 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180776 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:31:19.281406 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180778 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:31:19.281406 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180786 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:31:19.281406 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180789 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:31:19.281406 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180791 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:31:19.281406 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180794 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:31:19.281406 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180797 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:31:19.281406 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180800 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:31:19.281406 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180803 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:31:19.281406 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180805 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:31:19.282157 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180808 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:31:19.282157 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180811 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:31:19.282157 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180814 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:31:19.282157 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180816 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:31:19.282157 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180819 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:31:19.282157 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180822 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:31:19.282157 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180824 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:31:19.282157 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180827 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:31:19.282157 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180830 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:31:19.282157 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180832 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:31:19.282157 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180835 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:31:19.282157 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180838 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:31:19.282157 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180840 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:31:19.282157 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180843 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:31:19.282157 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180845 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:31:19.282157 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180848 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:31:19.282157 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180851 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:31:19.282157 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180853 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:31:19.282157 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180857 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:31:19.282157 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180859 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:31:19.282765 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180862 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:31:19.282765 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180864 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:31:19.282765 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180867 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:31:19.282765 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180872 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:31:19.282765 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180878 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:31:19.282765 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180881 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:31:19.282765 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180884 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:31:19.282765 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180887 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:31:19.282765 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180891 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:31:19.282765 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.180893 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:31:19.282765 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.180899 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 17:31:19.282765 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181035 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:31:19.282765 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181041 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:31:19.282765 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181044 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:31:19.283154 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181047 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:31:19.283154 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181050 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:31:19.283154 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181052 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:31:19.283154 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181055 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:31:19.283154 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181058 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:31:19.283154 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181061 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:31:19.283154 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181064 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:31:19.283154 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181066 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:31:19.283154 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181069 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:31:19.283154 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181072 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:31:19.283154 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181074 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:31:19.283154 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181077 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:31:19.283154 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181080 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:31:19.283154 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181082 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:31:19.283154 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181085 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:31:19.283154 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181088 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:31:19.283154 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181090 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:31:19.283154 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181094 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:31:19.283154 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181096 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:31:19.283154 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181099 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:31:19.284031 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181101 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:31:19.284031 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181104 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:31:19.284031 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181111 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:31:19.284031 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181116 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:31:19.284031 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181118 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:31:19.284031 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181121 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:31:19.284031 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181124 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:31:19.284031 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181127 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:31:19.284031 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181129 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:31:19.284031 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181133 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:31:19.284031 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181137 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:31:19.284031 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181140 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:31:19.284031 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181143 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:31:19.284031 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181146 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:31:19.284031 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181149 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:31:19.284031 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181152 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:31:19.284031 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181155 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:31:19.284031 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181158 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:31:19.284031 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181160 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:31:19.284863 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181163 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:31:19.284863 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181165 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:31:19.284863 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181168 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:31:19.284863 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181171 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:31:19.284863 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181174 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:31:19.284863 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181176 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:31:19.284863 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181179 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:31:19.284863 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181181 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:31:19.284863 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181184 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:31:19.284863 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181187 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:31:19.284863 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181189 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:31:19.284863 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181192 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:31:19.284863 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181195 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:31:19.284863 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181199 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:31:19.284863 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181203 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:31:19.284863 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181207 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:31:19.284863 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181211 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:31:19.284863 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181214 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:31:19.284863 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181217 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:31:19.284863 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181220 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:31:19.285800 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181222 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:31:19.285800 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181225 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:31:19.285800 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181228 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:31:19.285800 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181231 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:31:19.285800 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181233 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:31:19.285800 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181237 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:31:19.285800 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181239 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:31:19.285800 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181242 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:31:19.285800 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181245 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:31:19.285800 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181247 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:31:19.285800 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181250 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:31:19.285800 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181253 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:31:19.285800 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181255 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:31:19.285800 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181258 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:31:19.285800 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181260 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:31:19.285800 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181263 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:31:19.285800 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181266 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:31:19.285800 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181268 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:31:19.285800 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181271 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:31:19.285800 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181273 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:31:19.286679 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181276 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:31:19.286679 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181278 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:31:19.286679 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181281 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:31:19.286679 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181285 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:31:19.286679 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181290 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:31:19.286679 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181293 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:31:19.286679 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181296 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:31:19.286679 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181300 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:31:19.286679 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181304 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:31:19.286679 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181307 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:31:19.286679 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181309 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:31:19.286679 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181312 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:31:19.286679 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181315 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:31:19.286679 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181317 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:31:19.286679 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181320 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:31:19.286679 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181323 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:31:19.286679 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:19.181325 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:31:19.287198 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.181331 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 17:31:19.287198 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.181438 2553 server.go:962] "Client rotation is on, will bootstrap in background" Jul 17 17:31:19.287198 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.183551 2553 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 17 17:31:19.287198 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.184845 2553 server.go:1019] "Starting client certificate rotation" Jul 17 17:31:19.287198 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.184962 2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 17:31:19.287198 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.185002 2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 17:31:19.287198 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.187488 2553 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 17:31:19.287198 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.192443 2553 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 17:31:19.287198 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.205691 2553 log.go:25] "Validated CRI v1 runtime API" Jul 17 17:31:19.287198 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.208419 2553 log.go:25] "Validated CRI v1 image API" Jul 17 17:31:19.287198 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.209907 2553 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 17 17:31:19.287198 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.212341 2553 fs.go:135] Filesystem UUIDs: map[4191a475-60ee-4f12-a8d1-c71f18efca7e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9fb35531-5cb2-40f6-b59d-8472ff952cb5:/dev/nvme0n1p3] Jul 17 17:31:19.287513 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.212357 2553 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 17 17:31:19.287548 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.218495 2553 manager.go:216] Machine: {Timestamp:2026-07-17 17:31:19.21650577 +0000 UTC m=+0.335334044 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200567 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c21ca68926ffbcba8336470445b9f SystemUUID:ec2c21ca-6892-6ffb-cba8-336470445b9f BootID:1f7699c3-e411-4ebc-aa96-b73b1284edf8 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:75:f5:25:7e:e3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:75:f5:25:7e:e3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:bb:8c:eb:59:7e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 17 17:31:19.287548 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.218611 2553 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 17 17:31:19.287548 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.218696 2553 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 17 17:31:19.287548 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.220220 2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 17:31:19.287548 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.222702 2553 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 17 17:31:19.287548 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.222732 2553 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-43.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 17 17:31:19.287548 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.223486 2553 topology_manager.go:138] "Creating topology manager with none policy" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.223497 2553 container_manager_linux.go:309] "Creating device plugin manager" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.223505 2553 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.224265 2553 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.225610 2553 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.229842 2553 state_mem.go:36] "Initialized new in-memory state store" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.229996 2553 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.230026 2553 kubelet.go:486] "Attempting to sync node with API server" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.230037 2553 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.230054 2553 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.230065 2553 kubelet.go:398] "Adding apiserver pod source" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.230074 2553 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.230485 2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.230501 2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.232893 2553 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.234871 2553 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.234899 2553 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.235959 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.235981 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.235991 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.235998 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.236009 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.236016 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.236967 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.236975 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.236984 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.236990 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.237000 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.237008 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.237745 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.237751 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.241421 2553 server.go:1268] "Started kubelet" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.245885 2553 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.246211 2553 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.246300 2553 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.249886 2553 server.go:310] "Adding debug handlers to kubelet server" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:19.255369 2553 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.261201 2553 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.267083 2553 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.267133 2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:19.267433 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-43.ec2.internal\" not found" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.268099 2553 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.268232 2553 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.268246 2553 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 17 17:31:19.287766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.268257 2553 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 17 17:31:19.288872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.268274 2553 factory.go:55] Registering systemd factory Jul 17 17:31:19.288872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.268284 2553 factory.go:223] Registration of the systemd container factory successfully Jul 17 17:31:19.288872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.268389 2553 reconstruct.go:97] "Volume reconstruction finished" Jul 17 17:31:19.288872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.268398 2553 reconciler.go:29] "Reconciler: start to sync state" Jul 17 17:31:19.288872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.268624 2553 factory.go:153] Registering CRI-O factory Jul 17 17:31:19.288872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.268638 2553 factory.go:223] Registration of the crio container factory successfully Jul 17 17:31:19.288872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.268662 2553 factory.go:103] Registering Raw factory Jul 17 17:31:19.288872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.268679 2553 manager.go:1196] Started watching for new ooms in manager Jul 17 17:31:19.288872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.269350 2553 manager.go:349] Starting recovery of all containers Jul 17 17:31:19.288872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.274603 2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q47qk" Jul 17 17:31:19.288872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.280619 2553 manager.go:354] Recovery completed Jul 17 17:31:19.288872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.282227 2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q47qk" Jul 17 17:31:19.288872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.286604 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:31:19.290583 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.290565 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:31:19.290641 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.290597 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:31:19.290641 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.290611 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:31:19.291142 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.291129 2553 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 17 17:31:19.291199 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.291141 2553 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 17 17:31:19.291199 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.291188 2553 state_mem.go:36] "Initialized new in-memory state store" Jul 17 17:31:19.291297 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.291204 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:31:19.293353 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.293341 2553 policy_none.go:49] "None policy: Start" Jul 17 17:31:19.293394 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.293357 2553 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 17 17:31:19.293394 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.293368 2553 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 17 17:31:19.295443 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.295432 2553 policy_none.go:47] "Start" Jul 17 17:31:19.313355 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.313335 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:31:19.313435 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.313368 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:31:19.313435 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.313381 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:31:19.315052 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:19.315025 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 17 17:31:19.315171 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.315151 2553 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-43.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 17 17:31:19.315351 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:19.315329 2553 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 17 17:31:19.315938 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:19.315710 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 17 17:31:19.316004 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:19.315938 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 17 17:31:19.316594 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:19.315090 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-43.ec2.internal.18c32404878a3dd2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-43.ec2.internal,UID:ip-10-0-136-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-43.ec2.internal,},FirstTimestamp:2026-07-17 17:31:19.241301458 +0000 UTC m=+0.360129732,LastTimestamp:2026-07-17 17:31:19.241301458 +0000 UTC m=+0.360129732,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-43.ec2.internal,}" Jul 17 17:31:19.345541 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.345525 2553 manager.go:337] "Starting Device Plugin manager" Jul 17 17:31:19.345626 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:19.345561 2553 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 17 17:31:19.345626 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.345573 2553 server.go:87] "Starting device plugin registration server" Jul 17 17:31:19.345846 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.345836 2553 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 17 17:31:19.345891 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.345848 2553 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 17 17:31:19.346065 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.346044 2553 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 17 17:31:19.346152 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.346133 2553 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 17 17:31:19.346152 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.346147 2553 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 17 17:31:19.346633 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:19.346613 2553 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 17 17:31:19.346707 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:19.346669 2553 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-43.ec2.internal\" not found" Jul 17 17:31:19.446992 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.446945 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:31:19.450114 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.450088 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:31:19.450236 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.450136 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:31:19.450236 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.450153 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:31:19.450236 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.450191 2553 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-43.ec2.internal" Jul 17 17:31:19.455869 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.455822 2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 17 17:31:19.457021 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.457003 2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 17 17:31:19.457093 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.457054 2553 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 17 17:31:19.457093 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.457082 2553 kubelet.go:2448] "Starting kubelet main sync loop" Jul 17 17:31:19.457160 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:19.457123 2553 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 17 17:31:19.458270 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.458255 2553 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-43.ec2.internal" Jul 17 17:31:19.458362 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:19.458279 2553 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-43.ec2.internal\": node \"ip-10-0-136-43.ec2.internal\" not found" Jul 17 17:31:19.459571 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.459548 2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:31:19.495401 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:19.495376 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-43.ec2.internal\" not found" Jul 17 17:31:19.558375 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.558304 2553 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-43.ec2.internal"] Jul 17 17:31:19.558464 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.558379 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:31:19.560096 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.560079 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:31:19.560181 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.560110 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:31:19.560181 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.560122 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:31:19.562206 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.562194 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:31:19.562322 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.562306 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal" Jul 17 17:31:19.562371 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.562339 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:31:19.563687 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.563671 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:31:19.563770 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.563677 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:31:19.563770 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.563708 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:31:19.563770 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.563717 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:31:19.563770 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.563694 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:31:19.563979 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.563790 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:31:19.565832 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.565815 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-43.ec2.internal" Jul 17 17:31:19.565918 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.565845 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:31:19.566967 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.566946 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:31:19.567050 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.566977 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:31:19.567050 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.566990 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:31:19.570144 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.570125 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c86625a603c83bec1cac3688696a497-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal\" (UID: \"0c86625a603c83bec1cac3688696a497\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal" Jul 17 17:31:19.570291 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.570149 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c86625a603c83bec1cac3688696a497-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal\" (UID: \"0c86625a603c83bec1cac3688696a497\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal" Jul 17 17:31:19.570291 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.570168 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/33e1e8a8a145842c049c2ce22b537c78-config\") pod \"kube-apiserver-proxy-ip-10-0-136-43.ec2.internal\" (UID: \"33e1e8a8a145842c049c2ce22b537c78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-43.ec2.internal" Jul 17 17:31:19.580503 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:19.580484 2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-43.ec2.internal\" not found" node="ip-10-0-136-43.ec2.internal" Jul 17 17:31:19.583939 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:19.583917 2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-43.ec2.internal\" not found" node="ip-10-0-136-43.ec2.internal" Jul 17 17:31:19.596220 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:19.596199 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-43.ec2.internal\" not found" Jul 17 17:31:19.670936 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.670903 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c86625a603c83bec1cac3688696a497-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal\" (UID: \"0c86625a603c83bec1cac3688696a497\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal" Jul 17 17:31:19.670936 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.670937 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c86625a603c83bec1cac3688696a497-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal\" (UID: \"0c86625a603c83bec1cac3688696a497\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal" Jul 17 17:31:19.671146 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.670956 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/33e1e8a8a145842c049c2ce22b537c78-config\") pod \"kube-apiserver-proxy-ip-10-0-136-43.ec2.internal\" (UID: \"33e1e8a8a145842c049c2ce22b537c78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-43.ec2.internal" Jul 17 17:31:19.671146 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.671008 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c86625a603c83bec1cac3688696a497-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal\" (UID: \"0c86625a603c83bec1cac3688696a497\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal" Jul 17 17:31:19.671146 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.671013 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/33e1e8a8a145842c049c2ce22b537c78-config\") pod \"kube-apiserver-proxy-ip-10-0-136-43.ec2.internal\" (UID: \"33e1e8a8a145842c049c2ce22b537c78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-43.ec2.internal" Jul 17 17:31:19.671146 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.671073 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c86625a603c83bec1cac3688696a497-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal\" (UID: \"0c86625a603c83bec1cac3688696a497\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal" Jul 17 17:31:19.696579 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:19.696539 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-43.ec2.internal\" not found" Jul 17 17:31:19.798203 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:19.798167 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-43.ec2.internal\" not found" Jul 17 17:31:19.884072 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.883986 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal" Jul 17 17:31:19.887243 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:19.887225 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-43.ec2.internal" Jul 17 17:31:19.898452 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:19.898428 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-43.ec2.internal\" not found" Jul 17 17:31:20.000021 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:19.999980 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-43.ec2.internal\" not found" Jul 17 17:31:20.100952 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:20.100919 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-43.ec2.internal\" not found" Jul 17 17:31:20.184533 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:20.184462 2553 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 17 17:31:20.184656 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:20.184613 2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 17:31:20.202014 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:20.201994 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-43.ec2.internal\" not found" Jul 17 17:31:20.283921 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:20.283863 2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:31:20.287226 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:20.287199 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 17:26:19 +0000 UTC" deadline="2028-03-05 20:12:31.385969291 +0000 UTC" Jul 17 17:31:20.287226 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:20.287224 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14330h41m11.098748078s" Jul 17 17:31:20.302476 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:20.302450 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-43.ec2.internal\" not found" Jul 17 17:31:20.402945 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:20.402902 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-43.ec2.internal\" not found" Jul 17 17:31:20.503979 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:20.503943 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-43.ec2.internal\" not found" Jul 17 17:31:20.588001 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:20.587967 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e1e8a8a145842c049c2ce22b537c78.slice/crio-3e1da569192ea6e00dc9b091b4f1e0a99667e88921721b124aa0bcd34cd2ddb6 WatchSource:0}: Error finding container 3e1da569192ea6e00dc9b091b4f1e0a99667e88921721b124aa0bcd34cd2ddb6: Status 404 returned error can't find the container with id 3e1da569192ea6e00dc9b091b4f1e0a99667e88921721b124aa0bcd34cd2ddb6 Jul 17 17:31:20.588508 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:20.588484 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c86625a603c83bec1cac3688696a497.slice/crio-1d39237e4faefbc5af7c7fae308094939b12fbc9d08f74df7d4da9d765fa56d0 WatchSource:0}: Error finding container 1d39237e4faefbc5af7c7fae308094939b12fbc9d08f74df7d4da9d765fa56d0: Status 404 returned error can't find the container with id 1d39237e4faefbc5af7c7fae308094939b12fbc9d08f74df7d4da9d765fa56d0 Jul 17 17:31:20.594828 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:20.594795 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 17:31:20.604924 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:20.604903 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-43.ec2.internal\" not found" Jul 17 17:31:20.705500 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:20.705409 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-43.ec2.internal\" not found" Jul 17 17:31:20.754791 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:20.754766 2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:31:20.806022 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:20.805986 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-43.ec2.internal\" not found" Jul 17 17:31:20.894221 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:20.894196 2553 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:31:20.967654 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:20.967571 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal" Jul 17 17:31:20.982516 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:20.982492 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 17:31:20.983643 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:20.983629 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-43.ec2.internal" Jul 17 17:31:20.992530 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:20.992512 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 17:31:21.232055 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.231985 2553 apiserver.go:52] "Watching apiserver" Jul 17 17:31:21.241179 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.241153 2553 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 17 17:31:21.241689 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.241630 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-v5sbn","openshift-network-operator/iptables-alerter-vz64j","openshift-ovn-kubernetes/ovnkube-node-flxkv","kube-system/konnectivity-agent-7vkng","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb","openshift-cluster-node-tuning-operator/tuned-52rj6","openshift-image-registry/node-ca-xx4rh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal","openshift-multus/network-metrics-daemon-fvs2m","openshift-network-diagnostics/network-check-target-xf9wz","kube-system/global-pull-secret-syncer-czb2s","kube-system/kube-apiserver-proxy-ip-10-0-136-43.ec2.internal","openshift-dns/node-resolver-7w7jr","openshift-multus/multus-additional-cni-plugins-krv7b"] Jul 17 17:31:21.244367 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.244344 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xx4rh" Jul 17 17:31:21.246674 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.246652 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:21.246815 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:21.246783 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-czb2s" podUID="2758fa5d-c686-4179-af0b-bc2b26fcf0eb" Jul 17 17:31:21.247250 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.247229 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 17 17:31:21.247359 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.247335 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 17 17:31:21.247626 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.247602 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 17 17:31:21.248059 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.248040 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4f94m\"" Jul 17 17:31:21.250009 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.249795 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.252568 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.252441 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7vkng" Jul 17 17:31:21.254859 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.254837 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.255750 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.255713 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7g7l7\"" Jul 17 17:31:21.255879 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.255798 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 17 17:31:21.255879 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.255798 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 17 17:31:21.256296 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.256091 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 17 17:31:21.256296 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.256196 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cllkj\"" Jul 17 17:31:21.256649 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.256627 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 17 17:31:21.258280 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.257797 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 17 17:31:21.258280 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.258038 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.258280 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.258063 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 17 17:31:21.258280 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.258072 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 17 17:31:21.258491 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.258334 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 17 17:31:21.261450 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.260550 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 17 17:31:21.261450 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.260865 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-df4xv\"" Jul 17 17:31:21.261450 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.260983 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 17 17:31:21.261450 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.261233 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 17 17:31:21.261450 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.261246 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 17 17:31:21.261450 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.261298 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 17 17:31:21.262552 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.262246 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:21.262552 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:21.262353 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvs2m" podUID="8a107e73-1e71-4f81-9751-d19706139fad" Jul 17 17:31:21.265168 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.265034 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7w7jr" Jul 17 17:31:21.267334 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.267315 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vz64j" Jul 17 17:31:21.267913 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.267876 2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 17 17:31:21.270427 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.270409 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:21.270613 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:21.270589 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf9wz" podUID="d59d2582-9624-426a-a770-75a633dd1f0f" Jul 17 17:31:21.271959 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.271938 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 17 17:31:21.271959 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.271955 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 17 17:31:21.272106 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.272040 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 17 17:31:21.272162 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.272127 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 17 17:31:21.272162 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.272149 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xw4dz\"" Jul 17 17:31:21.272257 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.272185 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dsnk7\"" Jul 17 17:31:21.272420 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.272401 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 17 17:31:21.272518 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.272500 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.272573 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.272540 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 17 17:31:21.272682 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.272662 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 17 17:31:21.275103 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.275080 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.281761 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.280813 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-kubelet-config\") pod \"global-pull-secret-syncer-czb2s\" (UID: \"2758fa5d-c686-4179-af0b-bc2b26fcf0eb\") " pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:21.281761 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.280848 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-node-log\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.281761 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.280891 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-cni-netd\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.281761 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.280915 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-cnibin\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.281761 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.280939 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0f3eafc-0dd5-40d7-b50a-466253eadc33-iptables-alerter-script\") pod \"iptables-alerter-vz64j\" (UID: \"a0f3eafc-0dd5-40d7-b50a-466253eadc33\") " pod="openshift-network-operator/iptables-alerter-vz64j" Jul 17 17:31:21.281761 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.280964 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0f3eafc-0dd5-40d7-b50a-466253eadc33-host-slash\") pod \"iptables-alerter-vz64j\" (UID: \"a0f3eafc-0dd5-40d7-b50a-466253eadc33\") " pod="openshift-network-operator/iptables-alerter-vz64j" Jul 17 17:31:21.281761 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281002 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/04147e2d-052c-4d96-9693-802066c81adc-ovnkube-config\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.281761 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281029 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5177b409-481a-4b8f-ae82-c1e28eebd561-konnectivity-ca\") pod \"konnectivity-agent-7vkng\" (UID: \"5177b409-481a-4b8f-ae82-c1e28eebd561\") " pod="kube-system/konnectivity-agent-7vkng" Jul 17 17:31:21.281761 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281054 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-os-release\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.281761 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281081 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-systemd\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.281761 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281105 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-run\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.281761 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281129 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6ggp\" (UniqueName: \"kubernetes.io/projected/a0f3eafc-0dd5-40d7-b50a-466253eadc33-kube-api-access-j6ggp\") pod \"iptables-alerter-vz64j\" (UID: \"a0f3eafc-0dd5-40d7-b50a-466253eadc33\") " pod="openshift-network-operator/iptables-alerter-vz64j" Jul 17 17:31:21.281761 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281153 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-slash\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.281761 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281176 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-etc-openvswitch\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.282365 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281212 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-run-ovn-kubernetes\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.282365 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281236 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-host\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.282365 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281263 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bd6be5c2-6170-4336-9626-0e499a629268-hosts-file\") pod \"node-resolver-7w7jr\" (UID: \"bd6be5c2-6170-4336-9626-0e499a629268\") " pod="openshift-dns/node-resolver-7w7jr" Jul 17 17:31:21.282365 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281302 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret\") pod \"global-pull-secret-syncer-czb2s\" (UID: \"2758fa5d-c686-4179-af0b-bc2b26fcf0eb\") " pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:21.282365 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281326 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-sysctl-conf\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.282365 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281361 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-log-socket\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.282365 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281393 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.282365 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281418 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-tuned\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.282365 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281452 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-run-systemd\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.282365 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281471 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cnm8h\"" Jul 17 17:31:21.282365 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281474 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-sys\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.282365 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281540 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-systemd-units\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.282365 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281582 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f7f37cc-005b-44d5-a728-90cdb84b5d22-host\") pod \"node-ca-xx4rh\" (UID: \"8f7f37cc-005b-44d5-a728-90cdb84b5d22\") " pod="openshift-image-registry/node-ca-xx4rh" Jul 17 17:31:21.282365 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281641 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd6be5c2-6170-4336-9626-0e499a629268-tmp-dir\") pod \"node-resolver-7w7jr\" (UID: \"bd6be5c2-6170-4336-9626-0e499a629268\") " pod="openshift-dns/node-resolver-7w7jr" Jul 17 17:31:21.282840 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281665 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-dbus\") pod \"global-pull-secret-syncer-czb2s\" (UID: \"2758fa5d-c686-4179-af0b-bc2b26fcf0eb\") " pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:21.282840 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281695 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.282840 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281720 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-sysctl-d\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.282840 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281777 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4ppv\" (UniqueName: \"kubernetes.io/projected/04147e2d-052c-4d96-9693-802066c81adc-kube-api-access-p4ppv\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.282840 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281823 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nx9z\" (UniqueName: \"kubernetes.io/projected/9b365f43-0235-462f-a7ba-e38b1936bca4-kube-api-access-6nx9z\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.282840 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281877 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-lib-modules\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.282840 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281907 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw2sg\" (UniqueName: \"kubernetes.io/projected/bd6be5c2-6170-4336-9626-0e499a629268-kube-api-access-sw2sg\") pod \"node-resolver-7w7jr\" (UID: \"bd6be5c2-6170-4336-9626-0e499a629268\") " pod="openshift-dns/node-resolver-7w7jr" Jul 17 17:31:21.282840 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281945 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psv6p\" (UniqueName: \"kubernetes.io/projected/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-kube-api-access-psv6p\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.282840 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281974 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f7f37cc-005b-44d5-a728-90cdb84b5d22-serviceca\") pod \"node-ca-xx4rh\" (UID: \"8f7f37cc-005b-44d5-a728-90cdb84b5d22\") " pod="openshift-image-registry/node-ca-xx4rh" Jul 17 17:31:21.282840 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.281998 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-run-netns\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.282840 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282024 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/04147e2d-052c-4d96-9693-802066c81adc-ovnkube-script-lib\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.282840 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282049 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b365f43-0235-462f-a7ba-e38b1936bca4-tmp\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.282840 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282096 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-run-ovn\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.282840 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282134 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-modprobe-d\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.283507 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282160 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-var-lib-kubelet\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.283507 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282184 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/04147e2d-052c-4d96-9693-802066c81adc-env-overrides\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.283507 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282208 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5177b409-481a-4b8f-ae82-c1e28eebd561-agent-certs\") pod \"konnectivity-agent-7vkng\" (UID: \"5177b409-481a-4b8f-ae82-c1e28eebd561\") " pod="kube-system/konnectivity-agent-7vkng" Jul 17 17:31:21.283507 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282231 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-sysconfig\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.283507 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282250 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p8n2\" (UniqueName: \"kubernetes.io/projected/8a107e73-1e71-4f81-9751-d19706139fad-kube-api-access-5p8n2\") pod \"network-metrics-daemon-fvs2m\" (UID: \"8a107e73-1e71-4f81-9751-d19706139fad\") " pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:21.283507 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282269 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q7ds\" (UniqueName: \"kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds\") pod \"network-check-target-xf9wz\" (UID: \"d59d2582-9624-426a-a770-75a633dd1f0f\") " pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:21.283507 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282284 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-run-openvswitch\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.283507 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282316 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.283507 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282366 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs\") pod \"network-metrics-daemon-fvs2m\" (UID: \"8a107e73-1e71-4f81-9751-d19706139fad\") " pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:21.283507 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282401 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mnjp\" (UniqueName: \"kubernetes.io/projected/8f7f37cc-005b-44d5-a728-90cdb84b5d22-kube-api-access-5mnjp\") pod \"node-ca-xx4rh\" (UID: \"8f7f37cc-005b-44d5-a728-90cdb84b5d22\") " pod="openshift-image-registry/node-ca-xx4rh" Jul 17 17:31:21.283507 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282432 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-cni-bin\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.283507 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282461 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/04147e2d-052c-4d96-9693-802066c81adc-ovn-node-metrics-cert\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.283507 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282496 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-system-cni-dir\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.284872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282522 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.284872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282562 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-kubelet\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.284872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282588 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-var-lib-openvswitch\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.284872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282627 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.284872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.282653 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-kubernetes\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.284872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.284763 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 17 17:31:21.285195 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.284909 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-r9rvw\"" Jul 17 17:31:21.285195 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.285085 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4pt9v\"" Jul 17 17:31:21.285195 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.285171 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 17 17:31:21.285339 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.285302 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 17 17:31:21.285697 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.285543 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 17 17:31:21.289911 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.289847 2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 17:31:21.314735 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.314710 2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2spvs" Jul 17 17:31:21.321388 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.321081 2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2spvs" Jul 17 17:31:21.335025 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.334856 2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:31:21.368988 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.368951 2553 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 17 17:31:21.383284 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383240 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-var-lib-kubelet\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.383428 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383285 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-host-run-multus-certs\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.383428 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383318 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/04147e2d-052c-4d96-9693-802066c81adc-env-overrides\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.383428 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383347 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5177b409-481a-4b8f-ae82-c1e28eebd561-agent-certs\") pod \"konnectivity-agent-7vkng\" (UID: \"5177b409-481a-4b8f-ae82-c1e28eebd561\") " pod="kube-system/konnectivity-agent-7vkng" Jul 17 17:31:21.383428 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383372 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-sysconfig\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.383428 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383397 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p8n2\" (UniqueName: \"kubernetes.io/projected/8a107e73-1e71-4f81-9751-d19706139fad-kube-api-access-5p8n2\") pod \"network-metrics-daemon-fvs2m\" (UID: \"8a107e73-1e71-4f81-9751-d19706139fad\") " pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:21.383428 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383423 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-host-run-k8s-cni-cncf-io\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.383763 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383451 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlzmp\" (UniqueName: \"kubernetes.io/projected/169ea2a6-448e-4662-ac13-7b881a1566ef-kube-api-access-zlzmp\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.383763 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383478 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q7ds\" (UniqueName: \"kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds\") pod \"network-check-target-xf9wz\" (UID: \"d59d2582-9624-426a-a770-75a633dd1f0f\") " pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:21.383763 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383504 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-run-openvswitch\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.383763 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383529 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.383763 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383556 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs\") pod \"network-metrics-daemon-fvs2m\" (UID: \"8a107e73-1e71-4f81-9751-d19706139fad\") " pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:21.383763 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383587 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-system-cni-dir\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.383763 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383623 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-host-var-lib-cni-bin\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.383763 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383653 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mnjp\" (UniqueName: \"kubernetes.io/projected/8f7f37cc-005b-44d5-a728-90cdb84b5d22-kube-api-access-5mnjp\") pod \"node-ca-xx4rh\" (UID: \"8f7f37cc-005b-44d5-a728-90cdb84b5d22\") " pod="openshift-image-registry/node-ca-xx4rh" Jul 17 17:31:21.383763 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383682 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-cni-bin\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.383763 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383734 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/04147e2d-052c-4d96-9693-802066c81adc-ovn-node-metrics-cert\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.383763 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383753 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-cni-bin\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.384330 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383780 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-system-cni-dir\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.384330 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383811 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.384330 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383849 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-host-var-lib-cni-multus\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.384330 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383942 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-host-var-lib-kubelet\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.384330 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.383972 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-kubelet\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.384330 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.384000 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-var-lib-openvswitch\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.384330 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.384050 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.384330 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.384086 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-run-openvswitch\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.384723 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.384424 2553 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 17 17:31:21.384723 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.384689 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.384827 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.384715 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.384827 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.384807 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-var-lib-kubelet\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.384827 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:21.384821 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:31:21.384997 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:21.384903 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs podName:8a107e73-1e71-4f81-9751-d19706139fad nodeName:}" failed. No retries permitted until 2026-07-17 17:31:21.884881444 +0000 UTC m=+3.003709709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs") pod "network-metrics-daemon-fvs2m" (UID: "8a107e73-1e71-4f81-9751-d19706139fad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:31:21.384997 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.384931 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-var-lib-openvswitch\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.385630 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385136 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-sysconfig\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.385630 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385246 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-system-cni-dir\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.385630 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385414 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.385630 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385479 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-kubelet\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.385630 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385514 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-kubernetes\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.385630 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385561 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-host-run-netns\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.385630 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385586 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-etc-kubernetes\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.385630 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385612 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-kubelet-config\") pod \"global-pull-secret-syncer-czb2s\" (UID: \"2758fa5d-c686-4179-af0b-bc2b26fcf0eb\") " pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:21.386127 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385637 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-node-log\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.386127 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385643 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/04147e2d-052c-4d96-9693-802066c81adc-env-overrides\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.386127 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385660 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-cni-netd\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.386127 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385695 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-cni-netd\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.386127 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385699 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-cnibin\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.386127 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385730 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0f3eafc-0dd5-40d7-b50a-466253eadc33-iptables-alerter-script\") pod \"iptables-alerter-vz64j\" (UID: \"a0f3eafc-0dd5-40d7-b50a-466253eadc33\") " pod="openshift-network-operator/iptables-alerter-vz64j" Jul 17 17:31:21.386127 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385746 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-kubelet-config\") pod \"global-pull-secret-syncer-czb2s\" (UID: \"2758fa5d-c686-4179-af0b-bc2b26fcf0eb\") " pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:21.386127 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385755 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0f3eafc-0dd5-40d7-b50a-466253eadc33-host-slash\") pod \"iptables-alerter-vz64j\" (UID: \"a0f3eafc-0dd5-40d7-b50a-466253eadc33\") " pod="openshift-network-operator/iptables-alerter-vz64j" Jul 17 17:31:21.386127 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385773 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-kubernetes\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.386127 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385781 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/04147e2d-052c-4d96-9693-802066c81adc-ovnkube-config\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.386127 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385789 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-node-log\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.386127 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385818 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5177b409-481a-4b8f-ae82-c1e28eebd561-konnectivity-ca\") pod \"konnectivity-agent-7vkng\" (UID: \"5177b409-481a-4b8f-ae82-c1e28eebd561\") " pod="kube-system/konnectivity-agent-7vkng" Jul 17 17:31:21.386127 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385847 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-os-release\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.386127 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385874 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-systemd\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.386803 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385881 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0f3eafc-0dd5-40d7-b50a-466253eadc33-host-slash\") pod \"iptables-alerter-vz64j\" (UID: \"a0f3eafc-0dd5-40d7-b50a-466253eadc33\") " pod="openshift-network-operator/iptables-alerter-vz64j" Jul 17 17:31:21.386803 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385900 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-run\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.386803 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385947 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlrjv\" (UniqueName: \"kubernetes.io/projected/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-kube-api-access-hlrjv\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.386803 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385968 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-os-release\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.386803 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385842 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-cnibin\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.386803 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386050 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-systemd\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.386803 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386174 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-run\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.386803 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.385976 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6ggp\" (UniqueName: \"kubernetes.io/projected/a0f3eafc-0dd5-40d7-b50a-466253eadc33-kube-api-access-j6ggp\") pod \"iptables-alerter-vz64j\" (UID: \"a0f3eafc-0dd5-40d7-b50a-466253eadc33\") " pod="openshift-network-operator/iptables-alerter-vz64j" Jul 17 17:31:21.386803 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386221 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-slash\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.386803 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386247 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-etc-openvswitch\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.386803 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386254 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/04147e2d-052c-4d96-9693-802066c81adc-ovnkube-config\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.386803 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386275 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-run-ovn-kubernetes\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.386803 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386301 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-host\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.386803 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386322 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-slash\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.386803 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386329 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/169ea2a6-448e-4662-ac13-7b881a1566ef-device-dir\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.387529 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386361 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bd6be5c2-6170-4336-9626-0e499a629268-hosts-file\") pod \"node-resolver-7w7jr\" (UID: \"bd6be5c2-6170-4336-9626-0e499a629268\") " pod="openshift-dns/node-resolver-7w7jr" Jul 17 17:31:21.387529 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386375 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-run-ovn-kubernetes\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.387529 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386387 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret\") pod \"global-pull-secret-syncer-czb2s\" (UID: \"2758fa5d-c686-4179-af0b-bc2b26fcf0eb\") " pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:21.387529 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386414 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-sysctl-conf\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.387529 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386422 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-etc-openvswitch\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.387529 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386437 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5177b409-481a-4b8f-ae82-c1e28eebd561-konnectivity-ca\") pod \"konnectivity-agent-7vkng\" (UID: \"5177b409-481a-4b8f-ae82-c1e28eebd561\") " pod="kube-system/konnectivity-agent-7vkng" Jul 17 17:31:21.387529 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386443 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-cni-binary-copy\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.387529 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386476 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bd6be5c2-6170-4336-9626-0e499a629268-hosts-file\") pod \"node-resolver-7w7jr\" (UID: \"bd6be5c2-6170-4336-9626-0e499a629268\") " pod="openshift-dns/node-resolver-7w7jr" Jul 17 17:31:21.387529 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386504 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-multus-conf-dir\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.387529 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386526 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-host\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.387529 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386533 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-log-socket\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.387529 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386561 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.387529 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386593 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-sysctl-conf\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.387529 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386593 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-log-socket\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.387529 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386592 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-tuned\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.388271 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:21.386678 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:31:21.388271 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:21.386723 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret podName:2758fa5d-c686-4179-af0b-bc2b26fcf0eb nodeName:}" failed. No retries permitted until 2026-07-17 17:31:21.886708157 +0000 UTC m=+3.005536422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret") pod "global-pull-secret-syncer-czb2s" (UID: "2758fa5d-c686-4179-af0b-bc2b26fcf0eb") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:31:21.388271 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386752 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-multus-daemon-config\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.388271 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386778 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/169ea2a6-448e-4662-ac13-7b881a1566ef-sys-fs\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.388271 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386805 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-run-systemd\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.388271 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386830 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-sys\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.388271 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386830 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0f3eafc-0dd5-40d7-b50a-466253eadc33-iptables-alerter-script\") pod \"iptables-alerter-vz64j\" (UID: \"a0f3eafc-0dd5-40d7-b50a-466253eadc33\") " pod="openshift-network-operator/iptables-alerter-vz64j" Jul 17 17:31:21.388271 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386855 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-multus-cni-dir\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.388271 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386951 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-sys\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.388271 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.386989 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/169ea2a6-448e-4662-ac13-7b881a1566ef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.388271 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387025 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/169ea2a6-448e-4662-ac13-7b881a1566ef-socket-dir\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.388271 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387058 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-systemd-units\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.388271 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387072 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.388271 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387087 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f7f37cc-005b-44d5-a728-90cdb84b5d22-host\") pod \"node-ca-xx4rh\" (UID: \"8f7f37cc-005b-44d5-a728-90cdb84b5d22\") " pod="openshift-image-registry/node-ca-xx4rh" Jul 17 17:31:21.388977 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387127 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd6be5c2-6170-4336-9626-0e499a629268-tmp-dir\") pod \"node-resolver-7w7jr\" (UID: \"bd6be5c2-6170-4336-9626-0e499a629268\") " pod="openshift-dns/node-resolver-7w7jr" Jul 17 17:31:21.388977 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387128 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f7f37cc-005b-44d5-a728-90cdb84b5d22-host\") pod \"node-ca-xx4rh\" (UID: \"8f7f37cc-005b-44d5-a728-90cdb84b5d22\") " pod="openshift-image-registry/node-ca-xx4rh" Jul 17 17:31:21.388977 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387159 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-dbus\") pod \"global-pull-secret-syncer-czb2s\" (UID: \"2758fa5d-c686-4179-af0b-bc2b26fcf0eb\") " pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:21.388977 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387225 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.388977 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387253 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-sysctl-d\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.388977 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387279 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-hostroot\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.388977 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387279 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-dbus\") pod \"global-pull-secret-syncer-czb2s\" (UID: \"2758fa5d-c686-4179-af0b-bc2b26fcf0eb\") " pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:21.388977 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387321 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4ppv\" (UniqueName: \"kubernetes.io/projected/04147e2d-052c-4d96-9693-802066c81adc-kube-api-access-p4ppv\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.388977 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387324 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.388977 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387347 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nx9z\" (UniqueName: \"kubernetes.io/projected/9b365f43-0235-462f-a7ba-e38b1936bca4-kube-api-access-6nx9z\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.388977 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387370 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-run-systemd\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.388977 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387409 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-multus-socket-dir-parent\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.388977 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387431 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd6be5c2-6170-4336-9626-0e499a629268-tmp-dir\") pod \"node-resolver-7w7jr\" (UID: \"bd6be5c2-6170-4336-9626-0e499a629268\") " pod="openshift-dns/node-resolver-7w7jr" Jul 17 17:31:21.388977 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387438 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/169ea2a6-448e-4662-ac13-7b881a1566ef-etc-selinux\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.389636 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387468 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-lib-modules\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.389636 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387499 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw2sg\" (UniqueName: \"kubernetes.io/projected/bd6be5c2-6170-4336-9626-0e499a629268-kube-api-access-sw2sg\") pod \"node-resolver-7w7jr\" (UID: \"bd6be5c2-6170-4336-9626-0e499a629268\") " pod="openshift-dns/node-resolver-7w7jr" Jul 17 17:31:21.389636 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387551 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psv6p\" (UniqueName: \"kubernetes.io/projected/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-kube-api-access-psv6p\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.389636 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387567 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-sysctl-d\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.389636 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387580 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-os-release\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.389636 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387608 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f7f37cc-005b-44d5-a728-90cdb84b5d22-serviceca\") pod \"node-ca-xx4rh\" (UID: \"8f7f37cc-005b-44d5-a728-90cdb84b5d22\") " pod="openshift-image-registry/node-ca-xx4rh" Jul 17 17:31:21.389636 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387654 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-run-netns\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.389636 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387770 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/04147e2d-052c-4d96-9693-802066c81adc-ovnkube-script-lib\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.389636 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387808 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-host-run-netns\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.389636 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387816 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b365f43-0235-462f-a7ba-e38b1936bca4-tmp\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.389636 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387869 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-cnibin\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.389636 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387899 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/169ea2a6-448e-4662-ac13-7b881a1566ef-registration-dir\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.389636 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387922 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-lib-modules\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.389636 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387950 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-run-ovn\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.389636 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.387980 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-modprobe-d\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.390292 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.388111 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-modprobe-d\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.390292 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.388166 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-run-ovn\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.390292 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.388207 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/04147e2d-052c-4d96-9693-802066c81adc-systemd-units\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.390292 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.388670 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f7f37cc-005b-44d5-a728-90cdb84b5d22-serviceca\") pod \"node-ca-xx4rh\" (UID: \"8f7f37cc-005b-44d5-a728-90cdb84b5d22\") " pod="openshift-image-registry/node-ca-xx4rh" Jul 17 17:31:21.390292 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.388813 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/04147e2d-052c-4d96-9693-802066c81adc-ovnkube-script-lib\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.390292 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:21.389770 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:31:21.390292 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:21.389792 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:31:21.390292 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:21.389806 2553 projected.go:196] Error preparing data for projected volume kube-api-access-6q7ds for pod openshift-network-diagnostics/network-check-target-xf9wz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:31:21.390292 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:21.389870 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds podName:d59d2582-9624-426a-a770-75a633dd1f0f nodeName:}" failed. No retries permitted until 2026-07-17 17:31:21.889855163 +0000 UTC m=+3.008683426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6q7ds" (UniqueName: "kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds") pod "network-check-target-xf9wz" (UID: "d59d2582-9624-426a-a770-75a633dd1f0f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:31:21.390696 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.390593 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9b365f43-0235-462f-a7ba-e38b1936bca4-etc-tuned\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.390858 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.390835 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5177b409-481a-4b8f-ae82-c1e28eebd561-agent-certs\") pod \"konnectivity-agent-7vkng\" (UID: \"5177b409-481a-4b8f-ae82-c1e28eebd561\") " pod="kube-system/konnectivity-agent-7vkng" Jul 17 17:31:21.391138 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.391096 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/04147e2d-052c-4d96-9693-802066c81adc-ovn-node-metrics-cert\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.392710 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.392688 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b365f43-0235-462f-a7ba-e38b1936bca4-tmp\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.396812 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.396789 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mnjp\" (UniqueName: \"kubernetes.io/projected/8f7f37cc-005b-44d5-a728-90cdb84b5d22-kube-api-access-5mnjp\") pod \"node-ca-xx4rh\" (UID: \"8f7f37cc-005b-44d5-a728-90cdb84b5d22\") " pod="openshift-image-registry/node-ca-xx4rh" Jul 17 17:31:21.401342 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.401290 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4ppv\" (UniqueName: \"kubernetes.io/projected/04147e2d-052c-4d96-9693-802066c81adc-kube-api-access-p4ppv\") pod \"ovnkube-node-flxkv\" (UID: \"04147e2d-052c-4d96-9693-802066c81adc\") " pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.402274 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.402202 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psv6p\" (UniqueName: \"kubernetes.io/projected/2fb88f09-39c9-4e4b-91c4-7c3d21b071ba-kube-api-access-psv6p\") pod \"multus-additional-cni-plugins-krv7b\" (UID: \"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba\") " pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.402448 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.402401 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6ggp\" (UniqueName: \"kubernetes.io/projected/a0f3eafc-0dd5-40d7-b50a-466253eadc33-kube-api-access-j6ggp\") pod \"iptables-alerter-vz64j\" (UID: \"a0f3eafc-0dd5-40d7-b50a-466253eadc33\") " pod="openshift-network-operator/iptables-alerter-vz64j" Jul 17 17:31:21.402703 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.402624 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p8n2\" (UniqueName: \"kubernetes.io/projected/8a107e73-1e71-4f81-9751-d19706139fad-kube-api-access-5p8n2\") pod \"network-metrics-daemon-fvs2m\" (UID: \"8a107e73-1e71-4f81-9751-d19706139fad\") " pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:21.404952 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.404286 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nx9z\" (UniqueName: \"kubernetes.io/projected/9b365f43-0235-462f-a7ba-e38b1936bca4-kube-api-access-6nx9z\") pod \"tuned-52rj6\" (UID: \"9b365f43-0235-462f-a7ba-e38b1936bca4\") " pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.407857 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.407812 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw2sg\" (UniqueName: \"kubernetes.io/projected/bd6be5c2-6170-4336-9626-0e499a629268-kube-api-access-sw2sg\") pod \"node-resolver-7w7jr\" (UID: \"bd6be5c2-6170-4336-9626-0e499a629268\") " pod="openshift-dns/node-resolver-7w7jr" Jul 17 17:31:21.463952 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.463876 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-43.ec2.internal" event={"ID":"33e1e8a8a145842c049c2ce22b537c78","Type":"ContainerStarted","Data":"3e1da569192ea6e00dc9b091b4f1e0a99667e88921721b124aa0bcd34cd2ddb6"} Jul 17 17:31:21.465752 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.465718 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal" event={"ID":"0c86625a603c83bec1cac3688696a497","Type":"ContainerStarted","Data":"1d39237e4faefbc5af7c7fae308094939b12fbc9d08f74df7d4da9d765fa56d0"} Jul 17 17:31:21.489220 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489094 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-host-run-k8s-cni-cncf-io\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.489220 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489138 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlzmp\" (UniqueName: \"kubernetes.io/projected/169ea2a6-448e-4662-ac13-7b881a1566ef-kube-api-access-zlzmp\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.489540 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489215 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-host-run-k8s-cni-cncf-io\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.489540 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489283 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-system-cni-dir\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.489540 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489314 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-host-var-lib-cni-bin\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.489540 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489347 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-host-var-lib-cni-multus\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.489540 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489374 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-host-var-lib-kubelet\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.489540 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489379 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-system-cni-dir\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.489540 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489403 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-host-run-netns\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.489540 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489429 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-etc-kubernetes\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.489540 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489463 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-host-var-lib-kubelet\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.489540 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489429 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-host-var-lib-cni-multus\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.489540 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489464 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-host-run-netns\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.489540 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489467 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlrjv\" (UniqueName: \"kubernetes.io/projected/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-kube-api-access-hlrjv\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.489540 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489527 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/169ea2a6-448e-4662-ac13-7b881a1566ef-device-dir\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.490251 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489578 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-cni-binary-copy\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.490251 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489604 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-multus-conf-dir\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.490251 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489620 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/169ea2a6-448e-4662-ac13-7b881a1566ef-device-dir\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.490251 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489633 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-multus-daemon-config\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.490251 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489495 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-etc-kubernetes\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.490251 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489658 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/169ea2a6-448e-4662-ac13-7b881a1566ef-sys-fs\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.490251 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489428 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-host-var-lib-cni-bin\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.490251 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489686 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-multus-cni-dir\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.490251 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489713 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/169ea2a6-448e-4662-ac13-7b881a1566ef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.490251 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489739 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/169ea2a6-448e-4662-ac13-7b881a1566ef-socket-dir\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.490251 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489771 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-hostroot\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.490251 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489800 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-multus-socket-dir-parent\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.490251 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489825 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/169ea2a6-448e-4662-ac13-7b881a1566ef-etc-selinux\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.490251 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489851 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-os-release\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.490251 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489878 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-cnibin\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.491005 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489902 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/169ea2a6-448e-4662-ac13-7b881a1566ef-registration-dir\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.491005 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.489948 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-host-run-multus-certs\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.491005 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.490000 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-multus-conf-dir\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.491005 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.490044 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-host-run-multus-certs\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.491005 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.490061 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-hostroot\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.491005 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.490110 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/169ea2a6-448e-4662-ac13-7b881a1566ef-sys-fs\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.491005 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.490182 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-multus-socket-dir-parent\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.491005 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.490187 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/169ea2a6-448e-4662-ac13-7b881a1566ef-etc-selinux\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.491005 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.490243 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-multus-cni-dir\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.491005 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.490260 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-os-release\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.491005 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.490310 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/169ea2a6-448e-4662-ac13-7b881a1566ef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.491005 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.490320 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/169ea2a6-448e-4662-ac13-7b881a1566ef-registration-dir\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.491005 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.490321 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-cni-binary-copy\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.491681 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.491321 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-cnibin\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.491681 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.491416 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/169ea2a6-448e-4662-ac13-7b881a1566ef-socket-dir\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.491681 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.491660 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-multus-daemon-config\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.502558 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.502499 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlrjv\" (UniqueName: \"kubernetes.io/projected/b54d03ca-d903-45ac-b3e5-c4c70d5574cf-kube-api-access-hlrjv\") pod \"multus-v5sbn\" (UID: \"b54d03ca-d903-45ac-b3e5-c4c70d5574cf\") " pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.503185 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.503130 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlzmp\" (UniqueName: \"kubernetes.io/projected/169ea2a6-448e-4662-ac13-7b881a1566ef-kube-api-access-zlzmp\") pod \"aws-ebs-csi-driver-node-hmzkb\" (UID: \"169ea2a6-448e-4662-ac13-7b881a1566ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.558191 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.557977 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xx4rh" Jul 17 17:31:21.567769 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.567495 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:21.582605 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.581853 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-52rj6" Jul 17 17:31:21.589438 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.589383 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7vkng" Jul 17 17:31:21.596847 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:21.595498 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b365f43_0235_462f_a7ba_e38b1936bca4.slice/crio-8d48b8bd2bb77dd5a964cb62cc42e0cf84bfa5584c4ed9e23086b198b876df08 WatchSource:0}: Error finding container 8d48b8bd2bb77dd5a964cb62cc42e0cf84bfa5584c4ed9e23086b198b876df08: Status 404 returned error can't find the container with id 8d48b8bd2bb77dd5a964cb62cc42e0cf84bfa5584c4ed9e23086b198b876df08 Jul 17 17:31:21.597050 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.596984 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-krv7b" Jul 17 17:31:21.602594 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:21.602569 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5177b409_481a_4b8f_ae82_c1e28eebd561.slice/crio-9c9ce639727d2c2f159d27a3d6c8e72ff4372f77481665753276cf2a0471fd6a WatchSource:0}: Error finding container 9c9ce639727d2c2f159d27a3d6c8e72ff4372f77481665753276cf2a0471fd6a: Status 404 returned error can't find the container with id 9c9ce639727d2c2f159d27a3d6c8e72ff4372f77481665753276cf2a0471fd6a Jul 17 17:31:21.605747 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.605727 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7w7jr" Jul 17 17:31:21.610430 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:21.610392 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb88f09_39c9_4e4b_91c4_7c3d21b071ba.slice/crio-dfbb3d22902d2616c5b5c3393efd4a34556fc523e285ea9538027dfef1b81d77 WatchSource:0}: Error finding container dfbb3d22902d2616c5b5c3393efd4a34556fc523e285ea9538027dfef1b81d77: Status 404 returned error can't find the container with id dfbb3d22902d2616c5b5c3393efd4a34556fc523e285ea9538027dfef1b81d77 Jul 17 17:31:21.613637 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.613621 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vz64j" Jul 17 17:31:21.616893 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:21.616823 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6be5c2_6170_4336_9626_0e499a629268.slice/crio-1494dc30ca42255cf548b36aa2619a7795f3ab9226a64991d369c672348a7425 WatchSource:0}: Error finding container 1494dc30ca42255cf548b36aa2619a7795f3ab9226a64991d369c672348a7425: Status 404 returned error can't find the container with id 1494dc30ca42255cf548b36aa2619a7795f3ab9226a64991d369c672348a7425 Jul 17 17:31:21.624885 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:21.624765 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f3eafc_0dd5_40d7_b50a_466253eadc33.slice/crio-44bf1e3a95f8474c2248f17d63db7ad610a68792d525eca7f379ff2c0bb78f5a WatchSource:0}: Error finding container 44bf1e3a95f8474c2248f17d63db7ad610a68792d525eca7f379ff2c0bb78f5a: Status 404 returned error can't find the container with id 44bf1e3a95f8474c2248f17d63db7ad610a68792d525eca7f379ff2c0bb78f5a Jul 17 17:31:21.624885 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.624833 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v5sbn" Jul 17 17:31:21.630044 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.630022 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" Jul 17 17:31:21.641083 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:21.641059 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54d03ca_d903_45ac_b3e5_c4c70d5574cf.slice/crio-e9c478d8586844610bed6d5fbc7db34438600526ad636073e1ac762669a38d19 WatchSource:0}: Error finding container e9c478d8586844610bed6d5fbc7db34438600526ad636073e1ac762669a38d19: Status 404 returned error can't find the container with id e9c478d8586844610bed6d5fbc7db34438600526ad636073e1ac762669a38d19 Jul 17 17:31:21.647266 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:21.647139 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169ea2a6_448e_4662_ac13_7b881a1566ef.slice/crio-fb052459f611fd0d2940cbb9f4345ae05d317076ac3a43a80bec8487ae470492 WatchSource:0}: Error finding container fb052459f611fd0d2940cbb9f4345ae05d317076ac3a43a80bec8487ae470492: Status 404 returned error can't find the container with id fb052459f611fd0d2940cbb9f4345ae05d317076ac3a43a80bec8487ae470492 Jul 17 17:31:21.893183 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.893100 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q7ds\" (UniqueName: \"kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds\") pod \"network-check-target-xf9wz\" (UID: \"d59d2582-9624-426a-a770-75a633dd1f0f\") " pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:21.893183 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.893139 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs\") pod \"network-metrics-daemon-fvs2m\" (UID: \"8a107e73-1e71-4f81-9751-d19706139fad\") " pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:21.893183 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:21.893174 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret\") pod \"global-pull-secret-syncer-czb2s\" (UID: \"2758fa5d-c686-4179-af0b-bc2b26fcf0eb\") " pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:21.893434 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:21.893274 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:31:21.893434 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:21.893282 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:31:21.893434 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:21.893303 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:31:21.893434 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:21.893330 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:31:21.893434 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:21.893343 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret podName:2758fa5d-c686-4179-af0b-bc2b26fcf0eb nodeName:}" failed. No retries permitted until 2026-07-17 17:31:22.893321425 +0000 UTC m=+4.012149696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret") pod "global-pull-secret-syncer-czb2s" (UID: "2758fa5d-c686-4179-af0b-bc2b26fcf0eb") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:31:21.893434 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:21.893363 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs podName:8a107e73-1e71-4f81-9751-d19706139fad nodeName:}" failed. No retries permitted until 2026-07-17 17:31:22.893353576 +0000 UTC m=+4.012181841 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs") pod "network-metrics-daemon-fvs2m" (UID: "8a107e73-1e71-4f81-9751-d19706139fad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:31:21.893434 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:21.893344 2553 projected.go:196] Error preparing data for projected volume kube-api-access-6q7ds for pod openshift-network-diagnostics/network-check-target-xf9wz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:31:21.893779 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:21.893455 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds podName:d59d2582-9624-426a-a770-75a633dd1f0f nodeName:}" failed. No retries permitted until 2026-07-17 17:31:22.89343962 +0000 UTC m=+4.012267881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6q7ds" (UniqueName: "kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds") pod "network-check-target-xf9wz" (UID: "d59d2582-9624-426a-a770-75a633dd1f0f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:31:22.324766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:22.324625 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 17:26:21 +0000 UTC" deadline="2028-04-22 09:46:57.809703344 +0000 UTC" Jul 17 17:31:22.324766 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:22.324707 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15472h15m35.485001342s" Jul 17 17:31:22.458307 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:22.458273 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:22.458498 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:22.458427 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf9wz" podUID="d59d2582-9624-426a-a770-75a633dd1f0f" Jul 17 17:31:22.458498 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:22.458462 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:22.458618 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:22.458579 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-czb2s" podUID="2758fa5d-c686-4179-af0b-bc2b26fcf0eb" Jul 17 17:31:22.458675 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:22.458648 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:22.458750 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:22.458732 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvs2m" podUID="8a107e73-1e71-4f81-9751-d19706139fad" Jul 17 17:31:22.471520 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:22.471407 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" event={"ID":"169ea2a6-448e-4662-ac13-7b881a1566ef","Type":"ContainerStarted","Data":"fb052459f611fd0d2940cbb9f4345ae05d317076ac3a43a80bec8487ae470492"} Jul 17 17:31:22.474416 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:22.474338 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v5sbn" event={"ID":"b54d03ca-d903-45ac-b3e5-c4c70d5574cf","Type":"ContainerStarted","Data":"e9c478d8586844610bed6d5fbc7db34438600526ad636073e1ac762669a38d19"} Jul 17 17:31:22.476510 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:22.476439 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7w7jr" event={"ID":"bd6be5c2-6170-4336-9626-0e499a629268","Type":"ContainerStarted","Data":"1494dc30ca42255cf548b36aa2619a7795f3ab9226a64991d369c672348a7425"} Jul 17 17:31:22.478322 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:22.478249 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-52rj6" event={"ID":"9b365f43-0235-462f-a7ba-e38b1936bca4","Type":"ContainerStarted","Data":"8d48b8bd2bb77dd5a964cb62cc42e0cf84bfa5584c4ed9e23086b198b876df08"} Jul 17 17:31:22.479636 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:22.479575 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xx4rh" event={"ID":"8f7f37cc-005b-44d5-a728-90cdb84b5d22","Type":"ContainerStarted","Data":"de686aefd45de2e6b49febc3066b602826df27d946a625f43cef342af727ff19"} Jul 17 17:31:22.481900 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:22.481807 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vz64j" event={"ID":"a0f3eafc-0dd5-40d7-b50a-466253eadc33","Type":"ContainerStarted","Data":"44bf1e3a95f8474c2248f17d63db7ad610a68792d525eca7f379ff2c0bb78f5a"} Jul 17 17:31:22.484021 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:22.483786 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krv7b" event={"ID":"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba","Type":"ContainerStarted","Data":"dfbb3d22902d2616c5b5c3393efd4a34556fc523e285ea9538027dfef1b81d77"} Jul 17 17:31:22.485639 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:22.485573 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7vkng" event={"ID":"5177b409-481a-4b8f-ae82-c1e28eebd561","Type":"ContainerStarted","Data":"9c9ce639727d2c2f159d27a3d6c8e72ff4372f77481665753276cf2a0471fd6a"} Jul 17 17:31:22.487154 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:22.487002 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" event={"ID":"04147e2d-052c-4d96-9693-802066c81adc","Type":"ContainerStarted","Data":"e5781eb1bd95e683810f4a9b967c0279e5658561e37d908757ace45c78b36838"} Jul 17 17:31:22.904029 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:22.903675 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q7ds\" (UniqueName: \"kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds\") pod \"network-check-target-xf9wz\" (UID: \"d59d2582-9624-426a-a770-75a633dd1f0f\") " pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:22.904029 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:22.903727 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs\") pod \"network-metrics-daemon-fvs2m\" (UID: \"8a107e73-1e71-4f81-9751-d19706139fad\") " pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:22.904029 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:22.903778 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret\") pod \"global-pull-secret-syncer-czb2s\" (UID: \"2758fa5d-c686-4179-af0b-bc2b26fcf0eb\") " pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:22.904029 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:22.903893 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:31:22.904029 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:22.903885 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:31:22.904029 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:22.903894 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:31:22.904029 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:22.903955 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:31:22.904029 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:22.903963 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs podName:8a107e73-1e71-4f81-9751-d19706139fad nodeName:}" failed. No retries permitted until 2026-07-17 17:31:24.90394304 +0000 UTC m=+6.022771322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs") pod "network-metrics-daemon-fvs2m" (UID: "8a107e73-1e71-4f81-9751-d19706139fad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:31:22.904029 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:22.903968 2553 projected.go:196] Error preparing data for projected volume kube-api-access-6q7ds for pod openshift-network-diagnostics/network-check-target-xf9wz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:31:22.904029 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:22.903984 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret podName:2758fa5d-c686-4179-af0b-bc2b26fcf0eb nodeName:}" failed. No retries permitted until 2026-07-17 17:31:24.903973726 +0000 UTC m=+6.022801987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret") pod "global-pull-secret-syncer-czb2s" (UID: "2758fa5d-c686-4179-af0b-bc2b26fcf0eb") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:31:22.904029 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:22.904006 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds podName:d59d2582-9624-426a-a770-75a633dd1f0f nodeName:}" failed. No retries permitted until 2026-07-17 17:31:24.903991718 +0000 UTC m=+6.022819980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6q7ds" (UniqueName: "kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds") pod "network-check-target-xf9wz" (UID: "d59d2582-9624-426a-a770-75a633dd1f0f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:31:23.325708 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:23.325619 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 17:26:21 +0000 UTC" deadline="2028-02-14 21:48:30.027830429 +0000 UTC" Jul 17 17:31:23.325708 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:23.325660 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13852h17m6.702175376s" Jul 17 17:31:24.458694 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:24.458372 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:24.459164 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:24.458428 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:24.459164 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:24.458478 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:24.459164 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:24.458841 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf9wz" podUID="d59d2582-9624-426a-a770-75a633dd1f0f" Jul 17 17:31:24.459164 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:24.458944 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-czb2s" podUID="2758fa5d-c686-4179-af0b-bc2b26fcf0eb" Jul 17 17:31:24.459164 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:24.459023 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvs2m" podUID="8a107e73-1e71-4f81-9751-d19706139fad" Jul 17 17:31:24.496162 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:24.496124 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal" event={"ID":"0c86625a603c83bec1cac3688696a497","Type":"ContainerStarted","Data":"2d8232e9fee7982bd829941eca2c067b1d331ef94a5471aaa0471f436e0651b3"} Jul 17 17:31:24.500472 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:24.500443 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-43.ec2.internal" event={"ID":"33e1e8a8a145842c049c2ce22b537c78","Type":"ContainerStarted","Data":"877f637d289dca3c23196f1d912fb2d0c1279964b4f2030a4121ca4f16534600"} Jul 17 17:31:24.923081 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:24.922789 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q7ds\" (UniqueName: \"kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds\") pod \"network-check-target-xf9wz\" (UID: \"d59d2582-9624-426a-a770-75a633dd1f0f\") " pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:24.923081 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:24.922839 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs\") pod \"network-metrics-daemon-fvs2m\" (UID: \"8a107e73-1e71-4f81-9751-d19706139fad\") " pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:24.923081 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:24.922908 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret\") pod \"global-pull-secret-syncer-czb2s\" (UID: \"2758fa5d-c686-4179-af0b-bc2b26fcf0eb\") " pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:24.923081 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:24.923024 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:31:24.923081 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:24.923035 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:31:24.923081 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:24.923049 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:31:24.923081 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:24.923060 2553 projected.go:196] Error preparing data for projected volume kube-api-access-6q7ds for pod openshift-network-diagnostics/network-check-target-xf9wz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:31:24.923081 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:24.923044 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:31:24.923585 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:24.923106 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs podName:8a107e73-1e71-4f81-9751-d19706139fad nodeName:}" failed. No retries permitted until 2026-07-17 17:31:28.923086303 +0000 UTC m=+10.041914576 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs") pod "network-metrics-daemon-fvs2m" (UID: "8a107e73-1e71-4f81-9751-d19706139fad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:31:24.923585 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:24.923169 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret podName:2758fa5d-c686-4179-af0b-bc2b26fcf0eb nodeName:}" failed. No retries permitted until 2026-07-17 17:31:28.923114442 +0000 UTC m=+10.041942719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret") pod "global-pull-secret-syncer-czb2s" (UID: "2758fa5d-c686-4179-af0b-bc2b26fcf0eb") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:31:24.923585 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:24.923192 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds podName:d59d2582-9624-426a-a770-75a633dd1f0f nodeName:}" failed. No retries permitted until 2026-07-17 17:31:28.923180649 +0000 UTC m=+10.042008922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6q7ds" (UniqueName: "kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds") pod "network-check-target-xf9wz" (UID: "d59d2582-9624-426a-a770-75a633dd1f0f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:31:26.457925 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:26.457834 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:26.458353 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:26.457976 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf9wz" podUID="d59d2582-9624-426a-a770-75a633dd1f0f" Jul 17 17:31:26.458353 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:26.458268 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:26.458450 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:26.458360 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-czb2s" podUID="2758fa5d-c686-4179-af0b-bc2b26fcf0eb" Jul 17 17:31:26.458450 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:26.458415 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:26.458555 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:26.458516 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvs2m" podUID="8a107e73-1e71-4f81-9751-d19706139fad" Jul 17 17:31:26.508684 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:26.508522 2553 generic.go:356] "Generic (PLEG): container finished" podID="0c86625a603c83bec1cac3688696a497" containerID="2d8232e9fee7982bd829941eca2c067b1d331ef94a5471aaa0471f436e0651b3" exitCode=0 Jul 17 17:31:26.508684 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:26.508575 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal" event={"ID":"0c86625a603c83bec1cac3688696a497","Type":"ContainerDied","Data":"2d8232e9fee7982bd829941eca2c067b1d331ef94a5471aaa0471f436e0651b3"} Jul 17 17:31:26.529126 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:26.528688 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-43.ec2.internal" podStartSLOduration=5.528661713 podStartE2EDuration="5.528661713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:31:24.52671192 +0000 UTC m=+5.645540202" watchObservedRunningTime="2026-07-17 17:31:26.528661713 +0000 UTC m=+7.647489983" Jul 17 17:31:28.458242 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:28.458107 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:28.458242 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:28.458136 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:28.458242 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:28.458147 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:28.458790 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:28.458265 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvs2m" podUID="8a107e73-1e71-4f81-9751-d19706139fad" Jul 17 17:31:28.458790 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:28.458439 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf9wz" podUID="d59d2582-9624-426a-a770-75a633dd1f0f" Jul 17 17:31:28.458790 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:28.458534 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-czb2s" podUID="2758fa5d-c686-4179-af0b-bc2b26fcf0eb" Jul 17 17:31:28.958146 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:28.958089 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q7ds\" (UniqueName: \"kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds\") pod \"network-check-target-xf9wz\" (UID: \"d59d2582-9624-426a-a770-75a633dd1f0f\") " pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:28.958146 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:28.958166 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs\") pod \"network-metrics-daemon-fvs2m\" (UID: \"8a107e73-1e71-4f81-9751-d19706139fad\") " pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:28.958477 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:28.958219 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret\") pod \"global-pull-secret-syncer-czb2s\" (UID: \"2758fa5d-c686-4179-af0b-bc2b26fcf0eb\") " pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:28.958477 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:28.958239 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:31:28.958477 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:28.958266 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:31:28.958477 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:28.958281 2553 projected.go:196] Error preparing data for projected volume kube-api-access-6q7ds for pod openshift-network-diagnostics/network-check-target-xf9wz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:31:28.958477 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:28.958316 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:31:28.958477 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:28.958349 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds podName:d59d2582-9624-426a-a770-75a633dd1f0f nodeName:}" failed. No retries permitted until 2026-07-17 17:31:36.958328956 +0000 UTC m=+18.077157222 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6q7ds" (UniqueName: "kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds") pod "network-check-target-xf9wz" (UID: "d59d2582-9624-426a-a770-75a633dd1f0f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:31:28.958477 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:28.958373 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs podName:8a107e73-1e71-4f81-9751-d19706139fad nodeName:}" failed. No retries permitted until 2026-07-17 17:31:36.958357201 +0000 UTC m=+18.077185475 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs") pod "network-metrics-daemon-fvs2m" (UID: "8a107e73-1e71-4f81-9751-d19706139fad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:31:28.958477 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:28.958316 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:31:28.958477 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:28.958411 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret podName:2758fa5d-c686-4179-af0b-bc2b26fcf0eb nodeName:}" failed. No retries permitted until 2026-07-17 17:31:36.958401082 +0000 UTC m=+18.077229356 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret") pod "global-pull-secret-syncer-czb2s" (UID: "2758fa5d-c686-4179-af0b-bc2b26fcf0eb") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:31:30.458638 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:30.458110 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:30.458638 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:30.458252 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf9wz" podUID="d59d2582-9624-426a-a770-75a633dd1f0f" Jul 17 17:31:30.458638 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:30.458615 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:30.459237 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:30.458726 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvs2m" podUID="8a107e73-1e71-4f81-9751-d19706139fad" Jul 17 17:31:30.459237 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:30.458620 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:30.459237 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:30.458815 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-czb2s" podUID="2758fa5d-c686-4179-af0b-bc2b26fcf0eb" Jul 17 17:31:32.457369 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:32.457340 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:32.457369 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:32.457355 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:32.457369 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:32.457378 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:32.457843 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:32.457461 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-czb2s" podUID="2758fa5d-c686-4179-af0b-bc2b26fcf0eb" Jul 17 17:31:32.457843 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:32.457548 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvs2m" podUID="8a107e73-1e71-4f81-9751-d19706139fad" Jul 17 17:31:32.457843 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:32.457631 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf9wz" podUID="d59d2582-9624-426a-a770-75a633dd1f0f" Jul 17 17:31:32.522071 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:32.522036 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal" event={"ID":"0c86625a603c83bec1cac3688696a497","Type":"ContainerStarted","Data":"43a3fe4b64ff9d02d9941b3129787f2eac1aa6e01cf55761174b1add1290e554"} Jul 17 17:31:32.523661 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:32.523622 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7vkng" event={"ID":"5177b409-481a-4b8f-ae82-c1e28eebd561","Type":"ContainerStarted","Data":"9ada68254b2860266a16b0248b0dfcd331e7607d5b83ed76b7ff2775a10f95b5"} Jul 17 17:31:32.525213 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:32.525185 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" event={"ID":"169ea2a6-448e-4662-ac13-7b881a1566ef","Type":"ContainerStarted","Data":"acd079843f3ee187ba7245945c865187c2bdf804b45cab64b193fed415979da3"} Jul 17 17:31:32.526926 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:32.526898 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7w7jr" event={"ID":"bd6be5c2-6170-4336-9626-0e499a629268","Type":"ContainerStarted","Data":"f86be0e53a33d3457d4a092bdd0b99501c4418844bf7e5da6f3540573283b8ca"} Jul 17 17:31:32.536669 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:32.536615 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-43.ec2.internal" podStartSLOduration=11.536603212 podStartE2EDuration="11.536603212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:31:32.535903374 +0000 UTC m=+13.654731653" watchObservedRunningTime="2026-07-17 17:31:32.536603212 +0000 UTC m=+13.655431492" Jul 17 17:31:32.551360 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:32.551317 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7vkng" podStartSLOduration=3.060765039 podStartE2EDuration="13.551303203s" totalImagesPullingTime="10.490538164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:31:32.550663937 +0000 UTC m=+13.669492219" watchObservedRunningTime="2026-07-17 17:31:32.551303203 +0000 UTC m=+13.670131483" Jul 17 17:31:32.564835 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:32.564743 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7w7jr" podStartSLOduration=3.09021909 podStartE2EDuration="13.56472636s" totalImagesPullingTime="10.47450727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:31:32.564574372 +0000 UTC m=+13.683402655" watchObservedRunningTime="2026-07-17 17:31:32.56472636 +0000 UTC m=+13.683554631" Jul 17 17:31:32.711238 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:32.711206 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:32.895446 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:32.895365 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:33.297113 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:33.297072 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:33.356359 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:33.356329 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7vkng" Jul 17 17:31:33.356921 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:33.356903 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7vkng" Jul 17 17:31:33.500368 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:33.500169 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:33.530119 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:33.530070 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vz64j" event={"ID":"a0f3eafc-0dd5-40d7-b50a-466253eadc33","Type":"ContainerStarted","Data":"efd3bc880afae776186e353077ef0d241e91d3d43d9afb64aca1bea8a2d09d04"} Jul 17 17:31:33.698694 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:33.698608 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:34.297468 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:34.297438 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:34.457423 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:34.457388 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:34.457423 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:34.457410 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:34.457612 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:34.457506 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:34.457612 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:34.457513 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-czb2s" podUID="2758fa5d-c686-4179-af0b-bc2b26fcf0eb" Jul 17 17:31:34.457712 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:34.457599 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf9wz" podUID="d59d2582-9624-426a-a770-75a633dd1f0f" Jul 17 17:31:34.457712 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:34.457691 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvs2m" podUID="8a107e73-1e71-4f81-9751-d19706139fad" Jul 17 17:31:34.492991 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:34.492959 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:34.532245 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:34.532218 2553 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 17 17:31:34.848568 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:34.848539 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7vkng" Jul 17 17:31:34.849143 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:34.849124 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7vkng" Jul 17 17:31:34.864084 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:34.864033 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vz64j" podStartSLOduration=5.395379517 podStartE2EDuration="15.864016523s" totalImagesPullingTime="10.468637006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:31:33.549344032 +0000 UTC m=+14.668172312" watchObservedRunningTime="2026-07-17 17:31:34.864016523 +0000 UTC m=+15.982844805" Jul 17 17:31:36.319532 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:36.319505 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:36.371545 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:36.371510 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:36.457985 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:36.457953 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:36.458134 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:36.458057 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:36.458134 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:36.458068 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf9wz" podUID="d59d2582-9624-426a-a770-75a633dd1f0f" Jul 17 17:31:36.458134 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:36.458082 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:36.458289 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:36.458166 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-czb2s" podUID="2758fa5d-c686-4179-af0b-bc2b26fcf0eb" Jul 17 17:31:36.458289 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:36.458252 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvs2m" podUID="8a107e73-1e71-4f81-9751-d19706139fad" Jul 17 17:31:37.024138 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:37.024102 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q7ds\" (UniqueName: \"kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds\") pod \"network-check-target-xf9wz\" (UID: \"d59d2582-9624-426a-a770-75a633dd1f0f\") " pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:37.024326 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:37.024147 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs\") pod \"network-metrics-daemon-fvs2m\" (UID: \"8a107e73-1e71-4f81-9751-d19706139fad\") " pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:37.024326 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:37.024191 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret\") pod \"global-pull-secret-syncer-czb2s\" (UID: \"2758fa5d-c686-4179-af0b-bc2b26fcf0eb\") " pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:37.024326 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:37.024294 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:31:37.024326 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:37.024315 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:31:37.024518 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:37.024323 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:31:37.024518 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:37.024383 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs podName:8a107e73-1e71-4f81-9751-d19706139fad nodeName:}" failed. No retries permitted until 2026-07-17 17:31:53.02436387 +0000 UTC m=+34.143192145 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs") pod "network-metrics-daemon-fvs2m" (UID: "8a107e73-1e71-4f81-9751-d19706139fad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:31:37.024518 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:37.024395 2553 projected.go:196] Error preparing data for projected volume kube-api-access-6q7ds for pod openshift-network-diagnostics/network-check-target-xf9wz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:31:37.024518 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:37.024449 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds podName:d59d2582-9624-426a-a770-75a633dd1f0f nodeName:}" failed. No retries permitted until 2026-07-17 17:31:53.024432505 +0000 UTC m=+34.143260787 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6q7ds" (UniqueName: "kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds") pod "network-check-target-xf9wz" (UID: "d59d2582-9624-426a-a770-75a633dd1f0f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:31:37.024518 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:37.024300 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:31:37.024518 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:37.024500 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret podName:2758fa5d-c686-4179-af0b-bc2b26fcf0eb nodeName:}" failed. No retries permitted until 2026-07-17 17:31:53.02448911 +0000 UTC m=+34.143317400 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret") pod "global-pull-secret-syncer-czb2s" (UID: "2758fa5d-c686-4179-af0b-bc2b26fcf0eb") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:31:38.457467 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:38.457440 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:38.458054 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:38.457540 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:38.458054 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:38.457551 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf9wz" podUID="d59d2582-9624-426a-a770-75a633dd1f0f" Jul 17 17:31:38.458054 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:38.457572 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:38.458054 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:38.457672 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-czb2s" podUID="2758fa5d-c686-4179-af0b-bc2b26fcf0eb" Jul 17 17:31:38.458054 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:38.457768 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvs2m" podUID="8a107e73-1e71-4f81-9751-d19706139fad" Jul 17 17:31:40.458038 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:40.458013 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:40.458639 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:40.458060 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:40.458639 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:40.458131 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf9wz" podUID="d59d2582-9624-426a-a770-75a633dd1f0f" Jul 17 17:31:40.458639 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:40.458014 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:40.458639 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:40.458228 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvs2m" podUID="8a107e73-1e71-4f81-9751-d19706139fad" Jul 17 17:31:40.458639 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:40.458324 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-czb2s" podUID="2758fa5d-c686-4179-af0b-bc2b26fcf0eb" Jul 17 17:31:40.590032 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:40.587260 2553 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 17 17:31:41.358380 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:41.358119 2553 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T17:31:40.587472461Z","UUID":"08675dfa-3268-48d4-9606-1ec80db602ce","Handler":null,"Name":"","Endpoint":""} Jul 17 17:31:41.359639 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:41.359616 2553 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 17 17:31:41.359742 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:41.359662 2553 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 17 17:31:41.548605 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:41.548573 2553 generic.go:356] "Generic (PLEG): container finished" podID="2fb88f09-39c9-4e4b-91c4-7c3d21b071ba" containerID="c468d635d24dab9497377585b4d7347a86310260ecb1b5ed31f77bbf3fb32570" exitCode=0 Jul 17 17:31:41.549025 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:41.548670 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krv7b" event={"ID":"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba","Type":"ContainerDied","Data":"c468d635d24dab9497377585b4d7347a86310260ecb1b5ed31f77bbf3fb32570"} Jul 17 17:31:41.551773 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:41.551750 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" event={"ID":"04147e2d-052c-4d96-9693-802066c81adc","Type":"ContainerStarted","Data":"66c6d15bb2740f5858bc81b5900df2b0025963fbbb03a5ab48f6511d22697db7"} Jul 17 17:31:41.551903 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:41.551820 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" event={"ID":"04147e2d-052c-4d96-9693-802066c81adc","Type":"ContainerStarted","Data":"71d13a094f2ea46c042bc55cc03c5ca7442a7bd7d3dea3ea78bdb27a5157657d"} Jul 17 17:31:41.551903 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:41.551884 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" event={"ID":"04147e2d-052c-4d96-9693-802066c81adc","Type":"ContainerStarted","Data":"db5d02a00952afac7b11657c32cd420a3bd1049de3f13b601df913c6001d6794"} Jul 17 17:31:41.552026 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:41.551918 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" event={"ID":"04147e2d-052c-4d96-9693-802066c81adc","Type":"ContainerStarted","Data":"ca9b48e22d8331c1054f3d181c3bf0242a2dd633674a8402c763bf1d088843a6"} Jul 17 17:31:41.552026 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:41.551958 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" event={"ID":"04147e2d-052c-4d96-9693-802066c81adc","Type":"ContainerStarted","Data":"95f5b1e224edb2c8cd4452ab53e7f62f9f3c5c0b538cd9910637e58673eeb438"} Jul 17 17:31:41.552026 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:41.551996 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" event={"ID":"04147e2d-052c-4d96-9693-802066c81adc","Type":"ContainerStarted","Data":"76d2bd4bf59259b7774fc68e4153871d413d4ba622a34c3f368c2f47637de4e4"} Jul 17 17:31:41.553445 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:41.553409 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" event={"ID":"169ea2a6-448e-4662-ac13-7b881a1566ef","Type":"ContainerStarted","Data":"0d7df0bf4ce6850f98809b89ed038fb97e4d6d0fb8c75fe71746a88865e8a4b1"} Jul 17 17:31:41.554767 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:41.554748 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v5sbn" event={"ID":"b54d03ca-d903-45ac-b3e5-c4c70d5574cf","Type":"ContainerStarted","Data":"081a05ae2a3952b690e825528bc17655a0cbfe7fd112bacc8fef9856c3db9ee9"} Jul 17 17:31:41.556422 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:41.556399 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-52rj6" event={"ID":"9b365f43-0235-462f-a7ba-e38b1936bca4","Type":"ContainerStarted","Data":"87e57092eed0c0d92a0a8c08246d2017b1bb5b334fc0bd534825921ad3cc06fa"} Jul 17 17:31:41.557646 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:41.557627 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xx4rh" event={"ID":"8f7f37cc-005b-44d5-a728-90cdb84b5d22","Type":"ContainerStarted","Data":"6df0425cefb95255ba14a4e1ee26af21043cb58d8ffff631c27a6ac9e3d21f5d"} Jul 17 17:31:41.599060 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:41.599007 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xx4rh" podStartSLOduration=12.075562811 podStartE2EDuration="22.598992118s" totalImagesPullingTime="10.523429307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:31:41.59893355 +0000 UTC m=+22.717761833" watchObservedRunningTime="2026-07-17 17:31:41.598992118 +0000 UTC m=+22.717820400" Jul 17 17:31:41.599303 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:41.599283 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-v5sbn" podStartSLOduration=3.7957393980000003 podStartE2EDuration="22.599276474s" totalImagesPullingTime="18.803537076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:31:41.585664451 +0000 UTC m=+22.704492732" watchObservedRunningTime="2026-07-17 17:31:41.599276474 +0000 UTC m=+22.718104754" Jul 17 17:31:41.614694 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:41.614636 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-52rj6" podStartSLOduration=3.765774494 podStartE2EDuration="22.614626234s" totalImagesPullingTime="18.84885174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:31:41.61423121 +0000 UTC m=+22.733059501" watchObservedRunningTime="2026-07-17 17:31:41.614626234 +0000 UTC m=+22.733454514" Jul 17 17:31:41.833703 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:41.833674 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:41.933626 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:41.933589 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:42.116256 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:42.116225 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:42.312799 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:42.312764 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:42.423699 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:42.423650 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:42.457647 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:42.457621 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:42.457817 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:42.457621 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:42.457817 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:42.457748 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvs2m" podUID="8a107e73-1e71-4f81-9751-d19706139fad" Jul 17 17:31:42.457817 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:42.457629 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:42.457982 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:42.457819 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf9wz" podUID="d59d2582-9624-426a-a770-75a633dd1f0f" Jul 17 17:31:42.457982 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:42.457898 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-czb2s" podUID="2758fa5d-c686-4179-af0b-bc2b26fcf0eb" Jul 17 17:31:42.522154 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:42.522125 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:42.561969 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:42.561938 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" event={"ID":"169ea2a6-448e-4662-ac13-7b881a1566ef","Type":"ContainerStarted","Data":"a4f372277655859083f21f0cb7c160ab3d30bc12ff02e892f9d4e3648446246b"} Jul 17 17:31:42.596061 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:42.595998 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:42.678661 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:42.678630 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:42.774287 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:42.774261 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:42.836145 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:42.836119 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:42.898052 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:42.897960 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:43.080069 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:43.080044 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:43.135883 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:43.135840 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:43.428569 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:43.428493 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:43.557546 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:43.557516 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:43.567561 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:43.567529 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" event={"ID":"04147e2d-052c-4d96-9693-802066c81adc","Type":"ContainerStarted","Data":"fb2497dd2fda266478887781798b6cfe0e261dff5fe60a15b082d77999fe7dec"} Jul 17 17:31:43.706794 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:43.706702 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:43.800937 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:43.800903 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:44.457598 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:44.457574 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:44.457598 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:44.457594 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:44.457841 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:44.457574 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:44.457841 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:44.457693 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-czb2s" podUID="2758fa5d-c686-4179-af0b-bc2b26fcf0eb" Jul 17 17:31:44.457841 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:44.457767 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvs2m" podUID="8a107e73-1e71-4f81-9751-d19706139fad" Jul 17 17:31:44.457841 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:44.457821 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf9wz" podUID="d59d2582-9624-426a-a770-75a633dd1f0f" Jul 17 17:31:44.537582 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:44.537541 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:45.117139 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:45.117087 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:45.165395 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:45.165364 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:45.577607 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:45.577424 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" event={"ID":"04147e2d-052c-4d96-9693-802066c81adc","Type":"ContainerStarted","Data":"c7c2daf817129bf9de167db60c31913d1e6647b4442fa9bbd1ee9e4d8ee81e80"} Jul 17 17:31:45.577763 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:45.577675 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:45.577763 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:45.577715 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:45.592571 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:45.592545 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:45.604602 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:45.604556 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmzkb" podStartSLOduration=5.788401238 podStartE2EDuration="25.604539278s" totalImagesPullingTime="19.81613804s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:31:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:31:42.583236543 +0000 UTC m=+23.702064825" watchObservedRunningTime="2026-07-17 17:31:45.604539278 +0000 UTC m=+26.723367560" Jul 17 17:31:45.605150 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:45.605120 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" podStartSLOduration=7.740680595 podStartE2EDuration="26.605112471s" totalImagesPullingTime="18.864431876s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:31:45.604383419 +0000 UTC m=+26.723211738" watchObservedRunningTime="2026-07-17 17:31:45.605112471 +0000 UTC m=+26.723940751" Jul 17 17:31:46.457827 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:46.457791 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:46.457827 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:46.457826 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:46.458598 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:46.457911 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:46.458598 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:46.457923 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf9wz" podUID="d59d2582-9624-426a-a770-75a633dd1f0f" Jul 17 17:31:46.458598 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:46.457991 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-czb2s" podUID="2758fa5d-c686-4179-af0b-bc2b26fcf0eb" Jul 17 17:31:46.458598 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:46.458080 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvs2m" podUID="8a107e73-1e71-4f81-9751-d19706139fad" Jul 17 17:31:46.588791 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:46.588751 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:46.605749 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:46.605661 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:31:47.342618 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:47.342546 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-czb2s"] Jul 17 17:31:47.342772 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:47.342658 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:47.342772 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:47.342744 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-czb2s" podUID="2758fa5d-c686-4179-af0b-bc2b26fcf0eb" Jul 17 17:31:47.343079 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:47.343060 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fvs2m"] Jul 17 17:31:47.343160 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:47.343149 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:47.343255 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:47.343237 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvs2m" podUID="8a107e73-1e71-4f81-9751-d19706139fad" Jul 17 17:31:47.344208 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:47.344186 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xf9wz"] Jul 17 17:31:47.344293 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:47.344282 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:47.344378 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:47.344355 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf9wz" podUID="d59d2582-9624-426a-a770-75a633dd1f0f" Jul 17 17:31:48.265397 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:48.265368 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:48.398153 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:48.398116 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:48.458102 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:48.458074 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:48.458245 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:48.458167 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf9wz" podUID="d59d2582-9624-426a-a770-75a633dd1f0f" Jul 17 17:31:48.493046 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:48.493009 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:48.594126 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:48.594036 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:48.594500 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:48.594475 2553 generic.go:356] "Generic (PLEG): container finished" podID="2fb88f09-39c9-4e4b-91c4-7c3d21b071ba" containerID="444e3cf1fe6e85e3b669a0dc51be009e77a9f77375f9fec6ed79b8d87d053642" exitCode=0 Jul 17 17:31:48.594576 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:48.594554 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krv7b" event={"ID":"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba","Type":"ContainerDied","Data":"444e3cf1fe6e85e3b669a0dc51be009e77a9f77375f9fec6ed79b8d87d053642"} Jul 17 17:31:49.261750 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:49.261573 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:49.338736 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:49.338707 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:49.459086 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:49.459065 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:49.459193 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:49.459174 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-czb2s" podUID="2758fa5d-c686-4179-af0b-bc2b26fcf0eb" Jul 17 17:31:49.459247 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:49.459235 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:49.459360 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:49.459335 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvs2m" podUID="8a107e73-1e71-4f81-9751-d19706139fad" Jul 17 17:31:49.532646 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:49.532585 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:49.598496 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:49.598462 2553 generic.go:356] "Generic (PLEG): container finished" podID="2fb88f09-39c9-4e4b-91c4-7c3d21b071ba" containerID="5494d2aa9301292c0bb7b2efeef7f15f776f97cd7b1b6251328d53a386ea2121" exitCode=0 Jul 17 17:31:49.598645 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:49.598505 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krv7b" event={"ID":"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba","Type":"ContainerDied","Data":"5494d2aa9301292c0bb7b2efeef7f15f776f97cd7b1b6251328d53a386ea2121"} Jul 17 17:31:49.644493 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:49.644455 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:49.740242 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:49.740216 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:49.800913 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:49.800815 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:50.053816 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:50.053739 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:50.155117 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:50.155083 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:50.457608 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:50.457587 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:50.458114 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:50.457674 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf9wz" podUID="d59d2582-9624-426a-a770-75a633dd1f0f" Jul 17 17:31:50.602676 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:50.602594 2553 generic.go:356] "Generic (PLEG): container finished" podID="2fb88f09-39c9-4e4b-91c4-7c3d21b071ba" containerID="106a83fd258c23d2570979f5e5d4944d7707544e46db0dbc850a952ef2128025" exitCode=0 Jul 17 17:31:50.602676 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:50.602652 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krv7b" event={"ID":"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba","Type":"ContainerDied","Data":"106a83fd258c23d2570979f5e5d4944d7707544e46db0dbc850a952ef2128025"} Jul 17 17:31:51.321132 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:51.321097 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:51.421198 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:51.421162 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:51.458145 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:51.458121 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:51.458145 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:51.458138 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:51.458608 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:51.458245 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-czb2s" podUID="2758fa5d-c686-4179-af0b-bc2b26fcf0eb" Jul 17 17:31:51.458608 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:51.458367 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fvs2m" podUID="8a107e73-1e71-4f81-9751-d19706139fad" Jul 17 17:31:52.457360 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:52.457329 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:52.457528 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:52.457439 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf9wz" podUID="d59d2582-9624-426a-a770-75a633dd1f0f" Jul 17 17:31:53.051507 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.051466 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q7ds\" (UniqueName: \"kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds\") pod \"network-check-target-xf9wz\" (UID: \"d59d2582-9624-426a-a770-75a633dd1f0f\") " pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:53.052074 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.051518 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs\") pod \"network-metrics-daemon-fvs2m\" (UID: \"8a107e73-1e71-4f81-9751-d19706139fad\") " pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:53.052074 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.051565 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret\") pod \"global-pull-secret-syncer-czb2s\" (UID: \"2758fa5d-c686-4179-af0b-bc2b26fcf0eb\") " pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:53.052074 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:53.051656 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:31:53.052074 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:53.051686 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:31:53.052074 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:53.051699 2553 projected.go:196] Error preparing data for projected volume kube-api-access-6q7ds for pod openshift-network-diagnostics/network-check-target-xf9wz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:31:53.052074 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:53.051705 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:31:53.052074 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:53.051752 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:31:53.052074 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:53.051760 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds podName:d59d2582-9624-426a-a770-75a633dd1f0f nodeName:}" failed. No retries permitted until 2026-07-17 17:32:25.051739793 +0000 UTC m=+66.170568070 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6q7ds" (UniqueName: "kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds") pod "network-check-target-xf9wz" (UID: "d59d2582-9624-426a-a770-75a633dd1f0f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:31:53.052074 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:53.051783 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs podName:8a107e73-1e71-4f81-9751-d19706139fad nodeName:}" failed. No retries permitted until 2026-07-17 17:32:25.051771662 +0000 UTC m=+66.170599928 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs") pod "network-metrics-daemon-fvs2m" (UID: "8a107e73-1e71-4f81-9751-d19706139fad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:31:53.052074 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:31:53.051797 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret podName:2758fa5d-c686-4179-af0b-bc2b26fcf0eb nodeName:}" failed. No retries permitted until 2026-07-17 17:32:25.051788932 +0000 UTC m=+66.170617197 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret") pod "global-pull-secret-syncer-czb2s" (UID: "2758fa5d-c686-4179-af0b-bc2b26fcf0eb") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:31:53.083330 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.083305 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-43.ec2.internal" event="NodeReady" Jul 17 17:31:53.083470 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.083460 2553 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 17 17:31:53.119453 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.119420 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56bb5f997f-5jxmw"] Jul 17 17:31:53.122898 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.122871 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.125644 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.125577 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 17 17:31:53.125747 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.125682 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qgkjr\"" Jul 17 17:31:53.125827 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.125605 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 17 17:31:53.125945 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.125666 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 17 17:31:53.130478 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.130449 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8jchv"] Jul 17 17:31:53.131920 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.131635 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 17 17:31:53.133893 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.133871 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mhq7k"] Jul 17 17:31:53.134456 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.134439 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8jchv" Jul 17 17:31:53.136909 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.136889 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dpl9j\"" Jul 17 17:31:53.137101 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.136934 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 17 17:31:53.137101 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.136943 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 17 17:31:53.137101 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.136995 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 17 17:31:53.137263 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.137242 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56bb5f997f-5jxmw"] Jul 17 17:31:53.137356 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.137341 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mhq7k" Jul 17 17:31:53.139670 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.139651 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 17 17:31:53.140032 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.139998 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 17 17:31:53.140032 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.140012 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 17 17:31:53.140152 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.140012 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7jwcc\"" Jul 17 17:31:53.140152 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.140013 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 17 17:31:53.141949 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.141927 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8jchv"] Jul 17 17:31:53.145591 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.145281 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mhq7k"] Jul 17 17:31:53.151425 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.151401 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-phcbk"] Jul 17 17:31:53.154773 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.154754 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-phcbk" Jul 17 17:31:53.157524 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.157506 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 17 17:31:53.158001 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.157981 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 17 17:31:53.158204 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.158190 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c4wmq\"" Jul 17 17:31:53.162282 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.162227 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-phcbk"] Jul 17 17:31:53.252384 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.252352 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxznt\" (UniqueName: \"kubernetes.io/projected/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-kube-api-access-cxznt\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.252569 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.252393 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7dsp\" (UniqueName: \"kubernetes.io/projected/8d6c365e-40fc-4cee-ad65-e984c0686bfe-kube-api-access-s7dsp\") pod \"ingress-canary-8jchv\" (UID: \"8d6c365e-40fc-4cee-ad65-e984c0686bfe\") " pod="openshift-ingress-canary/ingress-canary-8jchv" Jul 17 17:31:53.252569 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.252443 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5657e768-43ad-48c2-b520-adf83fbab4d4-crio-socket\") pod \"insights-runtime-extractor-mhq7k\" (UID: \"5657e768-43ad-48c2-b520-adf83fbab4d4\") " pod="openshift-insights/insights-runtime-extractor-mhq7k" Jul 17 17:31:53.252569 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.252468 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5657e768-43ad-48c2-b520-adf83fbab4d4-data-volume\") pod \"insights-runtime-extractor-mhq7k\" (UID: \"5657e768-43ad-48c2-b520-adf83fbab4d4\") " pod="openshift-insights/insights-runtime-extractor-mhq7k" Jul 17 17:31:53.252569 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.252524 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-registry-certificates\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.252569 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.252560 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5657e768-43ad-48c2-b520-adf83fbab4d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mhq7k\" (UID: \"5657e768-43ad-48c2-b520-adf83fbab4d4\") " pod="openshift-insights/insights-runtime-extractor-mhq7k" Jul 17 17:31:53.252739 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.252578 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-registry-tls\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.252739 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.252629 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d6c365e-40fc-4cee-ad65-e984c0686bfe-cert\") pod \"ingress-canary-8jchv\" (UID: \"8d6c365e-40fc-4cee-ad65-e984c0686bfe\") " pod="openshift-ingress-canary/ingress-canary-8jchv" Jul 17 17:31:53.252739 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.252667 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5657e768-43ad-48c2-b520-adf83fbab4d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mhq7k\" (UID: \"5657e768-43ad-48c2-b520-adf83fbab4d4\") " pod="openshift-insights/insights-runtime-extractor-mhq7k" Jul 17 17:31:53.252739 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.252687 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66ba0e1c-5d64-45ba-8db5-69fbd6715a14-metrics-tls\") pod \"dns-default-phcbk\" (UID: \"66ba0e1c-5d64-45ba-8db5-69fbd6715a14\") " pod="openshift-dns/dns-default-phcbk" Jul 17 17:31:53.252739 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.252724 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-bound-sa-token\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.252922 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.252754 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fbtd\" (UniqueName: \"kubernetes.io/projected/5657e768-43ad-48c2-b520-adf83fbab4d4-kube-api-access-7fbtd\") pod \"insights-runtime-extractor-mhq7k\" (UID: \"5657e768-43ad-48c2-b520-adf83fbab4d4\") " pod="openshift-insights/insights-runtime-extractor-mhq7k" Jul 17 17:31:53.252922 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.252775 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66ba0e1c-5d64-45ba-8db5-69fbd6715a14-tmp-dir\") pod \"dns-default-phcbk\" (UID: \"66ba0e1c-5d64-45ba-8db5-69fbd6715a14\") " pod="openshift-dns/dns-default-phcbk" Jul 17 17:31:53.252922 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.252791 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-image-registry-private-configuration\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.252922 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.252815 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-trusted-ca\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.252922 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.252879 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnjbl\" (UniqueName: \"kubernetes.io/projected/66ba0e1c-5d64-45ba-8db5-69fbd6715a14-kube-api-access-rnjbl\") pod \"dns-default-phcbk\" (UID: \"66ba0e1c-5d64-45ba-8db5-69fbd6715a14\") " pod="openshift-dns/dns-default-phcbk" Jul 17 17:31:53.252922 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.252910 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/66ba0e1c-5d64-45ba-8db5-69fbd6715a14-config-volume\") pod \"dns-default-phcbk\" (UID: \"66ba0e1c-5d64-45ba-8db5-69fbd6715a14\") " pod="openshift-dns/dns-default-phcbk" Jul 17 17:31:53.253126 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.252925 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-ca-trust-extracted\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.253126 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.252940 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-installation-pull-secrets\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.261701 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.261672 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:53.353586 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.353554 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5657e768-43ad-48c2-b520-adf83fbab4d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mhq7k\" (UID: \"5657e768-43ad-48c2-b520-adf83fbab4d4\") " pod="openshift-insights/insights-runtime-extractor-mhq7k" Jul 17 17:31:53.353586 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.353585 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66ba0e1c-5d64-45ba-8db5-69fbd6715a14-metrics-tls\") pod \"dns-default-phcbk\" (UID: \"66ba0e1c-5d64-45ba-8db5-69fbd6715a14\") " pod="openshift-dns/dns-default-phcbk" Jul 17 17:31:53.353807 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.353717 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-bound-sa-token\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.353807 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.353759 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fbtd\" (UniqueName: \"kubernetes.io/projected/5657e768-43ad-48c2-b520-adf83fbab4d4-kube-api-access-7fbtd\") pod \"insights-runtime-extractor-mhq7k\" (UID: \"5657e768-43ad-48c2-b520-adf83fbab4d4\") " pod="openshift-insights/insights-runtime-extractor-mhq7k" Jul 17 17:31:53.353902 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.353882 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66ba0e1c-5d64-45ba-8db5-69fbd6715a14-tmp-dir\") pod \"dns-default-phcbk\" (UID: \"66ba0e1c-5d64-45ba-8db5-69fbd6715a14\") " pod="openshift-dns/dns-default-phcbk" Jul 17 17:31:53.353950 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.353919 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-image-registry-private-configuration\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.354000 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.353953 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-trusted-ca\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.354000 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.353987 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnjbl\" (UniqueName: \"kubernetes.io/projected/66ba0e1c-5d64-45ba-8db5-69fbd6715a14-kube-api-access-rnjbl\") pod \"dns-default-phcbk\" (UID: \"66ba0e1c-5d64-45ba-8db5-69fbd6715a14\") " pod="openshift-dns/dns-default-phcbk" Jul 17 17:31:53.354093 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.354020 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/66ba0e1c-5d64-45ba-8db5-69fbd6715a14-config-volume\") pod \"dns-default-phcbk\" (UID: \"66ba0e1c-5d64-45ba-8db5-69fbd6715a14\") " pod="openshift-dns/dns-default-phcbk" Jul 17 17:31:53.354093 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.354044 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-ca-trust-extracted\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.354093 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.354072 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-installation-pull-secrets\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.354229 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.354114 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxznt\" (UniqueName: \"kubernetes.io/projected/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-kube-api-access-cxznt\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.354229 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.354140 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7dsp\" (UniqueName: \"kubernetes.io/projected/8d6c365e-40fc-4cee-ad65-e984c0686bfe-kube-api-access-s7dsp\") pod \"ingress-canary-8jchv\" (UID: \"8d6c365e-40fc-4cee-ad65-e984c0686bfe\") " pod="openshift-ingress-canary/ingress-canary-8jchv" Jul 17 17:31:53.354229 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.354196 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5657e768-43ad-48c2-b520-adf83fbab4d4-crio-socket\") pod \"insights-runtime-extractor-mhq7k\" (UID: \"5657e768-43ad-48c2-b520-adf83fbab4d4\") " pod="openshift-insights/insights-runtime-extractor-mhq7k" Jul 17 17:31:53.354229 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.354222 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5657e768-43ad-48c2-b520-adf83fbab4d4-data-volume\") pod \"insights-runtime-extractor-mhq7k\" (UID: \"5657e768-43ad-48c2-b520-adf83fbab4d4\") " pod="openshift-insights/insights-runtime-extractor-mhq7k" Jul 17 17:31:53.354411 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.354236 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5657e768-43ad-48c2-b520-adf83fbab4d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mhq7k\" (UID: \"5657e768-43ad-48c2-b520-adf83fbab4d4\") " pod="openshift-insights/insights-runtime-extractor-mhq7k" Jul 17 17:31:53.354411 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.354253 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-registry-certificates\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.354411 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.354282 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5657e768-43ad-48c2-b520-adf83fbab4d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mhq7k\" (UID: \"5657e768-43ad-48c2-b520-adf83fbab4d4\") " pod="openshift-insights/insights-runtime-extractor-mhq7k" Jul 17 17:31:53.354411 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.354309 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-registry-tls\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.354411 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.354323 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66ba0e1c-5d64-45ba-8db5-69fbd6715a14-tmp-dir\") pod \"dns-default-phcbk\" (UID: \"66ba0e1c-5d64-45ba-8db5-69fbd6715a14\") " pod="openshift-dns/dns-default-phcbk" Jul 17 17:31:53.354411 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.354336 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d6c365e-40fc-4cee-ad65-e984c0686bfe-cert\") pod \"ingress-canary-8jchv\" (UID: \"8d6c365e-40fc-4cee-ad65-e984c0686bfe\") " pod="openshift-ingress-canary/ingress-canary-8jchv" Jul 17 17:31:53.354702 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.354476 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-ca-trust-extracted\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.354702 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.354606 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5657e768-43ad-48c2-b520-adf83fbab4d4-crio-socket\") pod \"insights-runtime-extractor-mhq7k\" (UID: \"5657e768-43ad-48c2-b520-adf83fbab4d4\") " pod="openshift-insights/insights-runtime-extractor-mhq7k" Jul 17 17:31:53.354997 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.354961 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/66ba0e1c-5d64-45ba-8db5-69fbd6715a14-config-volume\") pod \"dns-default-phcbk\" (UID: \"66ba0e1c-5d64-45ba-8db5-69fbd6715a14\") " pod="openshift-dns/dns-default-phcbk" Jul 17 17:31:53.355106 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.354992 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-trusted-ca\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.356309 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.355226 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5657e768-43ad-48c2-b520-adf83fbab4d4-data-volume\") pod \"insights-runtime-extractor-mhq7k\" (UID: \"5657e768-43ad-48c2-b520-adf83fbab4d4\") " pod="openshift-insights/insights-runtime-extractor-mhq7k" Jul 17 17:31:53.356309 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.355543 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-registry-certificates\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.358538 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.358517 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66ba0e1c-5d64-45ba-8db5-69fbd6715a14-metrics-tls\") pod \"dns-default-phcbk\" (UID: \"66ba0e1c-5d64-45ba-8db5-69fbd6715a14\") " pod="openshift-dns/dns-default-phcbk" Jul 17 17:31:53.358644 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.358626 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-installation-pull-secrets\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.358748 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.358725 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-image-registry-private-configuration\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.358790 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.358745 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5657e768-43ad-48c2-b520-adf83fbab4d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mhq7k\" (UID: \"5657e768-43ad-48c2-b520-adf83fbab4d4\") " pod="openshift-insights/insights-runtime-extractor-mhq7k" Jul 17 17:31:53.358790 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.358755 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d6c365e-40fc-4cee-ad65-e984c0686bfe-cert\") pod \"ingress-canary-8jchv\" (UID: \"8d6c365e-40fc-4cee-ad65-e984c0686bfe\") " pod="openshift-ingress-canary/ingress-canary-8jchv" Jul 17 17:31:53.358899 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.358881 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-registry-tls\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.361891 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.361871 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnjbl\" (UniqueName: \"kubernetes.io/projected/66ba0e1c-5d64-45ba-8db5-69fbd6715a14-kube-api-access-rnjbl\") pod \"dns-default-phcbk\" (UID: \"66ba0e1c-5d64-45ba-8db5-69fbd6715a14\") " pod="openshift-dns/dns-default-phcbk" Jul 17 17:31:53.361988 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.361969 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxznt\" (UniqueName: \"kubernetes.io/projected/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-kube-api-access-cxznt\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.362097 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.362075 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fbtd\" (UniqueName: \"kubernetes.io/projected/5657e768-43ad-48c2-b520-adf83fbab4d4-kube-api-access-7fbtd\") pod \"insights-runtime-extractor-mhq7k\" (UID: \"5657e768-43ad-48c2-b520-adf83fbab4d4\") " pod="openshift-insights/insights-runtime-extractor-mhq7k" Jul 17 17:31:53.362642 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.362614 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f7a3c2f-74e0-4ae2-982a-5abf88bf41af-bound-sa-token\") pod \"image-registry-56bb5f997f-5jxmw\" (UID: \"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af\") " pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.362964 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.362919 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7dsp\" (UniqueName: \"kubernetes.io/projected/8d6c365e-40fc-4cee-ad65-e984c0686bfe-kube-api-access-s7dsp\") pod \"ingress-canary-8jchv\" (UID: \"8d6c365e-40fc-4cee-ad65-e984c0686bfe\") " pod="openshift-ingress-canary/ingress-canary-8jchv" Jul 17 17:31:53.435190 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.435118 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:53.446208 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.446183 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8jchv" Jul 17 17:31:53.453258 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.453096 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mhq7k" Jul 17 17:31:53.460018 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.458200 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:31:53.460018 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.459024 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:31:53.461299 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.461279 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 17:31:53.461538 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.461519 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 17:31:53.461614 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.461605 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lgt4f\"" Jul 17 17:31:53.481549 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.481517 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-phcbk" Jul 17 17:31:53.629280 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.629229 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8jchv"] Jul 17 17:31:53.634573 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:53.634536 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d6c365e_40fc_4cee_ad65_e984c0686bfe.slice/crio-ca11ed7cdbac83c8e7a34b3d423be27a1d6afd182697a00697347a3a1af3dd16 WatchSource:0}: Error finding container ca11ed7cdbac83c8e7a34b3d423be27a1d6afd182697a00697347a3a1af3dd16: Status 404 returned error can't find the container with id ca11ed7cdbac83c8e7a34b3d423be27a1d6afd182697a00697347a3a1af3dd16 Jul 17 17:31:53.638121 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.637636 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56bb5f997f-5jxmw"] Jul 17 17:31:53.640785 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.639982 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mhq7k"] Jul 17 17:31:53.641055 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:53.641030 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7a3c2f_74e0_4ae2_982a_5abf88bf41af.slice/crio-71caf01d809b463053edae2f876909918f2690175c811baa02e15675d83b4ce7 WatchSource:0}: Error finding container 71caf01d809b463053edae2f876909918f2690175c811baa02e15675d83b4ce7: Status 404 returned error can't find the container with id 71caf01d809b463053edae2f876909918f2690175c811baa02e15675d83b4ce7 Jul 17 17:31:53.642943 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:53.642912 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5657e768_43ad_48c2_b520_adf83fbab4d4.slice/crio-95c436fb39b301145872ec77c182b71824a57e4a7267e5ee2ab3ab820f13151f WatchSource:0}: Error finding container 95c436fb39b301145872ec77c182b71824a57e4a7267e5ee2ab3ab820f13151f: Status 404 returned error can't find the container with id 95c436fb39b301145872ec77c182b71824a57e4a7267e5ee2ab3ab820f13151f Jul 17 17:31:53.653377 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:53.653349 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-phcbk"] Jul 17 17:31:53.657589 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:31:53.657539 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ba0e1c_5d64_45ba_8db5_69fbd6715a14.slice/crio-730fb9de0111fb34383324ff332c0e58862da27cd918dc1f638b3e9f9cbec3ae WatchSource:0}: Error finding container 730fb9de0111fb34383324ff332c0e58862da27cd918dc1f638b3e9f9cbec3ae: Status 404 returned error can't find the container with id 730fb9de0111fb34383324ff332c0e58862da27cd918dc1f638b3e9f9cbec3ae Jul 17 17:31:54.457999 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:54.457965 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:31:54.460711 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:54.460689 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 17:31:54.461648 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:54.461628 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fwmkw\"" Jul 17 17:31:54.461731 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:54.461673 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 17:31:54.614635 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:54.614594 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mhq7k" event={"ID":"5657e768-43ad-48c2-b520-adf83fbab4d4","Type":"ContainerStarted","Data":"95c436fb39b301145872ec77c182b71824a57e4a7267e5ee2ab3ab820f13151f"} Jul 17 17:31:54.616177 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:54.616148 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" event={"ID":"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af","Type":"ContainerStarted","Data":"2360483bd23ab2868edde0234ba7d4eec5716f99bedb840fc8e557a6b39d0bbd"} Jul 17 17:31:54.616325 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:54.616192 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" event={"ID":"5f7a3c2f-74e0-4ae2-982a-5abf88bf41af","Type":"ContainerStarted","Data":"71caf01d809b463053edae2f876909918f2690175c811baa02e15675d83b4ce7"} Jul 17 17:31:54.616325 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:54.616265 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:31:54.617518 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:54.617486 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-phcbk" event={"ID":"66ba0e1c-5d64-45ba-8db5-69fbd6715a14","Type":"ContainerStarted","Data":"730fb9de0111fb34383324ff332c0e58862da27cd918dc1f638b3e9f9cbec3ae"} Jul 17 17:31:54.618562 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:54.618541 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8jchv" event={"ID":"8d6c365e-40fc-4cee-ad65-e984c0686bfe","Type":"ContainerStarted","Data":"ca11ed7cdbac83c8e7a34b3d423be27a1d6afd182697a00697347a3a1af3dd16"} Jul 17 17:31:54.634503 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:54.634457 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" podStartSLOduration=4.634435957 podStartE2EDuration="4.634435957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:31:54.633406038 +0000 UTC m=+35.752234335" watchObservedRunningTime="2026-07-17 17:31:54.634435957 +0000 UTC m=+35.753264240" Jul 17 17:31:56.318069 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:56.317751 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:56.371041 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:56.370962 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:56.627768 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:56.627601 2553 generic.go:356] "Generic (PLEG): container finished" podID="2fb88f09-39c9-4e4b-91c4-7c3d21b071ba" containerID="d9aa3307472d1b4d1697bae2f6064d299faab95f1b6537cc2b7e2502b1206a25" exitCode=0 Jul 17 17:31:56.627768 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:56.627667 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krv7b" event={"ID":"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba","Type":"ContainerDied","Data":"d9aa3307472d1b4d1697bae2f6064d299faab95f1b6537cc2b7e2502b1206a25"} Jul 17 17:31:57.633721 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:57.633683 2553 generic.go:356] "Generic (PLEG): container finished" podID="2fb88f09-39c9-4e4b-91c4-7c3d21b071ba" containerID="284feb14c8c90ed6ca0af9e88f44e0517685fcd651aba637b9fd52c95cbb68da" exitCode=0 Jul 17 17:31:57.634287 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:57.633756 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krv7b" event={"ID":"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba","Type":"ContainerDied","Data":"284feb14c8c90ed6ca0af9e88f44e0517685fcd651aba637b9fd52c95cbb68da"} Jul 17 17:31:57.923262 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:57.923179 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56598579b-6kwmh"] Jul 17 17:31:57.933116 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:57.933087 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:57.936638 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:57.936606 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 17 17:31:57.936774 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:57.936706 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qd42z\"" Jul 17 17:31:57.937466 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:57.937447 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 17 17:31:57.937814 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:57.937741 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 17 17:31:57.937814 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:57.937769 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 17 17:31:57.937814 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:57.937803 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 17 17:31:57.938032 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:57.937821 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 17 17:31:57.938032 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:57.937804 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 17 17:31:57.944644 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:57.944616 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56598579b-6kwmh"] Jul 17 17:31:57.948394 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:57.948374 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 17 17:31:58.091002 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.090968 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-oauth-serving-cert\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.091169 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.091069 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-console-serving-cert\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.091169 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.091091 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-trusted-ca-bundle\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.091281 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.091171 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7cxz\" (UniqueName: \"kubernetes.io/projected/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-kube-api-access-w7cxz\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.091281 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.091260 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-console-oauth-config\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.091382 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.091313 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-console-config\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.091382 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.091371 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-service-ca\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.192203 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.192121 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-console-oauth-config\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.192203 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.192165 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-console-config\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.192203 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.192195 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-service-ca\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.192466 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.192218 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-oauth-serving-cert\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.192466 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.192284 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-console-serving-cert\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.192466 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.192308 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-trusted-ca-bundle\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.192466 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.192361 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7cxz\" (UniqueName: \"kubernetes.io/projected/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-kube-api-access-w7cxz\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.193064 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.193032 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-console-config\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.193296 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.193278 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-oauth-serving-cert\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.193361 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.193318 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-service-ca\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.193829 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.193803 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-trusted-ca-bundle\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.196823 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.196804 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-console-oauth-config\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.197000 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.196978 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-console-serving-cert\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.200526 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.200503 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7cxz\" (UniqueName: \"kubernetes.io/projected/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-kube-api-access-w7cxz\") pod \"console-56598579b-6kwmh\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:58.243058 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:58.243023 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:31:59.034873 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:59.034821 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:59.324570 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:59.324482 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:31:59.397975 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:31:59.397935 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:00.930654 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:00.930608 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:01.034129 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:01.034045 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:01.621087 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:01.621041 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:01.676498 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:01.676433 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:01.807554 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:01.807519 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:01.812290 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:01.812267 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-cb46q"] Jul 17 17:32:01.832184 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:01.832162 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-cb46q"] Jul 17 17:32:01.832298 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:01.832255 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-cb46q" Jul 17 17:32:01.835218 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:01.835194 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 17 17:32:01.836418 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:01.836350 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lmq67\"" Jul 17 17:32:01.836418 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:01.836366 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 17 17:32:01.836614 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:01.836594 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 17 17:32:01.836694 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:01.836675 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 17 17:32:01.836749 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:01.836707 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 17 17:32:01.866716 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:01.866689 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:01.923347 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:01.922749 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73e0afbd-63ea-4c9a-ae7b-3db773f2605c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-cb46q\" (UID: \"73e0afbd-63ea-4c9a-ae7b-3db773f2605c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cb46q" Jul 17 17:32:01.923347 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:01.922808 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4px5\" (UniqueName: \"kubernetes.io/projected/73e0afbd-63ea-4c9a-ae7b-3db773f2605c-kube-api-access-k4px5\") pod \"prometheus-operator-5945b5cfd-cb46q\" (UID: \"73e0afbd-63ea-4c9a-ae7b-3db773f2605c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cb46q" Jul 17 17:32:01.923347 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:01.922841 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/73e0afbd-63ea-4c9a-ae7b-3db773f2605c-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-cb46q\" (UID: \"73e0afbd-63ea-4c9a-ae7b-3db773f2605c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cb46q" Jul 17 17:32:01.923347 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:01.922888 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73e0afbd-63ea-4c9a-ae7b-3db773f2605c-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-cb46q\" (UID: \"73e0afbd-63ea-4c9a-ae7b-3db773f2605c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cb46q" Jul 17 17:32:01.971458 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:01.971438 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56598579b-6kwmh"] Jul 17 17:32:01.973499 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:32:01.973476 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59d6ab6_e518_44b8_acc9_6df2260cb7ef.slice/crio-f1fbf705608defae6e7cdf87f82950d4fbb3efae154965303cae4304770f3159 WatchSource:0}: Error finding container f1fbf705608defae6e7cdf87f82950d4fbb3efae154965303cae4304770f3159: Status 404 returned error can't find the container with id f1fbf705608defae6e7cdf87f82950d4fbb3efae154965303cae4304770f3159 Jul 17 17:32:02.023458 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.023428 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73e0afbd-63ea-4c9a-ae7b-3db773f2605c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-cb46q\" (UID: \"73e0afbd-63ea-4c9a-ae7b-3db773f2605c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cb46q" Jul 17 17:32:02.023792 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.023472 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4px5\" (UniqueName: \"kubernetes.io/projected/73e0afbd-63ea-4c9a-ae7b-3db773f2605c-kube-api-access-k4px5\") pod \"prometheus-operator-5945b5cfd-cb46q\" (UID: \"73e0afbd-63ea-4c9a-ae7b-3db773f2605c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cb46q" Jul 17 17:32:02.023792 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.023491 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/73e0afbd-63ea-4c9a-ae7b-3db773f2605c-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-cb46q\" (UID: \"73e0afbd-63ea-4c9a-ae7b-3db773f2605c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cb46q" Jul 17 17:32:02.023792 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.023508 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73e0afbd-63ea-4c9a-ae7b-3db773f2605c-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-cb46q\" (UID: \"73e0afbd-63ea-4c9a-ae7b-3db773f2605c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cb46q" Jul 17 17:32:02.024170 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.024149 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73e0afbd-63ea-4c9a-ae7b-3db773f2605c-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-cb46q\" (UID: \"73e0afbd-63ea-4c9a-ae7b-3db773f2605c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cb46q" Jul 17 17:32:02.024254 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:32:02.024239 2553 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 17 17:32:02.024308 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:32:02.024297 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73e0afbd-63ea-4c9a-ae7b-3db773f2605c-prometheus-operator-tls podName:73e0afbd-63ea-4c9a-ae7b-3db773f2605c nodeName:}" failed. No retries permitted until 2026-07-17 17:32:02.524277956 +0000 UTC m=+43.643106228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/73e0afbd-63ea-4c9a-ae7b-3db773f2605c-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-cb46q" (UID: "73e0afbd-63ea-4c9a-ae7b-3db773f2605c") : secret "prometheus-operator-tls" not found Jul 17 17:32:02.026266 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.026248 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73e0afbd-63ea-4c9a-ae7b-3db773f2605c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-cb46q\" (UID: \"73e0afbd-63ea-4c9a-ae7b-3db773f2605c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cb46q" Jul 17 17:32:02.034450 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.034429 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4px5\" (UniqueName: \"kubernetes.io/projected/73e0afbd-63ea-4c9a-ae7b-3db773f2605c-kube-api-access-k4px5\") pod \"prometheus-operator-5945b5cfd-cb46q\" (UID: \"73e0afbd-63ea-4c9a-ae7b-3db773f2605c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cb46q" Jul 17 17:32:02.253407 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.253377 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:02.527909 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.527874 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/73e0afbd-63ea-4c9a-ae7b-3db773f2605c-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-cb46q\" (UID: \"73e0afbd-63ea-4c9a-ae7b-3db773f2605c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cb46q" Jul 17 17:32:02.530387 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.530363 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/73e0afbd-63ea-4c9a-ae7b-3db773f2605c-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-cb46q\" (UID: \"73e0afbd-63ea-4c9a-ae7b-3db773f2605c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cb46q" Jul 17 17:32:02.649425 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.649368 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mhq7k" event={"ID":"5657e768-43ad-48c2-b520-adf83fbab4d4","Type":"ContainerStarted","Data":"9c03109e775623cf744ae8ee3fa58844d14154a56ecadc1a4641aca5650025ef"} Jul 17 17:32:02.652988 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.652957 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krv7b" event={"ID":"2fb88f09-39c9-4e4b-91c4-7c3d21b071ba","Type":"ContainerStarted","Data":"b4ed7231fad52fd0f4b6d61e9ffa882be43e9936bfce2a718437af61cad31b88"} Jul 17 17:32:02.654838 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.654804 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-phcbk" event={"ID":"66ba0e1c-5d64-45ba-8db5-69fbd6715a14","Type":"ContainerStarted","Data":"a486f6a741b36e1403a4f46212b770ea821abf015195381d34dd981db5de19b8"} Jul 17 17:32:02.654965 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.654887 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-phcbk" Jul 17 17:32:02.654965 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.654912 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-phcbk" event={"ID":"66ba0e1c-5d64-45ba-8db5-69fbd6715a14","Type":"ContainerStarted","Data":"77340b11f184c45f7f9be00a2ed5439db7dd3b63679af6ccd9beba1721b2cba1"} Jul 17 17:32:02.656489 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.656465 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8jchv" event={"ID":"8d6c365e-40fc-4cee-ad65-e984c0686bfe","Type":"ContainerStarted","Data":"1e5e4dddd07a80fbd0c089b195b22e374b2a17a409a57457ccf8c5061d90d308"} Jul 17 17:32:02.657816 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.657793 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56598579b-6kwmh" event={"ID":"d59d6ab6-e518-44b8-acc9-6df2260cb7ef","Type":"ContainerStarted","Data":"f1fbf705608defae6e7cdf87f82950d4fbb3efae154965303cae4304770f3159"} Jul 17 17:32:02.676272 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.676170 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-krv7b" podStartSLOduration=20.333925228 podStartE2EDuration="43.676151348s" totalImagesPullingTime="23.34222612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:31:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:02.674594454 +0000 UTC m=+43.793422742" watchObservedRunningTime="2026-07-17 17:32:02.676151348 +0000 UTC m=+43.794979632" Jul 17 17:32:02.690063 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.690008 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8jchv" podStartSLOduration=1.493983715 podStartE2EDuration="9.689988499s" totalImagesPullingTime="8.196004784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:31:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:02.689010699 +0000 UTC m=+43.807838981" watchObservedRunningTime="2026-07-17 17:32:02.689988499 +0000 UTC m=+43.808816781" Jul 17 17:32:02.705446 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.705177 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-phcbk" podStartSLOduration=1.927354084 podStartE2EDuration="9.705155064s" totalImagesPullingTime="7.77780098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:31:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:02.705022188 +0000 UTC m=+43.823850469" watchObservedRunningTime="2026-07-17 17:32:02.705155064 +0000 UTC m=+43.823983347" Jul 17 17:32:02.753444 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.753414 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-cb46q" Jul 17 17:32:02.801934 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.801695 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:02.857022 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.856901 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:02.908891 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:02.908835 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-cb46q"] Jul 17 17:32:02.914020 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:32:02.913986 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e0afbd_63ea_4c9a_ae7b_3db773f2605c.slice/crio-062cfbef7035281c5cacfee20de3bb4860ea0a4917afdb1a573a1b204e85b937 WatchSource:0}: Error finding container 062cfbef7035281c5cacfee20de3bb4860ea0a4917afdb1a573a1b204e85b937: Status 404 returned error can't find the container with id 062cfbef7035281c5cacfee20de3bb4860ea0a4917afdb1a573a1b204e85b937 Jul 17 17:32:03.440661 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:03.440596 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:03.540154 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:03.540085 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:03.663875 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:03.663761 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-cb46q" event={"ID":"73e0afbd-63ea-4c9a-ae7b-3db773f2605c","Type":"ContainerStarted","Data":"062cfbef7035281c5cacfee20de3bb4860ea0a4917afdb1a573a1b204e85b937"} Jul 17 17:32:06.325052 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:06.325017 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:06.395013 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:06.394981 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:08.436166 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:08.436132 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:08.680048 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:08.679957 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mhq7k" event={"ID":"5657e768-43ad-48c2-b520-adf83fbab4d4","Type":"ContainerStarted","Data":"3379f3058970cb67aa0f838becfdf6da8590149a2529c4163b924db123faad3f"} Jul 17 17:32:08.681907 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:08.681876 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-cb46q" event={"ID":"73e0afbd-63ea-4c9a-ae7b-3db773f2605c","Type":"ContainerStarted","Data":"0b8b5cf7403c8ce9cb96cf9278ced7d8831f2243373e2732ab1603b4e92e391f"} Jul 17 17:32:08.682023 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:08.681925 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-cb46q" event={"ID":"73e0afbd-63ea-4c9a-ae7b-3db773f2605c","Type":"ContainerStarted","Data":"4ba4e16e46c8d867379d595f66ae7e812adf95ee663d4bf380e90dc85d51e7f9"} Jul 17 17:32:08.683325 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:08.683302 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56598579b-6kwmh" event={"ID":"d59d6ab6-e518-44b8-acc9-6df2260cb7ef","Type":"ContainerStarted","Data":"44e1b6f7c547357cffd02e38a08e6faed2aa9f485ece1a8c45c56f2975bf2841"} Jul 17 17:32:08.700828 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:08.700786 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-cb46q" podStartSLOduration=2.459687114 podStartE2EDuration="7.700773606s" totalImagesPullingTime="5.241086492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:32:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:08.699475141 +0000 UTC m=+49.818303432" watchObservedRunningTime="2026-07-17 17:32:08.700773606 +0000 UTC m=+49.819601878" Jul 17 17:32:08.719067 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:08.719027 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56598579b-6kwmh" podStartSLOduration=5.540816109 podStartE2EDuration="11.719011636s" totalImagesPullingTime="6.178195527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:08.717872248 +0000 UTC m=+49.836700522" watchObservedRunningTime="2026-07-17 17:32:08.719011636 +0000 UTC m=+49.837839895" Jul 17 17:32:09.918001 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:09.917959 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:10.147980 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.147947 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:10.175943 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.175878 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz"] Jul 17 17:32:10.188810 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.188786 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz" Jul 17 17:32:10.193552 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.193526 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 17 17:32:10.193944 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.193926 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-94clf\"" Jul 17 17:32:10.194591 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.194564 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz"] Jul 17 17:32:10.197861 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.197834 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 17 17:32:10.213090 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.213069 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-7nnmr"] Jul 17 17:32:10.226535 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.226517 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:10.230659 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.230636 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 17 17:32:10.230760 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.230645 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pqckw\"" Jul 17 17:32:10.232440 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.232291 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 17 17:32:10.232440 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.232302 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 17 17:32:10.233568 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.233538 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xrsqr"] Jul 17 17:32:10.244771 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.244750 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-7nnmr"] Jul 17 17:32:10.244913 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.244897 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.247543 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.247494 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sbz59\"" Jul 17 17:32:10.247687 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.247626 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 17 17:32:10.247687 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.247637 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 17 17:32:10.247806 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.247718 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 17 17:32:10.263715 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.263688 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:10.283679 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.283646 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5dnk\" (UniqueName: \"kubernetes.io/projected/084294fe-58d0-4e65-8405-f234886b8e9f-kube-api-access-r5dnk\") pod \"kube-state-metrics-56c5594674-7nnmr\" (UID: \"084294fe-58d0-4e65-8405-f234886b8e9f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:10.283820 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.283699 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/084294fe-58d0-4e65-8405-f234886b8e9f-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-7nnmr\" (UID: \"084294fe-58d0-4e65-8405-f234886b8e9f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:10.283820 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.283794 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/084294fe-58d0-4e65-8405-f234886b8e9f-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-7nnmr\" (UID: \"084294fe-58d0-4e65-8405-f234886b8e9f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:10.283960 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.283830 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/084294fe-58d0-4e65-8405-f234886b8e9f-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-7nnmr\" (UID: \"084294fe-58d0-4e65-8405-f234886b8e9f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:10.283960 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.283881 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/084294fe-58d0-4e65-8405-f234886b8e9f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-7nnmr\" (UID: \"084294fe-58d0-4e65-8405-f234886b8e9f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:10.283960 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.283929 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-tls\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.284116 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.283958 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-accelerators-collector-config\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.284116 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.283988 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-textfile\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.284116 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.284014 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5e65b113-df6f-45da-a128-5fa48d45f304-root\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.284116 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.284059 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7864aa4a-1e4e-449c-ab69-ac18ec25a4d6-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-kwgnz\" (UID: \"7864aa4a-1e4e-449c-ab69-ac18ec25a4d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz" Jul 17 17:32:10.284116 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.284082 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-wtmp\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.284313 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.284116 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mksv2\" (UniqueName: \"kubernetes.io/projected/7864aa4a-1e4e-449c-ab69-ac18ec25a4d6-kube-api-access-mksv2\") pod \"openshift-state-metrics-79f6958cdf-kwgnz\" (UID: \"7864aa4a-1e4e-449c-ab69-ac18ec25a4d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz" Jul 17 17:32:10.284313 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.284141 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e65b113-df6f-45da-a128-5fa48d45f304-sys\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.284313 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.284186 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.284313 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.284238 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/084294fe-58d0-4e65-8405-f234886b8e9f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-7nnmr\" (UID: \"084294fe-58d0-4e65-8405-f234886b8e9f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:10.284313 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.284268 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e65b113-df6f-45da-a128-5fa48d45f304-metrics-client-ca\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.284493 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.284313 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7864aa4a-1e4e-449c-ab69-ac18ec25a4d6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-kwgnz\" (UID: \"7864aa4a-1e4e-449c-ab69-ac18ec25a4d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz" Jul 17 17:32:10.284493 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.284333 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sjqj\" (UniqueName: \"kubernetes.io/projected/5e65b113-df6f-45da-a128-5fa48d45f304-kube-api-access-2sjqj\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.284493 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.284372 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7864aa4a-1e4e-449c-ab69-ac18ec25a4d6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-kwgnz\" (UID: \"7864aa4a-1e4e-449c-ab69-ac18ec25a4d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz" Jul 17 17:32:10.340721 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.340685 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:10.385116 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385079 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/084294fe-58d0-4e65-8405-f234886b8e9f-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-7nnmr\" (UID: \"084294fe-58d0-4e65-8405-f234886b8e9f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:10.385291 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385129 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/084294fe-58d0-4e65-8405-f234886b8e9f-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-7nnmr\" (UID: \"084294fe-58d0-4e65-8405-f234886b8e9f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:10.385291 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385154 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/084294fe-58d0-4e65-8405-f234886b8e9f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-7nnmr\" (UID: \"084294fe-58d0-4e65-8405-f234886b8e9f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:10.385291 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385184 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-tls\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.385291 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385215 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-accelerators-collector-config\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.385291 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385256 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-textfile\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.385291 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385281 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5e65b113-df6f-45da-a128-5fa48d45f304-root\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.385596 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385382 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5e65b113-df6f-45da-a128-5fa48d45f304-root\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.385596 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:32:10.385483 2553 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 17 17:32:10.385596 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:32:10.385540 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-tls podName:5e65b113-df6f-45da-a128-5fa48d45f304 nodeName:}" failed. No retries permitted until 2026-07-17 17:32:10.885520654 +0000 UTC m=+52.004348922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-tls") pod "node-exporter-xrsqr" (UID: "5e65b113-df6f-45da-a128-5fa48d45f304") : secret "node-exporter-tls" not found Jul 17 17:32:10.385596 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385540 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/084294fe-58d0-4e65-8405-f234886b8e9f-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-7nnmr\" (UID: \"084294fe-58d0-4e65-8405-f234886b8e9f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:10.385806 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385630 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-textfile\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.385806 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385667 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7864aa4a-1e4e-449c-ab69-ac18ec25a4d6-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-kwgnz\" (UID: \"7864aa4a-1e4e-449c-ab69-ac18ec25a4d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz" Jul 17 17:32:10.385806 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385702 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-wtmp\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.385806 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385735 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mksv2\" (UniqueName: \"kubernetes.io/projected/7864aa4a-1e4e-449c-ab69-ac18ec25a4d6-kube-api-access-mksv2\") pod \"openshift-state-metrics-79f6958cdf-kwgnz\" (UID: \"7864aa4a-1e4e-449c-ab69-ac18ec25a4d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz" Jul 17 17:32:10.385806 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385751 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e65b113-df6f-45da-a128-5fa48d45f304-sys\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.385806 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385770 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.385806 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385794 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/084294fe-58d0-4e65-8405-f234886b8e9f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-7nnmr\" (UID: \"084294fe-58d0-4e65-8405-f234886b8e9f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:10.386157 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385814 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e65b113-df6f-45da-a128-5fa48d45f304-metrics-client-ca\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.386157 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385826 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/084294fe-58d0-4e65-8405-f234886b8e9f-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-7nnmr\" (UID: \"084294fe-58d0-4e65-8405-f234886b8e9f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:10.386157 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385831 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-accelerators-collector-config\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.386157 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385868 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7864aa4a-1e4e-449c-ab69-ac18ec25a4d6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-kwgnz\" (UID: \"7864aa4a-1e4e-449c-ab69-ac18ec25a4d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz" Jul 17 17:32:10.386157 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385879 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e65b113-df6f-45da-a128-5fa48d45f304-sys\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.386157 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385911 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sjqj\" (UniqueName: \"kubernetes.io/projected/5e65b113-df6f-45da-a128-5fa48d45f304-kube-api-access-2sjqj\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.386157 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385968 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7864aa4a-1e4e-449c-ab69-ac18ec25a4d6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-kwgnz\" (UID: \"7864aa4a-1e4e-449c-ab69-ac18ec25a4d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz" Jul 17 17:32:10.386157 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385966 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/084294fe-58d0-4e65-8405-f234886b8e9f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-7nnmr\" (UID: \"084294fe-58d0-4e65-8405-f234886b8e9f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:10.386157 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385975 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-wtmp\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.386157 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.385998 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5dnk\" (UniqueName: \"kubernetes.io/projected/084294fe-58d0-4e65-8405-f234886b8e9f-kube-api-access-r5dnk\") pod \"kube-state-metrics-56c5594674-7nnmr\" (UID: \"084294fe-58d0-4e65-8405-f234886b8e9f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:10.386157 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.386037 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/084294fe-58d0-4e65-8405-f234886b8e9f-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-7nnmr\" (UID: \"084294fe-58d0-4e65-8405-f234886b8e9f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:10.386157 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:32:10.386160 2553 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 17 17:32:10.386693 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:32:10.386214 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/084294fe-58d0-4e65-8405-f234886b8e9f-kube-state-metrics-tls podName:084294fe-58d0-4e65-8405-f234886b8e9f nodeName:}" failed. No retries permitted until 2026-07-17 17:32:10.886200006 +0000 UTC m=+52.005028269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/084294fe-58d0-4e65-8405-f234886b8e9f-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-7nnmr" (UID: "084294fe-58d0-4e65-8405-f234886b8e9f") : secret "kube-state-metrics-tls" not found Jul 17 17:32:10.386693 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:32:10.386275 2553 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 17 17:32:10.386693 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:32:10.386313 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7864aa4a-1e4e-449c-ab69-ac18ec25a4d6-openshift-state-metrics-tls podName:7864aa4a-1e4e-449c-ab69-ac18ec25a4d6 nodeName:}" failed. No retries permitted until 2026-07-17 17:32:10.886300091 +0000 UTC m=+52.005128354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/7864aa4a-1e4e-449c-ab69-ac18ec25a4d6-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-kwgnz" (UID: "7864aa4a-1e4e-449c-ab69-ac18ec25a4d6") : secret "openshift-state-metrics-tls" not found Jul 17 17:32:10.386693 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.386366 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7864aa4a-1e4e-449c-ab69-ac18ec25a4d6-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-kwgnz\" (UID: \"7864aa4a-1e4e-449c-ab69-ac18ec25a4d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz" Jul 17 17:32:10.386693 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.386456 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e65b113-df6f-45da-a128-5fa48d45f304-metrics-client-ca\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.388428 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.388401 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/084294fe-58d0-4e65-8405-f234886b8e9f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-7nnmr\" (UID: \"084294fe-58d0-4e65-8405-f234886b8e9f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:10.388521 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.388504 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7864aa4a-1e4e-449c-ab69-ac18ec25a4d6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-kwgnz\" (UID: \"7864aa4a-1e4e-449c-ab69-ac18ec25a4d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz" Jul 17 17:32:10.388521 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.388509 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.396691 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.396669 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5dnk\" (UniqueName: \"kubernetes.io/projected/084294fe-58d0-4e65-8405-f234886b8e9f-kube-api-access-r5dnk\") pod \"kube-state-metrics-56c5594674-7nnmr\" (UID: \"084294fe-58d0-4e65-8405-f234886b8e9f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:10.400310 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.400288 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sjqj\" (UniqueName: \"kubernetes.io/projected/5e65b113-df6f-45da-a128-5fa48d45f304-kube-api-access-2sjqj\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.400431 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.400292 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mksv2\" (UniqueName: \"kubernetes.io/projected/7864aa4a-1e4e-449c-ab69-ac18ec25a4d6-kube-api-access-mksv2\") pod \"openshift-state-metrics-79f6958cdf-kwgnz\" (UID: \"7864aa4a-1e4e-449c-ab69-ac18ec25a4d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz" Jul 17 17:32:10.545227 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.545185 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:10.653071 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.653036 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:10.806083 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.805998 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:10.891064 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.891029 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7864aa4a-1e4e-449c-ab69-ac18ec25a4d6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-kwgnz\" (UID: \"7864aa4a-1e4e-449c-ab69-ac18ec25a4d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz" Jul 17 17:32:10.891217 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.891084 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/084294fe-58d0-4e65-8405-f234886b8e9f-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-7nnmr\" (UID: \"084294fe-58d0-4e65-8405-f234886b8e9f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:10.891217 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.891140 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-tls\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:10.891333 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:32:10.891288 2553 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 17 17:32:10.891386 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:32:10.891360 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-tls podName:5e65b113-df6f-45da-a128-5fa48d45f304 nodeName:}" failed. No retries permitted until 2026-07-17 17:32:11.891341286 +0000 UTC m=+53.010169550 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-tls") pod "node-exporter-xrsqr" (UID: "5e65b113-df6f-45da-a128-5fa48d45f304") : secret "node-exporter-tls" not found Jul 17 17:32:10.893744 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.893684 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/084294fe-58d0-4e65-8405-f234886b8e9f-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-7nnmr\" (UID: \"084294fe-58d0-4e65-8405-f234886b8e9f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:10.893744 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.893697 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7864aa4a-1e4e-449c-ab69-ac18ec25a4d6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-kwgnz\" (UID: \"7864aa4a-1e4e-449c-ab69-ac18ec25a4d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz" Jul 17 17:32:10.918040 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:10.918008 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:11.032589 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.032541 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:11.098626 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.098544 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz" Jul 17 17:32:11.136584 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.136542 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" Jul 17 17:32:11.172049 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.172009 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:11.272156 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.272080 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:11.277831 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.277807 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:32:11.324090 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.323003 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:32:11.324090 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.323176 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.327280 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.327253 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 17 17:32:11.327410 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.327259 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 17 17:32:11.328634 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.328389 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-crdtf\"" Jul 17 17:32:11.328634 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.328422 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 17 17:32:11.328634 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.328476 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 17 17:32:11.328634 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.328488 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 17 17:32:11.328634 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.328520 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 17 17:32:11.328634 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.328480 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 17 17:32:11.329011 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.328741 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 17 17:32:11.335184 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.335165 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 17 17:32:11.382375 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.382278 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:11.396266 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.396237 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.396266 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.396274 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97cc2052-a3b0-4457-bf60-f48d33225882-config-out\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.396464 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.396291 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.396464 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.396379 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.396464 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.396450 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97cc2052-a3b0-4457-bf60-f48d33225882-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.396595 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.396495 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/97cc2052-a3b0-4457-bf60-f48d33225882-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.396595 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.396531 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-web-config\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.396595 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.396555 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97cc2052-a3b0-4457-bf60-f48d33225882-tls-assets\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.396595 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.396591 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.396762 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.396670 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.396762 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.396693 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jg8t\" (UniqueName: \"kubernetes.io/projected/97cc2052-a3b0-4457-bf60-f48d33225882-kube-api-access-7jg8t\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.396762 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.396741 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97cc2052-a3b0-4457-bf60-f48d33225882-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.396762 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.396760 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-config-volume\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.481884 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.481798 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:11.497507 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.497480 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.497671 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.497527 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.497671 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.497549 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jg8t\" (UniqueName: \"kubernetes.io/projected/97cc2052-a3b0-4457-bf60-f48d33225882-kube-api-access-7jg8t\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.497671 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.497604 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97cc2052-a3b0-4457-bf60-f48d33225882-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.497671 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.497644 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-config-volume\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.497888 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.497677 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.497888 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.497707 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97cc2052-a3b0-4457-bf60-f48d33225882-config-out\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.497888 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.497733 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.497888 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.497778 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.497888 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.497823 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97cc2052-a3b0-4457-bf60-f48d33225882-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.498226 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.497912 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/97cc2052-a3b0-4457-bf60-f48d33225882-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.498226 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.497954 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-web-config\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.498226 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.497980 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97cc2052-a3b0-4457-bf60-f48d33225882-tls-assets\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.500785 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.499308 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97cc2052-a3b0-4457-bf60-f48d33225882-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.500785 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.500146 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/97cc2052-a3b0-4457-bf60-f48d33225882-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.500785 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.500742 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-config-volume\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.501680 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.501632 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97cc2052-a3b0-4457-bf60-f48d33225882-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.502028 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.502006 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97cc2052-a3b0-4457-bf60-f48d33225882-config-out\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.502111 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.502090 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.502322 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.502247 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97cc2052-a3b0-4457-bf60-f48d33225882-tls-assets\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.502472 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.502452 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.502648 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.502612 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-web-config\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.502768 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.502749 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.502768 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.502755 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.504523 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.504503 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.513230 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.513211 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jg8t\" (UniqueName: \"kubernetes.io/projected/97cc2052-a3b0-4457-bf60-f48d33225882-kube-api-access-7jg8t\") pod \"alertmanager-main-0\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.634223 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.634137 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:32:11.829798 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.829762 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:11.900583 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.900505 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-tls\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:11.902694 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.902666 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e65b113-df6f-45da-a128-5fa48d45f304-node-exporter-tls\") pod \"node-exporter-xrsqr\" (UID: \"5e65b113-df6f-45da-a128-5fa48d45f304\") " pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:11.930686 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:11.930660 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:12.044687 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:12.042978 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:12.054092 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:12.054062 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xrsqr" Jul 17 17:32:12.079329 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:32:12.079295 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e65b113_df6f_45da_a128_5fa48d45f304.slice/crio-76d6fde599499000fd9345259fc5daab1f181dc2cfc31ab795c282da49ea945a WatchSource:0}: Error finding container 76d6fde599499000fd9345259fc5daab1f181dc2cfc31ab795c282da49ea945a: Status 404 returned error can't find the container with id 76d6fde599499000fd9345259fc5daab1f181dc2cfc31ab795c282da49ea945a Jul 17 17:32:12.132387 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:12.132350 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz"] Jul 17 17:32:12.134962 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:32:12.134940 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7864aa4a_1e4e_449c_ab69_ac18ec25a4d6.slice/crio-ade13b7dbdaa593dc6a6297b2450d71be54211e53288ec728eb250a5adae5382 WatchSource:0}: Error finding container ade13b7dbdaa593dc6a6297b2450d71be54211e53288ec728eb250a5adae5382: Status 404 returned error can't find the container with id ade13b7dbdaa593dc6a6297b2450d71be54211e53288ec728eb250a5adae5382 Jul 17 17:32:12.345443 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:12.345387 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-7nnmr"] Jul 17 17:32:12.354268 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:12.354097 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:32:12.379192 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:32:12.379152 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084294fe_58d0_4e65_8405_f234886b8e9f.slice/crio-eece16df09a0b014a9f9fe8a778aad2922702b43a9b349f01c69051adbb1a163 WatchSource:0}: Error finding container eece16df09a0b014a9f9fe8a778aad2922702b43a9b349f01c69051adbb1a163: Status 404 returned error can't find the container with id eece16df09a0b014a9f9fe8a778aad2922702b43a9b349f01c69051adbb1a163 Jul 17 17:32:12.379777 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:32:12.379753 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97cc2052_a3b0_4457_bf60_f48d33225882.slice/crio-43655644b50859ce6a8b4f7dd26ff243e0f026d984bd31135e3b2e9948a8446e WatchSource:0}: Error finding container 43655644b50859ce6a8b4f7dd26ff243e0f026d984bd31135e3b2e9948a8446e: Status 404 returned error can't find the container with id 43655644b50859ce6a8b4f7dd26ff243e0f026d984bd31135e3b2e9948a8446e Jul 17 17:32:12.495275 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:12.495232 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:12.597368 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:12.597328 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:12.665638 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:12.665612 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-phcbk" Jul 17 17:32:12.699738 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:12.699626 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97cc2052-a3b0-4457-bf60-f48d33225882","Type":"ContainerStarted","Data":"43655644b50859ce6a8b4f7dd26ff243e0f026d984bd31135e3b2e9948a8446e"} Jul 17 17:32:12.702633 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:12.702496 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mhq7k" event={"ID":"5657e768-43ad-48c2-b520-adf83fbab4d4","Type":"ContainerStarted","Data":"2f8f65f06c5383c338608166c2216ba2dd8123b28d31c2a837f1ab0408ee1e2e"} Jul 17 17:32:12.709698 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:12.709671 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" event={"ID":"084294fe-58d0-4e65-8405-f234886b8e9f","Type":"ContainerStarted","Data":"eece16df09a0b014a9f9fe8a778aad2922702b43a9b349f01c69051adbb1a163"} Jul 17 17:32:12.711665 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:12.711639 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz" event={"ID":"7864aa4a-1e4e-449c-ab69-ac18ec25a4d6","Type":"ContainerStarted","Data":"7d503f9814c02d605f74d411a614d6afbc2447117046f5eacc69bb984caaccff"} Jul 17 17:32:12.711778 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:12.711687 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz" event={"ID":"7864aa4a-1e4e-449c-ab69-ac18ec25a4d6","Type":"ContainerStarted","Data":"546487052d1ae0189db454d3d7aa52495ec486b6c6f15659a11a56a144b580c9"} Jul 17 17:32:12.711778 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:12.711707 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz" event={"ID":"7864aa4a-1e4e-449c-ab69-ac18ec25a4d6","Type":"ContainerStarted","Data":"ade13b7dbdaa593dc6a6297b2450d71be54211e53288ec728eb250a5adae5382"} Jul 17 17:32:12.713021 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:12.712998 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xrsqr" event={"ID":"5e65b113-df6f-45da-a128-5fa48d45f304","Type":"ContainerStarted","Data":"76d6fde599499000fd9345259fc5daab1f181dc2cfc31ab795c282da49ea945a"} Jul 17 17:32:12.729468 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:12.729397 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mhq7k" podStartSLOduration=1.13850516 podStartE2EDuration="19.729376772s" totalImagesPullingTime="18.590871612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:31:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:12.727342809 +0000 UTC m=+53.846171083" watchObservedRunningTime="2026-07-17 17:32:12.729376772 +0000 UTC m=+53.848205054" Jul 17 17:32:13.082160 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.082122 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:13.181645 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.181588 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:13.207029 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.207000 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6bf948c56c-shknk"] Jul 17 17:32:13.209667 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.209645 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.212151 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.212098 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 17 17:32:13.212272 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.212179 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 17 17:32:13.212272 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.212259 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 17 17:32:13.212412 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.212395 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 17 17:32:13.212697 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.212520 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7ih7e754vem8v\"" Jul 17 17:32:13.212697 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.212617 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 17 17:32:13.212697 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.212680 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lbwpn\"" Jul 17 17:32:13.223629 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.223607 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bf948c56c-shknk"] Jul 17 17:32:13.315231 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.315142 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/faa4f800-8ebb-4d34-9920-41545b5a3b4b-secret-thanos-querier-tls\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.315231 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.315193 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/faa4f800-8ebb-4d34-9920-41545b5a3b4b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.315231 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.315225 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/faa4f800-8ebb-4d34-9920-41545b5a3b4b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.315485 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.315253 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/faa4f800-8ebb-4d34-9920-41545b5a3b4b-metrics-client-ca\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.315485 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.315313 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/faa4f800-8ebb-4d34-9920-41545b5a3b4b-secret-grpc-tls\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.315485 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.315338 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm48z\" (UniqueName: \"kubernetes.io/projected/faa4f800-8ebb-4d34-9920-41545b5a3b4b-kube-api-access-tm48z\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.315485 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.315386 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/faa4f800-8ebb-4d34-9920-41545b5a3b4b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.315669 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.315509 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/faa4f800-8ebb-4d34-9920-41545b5a3b4b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.328910 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.328814 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:13.416956 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.416924 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/faa4f800-8ebb-4d34-9920-41545b5a3b4b-secret-grpc-tls\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.416956 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.416961 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm48z\" (UniqueName: \"kubernetes.io/projected/faa4f800-8ebb-4d34-9920-41545b5a3b4b-kube-api-access-tm48z\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.417204 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.417003 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/faa4f800-8ebb-4d34-9920-41545b5a3b4b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.417204 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.417044 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/faa4f800-8ebb-4d34-9920-41545b5a3b4b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.417204 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.417101 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/faa4f800-8ebb-4d34-9920-41545b5a3b4b-secret-thanos-querier-tls\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.417204 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.417135 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/faa4f800-8ebb-4d34-9920-41545b5a3b4b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.417204 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.417165 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/faa4f800-8ebb-4d34-9920-41545b5a3b4b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.417204 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.417191 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/faa4f800-8ebb-4d34-9920-41545b5a3b4b-metrics-client-ca\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.417995 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.417937 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/faa4f800-8ebb-4d34-9920-41545b5a3b4b-metrics-client-ca\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.420216 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.420148 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/faa4f800-8ebb-4d34-9920-41545b5a3b4b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.420357 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.420287 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/faa4f800-8ebb-4d34-9920-41545b5a3b4b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.420428 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.420372 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/faa4f800-8ebb-4d34-9920-41545b5a3b4b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.420646 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.420615 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/faa4f800-8ebb-4d34-9920-41545b5a3b4b-secret-grpc-tls\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.420732 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.420648 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/faa4f800-8ebb-4d34-9920-41545b5a3b4b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.420786 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.420750 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/faa4f800-8ebb-4d34-9920-41545b5a3b4b-secret-thanos-querier-tls\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.427060 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.427038 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm48z\" (UniqueName: \"kubernetes.io/projected/faa4f800-8ebb-4d34-9920-41545b5a3b4b-kube-api-access-tm48z\") pod \"thanos-querier-6bf948c56c-shknk\" (UID: \"faa4f800-8ebb-4d34-9920-41545b5a3b4b\") " pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.430748 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.430721 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:13.521336 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.521298 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:13.718547 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.718465 2553 generic.go:356] "Generic (PLEG): container finished" podID="5e65b113-df6f-45da-a128-5fa48d45f304" containerID="a94c1c7f1a0f6fa2cb191ca98ec8e81060cb7fdf0c42e1d36c1015e6f819c663" exitCode=0 Jul 17 17:32:13.718708 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.718557 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xrsqr" event={"ID":"5e65b113-df6f-45da-a128-5fa48d45f304","Type":"ContainerDied","Data":"a94c1c7f1a0f6fa2cb191ca98ec8e81060cb7fdf0c42e1d36c1015e6f819c663"} Jul 17 17:32:13.877892 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:13.877841 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:14.205903 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.205866 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bf948c56c-shknk"] Jul 17 17:32:14.627169 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.627084 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6f8f95d894-jn6s5"] Jul 17 17:32:14.631128 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.629770 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.632750 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.632726 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 17 17:32:14.632937 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.632917 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-qpvcn\"" Jul 17 17:32:14.633019 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.632952 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 17 17:32:14.633924 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.633906 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 17 17:32:14.633924 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.633913 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 17 17:32:14.634110 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.633974 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2i67oqu7i2m6l\"" Jul 17 17:32:14.642899 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.642843 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6f8f95d894-jn6s5"] Jul 17 17:32:14.724467 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.724435 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xrsqr" event={"ID":"5e65b113-df6f-45da-a128-5fa48d45f304","Type":"ContainerStarted","Data":"ba4b1d64f8b5efd1f8128377c75c2956e1a8f03c70273dc4652aee6d4f564dd8"} Jul 17 17:32:14.724467 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.724462 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:14.724689 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.724504 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xrsqr" event={"ID":"5e65b113-df6f-45da-a128-5fa48d45f304","Type":"ContainerStarted","Data":"deb26ba8cfacaefa5e4cbd44184760291b4da4c5d6497e8b7aab26737027ea97"} Jul 17 17:32:14.726090 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.726067 2553 generic.go:356] "Generic (PLEG): container finished" podID="97cc2052-a3b0-4457-bf60-f48d33225882" containerID="46d591a9a8f2b7e3152764b4e5b0a9f6c0d7d7cdcab9672d422e7492b87be920" exitCode=0 Jul 17 17:32:14.726207 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.726127 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97cc2052-a3b0-4457-bf60-f48d33225882","Type":"ContainerDied","Data":"46d591a9a8f2b7e3152764b4e5b0a9f6c0d7d7cdcab9672d422e7492b87be920"} Jul 17 17:32:14.727389 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.727367 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" event={"ID":"faa4f800-8ebb-4d34-9920-41545b5a3b4b","Type":"ContainerStarted","Data":"ca305dd961b63e6c5739fb4feb2dc1d60248cf38d209ffe9f0753f296c999720"} Jul 17 17:32:14.728954 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.728934 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/93703e27-00df-46bb-a91b-c09de578a2f7-secret-metrics-server-client-certs\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.729020 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.728991 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/93703e27-00df-46bb-a91b-c09de578a2f7-metrics-server-audit-profiles\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.729070 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.729056 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/93703e27-00df-46bb-a91b-c09de578a2f7-secret-metrics-server-tls\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.729128 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.729092 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93703e27-00df-46bb-a91b-c09de578a2f7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.729128 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.729121 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/93703e27-00df-46bb-a91b-c09de578a2f7-audit-log\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.729287 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.729152 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/93703e27-00df-46bb-a91b-c09de578a2f7-client-ca-bundle\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.729287 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.729177 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nddwh\" (UniqueName: \"kubernetes.io/projected/93703e27-00df-46bb-a91b-c09de578a2f7-kube-api-access-nddwh\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.729835 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.729815 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" event={"ID":"084294fe-58d0-4e65-8405-f234886b8e9f","Type":"ContainerStarted","Data":"3db316eceeeab2c6b1b6a69a013a0222743b6bab5f1b49986ac5cd1306b2581e"} Jul 17 17:32:14.729925 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.729886 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" event={"ID":"084294fe-58d0-4e65-8405-f234886b8e9f","Type":"ContainerStarted","Data":"dd1278e57d9f1728d943f7916b0e788a5e8f1f96e692c7f0c361d1cbc8b4a6b1"} Jul 17 17:32:14.729925 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.729919 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" event={"ID":"084294fe-58d0-4e65-8405-f234886b8e9f","Type":"ContainerStarted","Data":"c78888b4a271749db2f697abd6edbf336c5487caf7284594c7344f4e57520cac"} Jul 17 17:32:14.731793 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.731772 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz" event={"ID":"7864aa4a-1e4e-449c-ab69-ac18ec25a4d6","Type":"ContainerStarted","Data":"18694c391e52b33ae07dfe61300f6d50724ecba01574f651c3c42b6105509ecc"} Jul 17 17:32:14.751810 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.751762 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xrsqr" podStartSLOduration=3.758675624 podStartE2EDuration="4.751747277s" totalImagesPullingTime="918.579162ms" totalInitContainerRuntime="74.492491ms" isStatefulPod=true podCreationTimestamp="2026-07-17 17:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:14.750218462 +0000 UTC m=+55.869046743" watchObservedRunningTime="2026-07-17 17:32:14.751747277 +0000 UTC m=+55.870575552" Jul 17 17:32:14.813632 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.813584 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-7nnmr" podStartSLOduration=3.189335642 podStartE2EDuration="4.813568372s" totalImagesPullingTime="1.62423273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:14.812394177 +0000 UTC m=+55.931222454" watchObservedRunningTime="2026-07-17 17:32:14.813568372 +0000 UTC m=+55.932396652" Jul 17 17:32:14.831115 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.830702 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/93703e27-00df-46bb-a91b-c09de578a2f7-secret-metrics-server-client-certs\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.831115 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.830890 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/93703e27-00df-46bb-a91b-c09de578a2f7-metrics-server-audit-profiles\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.831115 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.831009 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/93703e27-00df-46bb-a91b-c09de578a2f7-secret-metrics-server-tls\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.831115 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.831102 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93703e27-00df-46bb-a91b-c09de578a2f7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.831415 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.831164 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/93703e27-00df-46bb-a91b-c09de578a2f7-audit-log\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.831415 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.831214 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/93703e27-00df-46bb-a91b-c09de578a2f7-client-ca-bundle\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.831415 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.831241 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nddwh\" (UniqueName: \"kubernetes.io/projected/93703e27-00df-46bb-a91b-c09de578a2f7-kube-api-access-nddwh\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.831548 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.831489 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/93703e27-00df-46bb-a91b-c09de578a2f7-audit-log\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.831819 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.831729 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93703e27-00df-46bb-a91b-c09de578a2f7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.831997 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.831974 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/93703e27-00df-46bb-a91b-c09de578a2f7-metrics-server-audit-profiles\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.833445 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.833424 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/93703e27-00df-46bb-a91b-c09de578a2f7-secret-metrics-server-client-certs\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.833578 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.833561 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/93703e27-00df-46bb-a91b-c09de578a2f7-client-ca-bundle\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.833619 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.833592 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/93703e27-00df-46bb-a91b-c09de578a2f7-secret-metrics-server-tls\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.840276 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.840241 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kwgnz" podStartSLOduration=3.330528166 podStartE2EDuration="4.84022783s" totalImagesPullingTime="1.509699664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:14.83861275 +0000 UTC m=+55.957441026" watchObservedRunningTime="2026-07-17 17:32:14.84022783 +0000 UTC m=+55.959056111" Jul 17 17:32:14.842630 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.842614 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nddwh\" (UniqueName: \"kubernetes.io/projected/93703e27-00df-46bb-a91b-c09de578a2f7-kube-api-access-nddwh\") pod \"metrics-server-6f8f95d894-jn6s5\" (UID: \"93703e27-00df-46bb-a91b-c09de578a2f7\") " pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:14.940979 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:14.940902 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:15.002432 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.002398 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:15.014842 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.014804 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-6ft8n"] Jul 17 17:32:15.017459 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.017437 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56598579b-6kwmh"] Jul 17 17:32:15.017584 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.017541 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-6ft8n" Jul 17 17:32:15.024392 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.024010 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-f5nwh\"" Jul 17 17:32:15.024392 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.024256 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 17 17:32:15.033595 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.032905 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-6ft8n"] Jul 17 17:32:15.091349 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.091328 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6f8f95d894-jn6s5"] Jul 17 17:32:15.094357 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:32:15.094328 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93703e27_00df_46bb_a91b_c09de578a2f7.slice/crio-e27673986fc10fa8c04e4d07337c05aa21e472ec087a43ea5fd5aaf3fdd72d34 WatchSource:0}: Error finding container e27673986fc10fa8c04e4d07337c05aa21e472ec087a43ea5fd5aaf3fdd72d34: Status 404 returned error can't find the container with id e27673986fc10fa8c04e4d07337c05aa21e472ec087a43ea5fd5aaf3fdd72d34 Jul 17 17:32:15.134774 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.134740 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59604b04-8411-4c2b-98db-db26c020c9fa-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-6ft8n\" (UID: \"59604b04-8411-4c2b-98db-db26c020c9fa\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-6ft8n" Jul 17 17:32:15.134774 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.134775 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj9nv\" (UniqueName: \"kubernetes.io/projected/59604b04-8411-4c2b-98db-db26c020c9fa-kube-api-access-hj9nv\") pod \"monitoring-plugin-7b7b7bc8bf-6ft8n\" (UID: \"59604b04-8411-4c2b-98db-db26c020c9fa\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-6ft8n" Jul 17 17:32:15.165514 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.165487 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:15.236373 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.236346 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59604b04-8411-4c2b-98db-db26c020c9fa-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-6ft8n\" (UID: \"59604b04-8411-4c2b-98db-db26c020c9fa\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-6ft8n" Jul 17 17:32:15.236809 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.236381 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj9nv\" (UniqueName: \"kubernetes.io/projected/59604b04-8411-4c2b-98db-db26c020c9fa-kube-api-access-hj9nv\") pod \"monitoring-plugin-7b7b7bc8bf-6ft8n\" (UID: \"59604b04-8411-4c2b-98db-db26c020c9fa\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-6ft8n" Jul 17 17:32:15.236809 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:32:15.236514 2553 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 17 17:32:15.236809 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:32:15.236593 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59604b04-8411-4c2b-98db-db26c020c9fa-monitoring-plugin-cert podName:59604b04-8411-4c2b-98db-db26c020c9fa nodeName:}" failed. No retries permitted until 2026-07-17 17:32:15.736575632 +0000 UTC m=+56.855403892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/59604b04-8411-4c2b-98db-db26c020c9fa-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-6ft8n" (UID: "59604b04-8411-4c2b-98db-db26c020c9fa") : secret "monitoring-plugin-cert" not found Jul 17 17:32:15.251339 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.251316 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj9nv\" (UniqueName: \"kubernetes.io/projected/59604b04-8411-4c2b-98db-db26c020c9fa-kube-api-access-hj9nv\") pod \"monitoring-plugin-7b7b7bc8bf-6ft8n\" (UID: \"59604b04-8411-4c2b-98db-db26c020c9fa\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-6ft8n" Jul 17 17:32:15.469570 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.469517 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-66ffb784d-rg7z6"] Jul 17 17:32:15.472297 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.472275 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.475066 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.475044 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-9q58w\"" Jul 17 17:32:15.475218 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.475043 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 17 17:32:15.475323 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.475305 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 17 17:32:15.475781 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.475764 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 17 17:32:15.475885 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.475834 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 17 17:32:15.478590 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.478567 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 17 17:32:15.481117 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.481098 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 17 17:32:15.493278 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.493214 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-66ffb784d-rg7z6"] Jul 17 17:32:15.539272 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.539234 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f70096da-1921-4a36-b12b-082bcdf6d55d-secret-telemeter-client\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.539429 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.539319 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f70096da-1921-4a36-b12b-082bcdf6d55d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.539429 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.539357 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mftmk\" (UniqueName: \"kubernetes.io/projected/f70096da-1921-4a36-b12b-082bcdf6d55d-kube-api-access-mftmk\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.539524 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.539437 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f70096da-1921-4a36-b12b-082bcdf6d55d-federate-client-tls\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.539524 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.539498 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f70096da-1921-4a36-b12b-082bcdf6d55d-telemeter-client-tls\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.539618 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.539530 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70096da-1921-4a36-b12b-082bcdf6d55d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.539618 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.539560 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70096da-1921-4a36-b12b-082bcdf6d55d-serving-certs-ca-bundle\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.539711 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.539637 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f70096da-1921-4a36-b12b-082bcdf6d55d-metrics-client-ca\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.626058 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.626030 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56bb5f997f-5jxmw" Jul 17 17:32:15.640429 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.640397 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mftmk\" (UniqueName: \"kubernetes.io/projected/f70096da-1921-4a36-b12b-082bcdf6d55d-kube-api-access-mftmk\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.640599 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.640480 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f70096da-1921-4a36-b12b-082bcdf6d55d-federate-client-tls\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.640599 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.640552 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f70096da-1921-4a36-b12b-082bcdf6d55d-telemeter-client-tls\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.640599 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.640593 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70096da-1921-4a36-b12b-082bcdf6d55d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.640767 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.640637 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70096da-1921-4a36-b12b-082bcdf6d55d-serving-certs-ca-bundle\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.640767 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.640672 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f70096da-1921-4a36-b12b-082bcdf6d55d-metrics-client-ca\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.640767 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.640718 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f70096da-1921-4a36-b12b-082bcdf6d55d-secret-telemeter-client\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.640937 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.640783 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f70096da-1921-4a36-b12b-082bcdf6d55d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.641579 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.641523 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70096da-1921-4a36-b12b-082bcdf6d55d-serving-certs-ca-bundle\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.641689 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.641589 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f70096da-1921-4a36-b12b-082bcdf6d55d-metrics-client-ca\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.641689 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.641636 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70096da-1921-4a36-b12b-082bcdf6d55d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.643561 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.643536 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f70096da-1921-4a36-b12b-082bcdf6d55d-telemeter-client-tls\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.643561 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.643547 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f70096da-1921-4a36-b12b-082bcdf6d55d-federate-client-tls\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.644287 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.644260 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f70096da-1921-4a36-b12b-082bcdf6d55d-secret-telemeter-client\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.644727 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.644702 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f70096da-1921-4a36-b12b-082bcdf6d55d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.651094 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.651051 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mftmk\" (UniqueName: \"kubernetes.io/projected/f70096da-1921-4a36-b12b-082bcdf6d55d-kube-api-access-mftmk\") pod \"telemeter-client-66ffb784d-rg7z6\" (UID: \"f70096da-1921-4a36-b12b-082bcdf6d55d\") " pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.739927 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.739842 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" event={"ID":"93703e27-00df-46bb-a91b-c09de578a2f7","Type":"ContainerStarted","Data":"e27673986fc10fa8c04e4d07337c05aa21e472ec087a43ea5fd5aaf3fdd72d34"} Jul 17 17:32:15.741321 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.741293 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59604b04-8411-4c2b-98db-db26c020c9fa-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-6ft8n\" (UID: \"59604b04-8411-4c2b-98db-db26c020c9fa\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-6ft8n" Jul 17 17:32:15.741467 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:32:15.741451 2553 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 17 17:32:15.741526 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:32:15.741509 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59604b04-8411-4c2b-98db-db26c020c9fa-monitoring-plugin-cert podName:59604b04-8411-4c2b-98db-db26c020c9fa nodeName:}" failed. No retries permitted until 2026-07-17 17:32:16.74149223 +0000 UTC m=+57.860320509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/59604b04-8411-4c2b-98db-db26c020c9fa-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-6ft8n" (UID: "59604b04-8411-4c2b-98db-db26c020c9fa") : secret "monitoring-plugin-cert" not found Jul 17 17:32:15.783734 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.783665 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" Jul 17 17:32:15.793130 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.793102 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:15.887307 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.887250 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:15.955239 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.955148 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-66ffb784d-rg7z6"] Jul 17 17:32:15.959190 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:32:15.959159 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70096da_1921_4a36_b12b_082bcdf6d55d.slice/crio-e84c2b50cc3e6ea940c8373afa047410f188e931edd35f68ce19e3353421bf92 WatchSource:0}: Error finding container e84c2b50cc3e6ea940c8373afa047410f188e931edd35f68ce19e3353421bf92: Status 404 returned error can't find the container with id e84c2b50cc3e6ea940c8373afa047410f188e931edd35f68ce19e3353421bf92 Jul 17 17:32:15.986229 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:15.986069 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:16.369662 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.369611 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:16.472028 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.471972 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:16.527048 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.527007 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:32:16.530674 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.530652 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.534436 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.534090 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 17 17:32:16.534436 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.534158 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 17 17:32:16.534436 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.534306 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 17 17:32:16.534436 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.534414 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 17 17:32:16.534731 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.534468 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e3r2q2fd95gf2\"" Jul 17 17:32:16.534731 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.534605 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 17 17:32:16.535753 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.535610 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 17 17:32:16.535898 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.535775 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 17 17:32:16.535898 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.535871 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 17 17:32:16.536053 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.535951 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 17 17:32:16.536123 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.536066 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 17 17:32:16.536123 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.536083 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 17 17:32:16.536383 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.536142 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 17 17:32:16.536383 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.536333 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 17 17:32:16.536532 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.536485 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fdwwn\"" Jul 17 17:32:16.545026 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.539206 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 17 17:32:16.549359 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.549332 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:32:16.569019 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.568979 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:16.651081 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.650986 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.651081 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.651077 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-config\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.651323 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.651103 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-web-config\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.651323 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.651151 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-config-out\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.651323 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.651175 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.651323 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.651270 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.651323 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.651308 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.651576 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.651348 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.651576 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.651404 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.651576 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.651422 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.651576 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.651495 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.651576 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.651529 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.651576 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.651551 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.651877 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.651630 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.651877 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.651680 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn5p2\" (UniqueName: \"kubernetes.io/projected/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-kube-api-access-hn5p2\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.651877 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.651725 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.651877 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.651863 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.652115 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.651903 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.652115 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.651931 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.652115 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.651961 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.747226 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.747190 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" event={"ID":"f70096da-1921-4a36-b12b-082bcdf6d55d","Type":"ContainerStarted","Data":"e84c2b50cc3e6ea940c8373afa047410f188e931edd35f68ce19e3353421bf92"} Jul 17 17:32:16.752681 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.752649 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.752823 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.752724 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.752914 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.752870 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.752973 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.752920 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.752973 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.752946 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.753074 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.752969 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.753074 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.753021 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59604b04-8411-4c2b-98db-db26c020c9fa-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-6ft8n\" (UID: \"59604b04-8411-4c2b-98db-db26c020c9fa\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-6ft8n" Jul 17 17:32:16.753074 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.753054 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.753208 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.753078 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn5p2\" (UniqueName: \"kubernetes.io/projected/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-kube-api-access-hn5p2\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.753208 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.753104 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.753208 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.753130 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.753208 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.753154 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.753208 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.753182 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.753425 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.753209 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.753425 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.753266 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.753425 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.753336 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-config\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.753425 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.753360 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-web-config\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.753425 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.753390 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-config-out\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.753425 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.753415 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.753678 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.753457 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.755479 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.754063 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.755479 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.754563 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.755479 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.754634 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.756087 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.756047 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.756699 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.756660 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.757189 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.757164 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.758510 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.757890 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.758510 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.758160 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.758510 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.758161 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.758510 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.758409 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.758738 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.758514 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-config\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.758738 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.758540 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.758957 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.758936 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.759486 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.759385 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.759682 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.759640 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.760054 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.760014 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59604b04-8411-4c2b-98db-db26c020c9fa-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-6ft8n\" (UID: \"59604b04-8411-4c2b-98db-db26c020c9fa\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-6ft8n" Jul 17 17:32:16.760143 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.760068 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-config-out\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.760610 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.760568 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.761510 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.761443 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-web-config\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.761510 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.761456 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.763233 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.763193 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.767572 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.767550 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn5p2\" (UniqueName: \"kubernetes.io/projected/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-kube-api-access-hn5p2\") pod \"prometheus-k8s-0\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.847817 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.847779 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-6ft8n" Jul 17 17:32:16.849023 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.849000 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:16.990033 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:16.989999 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:17.094218 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:17.094184 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:17.686333 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:17.680714 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-6ft8n"] Jul 17 17:32:17.686333 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:32:17.686273 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59604b04_8411_4c2b_98db_db26c020c9fa.slice/crio-940f084d1ef846c14813ddbfc8221070388ccbd1d145cdd36fa6e4e4abe8c539 WatchSource:0}: Error finding container 940f084d1ef846c14813ddbfc8221070388ccbd1d145cdd36fa6e4e4abe8c539: Status 404 returned error can't find the container with id 940f084d1ef846c14813ddbfc8221070388ccbd1d145cdd36fa6e4e4abe8c539 Jul 17 17:32:17.731918 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:17.731881 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:32:17.759017 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:17.758922 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97cc2052-a3b0-4457-bf60-f48d33225882","Type":"ContainerStarted","Data":"ece8047b7625ac47e0110c13b81494a4eb01f5a0161fec7b2a50889409a2ab53"} Jul 17 17:32:17.759227 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:17.759148 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97cc2052-a3b0-4457-bf60-f48d33225882","Type":"ContainerStarted","Data":"944dc08fc975d30a95e711261319eeb44787516c71cb71e138308fba056663dd"} Jul 17 17:32:17.763187 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:17.763164 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" event={"ID":"faa4f800-8ebb-4d34-9920-41545b5a3b4b","Type":"ContainerStarted","Data":"41c7877afe5b9095bffe2abce2b3058cbd6f6667f9df10742daeb3f74fec688b"} Jul 17 17:32:17.763283 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:17.763234 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" event={"ID":"faa4f800-8ebb-4d34-9920-41545b5a3b4b","Type":"ContainerStarted","Data":"87fb44b024e2514394f821c5fdffba2df5234f2dc53fa200fa53e2964c8c557e"} Jul 17 17:32:17.764942 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:17.764822 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5200de1-7ea0-4a0a-84fd-6f78fd513b28","Type":"ContainerStarted","Data":"51ab4b292f0573dfa234fcbafe59aef4267ee699d6146e4a6111fb81d7ebc2b3"} Jul 17 17:32:17.766367 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:17.766295 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-6ft8n" event={"ID":"59604b04-8411-4c2b-98db-db26c020c9fa","Type":"ContainerStarted","Data":"940f084d1ef846c14813ddbfc8221070388ccbd1d145cdd36fa6e4e4abe8c539"} Jul 17 17:32:17.769674 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:17.769654 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" event={"ID":"93703e27-00df-46bb-a91b-c09de578a2f7","Type":"ContainerStarted","Data":"c1d020fefd29773ea26e4eac1eb1e5f20df3dfe6c16e77f4f6502134ed155ae9"} Jul 17 17:32:17.789079 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:17.789032 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" podStartSLOduration=1.372369283 podStartE2EDuration="3.789013726s" totalImagesPullingTime="2.416644443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:17.788474507 +0000 UTC m=+58.907302781" watchObservedRunningTime="2026-07-17 17:32:17.789013726 +0000 UTC m=+58.907841987" Jul 17 17:32:18.244181 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:18.244144 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:32:18.609954 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:18.609887 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-flxkv" Jul 17 17:32:18.764457 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:18.764433 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:18.777228 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:18.777189 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97cc2052-a3b0-4457-bf60-f48d33225882","Type":"ContainerStarted","Data":"785e739ef6606657e150a5dccaddf1365ecf80865adcecd83c3011fa9638ba8d"} Jul 17 17:32:18.777326 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:18.777261 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97cc2052-a3b0-4457-bf60-f48d33225882","Type":"ContainerStarted","Data":"57dbc304a18d361eab7617f4e7d7b859644076f682b0c002e9cd8c734bca1109"} Jul 17 17:32:18.777326 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:18.777298 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97cc2052-a3b0-4457-bf60-f48d33225882","Type":"ContainerStarted","Data":"fd6f64dd7825c5dedafa5859be34cf33a837d3bc7c9960c9242f1aea5bc63503"} Jul 17 17:32:18.779593 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:18.779572 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" event={"ID":"faa4f800-8ebb-4d34-9920-41545b5a3b4b","Type":"ContainerStarted","Data":"196f26231d798fd2739067d632b57d1d71c5d267ec3cc22e1cc51890a1031adc"} Jul 17 17:32:18.780972 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:18.780945 2553 generic.go:356] "Generic (PLEG): container finished" podID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerID="44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4" exitCode=0 Jul 17 17:32:18.781052 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:18.781025 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5200de1-7ea0-4a0a-84fd-6f78fd513b28","Type":"ContainerDied","Data":"44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4"} Jul 17 17:32:18.866713 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:18.866675 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:19.790173 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:19.790124 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" event={"ID":"faa4f800-8ebb-4d34-9920-41545b5a3b4b","Type":"ContainerStarted","Data":"e4e8c24d54f54c05c794f8f01925e45f12ef7cae10640050436cb76143c56c02"} Jul 17 17:32:19.790627 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:19.790211 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" event={"ID":"faa4f800-8ebb-4d34-9920-41545b5a3b4b","Type":"ContainerStarted","Data":"584f4ab4ec0ff579698d129296f94d2d77091f19d26e1cdfaf94391d505364a4"} Jul 17 17:32:19.792564 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:19.792532 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" event={"ID":"f70096da-1921-4a36-b12b-082bcdf6d55d","Type":"ContainerStarted","Data":"c118ed68a0c5b7fd86b8c91209919354e67f2cc54a6bba4c6bf04acb247da57d"} Jul 17 17:32:19.792712 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:19.792594 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" event={"ID":"f70096da-1921-4a36-b12b-082bcdf6d55d","Type":"ContainerStarted","Data":"800d94268788bbc75a080a68d2aa8c7e93f4b2c2c2b73ab4696f5816bcf1d2a6"} Jul 17 17:32:19.792712 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:19.792624 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" event={"ID":"f70096da-1921-4a36-b12b-082bcdf6d55d","Type":"ContainerStarted","Data":"0164d973d400392136e0b6e684e75cf2f31ec073316e2dd90c224649af763ae9"} Jul 17 17:32:19.797226 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:19.797199 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97cc2052-a3b0-4457-bf60-f48d33225882","Type":"ContainerStarted","Data":"a63e99ad894112016870833c8aa683c017c9eaddcb4d65d70e7d86b10375efe0"} Jul 17 17:32:19.816679 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:19.816621 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-66ffb784d-rg7z6" podStartSLOduration=2.059603761 podStartE2EDuration="4.816603841s" totalImagesPullingTime="2.75700008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:19.815957655 +0000 UTC m=+60.934785960" watchObservedRunningTime="2026-07-17 17:32:19.816603841 +0000 UTC m=+60.935432125" Jul 17 17:32:19.844503 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:19.844450 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.894255826 podStartE2EDuration="8.844432752s" totalImagesPullingTime="5.918825139s" totalInitContainerRuntime="31.351787ms" isStatefulPod=false podCreationTimestamp="2026-07-17 17:32:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:19.842551868 +0000 UTC m=+60.961380187" watchObservedRunningTime="2026-07-17 17:32:19.844432752 +0000 UTC m=+60.963261032" Jul 17 17:32:20.803806 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:20.803756 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-6ft8n" event={"ID":"59604b04-8411-4c2b-98db-db26c020c9fa","Type":"ContainerStarted","Data":"5fbf09269dd6961c7b0e4081551f1e5aa8ac84552631eec09930ab477a9f6afa"} Jul 17 17:32:20.804362 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:20.804009 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-6ft8n" Jul 17 17:32:20.807186 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:20.807151 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" event={"ID":"faa4f800-8ebb-4d34-9920-41545b5a3b4b","Type":"ContainerStarted","Data":"84bc74641009928039a55d1568a385e325e87ff32d73c6d34d0421f43e7b4207"} Jul 17 17:32:20.809472 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:20.809450 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-6ft8n" Jul 17 17:32:20.827618 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:20.827305 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-6ft8n" podStartSLOduration=4.597365591 podStartE2EDuration="6.827283687s" totalImagesPullingTime="2.229918096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:20.825582514 +0000 UTC m=+61.944410794" watchObservedRunningTime="2026-07-17 17:32:20.827283687 +0000 UTC m=+61.946111969" Jul 17 17:32:20.878769 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:20.878690 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" podStartSLOduration=2.920616936 podStartE2EDuration="7.878674989s" totalImagesPullingTime="4.958058053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:32:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:20.876697902 +0000 UTC m=+61.995526182" watchObservedRunningTime="2026-07-17 17:32:20.878674989 +0000 UTC m=+61.997503281" Jul 17 17:32:21.001060 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:21.001005 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:21.101528 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:21.101444 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:21.358065 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:21.357969 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:21.461631 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:21.461600 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:21.603130 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:21.602974 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:21.702501 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:21.702425 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:21.809507 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:21.809419 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:21.812264 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:21.812239 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:22.002124 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:22.002087 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:22.102095 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:22.102050 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:22.822032 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:22.822012 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6bf948c56c-shknk" Jul 17 17:32:23.822632 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:23.822598 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5200de1-7ea0-4a0a-84fd-6f78fd513b28","Type":"ContainerStarted","Data":"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7"} Jul 17 17:32:23.823038 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:23.822669 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5200de1-7ea0-4a0a-84fd-6f78fd513b28","Type":"ContainerStarted","Data":"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312"} Jul 17 17:32:23.823038 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:23.822720 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5200de1-7ea0-4a0a-84fd-6f78fd513b28","Type":"ContainerStarted","Data":"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c"} Jul 17 17:32:23.823038 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:23.822756 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5200de1-7ea0-4a0a-84fd-6f78fd513b28","Type":"ContainerStarted","Data":"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70"} Jul 17 17:32:23.823038 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:23.822789 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5200de1-7ea0-4a0a-84fd-6f78fd513b28","Type":"ContainerStarted","Data":"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94"} Jul 17 17:32:23.823038 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:23.822820 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5200de1-7ea0-4a0a-84fd-6f78fd513b28","Type":"ContainerStarted","Data":"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629"} Jul 17 17:32:23.858503 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:23.858449 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.854822397 podStartE2EDuration="7.858430164s" totalImagesPullingTime="3.977225198s" totalInitContainerRuntime="26.382569ms" isStatefulPod=false podCreationTimestamp="2026-07-17 17:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:23.855792546 +0000 UTC m=+64.974620829" watchObservedRunningTime="2026-07-17 17:32:23.858430164 +0000 UTC m=+64.977258444" Jul 17 17:32:25.144046 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.144013 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret\") pod \"global-pull-secret-syncer-czb2s\" (UID: \"2758fa5d-c686-4179-af0b-bc2b26fcf0eb\") " pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:32:25.144493 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.144064 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs\") pod \"network-metrics-daemon-fvs2m\" (UID: \"8a107e73-1e71-4f81-9751-d19706139fad\") " pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:32:25.144493 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.144112 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q7ds\" (UniqueName: \"kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds\") pod \"network-check-target-xf9wz\" (UID: \"d59d2582-9624-426a-a770-75a633dd1f0f\") " pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:32:25.147081 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.147063 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 17:32:25.147959 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.147946 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 17:32:25.148004 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.147954 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 17:32:25.156429 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.156406 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a107e73-1e71-4f81-9751-d19706139fad-metrics-certs\") pod \"network-metrics-daemon-fvs2m\" (UID: \"8a107e73-1e71-4f81-9751-d19706139fad\") " pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:32:25.156539 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.156453 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2758fa5d-c686-4179-af0b-bc2b26fcf0eb-original-pull-secret\") pod \"global-pull-secret-syncer-czb2s\" (UID: \"2758fa5d-c686-4179-af0b-bc2b26fcf0eb\") " pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:32:25.157044 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.157028 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 17:32:25.167337 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.167317 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q7ds\" (UniqueName: \"kubernetes.io/projected/d59d2582-9624-426a-a770-75a633dd1f0f-kube-api-access-6q7ds\") pod \"network-check-target-xf9wz\" (UID: \"d59d2582-9624-426a-a770-75a633dd1f0f\") " pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:32:25.283806 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.283774 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lgt4f\"" Jul 17 17:32:25.291092 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.291069 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fvs2m" Jul 17 17:32:25.291217 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.291202 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-czb2s" Jul 17 17:32:25.371917 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.371769 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fwmkw\"" Jul 17 17:32:25.378690 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.378649 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:32:25.440985 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.440912 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-czb2s"] Jul 17 17:32:25.443405 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:32:25.443374 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2758fa5d_c686_4179_af0b_bc2b26fcf0eb.slice/crio-6f562d717efe1999de59b3774bfa5883cd9b5f6d8a8f705bdf89205f7b118249 WatchSource:0}: Error finding container 6f562d717efe1999de59b3774bfa5883cd9b5f6d8a8f705bdf89205f7b118249: Status 404 returned error can't find the container with id 6f562d717efe1999de59b3774bfa5883cd9b5f6d8a8f705bdf89205f7b118249 Jul 17 17:32:25.464079 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.464059 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fvs2m"] Jul 17 17:32:25.465366 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:32:25.465343 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a107e73_1e71_4f81_9751_d19706139fad.slice/crio-ff54dca842c130049eba93c9faa3455928437c255796c37fd17b49aed3ba7834 WatchSource:0}: Error finding container ff54dca842c130049eba93c9faa3455928437c255796c37fd17b49aed3ba7834: Status 404 returned error can't find the container with id ff54dca842c130049eba93c9faa3455928437c255796c37fd17b49aed3ba7834 Jul 17 17:32:25.554013 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.553982 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xf9wz"] Jul 17 17:32:25.556688 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:32:25.556652 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59d2582_9624_426a_a770_75a633dd1f0f.slice/crio-a0b600c369ef01a51e123c7d59811f198768aaecd7b7d79802e563e83a1af285 WatchSource:0}: Error finding container a0b600c369ef01a51e123c7d59811f198768aaecd7b7d79802e563e83a1af285: Status 404 returned error can't find the container with id a0b600c369ef01a51e123c7d59811f198768aaecd7b7d79802e563e83a1af285 Jul 17 17:32:25.831202 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.831162 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xf9wz" event={"ID":"d59d2582-9624-426a-a770-75a633dd1f0f","Type":"ContainerStarted","Data":"a0b600c369ef01a51e123c7d59811f198768aaecd7b7d79802e563e83a1af285"} Jul 17 17:32:25.832268 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.832248 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fvs2m" event={"ID":"8a107e73-1e71-4f81-9751-d19706139fad","Type":"ContainerStarted","Data":"ff54dca842c130049eba93c9faa3455928437c255796c37fd17b49aed3ba7834"} Jul 17 17:32:25.833244 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:25.833227 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-czb2s" event={"ID":"2758fa5d-c686-4179-af0b-bc2b26fcf0eb","Type":"ContainerStarted","Data":"6f562d717efe1999de59b3774bfa5883cd9b5f6d8a8f705bdf89205f7b118249"} Jul 17 17:32:26.849258 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:26.849216 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:32:27.847517 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:27.847393 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fvs2m" event={"ID":"8a107e73-1e71-4f81-9751-d19706139fad","Type":"ContainerStarted","Data":"06ed89644104293ec3d9de7b8a808db624d67e9ca36770d4bf026e076a7e5c3b"} Jul 17 17:32:27.847517 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:27.847446 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fvs2m" event={"ID":"8a107e73-1e71-4f81-9751-d19706139fad","Type":"ContainerStarted","Data":"5c80d59efd1dbc73a75e84cfb5d0832e426d754f96fd359569028805580121fc"} Jul 17 17:32:27.868286 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:27.868066 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fvs2m" podStartSLOduration=67.429452538 podStartE2EDuration="1m8.868047904s" totalImagesPullingTime="1.438595366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:27.866391449 +0000 UTC m=+68.985219758" watchObservedRunningTime="2026-07-17 17:32:27.868047904 +0000 UTC m=+68.986876185" Jul 17 17:32:28.010289 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:28.010236 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:28.092370 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:28.092330 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:30.863605 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:30.863516 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xf9wz" event={"ID":"d59d2582-9624-426a-a770-75a633dd1f0f","Type":"ContainerStarted","Data":"4c3cd92cb8bc4e1bc408f6648500386322b1400c768bf605a818c9a35e3237ba"} Jul 17 17:32:30.864074 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:30.863625 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:32:30.865243 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:30.865215 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-czb2s" event={"ID":"2758fa5d-c686-4179-af0b-bc2b26fcf0eb","Type":"ContainerStarted","Data":"a97f2764fa813de6e28e8042748c3806b12ed9f69a160addde695d5f078fdda9"} Jul 17 17:32:30.908471 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:30.908420 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xf9wz" podStartSLOduration=66.924823034 podStartE2EDuration="1m11.908403917s" totalImagesPullingTime="4.983580883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:30.907211256 +0000 UTC m=+72.026039538" watchObservedRunningTime="2026-07-17 17:32:30.908403917 +0000 UTC m=+72.027232212" Jul 17 17:32:31.937492 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:31.937436 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:32.054066 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:32.054030 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:33.875224 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:33.875172 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:33.931549 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:33.931511 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:34.941337 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:34.941294 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:34.941721 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:34.941396 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:36.322082 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:36.322050 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:36.378204 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:36.378171 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:40.043348 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.043290 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56598579b-6kwmh" podUID="d59d6ab6-e518-44b8-acc9-6df2260cb7ef" containerName="console" containerID="cri-o://44e1b6f7c547357cffd02e38a08e6faed2aa9f485ece1a8c45c56f2975bf2841" gracePeriod=15 Jul 17 17:32:40.362842 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.362815 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:32:40.390291 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.390261 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-oauth-serving-cert\") pod \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " Jul 17 17:32:40.390291 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.390292 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-console-oauth-config\") pod \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " Jul 17 17:32:40.390493 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.390320 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-console-serving-cert\") pod \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " Jul 17 17:32:40.390493 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.390340 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-trusted-ca-bundle\") pod \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " Jul 17 17:32:40.390493 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.390387 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7cxz\" (UniqueName: \"kubernetes.io/projected/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-kube-api-access-w7cxz\") pod \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " Jul 17 17:32:40.390493 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.390422 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-console-config\") pod \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " Jul 17 17:32:40.390493 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.390484 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-service-ca\") pod \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\" (UID: \"d59d6ab6-e518-44b8-acc9-6df2260cb7ef\") " Jul 17 17:32:40.390734 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.390669 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d59d6ab6-e518-44b8-acc9-6df2260cb7ef" (UID: "d59d6ab6-e518-44b8-acc9-6df2260cb7ef"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:32:40.390842 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.390805 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d59d6ab6-e518-44b8-acc9-6df2260cb7ef" (UID: "d59d6ab6-e518-44b8-acc9-6df2260cb7ef"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:32:40.390990 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.390960 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-console-config" (OuterVolumeSpecName: "console-config") pod "d59d6ab6-e518-44b8-acc9-6df2260cb7ef" (UID: "d59d6ab6-e518-44b8-acc9-6df2260cb7ef"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:32:40.391065 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.391009 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-service-ca" (OuterVolumeSpecName: "service-ca") pod "d59d6ab6-e518-44b8-acc9-6df2260cb7ef" (UID: "d59d6ab6-e518-44b8-acc9-6df2260cb7ef"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:32:40.392731 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.392702 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d59d6ab6-e518-44b8-acc9-6df2260cb7ef" (UID: "d59d6ab6-e518-44b8-acc9-6df2260cb7ef"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:32:40.392825 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.392703 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-kube-api-access-w7cxz" (OuterVolumeSpecName: "kube-api-access-w7cxz") pod "d59d6ab6-e518-44b8-acc9-6df2260cb7ef" (UID: "d59d6ab6-e518-44b8-acc9-6df2260cb7ef"). InnerVolumeSpecName "kube-api-access-w7cxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:32:40.392825 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.392778 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d59d6ab6-e518-44b8-acc9-6df2260cb7ef" (UID: "d59d6ab6-e518-44b8-acc9-6df2260cb7ef"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:32:40.402816 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.402771 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-czb2s" podStartSLOduration=76.302322802 podStartE2EDuration="1m21.402757679s" totalImagesPullingTime="5.100434877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:30.960428583 +0000 UTC m=+72.079256863" watchObservedRunningTime="2026-07-17 17:32:40.402757679 +0000 UTC m=+81.521585960" Jul 17 17:32:40.491428 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.491395 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7cxz\" (UniqueName: \"kubernetes.io/projected/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-kube-api-access-w7cxz\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:32:40.491428 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.491421 2553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-console-config\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:32:40.491428 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.491430 2553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-service-ca\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:32:40.491428 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.491439 2553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-oauth-serving-cert\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:32:40.491678 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.491446 2553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-console-oauth-config\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:32:40.491678 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.491455 2553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-console-serving-cert\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:32:40.491678 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.491462 2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d59d6ab6-e518-44b8-acc9-6df2260cb7ef-trusted-ca-bundle\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:32:40.908549 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.908511 2553 generic.go:356] "Generic (PLEG): container finished" podID="d59d6ab6-e518-44b8-acc9-6df2260cb7ef" containerID="44e1b6f7c547357cffd02e38a08e6faed2aa9f485ece1a8c45c56f2975bf2841" exitCode=0 Jul 17 17:32:40.908720 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.908565 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56598579b-6kwmh" Jul 17 17:32:40.908720 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.908597 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56598579b-6kwmh" event={"ID":"d59d6ab6-e518-44b8-acc9-6df2260cb7ef","Type":"ContainerDied","Data":"44e1b6f7c547357cffd02e38a08e6faed2aa9f485ece1a8c45c56f2975bf2841"} Jul 17 17:32:40.908720 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.908648 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56598579b-6kwmh" event={"ID":"d59d6ab6-e518-44b8-acc9-6df2260cb7ef","Type":"ContainerDied","Data":"f1fbf705608defae6e7cdf87f82950d4fbb3efae154965303cae4304770f3159"} Jul 17 17:32:40.908720 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.908670 2553 scope.go:117] "RemoveContainer" containerID="44e1b6f7c547357cffd02e38a08e6faed2aa9f485ece1a8c45c56f2975bf2841" Jul 17 17:32:40.917609 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.917592 2553 scope.go:117] "RemoveContainer" containerID="44e1b6f7c547357cffd02e38a08e6faed2aa9f485ece1a8c45c56f2975bf2841" Jul 17 17:32:40.917896 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:32:40.917873 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e1b6f7c547357cffd02e38a08e6faed2aa9f485ece1a8c45c56f2975bf2841\": container with ID starting with 44e1b6f7c547357cffd02e38a08e6faed2aa9f485ece1a8c45c56f2975bf2841 not found: ID does not exist" containerID="44e1b6f7c547357cffd02e38a08e6faed2aa9f485ece1a8c45c56f2975bf2841" Jul 17 17:32:40.917995 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.917903 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e1b6f7c547357cffd02e38a08e6faed2aa9f485ece1a8c45c56f2975bf2841"} err="failed to get container status \"44e1b6f7c547357cffd02e38a08e6faed2aa9f485ece1a8c45c56f2975bf2841\": rpc error: code = NotFound desc = could not find container \"44e1b6f7c547357cffd02e38a08e6faed2aa9f485ece1a8c45c56f2975bf2841\": container with ID starting with 44e1b6f7c547357cffd02e38a08e6faed2aa9f485ece1a8c45c56f2975bf2841 not found: ID does not exist" Jul 17 17:32:40.930798 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.930767 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56598579b-6kwmh"] Jul 17 17:32:40.936490 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:40.936472 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56598579b-6kwmh"] Jul 17 17:32:41.462409 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:41.462378 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d59d6ab6-e518-44b8-acc9-6df2260cb7ef" path="/var/lib/kubelet/pods/d59d6ab6-e518-44b8-acc9-6df2260cb7ef/volumes" Jul 17 17:32:42.346080 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:42.346050 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:42.501122 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:42.501073 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:44.589222 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:44.589193 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:54.946287 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:54.946257 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:54.951133 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:54.951111 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6f8f95d894-jn6s5" Jul 17 17:32:58.853561 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:58.853526 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:32:58.916611 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:32:58.916573 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:01.872150 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:01.872119 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xf9wz" Jul 17 17:33:02.024164 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:02.024130 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:02.042709 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:02.042679 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:03.016202 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:03.016169 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:04.970070 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:04.970032 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:05.028103 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:05.028077 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:06.396927 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:06.396877 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:06.453412 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:06.453376 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:10.739591 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:10.739558 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:33:10.740133 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:10.739877 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="alertmanager" containerID="cri-o://944dc08fc975d30a95e711261319eeb44787516c71cb71e138308fba056663dd" gracePeriod=120 Jul 17 17:33:10.740133 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:10.739938 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="config-reloader" containerID="cri-o://ece8047b7625ac47e0110c13b81494a4eb01f5a0161fec7b2a50889409a2ab53" gracePeriod=120 Jul 17 17:33:10.740133 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:10.739934 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="kube-rbac-proxy-web" containerID="cri-o://fd6f64dd7825c5dedafa5859be34cf33a837d3bc7c9960c9242f1aea5bc63503" gracePeriod=120 Jul 17 17:33:10.740133 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:10.739981 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="kube-rbac-proxy" containerID="cri-o://57dbc304a18d361eab7617f4e7d7b859644076f682b0c002e9cd8c734bca1109" gracePeriod=120 Jul 17 17:33:10.740133 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:10.739934 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="kube-rbac-proxy-metric" containerID="cri-o://785e739ef6606657e150a5dccaddf1365ecf80865adcecd83c3011fa9638ba8d" gracePeriod=120 Jul 17 17:33:10.740133 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:10.740004 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="prom-label-proxy" containerID="cri-o://a63e99ad894112016870833c8aa683c017c9eaddcb4d65d70e7d86b10375efe0" gracePeriod=120 Jul 17 17:33:11.035753 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:11.035674 2553 generic.go:356] "Generic (PLEG): container finished" podID="97cc2052-a3b0-4457-bf60-f48d33225882" containerID="a63e99ad894112016870833c8aa683c017c9eaddcb4d65d70e7d86b10375efe0" exitCode=0 Jul 17 17:33:11.035753 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:11.035699 2553 generic.go:356] "Generic (PLEG): container finished" podID="97cc2052-a3b0-4457-bf60-f48d33225882" containerID="57dbc304a18d361eab7617f4e7d7b859644076f682b0c002e9cd8c734bca1109" exitCode=0 Jul 17 17:33:11.035753 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:11.035705 2553 generic.go:356] "Generic (PLEG): container finished" podID="97cc2052-a3b0-4457-bf60-f48d33225882" containerID="ece8047b7625ac47e0110c13b81494a4eb01f5a0161fec7b2a50889409a2ab53" exitCode=0 Jul 17 17:33:11.035753 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:11.035710 2553 generic.go:356] "Generic (PLEG): container finished" podID="97cc2052-a3b0-4457-bf60-f48d33225882" containerID="944dc08fc975d30a95e711261319eeb44787516c71cb71e138308fba056663dd" exitCode=0 Jul 17 17:33:11.035753 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:11.035741 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97cc2052-a3b0-4457-bf60-f48d33225882","Type":"ContainerDied","Data":"a63e99ad894112016870833c8aa683c017c9eaddcb4d65d70e7d86b10375efe0"} Jul 17 17:33:11.036030 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:11.035811 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97cc2052-a3b0-4457-bf60-f48d33225882","Type":"ContainerDied","Data":"57dbc304a18d361eab7617f4e7d7b859644076f682b0c002e9cd8c734bca1109"} Jul 17 17:33:11.036030 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:11.035840 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97cc2052-a3b0-4457-bf60-f48d33225882","Type":"ContainerDied","Data":"ece8047b7625ac47e0110c13b81494a4eb01f5a0161fec7b2a50889409a2ab53"} Jul 17 17:33:11.036030 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:11.035881 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97cc2052-a3b0-4457-bf60-f48d33225882","Type":"ContainerDied","Data":"944dc08fc975d30a95e711261319eeb44787516c71cb71e138308fba056663dd"} Jul 17 17:33:12.027474 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.027445 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:33:12.042946 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.042919 2553 generic.go:356] "Generic (PLEG): container finished" podID="97cc2052-a3b0-4457-bf60-f48d33225882" containerID="785e739ef6606657e150a5dccaddf1365ecf80865adcecd83c3011fa9638ba8d" exitCode=0 Jul 17 17:33:12.042946 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.042941 2553 generic.go:356] "Generic (PLEG): container finished" podID="97cc2052-a3b0-4457-bf60-f48d33225882" containerID="fd6f64dd7825c5dedafa5859be34cf33a837d3bc7c9960c9242f1aea5bc63503" exitCode=0 Jul 17 17:33:12.043115 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.042965 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97cc2052-a3b0-4457-bf60-f48d33225882","Type":"ContainerDied","Data":"785e739ef6606657e150a5dccaddf1365ecf80865adcecd83c3011fa9638ba8d"} Jul 17 17:33:12.043115 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.042990 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:33:12.043115 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.043034 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97cc2052-a3b0-4457-bf60-f48d33225882","Type":"ContainerDied","Data":"fd6f64dd7825c5dedafa5859be34cf33a837d3bc7c9960c9242f1aea5bc63503"} Jul 17 17:33:12.043115 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.043061 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97cc2052-a3b0-4457-bf60-f48d33225882","Type":"ContainerDied","Data":"43655644b50859ce6a8b4f7dd26ff243e0f026d984bd31135e3b2e9948a8446e"} Jul 17 17:33:12.043115 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.043094 2553 scope.go:117] "RemoveContainer" containerID="a63e99ad894112016870833c8aa683c017c9eaddcb4d65d70e7d86b10375efe0" Jul 17 17:33:12.050471 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.050449 2553 scope.go:117] "RemoveContainer" containerID="785e739ef6606657e150a5dccaddf1365ecf80865adcecd83c3011fa9638ba8d" Jul 17 17:33:12.057311 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.057287 2553 scope.go:117] "RemoveContainer" containerID="57dbc304a18d361eab7617f4e7d7b859644076f682b0c002e9cd8c734bca1109" Jul 17 17:33:12.063998 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.063967 2553 scope.go:117] "RemoveContainer" containerID="fd6f64dd7825c5dedafa5859be34cf33a837d3bc7c9960c9242f1aea5bc63503" Jul 17 17:33:12.072698 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.072678 2553 scope.go:117] "RemoveContainer" containerID="ece8047b7625ac47e0110c13b81494a4eb01f5a0161fec7b2a50889409a2ab53" Jul 17 17:33:12.080816 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.080797 2553 scope.go:117] "RemoveContainer" containerID="944dc08fc975d30a95e711261319eeb44787516c71cb71e138308fba056663dd" Jul 17 17:33:12.090203 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.090178 2553 scope.go:117] "RemoveContainer" containerID="46d591a9a8f2b7e3152764b4e5b0a9f6c0d7d7cdcab9672d422e7492b87be920" Jul 17 17:33:12.098455 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.098431 2553 scope.go:117] "RemoveContainer" containerID="a63e99ad894112016870833c8aa683c017c9eaddcb4d65d70e7d86b10375efe0" Jul 17 17:33:12.098754 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:33:12.098735 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a63e99ad894112016870833c8aa683c017c9eaddcb4d65d70e7d86b10375efe0\": container with ID starting with a63e99ad894112016870833c8aa683c017c9eaddcb4d65d70e7d86b10375efe0 not found: ID does not exist" containerID="a63e99ad894112016870833c8aa683c017c9eaddcb4d65d70e7d86b10375efe0" Jul 17 17:33:12.098874 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.098765 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a63e99ad894112016870833c8aa683c017c9eaddcb4d65d70e7d86b10375efe0"} err="failed to get container status \"a63e99ad894112016870833c8aa683c017c9eaddcb4d65d70e7d86b10375efe0\": rpc error: code = NotFound desc = could not find container \"a63e99ad894112016870833c8aa683c017c9eaddcb4d65d70e7d86b10375efe0\": container with ID starting with a63e99ad894112016870833c8aa683c017c9eaddcb4d65d70e7d86b10375efe0 not found: ID does not exist" Jul 17 17:33:12.098874 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.098784 2553 scope.go:117] "RemoveContainer" containerID="785e739ef6606657e150a5dccaddf1365ecf80865adcecd83c3011fa9638ba8d" Jul 17 17:33:12.099059 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:33:12.099042 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"785e739ef6606657e150a5dccaddf1365ecf80865adcecd83c3011fa9638ba8d\": container with ID starting with 785e739ef6606657e150a5dccaddf1365ecf80865adcecd83c3011fa9638ba8d not found: ID does not exist" containerID="785e739ef6606657e150a5dccaddf1365ecf80865adcecd83c3011fa9638ba8d" Jul 17 17:33:12.099127 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.099066 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"785e739ef6606657e150a5dccaddf1365ecf80865adcecd83c3011fa9638ba8d"} err="failed to get container status \"785e739ef6606657e150a5dccaddf1365ecf80865adcecd83c3011fa9638ba8d\": rpc error: code = NotFound desc = could not find container \"785e739ef6606657e150a5dccaddf1365ecf80865adcecd83c3011fa9638ba8d\": container with ID starting with 785e739ef6606657e150a5dccaddf1365ecf80865adcecd83c3011fa9638ba8d not found: ID does not exist" Jul 17 17:33:12.099127 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.099080 2553 scope.go:117] "RemoveContainer" containerID="57dbc304a18d361eab7617f4e7d7b859644076f682b0c002e9cd8c734bca1109" Jul 17 17:33:12.099322 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:33:12.099304 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57dbc304a18d361eab7617f4e7d7b859644076f682b0c002e9cd8c734bca1109\": container with ID starting with 57dbc304a18d361eab7617f4e7d7b859644076f682b0c002e9cd8c734bca1109 not found: ID does not exist" containerID="57dbc304a18d361eab7617f4e7d7b859644076f682b0c002e9cd8c734bca1109" Jul 17 17:33:12.099385 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.099327 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57dbc304a18d361eab7617f4e7d7b859644076f682b0c002e9cd8c734bca1109"} err="failed to get container status \"57dbc304a18d361eab7617f4e7d7b859644076f682b0c002e9cd8c734bca1109\": rpc error: code = NotFound desc = could not find container \"57dbc304a18d361eab7617f4e7d7b859644076f682b0c002e9cd8c734bca1109\": container with ID starting with 57dbc304a18d361eab7617f4e7d7b859644076f682b0c002e9cd8c734bca1109 not found: ID does not exist" Jul 17 17:33:12.099385 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.099347 2553 scope.go:117] "RemoveContainer" containerID="fd6f64dd7825c5dedafa5859be34cf33a837d3bc7c9960c9242f1aea5bc63503" Jul 17 17:33:12.099590 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:33:12.099573 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd6f64dd7825c5dedafa5859be34cf33a837d3bc7c9960c9242f1aea5bc63503\": container with ID starting with fd6f64dd7825c5dedafa5859be34cf33a837d3bc7c9960c9242f1aea5bc63503 not found: ID does not exist" containerID="fd6f64dd7825c5dedafa5859be34cf33a837d3bc7c9960c9242f1aea5bc63503" Jul 17 17:33:12.099655 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.099593 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6f64dd7825c5dedafa5859be34cf33a837d3bc7c9960c9242f1aea5bc63503"} err="failed to get container status \"fd6f64dd7825c5dedafa5859be34cf33a837d3bc7c9960c9242f1aea5bc63503\": rpc error: code = NotFound desc = could not find container \"fd6f64dd7825c5dedafa5859be34cf33a837d3bc7c9960c9242f1aea5bc63503\": container with ID starting with fd6f64dd7825c5dedafa5859be34cf33a837d3bc7c9960c9242f1aea5bc63503 not found: ID does not exist" Jul 17 17:33:12.099655 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.099606 2553 scope.go:117] "RemoveContainer" containerID="ece8047b7625ac47e0110c13b81494a4eb01f5a0161fec7b2a50889409a2ab53" Jul 17 17:33:12.099811 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:33:12.099796 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ece8047b7625ac47e0110c13b81494a4eb01f5a0161fec7b2a50889409a2ab53\": container with ID starting with ece8047b7625ac47e0110c13b81494a4eb01f5a0161fec7b2a50889409a2ab53 not found: ID does not exist" containerID="ece8047b7625ac47e0110c13b81494a4eb01f5a0161fec7b2a50889409a2ab53" Jul 17 17:33:12.100241 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.099813 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece8047b7625ac47e0110c13b81494a4eb01f5a0161fec7b2a50889409a2ab53"} err="failed to get container status \"ece8047b7625ac47e0110c13b81494a4eb01f5a0161fec7b2a50889409a2ab53\": rpc error: code = NotFound desc = could not find container \"ece8047b7625ac47e0110c13b81494a4eb01f5a0161fec7b2a50889409a2ab53\": container with ID starting with ece8047b7625ac47e0110c13b81494a4eb01f5a0161fec7b2a50889409a2ab53 not found: ID does not exist" Jul 17 17:33:12.100241 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.099824 2553 scope.go:117] "RemoveContainer" containerID="944dc08fc975d30a95e711261319eeb44787516c71cb71e138308fba056663dd" Jul 17 17:33:12.100241 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:33:12.100027 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944dc08fc975d30a95e711261319eeb44787516c71cb71e138308fba056663dd\": container with ID starting with 944dc08fc975d30a95e711261319eeb44787516c71cb71e138308fba056663dd not found: ID does not exist" containerID="944dc08fc975d30a95e711261319eeb44787516c71cb71e138308fba056663dd" Jul 17 17:33:12.100241 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.100042 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944dc08fc975d30a95e711261319eeb44787516c71cb71e138308fba056663dd"} err="failed to get container status \"944dc08fc975d30a95e711261319eeb44787516c71cb71e138308fba056663dd\": rpc error: code = NotFound desc = could not find container \"944dc08fc975d30a95e711261319eeb44787516c71cb71e138308fba056663dd\": container with ID starting with 944dc08fc975d30a95e711261319eeb44787516c71cb71e138308fba056663dd not found: ID does not exist" Jul 17 17:33:12.100241 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.100054 2553 scope.go:117] "RemoveContainer" containerID="46d591a9a8f2b7e3152764b4e5b0a9f6c0d7d7cdcab9672d422e7492b87be920" Jul 17 17:33:12.100515 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:33:12.100262 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46d591a9a8f2b7e3152764b4e5b0a9f6c0d7d7cdcab9672d422e7492b87be920\": container with ID starting with 46d591a9a8f2b7e3152764b4e5b0a9f6c0d7d7cdcab9672d422e7492b87be920 not found: ID does not exist" containerID="46d591a9a8f2b7e3152764b4e5b0a9f6c0d7d7cdcab9672d422e7492b87be920" Jul 17 17:33:12.100515 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.100278 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d591a9a8f2b7e3152764b4e5b0a9f6c0d7d7cdcab9672d422e7492b87be920"} err="failed to get container status \"46d591a9a8f2b7e3152764b4e5b0a9f6c0d7d7cdcab9672d422e7492b87be920\": rpc error: code = NotFound desc = could not find container \"46d591a9a8f2b7e3152764b4e5b0a9f6c0d7d7cdcab9672d422e7492b87be920\": container with ID starting with 46d591a9a8f2b7e3152764b4e5b0a9f6c0d7d7cdcab9672d422e7492b87be920 not found: ID does not exist" Jul 17 17:33:12.100515 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.100293 2553 scope.go:117] "RemoveContainer" containerID="a63e99ad894112016870833c8aa683c017c9eaddcb4d65d70e7d86b10375efe0" Jul 17 17:33:12.100666 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.100531 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a63e99ad894112016870833c8aa683c017c9eaddcb4d65d70e7d86b10375efe0"} err="failed to get container status \"a63e99ad894112016870833c8aa683c017c9eaddcb4d65d70e7d86b10375efe0\": rpc error: code = NotFound desc = could not find container \"a63e99ad894112016870833c8aa683c017c9eaddcb4d65d70e7d86b10375efe0\": container with ID starting with a63e99ad894112016870833c8aa683c017c9eaddcb4d65d70e7d86b10375efe0 not found: ID does not exist" Jul 17 17:33:12.100666 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.100545 2553 scope.go:117] "RemoveContainer" containerID="785e739ef6606657e150a5dccaddf1365ecf80865adcecd83c3011fa9638ba8d" Jul 17 17:33:12.100775 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.100752 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"785e739ef6606657e150a5dccaddf1365ecf80865adcecd83c3011fa9638ba8d"} err="failed to get container status \"785e739ef6606657e150a5dccaddf1365ecf80865adcecd83c3011fa9638ba8d\": rpc error: code = NotFound desc = could not find container \"785e739ef6606657e150a5dccaddf1365ecf80865adcecd83c3011fa9638ba8d\": container with ID starting with 785e739ef6606657e150a5dccaddf1365ecf80865adcecd83c3011fa9638ba8d not found: ID does not exist" Jul 17 17:33:12.100775 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.100767 2553 scope.go:117] "RemoveContainer" containerID="57dbc304a18d361eab7617f4e7d7b859644076f682b0c002e9cd8c734bca1109" Jul 17 17:33:12.101012 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.100996 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57dbc304a18d361eab7617f4e7d7b859644076f682b0c002e9cd8c734bca1109"} err="failed to get container status \"57dbc304a18d361eab7617f4e7d7b859644076f682b0c002e9cd8c734bca1109\": rpc error: code = NotFound desc = could not find container \"57dbc304a18d361eab7617f4e7d7b859644076f682b0c002e9cd8c734bca1109\": container with ID starting with 57dbc304a18d361eab7617f4e7d7b859644076f682b0c002e9cd8c734bca1109 not found: ID does not exist" Jul 17 17:33:12.101077 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.101012 2553 scope.go:117] "RemoveContainer" containerID="fd6f64dd7825c5dedafa5859be34cf33a837d3bc7c9960c9242f1aea5bc63503" Jul 17 17:33:12.101215 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.101200 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6f64dd7825c5dedafa5859be34cf33a837d3bc7c9960c9242f1aea5bc63503"} err="failed to get container status \"fd6f64dd7825c5dedafa5859be34cf33a837d3bc7c9960c9242f1aea5bc63503\": rpc error: code = NotFound desc = could not find container \"fd6f64dd7825c5dedafa5859be34cf33a837d3bc7c9960c9242f1aea5bc63503\": container with ID starting with fd6f64dd7825c5dedafa5859be34cf33a837d3bc7c9960c9242f1aea5bc63503 not found: ID does not exist" Jul 17 17:33:12.101215 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.101215 2553 scope.go:117] "RemoveContainer" containerID="ece8047b7625ac47e0110c13b81494a4eb01f5a0161fec7b2a50889409a2ab53" Jul 17 17:33:12.101438 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.101420 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece8047b7625ac47e0110c13b81494a4eb01f5a0161fec7b2a50889409a2ab53"} err="failed to get container status \"ece8047b7625ac47e0110c13b81494a4eb01f5a0161fec7b2a50889409a2ab53\": rpc error: code = NotFound desc = could not find container \"ece8047b7625ac47e0110c13b81494a4eb01f5a0161fec7b2a50889409a2ab53\": container with ID starting with ece8047b7625ac47e0110c13b81494a4eb01f5a0161fec7b2a50889409a2ab53 not found: ID does not exist" Jul 17 17:33:12.101438 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.101437 2553 scope.go:117] "RemoveContainer" containerID="944dc08fc975d30a95e711261319eeb44787516c71cb71e138308fba056663dd" Jul 17 17:33:12.101684 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.101658 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944dc08fc975d30a95e711261319eeb44787516c71cb71e138308fba056663dd"} err="failed to get container status \"944dc08fc975d30a95e711261319eeb44787516c71cb71e138308fba056663dd\": rpc error: code = NotFound desc = could not find container \"944dc08fc975d30a95e711261319eeb44787516c71cb71e138308fba056663dd\": container with ID starting with 944dc08fc975d30a95e711261319eeb44787516c71cb71e138308fba056663dd not found: ID does not exist" Jul 17 17:33:12.101749 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.101686 2553 scope.go:117] "RemoveContainer" containerID="46d591a9a8f2b7e3152764b4e5b0a9f6c0d7d7cdcab9672d422e7492b87be920" Jul 17 17:33:12.101923 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.101898 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d591a9a8f2b7e3152764b4e5b0a9f6c0d7d7cdcab9672d422e7492b87be920"} err="failed to get container status \"46d591a9a8f2b7e3152764b4e5b0a9f6c0d7d7cdcab9672d422e7492b87be920\": rpc error: code = NotFound desc = could not find container \"46d591a9a8f2b7e3152764b4e5b0a9f6c0d7d7cdcab9672d422e7492b87be920\": container with ID starting with 46d591a9a8f2b7e3152764b4e5b0a9f6c0d7d7cdcab9672d422e7492b87be920 not found: ID does not exist" Jul 17 17:33:12.151369 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.151337 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-cluster-tls-config\") pod \"97cc2052-a3b0-4457-bf60-f48d33225882\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " Jul 17 17:33:12.151369 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.151377 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97cc2052-a3b0-4457-bf60-f48d33225882-alertmanager-trusted-ca-bundle\") pod \"97cc2052-a3b0-4457-bf60-f48d33225882\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " Jul 17 17:33:12.151607 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.151403 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-kube-rbac-proxy\") pod \"97cc2052-a3b0-4457-bf60-f48d33225882\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " Jul 17 17:33:12.151607 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.151434 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97cc2052-a3b0-4457-bf60-f48d33225882-config-out\") pod \"97cc2052-a3b0-4457-bf60-f48d33225882\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " Jul 17 17:33:12.151607 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.151460 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jg8t\" (UniqueName: \"kubernetes.io/projected/97cc2052-a3b0-4457-bf60-f48d33225882-kube-api-access-7jg8t\") pod \"97cc2052-a3b0-4457-bf60-f48d33225882\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " Jul 17 17:33:12.151607 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.151495 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-kube-rbac-proxy-metric\") pod \"97cc2052-a3b0-4457-bf60-f48d33225882\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " Jul 17 17:33:12.151607 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.151540 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97cc2052-a3b0-4457-bf60-f48d33225882-metrics-client-ca\") pod \"97cc2052-a3b0-4457-bf60-f48d33225882\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " Jul 17 17:33:12.151607 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.151577 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-main-tls\") pod \"97cc2052-a3b0-4457-bf60-f48d33225882\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " Jul 17 17:33:12.151973 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.151619 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/97cc2052-a3b0-4457-bf60-f48d33225882-alertmanager-main-db\") pod \"97cc2052-a3b0-4457-bf60-f48d33225882\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " Jul 17 17:33:12.151973 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.151644 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-config-volume\") pod \"97cc2052-a3b0-4457-bf60-f48d33225882\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " Jul 17 17:33:12.151973 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.151682 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-kube-rbac-proxy-web\") pod \"97cc2052-a3b0-4457-bf60-f48d33225882\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " Jul 17 17:33:12.151973 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.151722 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-web-config\") pod \"97cc2052-a3b0-4457-bf60-f48d33225882\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " Jul 17 17:33:12.151973 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.151786 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97cc2052-a3b0-4457-bf60-f48d33225882-tls-assets\") pod \"97cc2052-a3b0-4457-bf60-f48d33225882\" (UID: \"97cc2052-a3b0-4457-bf60-f48d33225882\") " Jul 17 17:33:12.151973 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.151789 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97cc2052-a3b0-4457-bf60-f48d33225882-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "97cc2052-a3b0-4457-bf60-f48d33225882" (UID: "97cc2052-a3b0-4457-bf60-f48d33225882"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:33:12.152271 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.152078 2553 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97cc2052-a3b0-4457-bf60-f48d33225882-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:12.152271 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.152145 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97cc2052-a3b0-4457-bf60-f48d33225882-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "97cc2052-a3b0-4457-bf60-f48d33225882" (UID: "97cc2052-a3b0-4457-bf60-f48d33225882"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:33:12.152271 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.152233 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97cc2052-a3b0-4457-bf60-f48d33225882-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "97cc2052-a3b0-4457-bf60-f48d33225882" (UID: "97cc2052-a3b0-4457-bf60-f48d33225882"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:33:12.154757 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.154719 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97cc2052-a3b0-4457-bf60-f48d33225882-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "97cc2052-a3b0-4457-bf60-f48d33225882" (UID: "97cc2052-a3b0-4457-bf60-f48d33225882"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:33:12.154900 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.154801 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "97cc2052-a3b0-4457-bf60-f48d33225882" (UID: "97cc2052-a3b0-4457-bf60-f48d33225882"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:33:12.155057 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.155034 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "97cc2052-a3b0-4457-bf60-f48d33225882" (UID: "97cc2052-a3b0-4457-bf60-f48d33225882"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:33:12.155127 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.155103 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97cc2052-a3b0-4457-bf60-f48d33225882-config-out" (OuterVolumeSpecName: "config-out") pod "97cc2052-a3b0-4457-bf60-f48d33225882" (UID: "97cc2052-a3b0-4457-bf60-f48d33225882"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:33:12.155192 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.155125 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97cc2052-a3b0-4457-bf60-f48d33225882-kube-api-access-7jg8t" (OuterVolumeSpecName: "kube-api-access-7jg8t") pod "97cc2052-a3b0-4457-bf60-f48d33225882" (UID: "97cc2052-a3b0-4457-bf60-f48d33225882"). InnerVolumeSpecName "kube-api-access-7jg8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:33:12.155386 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.155357 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "97cc2052-a3b0-4457-bf60-f48d33225882" (UID: "97cc2052-a3b0-4457-bf60-f48d33225882"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:33:12.155476 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.155455 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "97cc2052-a3b0-4457-bf60-f48d33225882" (UID: "97cc2052-a3b0-4457-bf60-f48d33225882"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:33:12.156156 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.156134 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-config-volume" (OuterVolumeSpecName: "config-volume") pod "97cc2052-a3b0-4457-bf60-f48d33225882" (UID: "97cc2052-a3b0-4457-bf60-f48d33225882"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:33:12.159325 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.159208 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "97cc2052-a3b0-4457-bf60-f48d33225882" (UID: "97cc2052-a3b0-4457-bf60-f48d33225882"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:33:12.166109 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.166084 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-web-config" (OuterVolumeSpecName: "web-config") pod "97cc2052-a3b0-4457-bf60-f48d33225882" (UID: "97cc2052-a3b0-4457-bf60-f48d33225882"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:33:12.252811 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.252768 2553 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-main-tls\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:12.252811 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.252809 2553 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/97cc2052-a3b0-4457-bf60-f48d33225882-alertmanager-main-db\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:12.252811 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.252819 2553 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-config-volume\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:12.253021 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.252829 2553 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:12.253021 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.252879 2553 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-web-config\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:12.253021 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.252888 2553 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97cc2052-a3b0-4457-bf60-f48d33225882-tls-assets\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:12.253021 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.252895 2553 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-cluster-tls-config\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:12.253021 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.252904 2553 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:12.253021 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.252914 2553 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97cc2052-a3b0-4457-bf60-f48d33225882-config-out\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:12.253021 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.252923 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jg8t\" (UniqueName: \"kubernetes.io/projected/97cc2052-a3b0-4457-bf60-f48d33225882-kube-api-access-7jg8t\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:12.253021 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.252931 2553 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/97cc2052-a3b0-4457-bf60-f48d33225882-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:12.253021 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.252940 2553 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97cc2052-a3b0-4457-bf60-f48d33225882-metrics-client-ca\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:12.370253 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.370209 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:33:12.375031 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:12.375004 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:33:13.465339 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:13.465306 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" path="/var/lib/kubelet/pods/97cc2052-a3b0-4457-bf60-f48d33225882/volumes" Jul 17 17:33:14.982068 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:14.982034 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:33:14.982494 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:14.982343 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="prometheus" containerID="cri-o://19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629" gracePeriod=600 Jul 17 17:33:14.982494 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:14.982361 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="kube-rbac-proxy-web" containerID="cri-o://e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c" gracePeriod=600 Jul 17 17:33:14.982494 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:14.982381 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="thanos-sidecar" containerID="cri-o://fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70" gracePeriod=600 Jul 17 17:33:14.982647 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:14.982500 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="config-reloader" containerID="cri-o://ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94" gracePeriod=600 Jul 17 17:33:14.982647 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:14.982520 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="kube-rbac-proxy" containerID="cri-o://111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312" gracePeriod=600 Jul 17 17:33:14.982647 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:14.982500 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7" gracePeriod=600 Jul 17 17:33:15.266216 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.266192 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:15.380445 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.380414 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-configmap-serving-certs-ca-bundle\") pod \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " Jul 17 17:33:15.380445 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.380450 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " Jul 17 17:33:15.380692 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.380470 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-rulefiles-0\") pod \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " Jul 17 17:33:15.380692 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.380492 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-kube-rbac-proxy\") pod \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " Jul 17 17:33:15.380692 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.380512 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-db\") pod \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " Jul 17 17:33:15.380692 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.380583 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-web-config\") pod \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " Jul 17 17:33:15.380692 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.380626 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-configmap-metrics-client-ca\") pod \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " Jul 17 17:33:15.380692 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.380661 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn5p2\" (UniqueName: \"kubernetes.io/projected/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-kube-api-access-hn5p2\") pod \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " Jul 17 17:33:15.381038 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.380713 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-rulefiles-1\") pod \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " Jul 17 17:33:15.381038 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.380746 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-trusted-ca-bundle\") pod \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " Jul 17 17:33:15.381038 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.380771 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-tls-assets\") pod \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " Jul 17 17:33:15.381038 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.380804 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-grpc-tls\") pod \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " Jul 17 17:33:15.381038 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.380832 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-config-out\") pod \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " Jul 17 17:33:15.381038 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.380842 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "f5200de1-7ea0-4a0a-84fd-6f78fd513b28" (UID: "f5200de1-7ea0-4a0a-84fd-6f78fd513b28"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:33:15.381038 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.380896 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-thanos-prometheus-http-client-file\") pod \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " Jul 17 17:33:15.381038 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.380975 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-prometheus-k8s-tls\") pod \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " Jul 17 17:33:15.381038 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.381007 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-config\") pod \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " Jul 17 17:33:15.381038 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.381030 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " Jul 17 17:33:15.381509 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.381053 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-rulefiles-2\") pod \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " Jul 17 17:33:15.381509 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.381091 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-metrics-client-certs\") pod \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " Jul 17 17:33:15.381509 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.381122 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-configmap-kubelet-serving-ca-bundle\") pod \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\" (UID: \"f5200de1-7ea0-4a0a-84fd-6f78fd513b28\") " Jul 17 17:33:15.381509 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.381171 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "f5200de1-7ea0-4a0a-84fd-6f78fd513b28" (UID: "f5200de1-7ea0-4a0a-84fd-6f78fd513b28"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:33:15.381509 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.381385 2553 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:15.381509 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.381405 2553 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-configmap-metrics-client-ca\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:15.381800 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.381588 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "f5200de1-7ea0-4a0a-84fd-6f78fd513b28" (UID: "f5200de1-7ea0-4a0a-84fd-6f78fd513b28"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:33:15.382307 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.382276 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "f5200de1-7ea0-4a0a-84fd-6f78fd513b28" (UID: "f5200de1-7ea0-4a0a-84fd-6f78fd513b28"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:33:15.383036 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.383012 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "f5200de1-7ea0-4a0a-84fd-6f78fd513b28" (UID: "f5200de1-7ea0-4a0a-84fd-6f78fd513b28"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:33:15.383651 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.383625 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "f5200de1-7ea0-4a0a-84fd-6f78fd513b28" (UID: "f5200de1-7ea0-4a0a-84fd-6f78fd513b28"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:33:15.383764 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.383739 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "f5200de1-7ea0-4a0a-84fd-6f78fd513b28" (UID: "f5200de1-7ea0-4a0a-84fd-6f78fd513b28"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:33:15.383869 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.383833 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "f5200de1-7ea0-4a0a-84fd-6f78fd513b28" (UID: "f5200de1-7ea0-4a0a-84fd-6f78fd513b28"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:33:15.383980 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.383956 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f5200de1-7ea0-4a0a-84fd-6f78fd513b28" (UID: "f5200de1-7ea0-4a0a-84fd-6f78fd513b28"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:33:15.384176 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.384153 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "f5200de1-7ea0-4a0a-84fd-6f78fd513b28" (UID: "f5200de1-7ea0-4a0a-84fd-6f78fd513b28"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:33:15.384424 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.384393 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "f5200de1-7ea0-4a0a-84fd-6f78fd513b28" (UID: "f5200de1-7ea0-4a0a-84fd-6f78fd513b28"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:33:15.384802 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.384550 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "f5200de1-7ea0-4a0a-84fd-6f78fd513b28" (UID: "f5200de1-7ea0-4a0a-84fd-6f78fd513b28"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:33:15.385135 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.385043 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "f5200de1-7ea0-4a0a-84fd-6f78fd513b28" (UID: "f5200de1-7ea0-4a0a-84fd-6f78fd513b28"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:33:15.385258 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.385234 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-kube-api-access-hn5p2" (OuterVolumeSpecName: "kube-api-access-hn5p2") pod "f5200de1-7ea0-4a0a-84fd-6f78fd513b28" (UID: "f5200de1-7ea0-4a0a-84fd-6f78fd513b28"). InnerVolumeSpecName "kube-api-access-hn5p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:33:15.385488 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.385464 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "f5200de1-7ea0-4a0a-84fd-6f78fd513b28" (UID: "f5200de1-7ea0-4a0a-84fd-6f78fd513b28"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:33:15.385880 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.385840 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-config-out" (OuterVolumeSpecName: "config-out") pod "f5200de1-7ea0-4a0a-84fd-6f78fd513b28" (UID: "f5200de1-7ea0-4a0a-84fd-6f78fd513b28"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:33:15.386047 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.386029 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "f5200de1-7ea0-4a0a-84fd-6f78fd513b28" (UID: "f5200de1-7ea0-4a0a-84fd-6f78fd513b28"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:33:15.386318 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.386293 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-config" (OuterVolumeSpecName: "config") pod "f5200de1-7ea0-4a0a-84fd-6f78fd513b28" (UID: "f5200de1-7ea0-4a0a-84fd-6f78fd513b28"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:33:15.386404 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.386358 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "f5200de1-7ea0-4a0a-84fd-6f78fd513b28" (UID: "f5200de1-7ea0-4a0a-84fd-6f78fd513b28"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:33:15.393675 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.393655 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-web-config" (OuterVolumeSpecName: "web-config") pod "f5200de1-7ea0-4a0a-84fd-6f78fd513b28" (UID: "f5200de1-7ea0-4a0a-84fd-6f78fd513b28"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:33:15.482206 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.482178 2553 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:15.482206 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.482206 2553 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:15.482376 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.482220 2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:15.482376 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.482233 2553 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-kube-rbac-proxy\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:15.482376 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.482244 2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-db\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:15.482376 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.482256 2553 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-web-config\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:15.482376 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.482268 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn5p2\" (UniqueName: \"kubernetes.io/projected/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-kube-api-access-hn5p2\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:15.482376 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.482279 2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:15.482376 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.482292 2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:15.482376 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.482303 2553 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-tls-assets\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:15.482376 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.482314 2553 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-grpc-tls\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:15.482376 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.482324 2553 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-config-out\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:15.482376 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.482336 2553 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:15.482376 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.482350 2553 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:15.482376 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.482364 2553 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-config\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:15.482376 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.482378 2553 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:15.482790 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.482392 2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:15.482790 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:15.482405 2553 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f5200de1-7ea0-4a0a-84fd-6f78fd513b28-secret-metrics-client-certs\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:33:16.062457 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.062420 2553 generic.go:356] "Generic (PLEG): container finished" podID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerID="a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7" exitCode=0 Jul 17 17:33:16.062457 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.062448 2553 generic.go:356] "Generic (PLEG): container finished" podID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerID="111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312" exitCode=0 Jul 17 17:33:16.062457 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.062454 2553 generic.go:356] "Generic (PLEG): container finished" podID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerID="e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c" exitCode=0 Jul 17 17:33:16.062457 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.062459 2553 generic.go:356] "Generic (PLEG): container finished" podID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerID="fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70" exitCode=0 Jul 17 17:33:16.062457 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.062465 2553 generic.go:356] "Generic (PLEG): container finished" podID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerID="ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94" exitCode=0 Jul 17 17:33:16.063042 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.062470 2553 generic.go:356] "Generic (PLEG): container finished" podID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerID="19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629" exitCode=0 Jul 17 17:33:16.063042 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.062491 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.063042 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.062503 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5200de1-7ea0-4a0a-84fd-6f78fd513b28","Type":"ContainerDied","Data":"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7"} Jul 17 17:33:16.063042 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.062565 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5200de1-7ea0-4a0a-84fd-6f78fd513b28","Type":"ContainerDied","Data":"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312"} Jul 17 17:33:16.063042 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.062592 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5200de1-7ea0-4a0a-84fd-6f78fd513b28","Type":"ContainerDied","Data":"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c"} Jul 17 17:33:16.063042 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.062615 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5200de1-7ea0-4a0a-84fd-6f78fd513b28","Type":"ContainerDied","Data":"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70"} Jul 17 17:33:16.063042 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.062644 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5200de1-7ea0-4a0a-84fd-6f78fd513b28","Type":"ContainerDied","Data":"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94"} Jul 17 17:33:16.063042 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.062670 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5200de1-7ea0-4a0a-84fd-6f78fd513b28","Type":"ContainerDied","Data":"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629"} Jul 17 17:33:16.063042 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.062693 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f5200de1-7ea0-4a0a-84fd-6f78fd513b28","Type":"ContainerDied","Data":"51ab4b292f0573dfa234fcbafe59aef4267ee699d6146e4a6111fb81d7ebc2b3"} Jul 17 17:33:16.063042 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.062641 2553 scope.go:117] "RemoveContainer" containerID="a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7" Jul 17 17:33:16.070189 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.070171 2553 scope.go:117] "RemoveContainer" containerID="111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312" Jul 17 17:33:16.078048 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.078028 2553 scope.go:117] "RemoveContainer" containerID="e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c" Jul 17 17:33:16.085612 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.085592 2553 scope.go:117] "RemoveContainer" containerID="fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70" Jul 17 17:33:16.092750 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.092725 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:33:16.092860 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.092753 2553 scope.go:117] "RemoveContainer" containerID="ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94" Jul 17 17:33:16.097446 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.097424 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:33:16.099667 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.099647 2553 scope.go:117] "RemoveContainer" containerID="19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629" Jul 17 17:33:16.106580 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.106561 2553 scope.go:117] "RemoveContainer" containerID="44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4" Jul 17 17:33:16.112879 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.112829 2553 scope.go:117] "RemoveContainer" containerID="a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7" Jul 17 17:33:16.113131 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:33:16.113104 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7\": container with ID starting with a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7 not found: ID does not exist" containerID="a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7" Jul 17 17:33:16.113175 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.113140 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7"} err="failed to get container status \"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7\": rpc error: code = NotFound desc = could not find container \"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7\": container with ID starting with a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7 not found: ID does not exist" Jul 17 17:33:16.113175 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.113159 2553 scope.go:117] "RemoveContainer" containerID="111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312" Jul 17 17:33:16.113390 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:33:16.113373 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312\": container with ID starting with 111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312 not found: ID does not exist" containerID="111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312" Jul 17 17:33:16.113435 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.113396 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312"} err="failed to get container status \"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312\": rpc error: code = NotFound desc = could not find container \"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312\": container with ID starting with 111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312 not found: ID does not exist" Jul 17 17:33:16.113435 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.113411 2553 scope.go:117] "RemoveContainer" containerID="e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c" Jul 17 17:33:16.113659 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:33:16.113644 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c\": container with ID starting with e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c not found: ID does not exist" containerID="e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c" Jul 17 17:33:16.113696 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.113665 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c"} err="failed to get container status \"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c\": rpc error: code = NotFound desc = could not find container \"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c\": container with ID starting with e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c not found: ID does not exist" Jul 17 17:33:16.113728 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.113699 2553 scope.go:117] "RemoveContainer" containerID="fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70" Jul 17 17:33:16.113944 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:33:16.113928 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70\": container with ID starting with fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70 not found: ID does not exist" containerID="fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70" Jul 17 17:33:16.113988 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.113949 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70"} err="failed to get container status \"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70\": rpc error: code = NotFound desc = could not find container \"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70\": container with ID starting with fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70 not found: ID does not exist" Jul 17 17:33:16.113988 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.113963 2553 scope.go:117] "RemoveContainer" containerID="ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94" Jul 17 17:33:16.114168 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:33:16.114154 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94\": container with ID starting with ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94 not found: ID does not exist" containerID="ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94" Jul 17 17:33:16.114202 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.114176 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94"} err="failed to get container status \"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94\": rpc error: code = NotFound desc = could not find container \"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94\": container with ID starting with ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94 not found: ID does not exist" Jul 17 17:33:16.114202 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.114200 2553 scope.go:117] "RemoveContainer" containerID="19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629" Jul 17 17:33:16.114437 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:33:16.114423 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629\": container with ID starting with 19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629 not found: ID does not exist" containerID="19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629" Jul 17 17:33:16.114471 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.114442 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629"} err="failed to get container status \"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629\": rpc error: code = NotFound desc = could not find container \"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629\": container with ID starting with 19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629 not found: ID does not exist" Jul 17 17:33:16.114471 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.114455 2553 scope.go:117] "RemoveContainer" containerID="44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4" Jul 17 17:33:16.114684 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:33:16.114671 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4\": container with ID starting with 44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4 not found: ID does not exist" containerID="44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4" Jul 17 17:33:16.114715 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.114688 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4"} err="failed to get container status \"44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4\": rpc error: code = NotFound desc = could not find container \"44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4\": container with ID starting with 44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4 not found: ID does not exist" Jul 17 17:33:16.114715 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.114708 2553 scope.go:117] "RemoveContainer" containerID="a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7" Jul 17 17:33:16.114961 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.114942 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7"} err="failed to get container status \"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7\": rpc error: code = NotFound desc = could not find container \"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7\": container with ID starting with a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7 not found: ID does not exist" Jul 17 17:33:16.115022 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.114962 2553 scope.go:117] "RemoveContainer" containerID="111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312" Jul 17 17:33:16.115206 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.115190 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312"} err="failed to get container status \"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312\": rpc error: code = NotFound desc = could not find container \"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312\": container with ID starting with 111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312 not found: ID does not exist" Jul 17 17:33:16.115254 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.115206 2553 scope.go:117] "RemoveContainer" containerID="e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c" Jul 17 17:33:16.115403 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.115385 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c"} err="failed to get container status \"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c\": rpc error: code = NotFound desc = could not find container \"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c\": container with ID starting with e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c not found: ID does not exist" Jul 17 17:33:16.115455 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.115403 2553 scope.go:117] "RemoveContainer" containerID="fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70" Jul 17 17:33:16.115595 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.115579 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70"} err="failed to get container status \"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70\": rpc error: code = NotFound desc = could not find container \"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70\": container with ID starting with fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70 not found: ID does not exist" Jul 17 17:33:16.115637 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.115595 2553 scope.go:117] "RemoveContainer" containerID="ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94" Jul 17 17:33:16.115755 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.115735 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94"} err="failed to get container status \"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94\": rpc error: code = NotFound desc = could not find container \"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94\": container with ID starting with ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94 not found: ID does not exist" Jul 17 17:33:16.115805 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.115756 2553 scope.go:117] "RemoveContainer" containerID="19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629" Jul 17 17:33:16.115989 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.115971 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629"} err="failed to get container status \"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629\": rpc error: code = NotFound desc = could not find container \"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629\": container with ID starting with 19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629 not found: ID does not exist" Jul 17 17:33:16.116032 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.115990 2553 scope.go:117] "RemoveContainer" containerID="44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4" Jul 17 17:33:16.116194 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.116178 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4"} err="failed to get container status \"44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4\": rpc error: code = NotFound desc = could not find container \"44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4\": container with ID starting with 44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4 not found: ID does not exist" Jul 17 17:33:16.116234 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.116196 2553 scope.go:117] "RemoveContainer" containerID="a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7" Jul 17 17:33:16.116405 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.116388 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7"} err="failed to get container status \"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7\": rpc error: code = NotFound desc = could not find container \"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7\": container with ID starting with a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7 not found: ID does not exist" Jul 17 17:33:16.116405 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.116405 2553 scope.go:117] "RemoveContainer" containerID="111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312" Jul 17 17:33:16.116596 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.116580 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312"} err="failed to get container status \"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312\": rpc error: code = NotFound desc = could not find container \"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312\": container with ID starting with 111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312 not found: ID does not exist" Jul 17 17:33:16.116596 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.116595 2553 scope.go:117] "RemoveContainer" containerID="e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c" Jul 17 17:33:16.116809 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.116790 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c"} err="failed to get container status \"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c\": rpc error: code = NotFound desc = could not find container \"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c\": container with ID starting with e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c not found: ID does not exist" Jul 17 17:33:16.116866 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.116809 2553 scope.go:117] "RemoveContainer" containerID="fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70" Jul 17 17:33:16.117021 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.117005 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70"} err="failed to get container status \"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70\": rpc error: code = NotFound desc = could not find container \"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70\": container with ID starting with fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70 not found: ID does not exist" Jul 17 17:33:16.117074 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.117022 2553 scope.go:117] "RemoveContainer" containerID="ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94" Jul 17 17:33:16.117258 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.117220 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94"} err="failed to get container status \"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94\": rpc error: code = NotFound desc = could not find container \"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94\": container with ID starting with ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94 not found: ID does not exist" Jul 17 17:33:16.117258 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.117242 2553 scope.go:117] "RemoveContainer" containerID="19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629" Jul 17 17:33:16.117502 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.117482 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629"} err="failed to get container status \"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629\": rpc error: code = NotFound desc = could not find container \"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629\": container with ID starting with 19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629 not found: ID does not exist" Jul 17 17:33:16.117578 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.117504 2553 scope.go:117] "RemoveContainer" containerID="44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4" Jul 17 17:33:16.117762 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.117739 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4"} err="failed to get container status \"44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4\": rpc error: code = NotFound desc = could not find container \"44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4\": container with ID starting with 44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4 not found: ID does not exist" Jul 17 17:33:16.117762 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.117762 2553 scope.go:117] "RemoveContainer" containerID="a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7" Jul 17 17:33:16.118049 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.118027 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7"} err="failed to get container status \"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7\": rpc error: code = NotFound desc = could not find container \"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7\": container with ID starting with a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7 not found: ID does not exist" Jul 17 17:33:16.118102 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.118050 2553 scope.go:117] "RemoveContainer" containerID="111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312" Jul 17 17:33:16.118310 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.118293 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312"} err="failed to get container status \"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312\": rpc error: code = NotFound desc = could not find container \"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312\": container with ID starting with 111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312 not found: ID does not exist" Jul 17 17:33:16.118355 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.118311 2553 scope.go:117] "RemoveContainer" containerID="e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c" Jul 17 17:33:16.118508 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.118490 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c"} err="failed to get container status \"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c\": rpc error: code = NotFound desc = could not find container \"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c\": container with ID starting with e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c not found: ID does not exist" Jul 17 17:33:16.118544 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.118508 2553 scope.go:117] "RemoveContainer" containerID="fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70" Jul 17 17:33:16.118734 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.118713 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70"} err="failed to get container status \"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70\": rpc error: code = NotFound desc = could not find container \"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70\": container with ID starting with fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70 not found: ID does not exist" Jul 17 17:33:16.118734 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.118732 2553 scope.go:117] "RemoveContainer" containerID="ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94" Jul 17 17:33:16.118991 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.118970 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94"} err="failed to get container status \"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94\": rpc error: code = NotFound desc = could not find container \"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94\": container with ID starting with ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94 not found: ID does not exist" Jul 17 17:33:16.118991 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.118990 2553 scope.go:117] "RemoveContainer" containerID="19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629" Jul 17 17:33:16.119209 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.119191 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629"} err="failed to get container status \"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629\": rpc error: code = NotFound desc = could not find container \"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629\": container with ID starting with 19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629 not found: ID does not exist" Jul 17 17:33:16.119249 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.119210 2553 scope.go:117] "RemoveContainer" containerID="44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4" Jul 17 17:33:16.119424 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.119407 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4"} err="failed to get container status \"44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4\": rpc error: code = NotFound desc = could not find container \"44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4\": container with ID starting with 44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4 not found: ID does not exist" Jul 17 17:33:16.119491 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.119437 2553 scope.go:117] "RemoveContainer" containerID="a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7" Jul 17 17:33:16.119686 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.119670 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7"} err="failed to get container status \"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7\": rpc error: code = NotFound desc = could not find container \"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7\": container with ID starting with a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7 not found: ID does not exist" Jul 17 17:33:16.119733 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.119686 2553 scope.go:117] "RemoveContainer" containerID="111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312" Jul 17 17:33:16.119927 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.119910 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312"} err="failed to get container status \"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312\": rpc error: code = NotFound desc = could not find container \"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312\": container with ID starting with 111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312 not found: ID does not exist" Jul 17 17:33:16.120039 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.119929 2553 scope.go:117] "RemoveContainer" containerID="e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c" Jul 17 17:33:16.120175 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.120152 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c"} err="failed to get container status \"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c\": rpc error: code = NotFound desc = could not find container \"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c\": container with ID starting with e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c not found: ID does not exist" Jul 17 17:33:16.120213 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.120176 2553 scope.go:117] "RemoveContainer" containerID="fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70" Jul 17 17:33:16.120363 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.120346 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70"} err="failed to get container status \"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70\": rpc error: code = NotFound desc = could not find container \"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70\": container with ID starting with fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70 not found: ID does not exist" Jul 17 17:33:16.120398 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.120364 2553 scope.go:117] "RemoveContainer" containerID="ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94" Jul 17 17:33:16.120557 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.120543 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94"} err="failed to get container status \"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94\": rpc error: code = NotFound desc = could not find container \"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94\": container with ID starting with ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94 not found: ID does not exist" Jul 17 17:33:16.120607 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.120558 2553 scope.go:117] "RemoveContainer" containerID="19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629" Jul 17 17:33:16.120755 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.120740 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629"} err="failed to get container status \"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629\": rpc error: code = NotFound desc = could not find container \"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629\": container with ID starting with 19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629 not found: ID does not exist" Jul 17 17:33:16.120797 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.120754 2553 scope.go:117] "RemoveContainer" containerID="44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4" Jul 17 17:33:16.121071 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.121053 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4"} err="failed to get container status \"44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4\": rpc error: code = NotFound desc = could not find container \"44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4\": container with ID starting with 44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4 not found: ID does not exist" Jul 17 17:33:16.121151 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.121072 2553 scope.go:117] "RemoveContainer" containerID="a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7" Jul 17 17:33:16.121290 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.121275 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7"} err="failed to get container status \"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7\": rpc error: code = NotFound desc = could not find container \"a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7\": container with ID starting with a8f6b70cf9b0d24bbefb8734ac6c43c4d00900476f95c45c02bd73f6f083cde7 not found: ID does not exist" Jul 17 17:33:16.121348 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.121292 2553 scope.go:117] "RemoveContainer" containerID="111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312" Jul 17 17:33:16.121508 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.121493 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312"} err="failed to get container status \"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312\": rpc error: code = NotFound desc = could not find container \"111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312\": container with ID starting with 111488b126c1db34b0602c73ed08c1092c06bd3894cf8b606463c15c163be312 not found: ID does not exist" Jul 17 17:33:16.121571 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.121510 2553 scope.go:117] "RemoveContainer" containerID="e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c" Jul 17 17:33:16.121724 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.121710 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c"} err="failed to get container status \"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c\": rpc error: code = NotFound desc = could not find container \"e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c\": container with ID starting with e91363b66d9da89bc291f49f8e22047cad4efd85b485eefdf0d713e6eb89855c not found: ID does not exist" Jul 17 17:33:16.121788 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.121726 2553 scope.go:117] "RemoveContainer" containerID="fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70" Jul 17 17:33:16.121957 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.121939 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70"} err="failed to get container status \"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70\": rpc error: code = NotFound desc = could not find container \"fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70\": container with ID starting with fb92f86064cd764dbdbdbadd103fe421923c3cc464fae80f7aac7cb78294ab70 not found: ID does not exist" Jul 17 17:33:16.122029 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.121958 2553 scope.go:117] "RemoveContainer" containerID="ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94" Jul 17 17:33:16.122177 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.122160 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94"} err="failed to get container status \"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94\": rpc error: code = NotFound desc = could not find container \"ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94\": container with ID starting with ce44867d103f2673c3b0c292197cbee0a75fbf1e84b2b96ed4e862dfb722aa94 not found: ID does not exist" Jul 17 17:33:16.122230 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.122178 2553 scope.go:117] "RemoveContainer" containerID="19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629" Jul 17 17:33:16.122356 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.122340 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629"} err="failed to get container status \"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629\": rpc error: code = NotFound desc = could not find container \"19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629\": container with ID starting with 19a4a368fb972d9efa461f06ab4e6806f70ccad52adeb552985d41e32a138629 not found: ID does not exist" Jul 17 17:33:16.122402 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.122355 2553 scope.go:117] "RemoveContainer" containerID="44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4" Jul 17 17:33:16.122593 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.122574 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4"} err="failed to get container status \"44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4\": rpc error: code = NotFound desc = could not find container \"44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4\": container with ID starting with 44eb492d36603a04b840e736a6e178a9719e8c21ab711c62acb3f9bc06cd41a4 not found: ID does not exist" Jul 17 17:33:16.127041 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127020 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:33:16.127353 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127336 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="kube-rbac-proxy" Jul 17 17:33:16.127427 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127355 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="kube-rbac-proxy" Jul 17 17:33:16.127427 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127371 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="alertmanager" Jul 17 17:33:16.127427 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127379 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="alertmanager" Jul 17 17:33:16.127427 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127389 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="kube-rbac-proxy-metric" Jul 17 17:33:16.127427 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127398 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="kube-rbac-proxy-metric" Jul 17 17:33:16.127427 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127407 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="thanos-sidecar" Jul 17 17:33:16.127427 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127415 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="thanos-sidecar" Jul 17 17:33:16.127427 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127424 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="init-config-reloader" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127432 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="init-config-reloader" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127442 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="prom-label-proxy" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127450 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="prom-label-proxy" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127463 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="kube-rbac-proxy-web" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127470 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="kube-rbac-proxy-web" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127481 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="init-config-reloader" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127489 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="init-config-reloader" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127505 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="config-reloader" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127513 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="config-reloader" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127522 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d59d6ab6-e518-44b8-acc9-6df2260cb7ef" containerName="console" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127529 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d59d6ab6-e518-44b8-acc9-6df2260cb7ef" containerName="console" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127542 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="kube-rbac-proxy" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127549 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="kube-rbac-proxy" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127559 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="kube-rbac-proxy-web" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127567 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="kube-rbac-proxy-web" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127582 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="kube-rbac-proxy-thanos" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127590 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="kube-rbac-proxy-thanos" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127600 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="config-reloader" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127607 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="config-reloader" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127616 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="prometheus" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127624 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="prometheus" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127703 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="kube-rbac-proxy-metric" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127716 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="prometheus" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127724 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="prom-label-proxy" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127731 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="kube-rbac-proxy-thanos" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127739 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="kube-rbac-proxy" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127748 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="config-reloader" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127755 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="config-reloader" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127764 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="kube-rbac-proxy-web" Jul 17 17:33:16.127863 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127771 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d59d6ab6-e518-44b8-acc9-6df2260cb7ef" containerName="console" Jul 17 17:33:16.129187 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127778 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="kube-rbac-proxy" Jul 17 17:33:16.129187 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127788 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="alertmanager" Jul 17 17:33:16.129187 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127800 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="97cc2052-a3b0-4457-bf60-f48d33225882" containerName="kube-rbac-proxy-web" Jul 17 17:33:16.129187 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.127809 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" containerName="thanos-sidecar" Jul 17 17:33:16.134308 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.134291 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.137224 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.137201 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 17 17:33:16.137306 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.137201 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 17 17:33:16.137368 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.137344 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fdwwn\"" Jul 17 17:33:16.137713 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.137692 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 17 17:33:16.137797 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.137722 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 17 17:33:16.137797 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.137728 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 17 17:33:16.138234 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.138082 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 17 17:33:16.138234 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.138094 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 17 17:33:16.138234 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.138102 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 17 17:33:16.138234 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.138176 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 17 17:33:16.138234 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.138207 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e3r2q2fd95gf2\"" Jul 17 17:33:16.138813 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.138796 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 17 17:33:16.138957 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.138924 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 17 17:33:16.139424 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.139411 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 17 17:33:16.140568 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.140546 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 17 17:33:16.144040 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.144021 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 17 17:33:16.151010 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.150991 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:33:16.191622 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.191588 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.191879 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.191639 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.191879 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.191674 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.191879 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.191710 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.191879 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.191740 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.191879 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.191772 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.191879 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.191791 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.191879 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.191807 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.191879 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.191831 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.192290 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.192007 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.192290 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.192047 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.192290 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.192090 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.192290 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.192131 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb664\" (UniqueName: \"kubernetes.io/projected/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-kube-api-access-jb664\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.192290 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.192159 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.192290 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.192199 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.192290 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.192221 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.192290 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.192247 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.192290 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.192270 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.192608 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.192295 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.192608 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.192372 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-config\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.293366 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.293266 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.293366 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.293316 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.293595 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.293382 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.293595 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.293448 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.293595 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.293482 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.293595 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.293538 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-config\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.293595 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.293573 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.293841 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.293601 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.293841 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.293631 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.293841 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.293658 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.293841 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.293692 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.293841 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.293725 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.293841 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.293756 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.293841 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.293781 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.293841 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.293814 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.295103 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.293891 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.295103 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.293922 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.295103 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.293951 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.295103 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.293979 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb664\" (UniqueName: \"kubernetes.io/projected/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-kube-api-access-jb664\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.295103 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.294006 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.295103 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.294164 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.295103 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.294271 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.295103 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.294430 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.295103 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.294887 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.295103 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.295104 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.295576 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.295396 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.296090 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.296063 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.297042 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.296938 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.298312 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.297806 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.298312 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.297865 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.298312 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.297916 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.298312 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.298277 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.298714 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.298691 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-config\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.298816 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.298760 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.298998 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.298970 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.298998 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.298974 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.299435 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.299421 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.299471 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.299454 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.299524 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.299508 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.305393 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.305372 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb664\" (UniqueName: \"kubernetes.io/projected/fc697c1e-495a-4c6a-9daf-8c8b6670e4d7-kube-api-access-jb664\") pod \"prometheus-k8s-0\" (UID: \"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.444011 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.443973 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:16.796194 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:33:16.796166 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc697c1e_495a_4c6a_9daf_8c8b6670e4d7.slice/crio-eb4017edecb03152ca612264ab2e52977162f167fb9f979a350572f18c2c88f6 WatchSource:0}: Error finding container eb4017edecb03152ca612264ab2e52977162f167fb9f979a350572f18c2c88f6: Status 404 returned error can't find the container with id eb4017edecb03152ca612264ab2e52977162f167fb9f979a350572f18c2c88f6 Jul 17 17:33:16.796351 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:16.796328 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:33:17.068546 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:17.068473 2553 generic.go:356] "Generic (PLEG): container finished" podID="fc697c1e-495a-4c6a-9daf-8c8b6670e4d7" containerID="e5a4acbd4f29e2d0b4a757d626971a90ffd6b04250dae2538dd0aca19aec0699" exitCode=0 Jul 17 17:33:17.068546 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:17.068507 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7","Type":"ContainerDied","Data":"e5a4acbd4f29e2d0b4a757d626971a90ffd6b04250dae2538dd0aca19aec0699"} Jul 17 17:33:17.068987 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:17.068554 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7","Type":"ContainerStarted","Data":"eb4017edecb03152ca612264ab2e52977162f167fb9f979a350572f18c2c88f6"} Jul 17 17:33:17.465161 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:17.465127 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5200de1-7ea0-4a0a-84fd-6f78fd513b28" path="/var/lib/kubelet/pods/f5200de1-7ea0-4a0a-84fd-6f78fd513b28/volumes" Jul 17 17:33:18.077172 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:18.077137 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7","Type":"ContainerStarted","Data":"efc50d592bb553409b049eb29a2d2246f3456caab27edaa938f1a96dafdc1ee9"} Jul 17 17:33:18.077633 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:18.077211 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7","Type":"ContainerStarted","Data":"e3997074b731c17ae94847623efe099eae85c35e5b5f2af9a6a9c7d2cc9a5949"} Jul 17 17:33:18.077633 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:18.077249 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7","Type":"ContainerStarted","Data":"f85521f0af6c2cbf9d2c77f5a7d8685df11244310745b398c79fac3f2fb67c9b"} Jul 17 17:33:18.077633 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:18.077293 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7","Type":"ContainerStarted","Data":"1421b2a56f05ded44d892023e1916438f01ab430feaad9d78224113371eaf545"} Jul 17 17:33:18.077633 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:18.077334 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7","Type":"ContainerStarted","Data":"f89577b4fc9e27bd79acb860f30471e0465c7ef1c56e04547e21af5265abc1b1"} Jul 17 17:33:18.077633 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:18.077374 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc697c1e-495a-4c6a-9daf-8c8b6670e4d7","Type":"ContainerStarted","Data":"59aa965472681826e8da260aac8d6f3804374b0d85e8c79776affed24f17f950"} Jul 17 17:33:18.111032 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:18.110970 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.094488627 podStartE2EDuration="2.110948578s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.459951ms" isStatefulPod=false podCreationTimestamp="2026-07-17 17:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:33:18.109053565 +0000 UTC m=+119.227881848" watchObservedRunningTime="2026-07-17 17:33:18.110948578 +0000 UTC m=+119.229776862" Jul 17 17:33:18.304604 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:18.304573 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:18.405710 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:18.405624 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:21.161278 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:21.161239 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:21.348648 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:21.348607 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:21.444979 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:21.444904 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:33:21.448788 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:21.448756 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:21.561359 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:21.561329 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:30.159353 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.159317 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bc9f9794f-btxkk"] Jul 17 17:33:30.164275 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.164259 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.169583 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.169552 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 17 17:33:30.171138 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.171119 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 17 17:33:30.171679 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.171655 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 17 17:33:30.172334 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.172317 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 17 17:33:30.172619 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.172605 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 17 17:33:30.173604 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.173589 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 17 17:33:30.173867 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.173837 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 17 17:33:30.178054 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.178039 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 17 17:33:30.178501 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.178488 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qd42z\"" Jul 17 17:33:30.192221 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.192196 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bc9f9794f-btxkk"] Jul 17 17:33:30.249153 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.249109 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:30.325980 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.325950 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-oauth-serving-cert\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.326124 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.325983 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2gcp\" (UniqueName: \"kubernetes.io/projected/7db3deea-8080-44fe-b0c9-434a94a2d37f-kube-api-access-d2gcp\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.326124 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.326013 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7db3deea-8080-44fe-b0c9-434a94a2d37f-console-serving-cert\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.326193 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.326121 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7db3deea-8080-44fe-b0c9-434a94a2d37f-console-oauth-config\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.326193 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.326156 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-console-config\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.326251 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.326189 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-trusted-ca-bundle\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.326251 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.326220 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-service-ca\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.375123 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.375083 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:30.427497 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.427405 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-trusted-ca-bundle\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.427497 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.427456 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-service-ca\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.427704 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.427499 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-oauth-serving-cert\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.427704 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.427524 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2gcp\" (UniqueName: \"kubernetes.io/projected/7db3deea-8080-44fe-b0c9-434a94a2d37f-kube-api-access-d2gcp\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.427704 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.427579 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7db3deea-8080-44fe-b0c9-434a94a2d37f-console-serving-cert\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.427704 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.427627 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7db3deea-8080-44fe-b0c9-434a94a2d37f-console-oauth-config\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.427704 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.427658 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-console-config\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.428365 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.428332 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-oauth-serving-cert\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.428365 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.428347 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-service-ca\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.428498 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.428404 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-console-config\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.428557 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.428542 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-trusted-ca-bundle\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.430005 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.429987 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7db3deea-8080-44fe-b0c9-434a94a2d37f-console-oauth-config\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.430125 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.430109 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7db3deea-8080-44fe-b0c9-434a94a2d37f-console-serving-cert\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.444800 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.444776 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2gcp\" (UniqueName: \"kubernetes.io/projected/7db3deea-8080-44fe-b0c9-434a94a2d37f-kube-api-access-d2gcp\") pod \"console-5bc9f9794f-btxkk\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.473861 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.473825 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:30.822568 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:30.822544 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bc9f9794f-btxkk"] Jul 17 17:33:30.827211 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:33:30.827158 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db3deea_8080_44fe_b0c9_434a94a2d37f.slice/crio-ef609787d9461a69008e869e443e92a18a16f4b23c80b6871eceef33e2f068cd WatchSource:0}: Error finding container ef609787d9461a69008e869e443e92a18a16f4b23c80b6871eceef33e2f068cd: Status 404 returned error can't find the container with id ef609787d9461a69008e869e443e92a18a16f4b23c80b6871eceef33e2f068cd Jul 17 17:33:31.127679 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:31.127582 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc9f9794f-btxkk" event={"ID":"7db3deea-8080-44fe-b0c9-434a94a2d37f","Type":"ContainerStarted","Data":"ed2037d9342cf9c304a1f3b7c96421510783994ceafaa05aa4676fe09304cbd8"} Jul 17 17:33:31.127679 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:31.127633 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc9f9794f-btxkk" event={"ID":"7db3deea-8080-44fe-b0c9-434a94a2d37f","Type":"ContainerStarted","Data":"ef609787d9461a69008e869e443e92a18a16f4b23c80b6871eceef33e2f068cd"} Jul 17 17:33:31.150189 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:31.150131 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bc9f9794f-btxkk" podStartSLOduration=1.150108012 podStartE2EDuration="1.150108012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:33:31.149039952 +0000 UTC m=+132.267868232" watchObservedRunningTime="2026-07-17 17:33:31.150108012 +0000 UTC m=+132.268936290" Jul 17 17:33:40.474623 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:40.474578 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:40.475047 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:40.474659 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:40.479008 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:40.478987 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:41.167239 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:41.167210 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:33:41.409410 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:41.409379 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:33:41.644573 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:33:41.644537 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:08.459739 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:08.459698 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:08.559712 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:08.559679 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:16.444701 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:16.444600 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:34:16.459554 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:16.459528 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:34:17.315410 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:17.315382 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:34:30.438055 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:30.438022 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:31.374258 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:31.374221 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:34.437714 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:34.437663 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:34.535332 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:34.535296 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:36.317762 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:36.317720 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:36.419465 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:36.419430 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:36.622134 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:36.622044 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:36.758293 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:36.758260 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:38.553433 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:38.553397 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:39.325283 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:39.325237 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:41.227873 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:41.227816 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:41.329589 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:41.329550 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:46.929131 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:46.929092 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:48.877135 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:48.877097 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:49.055395 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:49.055358 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:54.094698 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:54.094660 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:54.258524 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:54.258490 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:56.252448 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:56.252413 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:56.476092 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:56.476061 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:34:56.583630 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:34:56.583546 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:04.902431 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:04.902397 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:05.003544 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:05.003514 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:11.511627 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:11.511587 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:11.606808 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:11.606771 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:12.097638 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.097600 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:12.457151 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.457056 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:12.560337 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.560302 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:12.617121 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.617085 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t"] Jul 17 17:35:12.620469 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.620449 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" Jul 17 17:35:12.625579 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.625559 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 17 17:35:12.625699 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.625606 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 17 17:35:12.625699 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.625664 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rvvj5\"" Jul 17 17:35:12.625992 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.625977 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 17 17:35:12.626159 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.626136 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 17 17:35:12.650385 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.650359 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t"] Jul 17 17:35:12.704663 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.704627 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65d884c4-b8c9-44bd-8e18-833eb3f38bd6-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5f54cb78-bf48t\" (UID: \"65d884c4-b8c9-44bd-8e18-833eb3f38bd6\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" Jul 17 17:35:12.704835 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.704667 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65d884c4-b8c9-44bd-8e18-833eb3f38bd6-tmp\") pod \"opendatahub-operator-controller-manager-7f5f54cb78-bf48t\" (UID: \"65d884c4-b8c9-44bd-8e18-833eb3f38bd6\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" Jul 17 17:35:12.704835 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.704695 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65d884c4-b8c9-44bd-8e18-833eb3f38bd6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5f54cb78-bf48t\" (UID: \"65d884c4-b8c9-44bd-8e18-833eb3f38bd6\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" Jul 17 17:35:12.704835 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.704737 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65d884c4-b8c9-44bd-8e18-833eb3f38bd6-manifests\") pod \"opendatahub-operator-controller-manager-7f5f54cb78-bf48t\" (UID: \"65d884c4-b8c9-44bd-8e18-833eb3f38bd6\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" Jul 17 17:35:12.705018 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.704831 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bphbl\" (UniqueName: \"kubernetes.io/projected/65d884c4-b8c9-44bd-8e18-833eb3f38bd6-kube-api-access-bphbl\") pod \"opendatahub-operator-controller-manager-7f5f54cb78-bf48t\" (UID: \"65d884c4-b8c9-44bd-8e18-833eb3f38bd6\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" Jul 17 17:35:12.806135 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.806101 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65d884c4-b8c9-44bd-8e18-833eb3f38bd6-manifests\") pod \"opendatahub-operator-controller-manager-7f5f54cb78-bf48t\" (UID: \"65d884c4-b8c9-44bd-8e18-833eb3f38bd6\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" Jul 17 17:35:12.806328 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.806159 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bphbl\" (UniqueName: \"kubernetes.io/projected/65d884c4-b8c9-44bd-8e18-833eb3f38bd6-kube-api-access-bphbl\") pod \"opendatahub-operator-controller-manager-7f5f54cb78-bf48t\" (UID: \"65d884c4-b8c9-44bd-8e18-833eb3f38bd6\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" Jul 17 17:35:12.806328 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.806198 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65d884c4-b8c9-44bd-8e18-833eb3f38bd6-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5f54cb78-bf48t\" (UID: \"65d884c4-b8c9-44bd-8e18-833eb3f38bd6\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" Jul 17 17:35:12.806328 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.806223 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65d884c4-b8c9-44bd-8e18-833eb3f38bd6-tmp\") pod \"opendatahub-operator-controller-manager-7f5f54cb78-bf48t\" (UID: \"65d884c4-b8c9-44bd-8e18-833eb3f38bd6\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" Jul 17 17:35:12.806328 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.806241 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65d884c4-b8c9-44bd-8e18-833eb3f38bd6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5f54cb78-bf48t\" (UID: \"65d884c4-b8c9-44bd-8e18-833eb3f38bd6\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" Jul 17 17:35:12.806549 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.806526 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65d884c4-b8c9-44bd-8e18-833eb3f38bd6-manifests\") pod \"opendatahub-operator-controller-manager-7f5f54cb78-bf48t\" (UID: \"65d884c4-b8c9-44bd-8e18-833eb3f38bd6\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" Jul 17 17:35:12.808435 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.808412 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65d884c4-b8c9-44bd-8e18-833eb3f38bd6-tmp\") pod \"opendatahub-operator-controller-manager-7f5f54cb78-bf48t\" (UID: \"65d884c4-b8c9-44bd-8e18-833eb3f38bd6\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" Jul 17 17:35:12.808616 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.808596 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65d884c4-b8c9-44bd-8e18-833eb3f38bd6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5f54cb78-bf48t\" (UID: \"65d884c4-b8c9-44bd-8e18-833eb3f38bd6\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" Jul 17 17:35:12.808676 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.808619 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65d884c4-b8c9-44bd-8e18-833eb3f38bd6-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5f54cb78-bf48t\" (UID: \"65d884c4-b8c9-44bd-8e18-833eb3f38bd6\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" Jul 17 17:35:12.819123 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.819100 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:12.819452 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.819427 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bphbl\" (UniqueName: \"kubernetes.io/projected/65d884c4-b8c9-44bd-8e18-833eb3f38bd6-kube-api-access-bphbl\") pod \"opendatahub-operator-controller-manager-7f5f54cb78-bf48t\" (UID: \"65d884c4-b8c9-44bd-8e18-833eb3f38bd6\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" Jul 17 17:35:12.917501 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.917462 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:12.930372 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:12.930353 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" Jul 17 17:35:13.016739 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:13.016708 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:13.073519 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:13.073487 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t"] Jul 17 17:35:13.076006 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:35:13.075983 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d884c4_b8c9_44bd_8e18_833eb3f38bd6.slice/crio-0c5c619a27fe30b788dc39bd4c271ab1ca8750d450fe6436e0c599a22270848f WatchSource:0}: Error finding container 0c5c619a27fe30b788dc39bd4c271ab1ca8750d450fe6436e0c599a22270848f: Status 404 returned error can't find the container with id 0c5c619a27fe30b788dc39bd4c271ab1ca8750d450fe6436e0c599a22270848f Jul 17 17:35:13.121019 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:13.120976 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:13.222281 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:13.222243 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:13.320714 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:13.320681 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:13.521184 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:13.521149 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" event={"ID":"65d884c4-b8c9-44bd-8e18-833eb3f38bd6","Type":"ContainerStarted","Data":"0c5c619a27fe30b788dc39bd4c271ab1ca8750d450fe6436e0c599a22270848f"} Jul 17 17:35:16.536935 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:16.536898 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" event={"ID":"65d884c4-b8c9-44bd-8e18-833eb3f38bd6","Type":"ContainerStarted","Data":"8359fd981b4c585a4f6dc1764c8ed38bbf47fc7eee1dab51382d038af13bea11"} Jul 17 17:35:17.542797 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:17.542762 2553 generic.go:356] "Generic (PLEG): container finished" podID="65d884c4-b8c9-44bd-8e18-833eb3f38bd6" containerID="8359fd981b4c585a4f6dc1764c8ed38bbf47fc7eee1dab51382d038af13bea11" exitCode=0 Jul 17 17:35:17.543211 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:17.542869 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" event={"ID":"65d884c4-b8c9-44bd-8e18-833eb3f38bd6","Type":"ContainerDied","Data":"8359fd981b4c585a4f6dc1764c8ed38bbf47fc7eee1dab51382d038af13bea11"} Jul 17 17:35:18.548864 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:18.548823 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" event={"ID":"65d884c4-b8c9-44bd-8e18-833eb3f38bd6","Type":"ContainerStarted","Data":"e1445e089f0c0714dac695aca7981cf581c9362ac303e3de4e612317f084ca3f"} Jul 17 17:35:18.549252 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:18.548966 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" Jul 17 17:35:18.573787 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:18.573731 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" podStartSLOduration=2.352653648 podStartE2EDuration="6.573713182s" totalImagesPullingTime="3.17040679s" totalInitContainerRuntime="1.050652744s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:35:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:35:18.573194338 +0000 UTC m=+239.692022632" watchObservedRunningTime="2026-07-17 17:35:18.573713182 +0000 UTC m=+239.692541528" Jul 17 17:35:21.925047 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:21.925008 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:22.075638 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:22.075607 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:29.555500 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:29.555463 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f5f54cb78-bf48t" Jul 17 17:35:31.637418 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:31.637379 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:31.825918 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:31.825886 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:33.002785 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:33.002755 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:33.437040 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:33.436956 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:33.790236 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:33.790203 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:34.511193 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:34.511159 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:34.543070 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:34.543032 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-56969c4fd6-2tqjl"] Jul 17 17:35:34.554016 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:34.553990 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-56969c4fd6-2tqjl" Jul 17 17:35:34.564161 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:34.564138 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 17 17:35:34.564395 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:34.564376 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 17 17:35:34.564786 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:34.564772 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xx7n4\"" Jul 17 17:35:34.567270 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:34.567257 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 17 17:35:34.568600 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:34.568585 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 17 17:35:34.602504 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:34.602480 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-56969c4fd6-2tqjl"] Jul 17 17:35:34.702000 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:34.701972 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0abe360-abcf-4529-82d4-f37d8caf0141-tmp\") pod \"kube-auth-proxy-56969c4fd6-2tqjl\" (UID: \"f0abe360-abcf-4529-82d4-f37d8caf0141\") " pod="openshift-ingress/kube-auth-proxy-56969c4fd6-2tqjl" Jul 17 17:35:34.702183 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:34.702014 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-579cs\" (UniqueName: \"kubernetes.io/projected/f0abe360-abcf-4529-82d4-f37d8caf0141-kube-api-access-579cs\") pod \"kube-auth-proxy-56969c4fd6-2tqjl\" (UID: \"f0abe360-abcf-4529-82d4-f37d8caf0141\") " pod="openshift-ingress/kube-auth-proxy-56969c4fd6-2tqjl" Jul 17 17:35:34.702183 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:34.702038 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0abe360-abcf-4529-82d4-f37d8caf0141-tls-certs\") pod \"kube-auth-proxy-56969c4fd6-2tqjl\" (UID: \"f0abe360-abcf-4529-82d4-f37d8caf0141\") " pod="openshift-ingress/kube-auth-proxy-56969c4fd6-2tqjl" Jul 17 17:35:34.803496 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:34.803414 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0abe360-abcf-4529-82d4-f37d8caf0141-tmp\") pod \"kube-auth-proxy-56969c4fd6-2tqjl\" (UID: \"f0abe360-abcf-4529-82d4-f37d8caf0141\") " pod="openshift-ingress/kube-auth-proxy-56969c4fd6-2tqjl" Jul 17 17:35:34.803496 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:34.803460 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-579cs\" (UniqueName: \"kubernetes.io/projected/f0abe360-abcf-4529-82d4-f37d8caf0141-kube-api-access-579cs\") pod \"kube-auth-proxy-56969c4fd6-2tqjl\" (UID: \"f0abe360-abcf-4529-82d4-f37d8caf0141\") " pod="openshift-ingress/kube-auth-proxy-56969c4fd6-2tqjl" Jul 17 17:35:34.803496 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:34.803480 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0abe360-abcf-4529-82d4-f37d8caf0141-tls-certs\") pod \"kube-auth-proxy-56969c4fd6-2tqjl\" (UID: \"f0abe360-abcf-4529-82d4-f37d8caf0141\") " pod="openshift-ingress/kube-auth-proxy-56969c4fd6-2tqjl" Jul 17 17:35:34.805743 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:34.805720 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0abe360-abcf-4529-82d4-f37d8caf0141-tmp\") pod \"kube-auth-proxy-56969c4fd6-2tqjl\" (UID: \"f0abe360-abcf-4529-82d4-f37d8caf0141\") " pod="openshift-ingress/kube-auth-proxy-56969c4fd6-2tqjl" Jul 17 17:35:34.805878 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:34.805835 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0abe360-abcf-4529-82d4-f37d8caf0141-tls-certs\") pod \"kube-auth-proxy-56969c4fd6-2tqjl\" (UID: \"f0abe360-abcf-4529-82d4-f37d8caf0141\") " pod="openshift-ingress/kube-auth-proxy-56969c4fd6-2tqjl" Jul 17 17:35:34.842200 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:34.842176 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-579cs\" (UniqueName: \"kubernetes.io/projected/f0abe360-abcf-4529-82d4-f37d8caf0141-kube-api-access-579cs\") pod \"kube-auth-proxy-56969c4fd6-2tqjl\" (UID: \"f0abe360-abcf-4529-82d4-f37d8caf0141\") " pod="openshift-ingress/kube-auth-proxy-56969c4fd6-2tqjl" Jul 17 17:35:34.863410 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:34.863385 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-56969c4fd6-2tqjl" Jul 17 17:35:35.028867 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:35.028821 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-56969c4fd6-2tqjl"] Jul 17 17:35:35.041631 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:35:35.041599 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0abe360_abcf_4529_82d4_f37d8caf0141.slice/crio-5fa5bfb8da8c315c7f01e3f7b912ea51d2e1e74e82588fcd0d4575fdf3396624 WatchSource:0}: Error finding container 5fa5bfb8da8c315c7f01e3f7b912ea51d2e1e74e82588fcd0d4575fdf3396624: Status 404 returned error can't find the container with id 5fa5bfb8da8c315c7f01e3f7b912ea51d2e1e74e82588fcd0d4575fdf3396624 Jul 17 17:35:35.619162 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:35.619131 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-56969c4fd6-2tqjl" event={"ID":"f0abe360-abcf-4529-82d4-f37d8caf0141","Type":"ContainerStarted","Data":"5fa5bfb8da8c315c7f01e3f7b912ea51d2e1e74e82588fcd0d4575fdf3396624"} Jul 17 17:35:35.936709 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:35.936616 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:37.140387 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:37.140353 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:37.319019 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:37.318984 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:38.415101 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.415066 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4"] Jul 17 17:35:38.417801 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.417781 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4" Jul 17 17:35:38.420904 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.420885 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 17 17:35:38.421629 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.421613 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 17 17:35:38.421985 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.421959 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 17 17:35:38.421985 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.421964 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 17 17:35:38.422253 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.421965 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vhxp4\"" Jul 17 17:35:38.432257 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.432234 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 17 17:35:38.450540 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.450514 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4"] Jul 17 17:35:38.534861 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.534809 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cb94d097-c65f-482f-ab4f-cbb930048f6c-manager-config\") pod \"lws-controller-manager-76cbd9fc7d-lnhf4\" (UID: \"cb94d097-c65f-482f-ab4f-cbb930048f6c\") " pod="openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4" Jul 17 17:35:38.535021 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.534884 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb94d097-c65f-482f-ab4f-cbb930048f6c-cert\") pod \"lws-controller-manager-76cbd9fc7d-lnhf4\" (UID: \"cb94d097-c65f-482f-ab4f-cbb930048f6c\") " pod="openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4" Jul 17 17:35:38.535074 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.535052 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk8vj\" (UniqueName: \"kubernetes.io/projected/cb94d097-c65f-482f-ab4f-cbb930048f6c-kube-api-access-lk8vj\") pod \"lws-controller-manager-76cbd9fc7d-lnhf4\" (UID: \"cb94d097-c65f-482f-ab4f-cbb930048f6c\") " pod="openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4" Jul 17 17:35:38.535197 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.535174 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb94d097-c65f-482f-ab4f-cbb930048f6c-metrics-cert\") pod \"lws-controller-manager-76cbd9fc7d-lnhf4\" (UID: \"cb94d097-c65f-482f-ab4f-cbb930048f6c\") " pod="openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4" Jul 17 17:35:38.634769 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.634737 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-56969c4fd6-2tqjl" event={"ID":"f0abe360-abcf-4529-82d4-f37d8caf0141","Type":"ContainerStarted","Data":"b058f4a0dbc127d528ab4c0ea0f04dc0488ca073fe96d730c748e7c2882c51b3"} Jul 17 17:35:38.636163 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.636146 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cb94d097-c65f-482f-ab4f-cbb930048f6c-manager-config\") pod \"lws-controller-manager-76cbd9fc7d-lnhf4\" (UID: \"cb94d097-c65f-482f-ab4f-cbb930048f6c\") " pod="openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4" Jul 17 17:35:38.636208 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.636182 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb94d097-c65f-482f-ab4f-cbb930048f6c-cert\") pod \"lws-controller-manager-76cbd9fc7d-lnhf4\" (UID: \"cb94d097-c65f-482f-ab4f-cbb930048f6c\") " pod="openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4" Jul 17 17:35:38.636250 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.636208 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk8vj\" (UniqueName: \"kubernetes.io/projected/cb94d097-c65f-482f-ab4f-cbb930048f6c-kube-api-access-lk8vj\") pod \"lws-controller-manager-76cbd9fc7d-lnhf4\" (UID: \"cb94d097-c65f-482f-ab4f-cbb930048f6c\") " pod="openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4" Jul 17 17:35:38.636250 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.636243 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb94d097-c65f-482f-ab4f-cbb930048f6c-metrics-cert\") pod \"lws-controller-manager-76cbd9fc7d-lnhf4\" (UID: \"cb94d097-c65f-482f-ab4f-cbb930048f6c\") " pod="openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4" Jul 17 17:35:38.636894 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.636870 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cb94d097-c65f-482f-ab4f-cbb930048f6c-manager-config\") pod \"lws-controller-manager-76cbd9fc7d-lnhf4\" (UID: \"cb94d097-c65f-482f-ab4f-cbb930048f6c\") " pod="openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4" Jul 17 17:35:38.638592 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.638571 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb94d097-c65f-482f-ab4f-cbb930048f6c-cert\") pod \"lws-controller-manager-76cbd9fc7d-lnhf4\" (UID: \"cb94d097-c65f-482f-ab4f-cbb930048f6c\") " pod="openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4" Jul 17 17:35:38.638702 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.638627 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb94d097-c65f-482f-ab4f-cbb930048f6c-metrics-cert\") pod \"lws-controller-manager-76cbd9fc7d-lnhf4\" (UID: \"cb94d097-c65f-482f-ab4f-cbb930048f6c\") " pod="openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4" Jul 17 17:35:38.652883 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.652839 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk8vj\" (UniqueName: \"kubernetes.io/projected/cb94d097-c65f-482f-ab4f-cbb930048f6c-kube-api-access-lk8vj\") pod \"lws-controller-manager-76cbd9fc7d-lnhf4\" (UID: \"cb94d097-c65f-482f-ab4f-cbb930048f6c\") " pod="openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4" Jul 17 17:35:38.666709 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.666637 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-56969c4fd6-2tqjl" podStartSLOduration=2.022394523 podStartE2EDuration="4.666623415s" totalImagesPullingTime="2.644228892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:35:38.665607874 +0000 UTC m=+259.784436201" watchObservedRunningTime="2026-07-17 17:35:38.666623415 +0000 UTC m=+259.785451695" Jul 17 17:35:38.727320 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.727291 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4" Jul 17 17:35:38.867516 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:38.867478 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4"] Jul 17 17:35:38.868395 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:35:38.868366 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb94d097_c65f_482f_ab4f_cbb930048f6c.slice/crio-92f538271409944e5dc1334156250234868308895d71d361b42656f09764bb3d WatchSource:0}: Error finding container 92f538271409944e5dc1334156250234868308895d71d361b42656f09764bb3d: Status 404 returned error can't find the container with id 92f538271409944e5dc1334156250234868308895d71d361b42656f09764bb3d Jul 17 17:35:39.115563 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:39.115528 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:39.642157 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:39.642041 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4" event={"ID":"cb94d097-c65f-482f-ab4f-cbb930048f6c","Type":"ContainerStarted","Data":"92f538271409944e5dc1334156250234868308895d71d361b42656f09764bb3d"} Jul 17 17:35:41.652550 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:41.652502 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4" event={"ID":"cb94d097-c65f-482f-ab4f-cbb930048f6c","Type":"ContainerStarted","Data":"c46dce6fce829d8f8f66b89ffe7fa04159680cc13a25bddff235cf50e145a9e1"} Jul 17 17:35:41.652934 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:41.652590 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4" Jul 17 17:35:41.708351 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:41.708297 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4" podStartSLOduration=1.361959724 podStartE2EDuration="3.708282147s" totalImagesPullingTime="2.346322423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:35:41.706143419 +0000 UTC m=+262.824971710" watchObservedRunningTime="2026-07-17 17:35:41.708282147 +0000 UTC m=+262.827110427" Jul 17 17:35:41.727137 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:41.727106 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:41.878437 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:41.878407 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:43.386330 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:43.386290 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:43.530433 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:43.530390 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:43.838559 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:43.838519 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:44.349686 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:44.349645 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:44.551829 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:44.551790 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:44.688769 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:44.688675 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:45.127725 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:45.127689 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:45.518786 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:45.518757 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:45.913774 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:45.913684 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:46.075053 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:46.075014 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:35:52.659064 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:52.659035 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-76cbd9fc7d-lnhf4" Jul 17 17:35:59.298460 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:35:59.298408 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:36:19.283530 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:36:19.283503 2553 kubelet.go:1648] "Image garbage collection succeeded" Jul 17 17:36:33.132275 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:36:33.132241 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:36:33.224097 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:36:33.224056 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:36:33.317142 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:36:33.317106 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:01.561687 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:01.561629 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:02.165696 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:02.165661 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:03.222393 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:03.222363 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:03.792954 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:03.792918 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:03.893451 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:03.893414 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:05.858021 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:05.857979 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:05.954437 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:05.954394 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:06.780145 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:06.780099 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:08.482060 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:08.482022 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-mzljg"] Jul 17 17:37:08.485396 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:08.485380 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mzljg" Jul 17 17:37:08.488592 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:08.488561 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 17:37:08.488716 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:08.488636 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 17 17:37:08.489393 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:08.489363 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-4zgg8\"" Jul 17 17:37:08.489491 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:08.489444 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 17:37:08.499103 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:08.499080 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-mzljg"] Jul 17 17:37:08.609366 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:08.609330 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2sdt\" (UniqueName: \"kubernetes.io/projected/e032c1ee-dd72-4343-bafb-1a5ff6c0f538-kube-api-access-d2sdt\") pod \"dns-operator-controller-manager-864dd78c4c-mzljg\" (UID: \"e032c1ee-dd72-4343-bafb-1a5ff6c0f538\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mzljg" Jul 17 17:37:08.710712 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:08.710674 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2sdt\" (UniqueName: \"kubernetes.io/projected/e032c1ee-dd72-4343-bafb-1a5ff6c0f538-kube-api-access-d2sdt\") pod \"dns-operator-controller-manager-864dd78c4c-mzljg\" (UID: \"e032c1ee-dd72-4343-bafb-1a5ff6c0f538\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mzljg" Jul 17 17:37:08.726129 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:08.726105 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2sdt\" (UniqueName: \"kubernetes.io/projected/e032c1ee-dd72-4343-bafb-1a5ff6c0f538-kube-api-access-d2sdt\") pod \"dns-operator-controller-manager-864dd78c4c-mzljg\" (UID: \"e032c1ee-dd72-4343-bafb-1a5ff6c0f538\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mzljg" Jul 17 17:37:08.796377 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:08.796287 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mzljg" Jul 17 17:37:08.935149 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:08.935091 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-mzljg"] Jul 17 17:37:08.938662 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:37:08.938631 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode032c1ee_dd72_4343_bafb_1a5ff6c0f538.slice/crio-3eb5e95c41da9c3e07e904f03e1828e150842ccb2b34404c8623d29a10e3d33b WatchSource:0}: Error finding container 3eb5e95c41da9c3e07e904f03e1828e150842ccb2b34404c8623d29a10e3d33b: Status 404 returned error can't find the container with id 3eb5e95c41da9c3e07e904f03e1828e150842ccb2b34404c8623d29a10e3d33b Jul 17 17:37:08.941120 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:08.941101 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 17:37:08.966544 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:08.966504 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:09.019989 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:09.019958 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mzljg" event={"ID":"e032c1ee-dd72-4343-bafb-1a5ff6c0f538","Type":"ContainerStarted","Data":"3eb5e95c41da9c3e07e904f03e1828e150842ccb2b34404c8623d29a10e3d33b"} Jul 17 17:37:09.240317 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:09.240278 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:10.250453 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:10.250418 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:11.030237 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:11.030204 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mzljg" event={"ID":"e032c1ee-dd72-4343-bafb-1a5ff6c0f538","Type":"ContainerStarted","Data":"37d54e1a8bd76ca6ac92b47cf58a3fb97212cdf1c8d543dafe29a263632f95fa"} Jul 17 17:37:11.030412 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:11.030327 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mzljg" Jul 17 17:37:11.088099 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:11.088043 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mzljg" podStartSLOduration=1.08930128 podStartE2EDuration="3.088023978s" totalImagesPullingTime="1.998722698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:37:11.085970228 +0000 UTC m=+352.204798521" watchObservedRunningTime="2026-07-17 17:37:11.088023978 +0000 UTC m=+352.206852259" Jul 17 17:37:12.458167 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:12.458132 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v"] Jul 17 17:37:12.461644 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:12.461625 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:12.464202 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:12.464182 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2fmjr\"" Jul 17 17:37:12.476297 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:12.476274 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v"] Jul 17 17:37:12.541609 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:12.541576 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vkdh\" (UniqueName: \"kubernetes.io/projected/b535fef9-0683-4e72-adff-57e9c913ff65-kube-api-access-4vkdh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5586v\" (UID: \"b535fef9-0683-4e72-adff-57e9c913ff65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:12.541609 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:12.541617 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b535fef9-0683-4e72-adff-57e9c913ff65-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5586v\" (UID: \"b535fef9-0683-4e72-adff-57e9c913ff65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:12.642576 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:12.642540 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b535fef9-0683-4e72-adff-57e9c913ff65-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5586v\" (UID: \"b535fef9-0683-4e72-adff-57e9c913ff65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:12.642725 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:12.642627 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vkdh\" (UniqueName: \"kubernetes.io/projected/b535fef9-0683-4e72-adff-57e9c913ff65-kube-api-access-4vkdh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5586v\" (UID: \"b535fef9-0683-4e72-adff-57e9c913ff65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:12.642962 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:12.642939 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b535fef9-0683-4e72-adff-57e9c913ff65-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5586v\" (UID: \"b535fef9-0683-4e72-adff-57e9c913ff65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:12.657978 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:12.657953 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vkdh\" (UniqueName: \"kubernetes.io/projected/b535fef9-0683-4e72-adff-57e9c913ff65-kube-api-access-4vkdh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5586v\" (UID: \"b535fef9-0683-4e72-adff-57e9c913ff65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:12.771363 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:12.771326 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:12.906566 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:12.906535 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v"] Jul 17 17:37:12.909035 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:37:12.909002 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb535fef9_0683_4e72_adff_57e9c913ff65.slice/crio-885eee77f802226eb3a04cd49634a54a993b05caf89fd45ed5bad8664fc26602 WatchSource:0}: Error finding container 885eee77f802226eb3a04cd49634a54a993b05caf89fd45ed5bad8664fc26602: Status 404 returned error can't find the container with id 885eee77f802226eb3a04cd49634a54a993b05caf89fd45ed5bad8664fc26602 Jul 17 17:37:12.982013 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:12.981981 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:13.039329 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:13.039252 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" event={"ID":"b535fef9-0683-4e72-adff-57e9c913ff65","Type":"ContainerStarted","Data":"885eee77f802226eb3a04cd49634a54a993b05caf89fd45ed5bad8664fc26602"} Jul 17 17:37:14.014998 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:14.014955 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:15.973679 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:15.973643 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:16.028145 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:16.028079 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:16.379705 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:16.379624 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:16.450968 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:16.450932 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:20.075719 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:20.075685 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" event={"ID":"b535fef9-0683-4e72-adff-57e9c913ff65","Type":"ContainerStarted","Data":"3e9f0f072a98398e746b71ddc3edf7df2e52cb8ce387014d186c6043fa124976"} Jul 17 17:37:20.076143 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:20.075943 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:20.095494 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:20.095444 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" podStartSLOduration=1.56303905 podStartE2EDuration="8.095429937s" totalImagesPullingTime="6.532390887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:37:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:37:20.093810238 +0000 UTC m=+361.212638519" watchObservedRunningTime="2026-07-17 17:37:20.095429937 +0000 UTC m=+361.214258217" Jul 17 17:37:20.196104 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:20.196068 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:20.298051 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:20.297988 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:22.036944 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.036908 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mzljg" Jul 17 17:37:22.104785 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.104755 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pxfjc"] Jul 17 17:37:22.108450 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.108431 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pxfjc" Jul 17 17:37:22.111362 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.111336 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 17 17:37:22.111503 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.111368 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 17 17:37:22.111503 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.111400 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-p476s\"" Jul 17 17:37:22.118716 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.118685 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pxfjc"] Jul 17 17:37:22.119024 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.119002 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:22.229078 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.229039 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2a4854fc-38d1-435f-aaad-181f75bb9cea-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pxfjc\" (UID: \"2a4854fc-38d1-435f-aaad-181f75bb9cea\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pxfjc" Jul 17 17:37:22.229078 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.229078 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6ppz\" (UniqueName: \"kubernetes.io/projected/2a4854fc-38d1-435f-aaad-181f75bb9cea-kube-api-access-c6ppz\") pod \"kuadrant-console-plugin-66b5774985-pxfjc\" (UID: \"2a4854fc-38d1-435f-aaad-181f75bb9cea\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pxfjc" Jul 17 17:37:22.229288 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.229103 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a4854fc-38d1-435f-aaad-181f75bb9cea-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pxfjc\" (UID: \"2a4854fc-38d1-435f-aaad-181f75bb9cea\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pxfjc" Jul 17 17:37:22.330824 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.330735 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2a4854fc-38d1-435f-aaad-181f75bb9cea-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pxfjc\" (UID: \"2a4854fc-38d1-435f-aaad-181f75bb9cea\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pxfjc" Jul 17 17:37:22.330824 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.330773 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6ppz\" (UniqueName: \"kubernetes.io/projected/2a4854fc-38d1-435f-aaad-181f75bb9cea-kube-api-access-c6ppz\") pod \"kuadrant-console-plugin-66b5774985-pxfjc\" (UID: \"2a4854fc-38d1-435f-aaad-181f75bb9cea\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pxfjc" Jul 17 17:37:22.330824 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.330809 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a4854fc-38d1-435f-aaad-181f75bb9cea-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pxfjc\" (UID: \"2a4854fc-38d1-435f-aaad-181f75bb9cea\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pxfjc" Jul 17 17:37:22.331438 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.331415 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2a4854fc-38d1-435f-aaad-181f75bb9cea-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pxfjc\" (UID: \"2a4854fc-38d1-435f-aaad-181f75bb9cea\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pxfjc" Jul 17 17:37:22.333304 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.333282 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a4854fc-38d1-435f-aaad-181f75bb9cea-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pxfjc\" (UID: \"2a4854fc-38d1-435f-aaad-181f75bb9cea\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pxfjc" Jul 17 17:37:22.348794 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.348759 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6ppz\" (UniqueName: \"kubernetes.io/projected/2a4854fc-38d1-435f-aaad-181f75bb9cea-kube-api-access-c6ppz\") pod \"kuadrant-console-plugin-66b5774985-pxfjc\" (UID: \"2a4854fc-38d1-435f-aaad-181f75bb9cea\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pxfjc" Jul 17 17:37:22.374645 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.374613 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:22.424507 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.424466 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pxfjc" Jul 17 17:37:22.581485 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.581462 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pxfjc"] Jul 17 17:37:22.583558 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:37:22.583522 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a4854fc_38d1_435f_aaad_181f75bb9cea.slice/crio-6ecc9858ad4a3b980e142a6ad0253de494404e043b7874df6a4ce3d49062e36a WatchSource:0}: Error finding container 6ecc9858ad4a3b980e142a6ad0253de494404e043b7874df6a4ce3d49062e36a: Status 404 returned error can't find the container with id 6ecc9858ad4a3b980e142a6ad0253de494404e043b7874df6a4ce3d49062e36a Jul 17 17:37:22.858279 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.858189 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:22.958773 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:22.958738 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:23.091128 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:23.091090 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pxfjc" event={"ID":"2a4854fc-38d1-435f-aaad-181f75bb9cea","Type":"ContainerStarted","Data":"6ecc9858ad4a3b980e142a6ad0253de494404e043b7874df6a4ce3d49062e36a"} Jul 17 17:37:25.588916 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:25.588842 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:25.670051 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:25.670012 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:31.083353 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:31.083315 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:31.857589 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:31.857542 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:31.989592 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:31.989438 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:32.798496 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:32.798463 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8sk6q"] Jul 17 17:37:32.825865 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:32.825821 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8sk6q"] Jul 17 17:37:32.826037 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:32.825971 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8sk6q" Jul 17 17:37:32.939332 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:32.939295 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/655171cf-3039-43ad-a635-3a1062d1f4f6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8sk6q\" (UID: \"655171cf-3039-43ad-a635-3a1062d1f4f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8sk6q" Jul 17 17:37:32.939501 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:32.939420 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skgwr\" (UniqueName: \"kubernetes.io/projected/655171cf-3039-43ad-a635-3a1062d1f4f6-kube-api-access-skgwr\") pod \"kuadrant-operator-controller-manager-86878c769f-8sk6q\" (UID: \"655171cf-3039-43ad-a635-3a1062d1f4f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8sk6q" Jul 17 17:37:32.976545 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:32.976248 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v"] Jul 17 17:37:32.976737 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:32.976678 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" podUID="b535fef9-0683-4e72-adff-57e9c913ff65" containerName="manager" containerID="cri-o://3e9f0f072a98398e746b71ddc3edf7df2e52cb8ce387014d186c6043fa124976" gracePeriod=2 Jul 17 17:37:32.978454 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:32.978428 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v"] Jul 17 17:37:32.980893 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:37:32.980686 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5586v\" is forbidden: User \"system:node:ip-10-0-136-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-43.ec2.internal' and this object" podUID="b535fef9-0683-4e72-adff-57e9c913ff65" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:33.001634 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.001588 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8sk6q"] Jul 17 17:37:33.001961 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:37:33.001938 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-skgwr], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8sk6q" podUID="655171cf-3039-43ad-a635-3a1062d1f4f6" Jul 17 17:37:33.015693 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.015351 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b"] Jul 17 17:37:33.020129 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.020106 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b535fef9-0683-4e72-adff-57e9c913ff65" containerName="manager" Jul 17 17:37:33.020129 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.020132 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b535fef9-0683-4e72-adff-57e9c913ff65" containerName="manager" Jul 17 17:37:33.020335 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.020298 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b535fef9-0683-4e72-adff-57e9c913ff65" containerName="manager" Jul 17 17:37:33.024194 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.024109 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b" Jul 17 17:37:33.027944 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.027910 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8sk6q"] Jul 17 17:37:33.028129 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:37:33.028015 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5586v\" is forbidden: User \"system:node:ip-10-0-136-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-43.ec2.internal' and this object" podUID="b535fef9-0683-4e72-adff-57e9c913ff65" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:33.030994 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.030968 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch"] Jul 17 17:37:33.035542 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.035522 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b"] Jul 17 17:37:33.035713 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.035693 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch" Jul 17 17:37:33.040260 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.040236 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/655171cf-3039-43ad-a635-3a1062d1f4f6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8sk6q\" (UID: \"655171cf-3039-43ad-a635-3a1062d1f4f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8sk6q" Jul 17 17:37:33.040375 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.040302 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skgwr\" (UniqueName: \"kubernetes.io/projected/655171cf-3039-43ad-a635-3a1062d1f4f6-kube-api-access-skgwr\") pod \"kuadrant-operator-controller-manager-86878c769f-8sk6q\" (UID: \"655171cf-3039-43ad-a635-3a1062d1f4f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8sk6q" Jul 17 17:37:33.040694 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.040673 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/655171cf-3039-43ad-a635-3a1062d1f4f6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8sk6q\" (UID: \"655171cf-3039-43ad-a635-3a1062d1f4f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8sk6q" Jul 17 17:37:33.046717 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:37:33.046696 2553 projected.go:196] Error preparing data for projected volume kube-api-access-skgwr for pod kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8sk6q: failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-8sk6q" not found Jul 17 17:37:33.046831 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:37:33.046781 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/655171cf-3039-43ad-a635-3a1062d1f4f6-kube-api-access-skgwr podName:655171cf-3039-43ad-a635-3a1062d1f4f6 nodeName:}" failed. No retries permitted until 2026-07-17 17:37:33.546757958 +0000 UTC m=+374.665586218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-skgwr" (UniqueName: "kubernetes.io/projected/655171cf-3039-43ad-a635-3a1062d1f4f6-kube-api-access-skgwr") pod "kuadrant-operator-controller-manager-86878c769f-8sk6q" (UID: "655171cf-3039-43ad-a635-3a1062d1f4f6") : failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-8sk6q" not found Jul 17 17:37:33.052086 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.051976 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch"] Jul 17 17:37:33.059277 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:37:33.059236 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5586v\" is forbidden: User \"system:node:ip-10-0-136-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-43.ec2.internal' and this object" podUID="b535fef9-0683-4e72-adff-57e9c913ff65" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:33.141666 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.141614 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb8rw\" (UniqueName: \"kubernetes.io/projected/f61de066-e6e8-4264-b3a8-50c60e77ad80-kube-api-access-xb8rw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fpt9b\" (UID: \"f61de066-e6e8-4264-b3a8-50c60e77ad80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b" Jul 17 17:37:33.141666 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.141666 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73bf37df-8a00-4129-8ffc-957752068836-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-swgch\" (UID: \"73bf37df-8a00-4129-8ffc-957752068836\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch" Jul 17 17:37:33.141929 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.141704 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f61de066-e6e8-4264-b3a8-50c60e77ad80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fpt9b\" (UID: \"f61de066-e6e8-4264-b3a8-50c60e77ad80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b" Jul 17 17:37:33.141929 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.141759 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgg7t\" (UniqueName: \"kubernetes.io/projected/73bf37df-8a00-4129-8ffc-957752068836-kube-api-access-kgg7t\") pod \"kuadrant-operator-controller-manager-86878c769f-swgch\" (UID: \"73bf37df-8a00-4129-8ffc-957752068836\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch" Jul 17 17:37:33.143581 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.143556 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8sk6q" Jul 17 17:37:33.146193 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:37:33.146160 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-8sk6q\" is forbidden: User \"system:node:ip-10-0-136-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-43.ec2.internal' and this object" podUID="655171cf-3039-43ad-a635-3a1062d1f4f6" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8sk6q" Jul 17 17:37:33.148520 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:37:33.148469 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5586v\" is forbidden: User \"system:node:ip-10-0-136-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-43.ec2.internal' and this object" podUID="b535fef9-0683-4e72-adff-57e9c913ff65" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:33.153251 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.153225 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8sk6q" Jul 17 17:37:33.155763 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:37:33.155733 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-8sk6q\" is forbidden: User \"system:node:ip-10-0-136-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-43.ec2.internal' and this object" podUID="655171cf-3039-43ad-a635-3a1062d1f4f6" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8sk6q" Jul 17 17:37:33.158399 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:37:33.158370 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5586v\" is forbidden: User \"system:node:ip-10-0-136-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-43.ec2.internal' and this object" podUID="b535fef9-0683-4e72-adff-57e9c913ff65" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:33.242735 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.242697 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f61de066-e6e8-4264-b3a8-50c60e77ad80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fpt9b\" (UID: \"f61de066-e6e8-4264-b3a8-50c60e77ad80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b" Jul 17 17:37:33.242917 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.242762 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgg7t\" (UniqueName: \"kubernetes.io/projected/73bf37df-8a00-4129-8ffc-957752068836-kube-api-access-kgg7t\") pod \"kuadrant-operator-controller-manager-86878c769f-swgch\" (UID: \"73bf37df-8a00-4129-8ffc-957752068836\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch" Jul 17 17:37:33.242977 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.242921 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb8rw\" (UniqueName: \"kubernetes.io/projected/f61de066-e6e8-4264-b3a8-50c60e77ad80-kube-api-access-xb8rw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fpt9b\" (UID: \"f61de066-e6e8-4264-b3a8-50c60e77ad80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b" Jul 17 17:37:33.242977 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.242954 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73bf37df-8a00-4129-8ffc-957752068836-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-swgch\" (UID: \"73bf37df-8a00-4129-8ffc-957752068836\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch" Jul 17 17:37:33.243175 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.243126 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f61de066-e6e8-4264-b3a8-50c60e77ad80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fpt9b\" (UID: \"f61de066-e6e8-4264-b3a8-50c60e77ad80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b" Jul 17 17:37:33.243318 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.243297 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73bf37df-8a00-4129-8ffc-957752068836-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-swgch\" (UID: \"73bf37df-8a00-4129-8ffc-957752068836\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch" Jul 17 17:37:33.253919 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.253893 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb8rw\" (UniqueName: \"kubernetes.io/projected/f61de066-e6e8-4264-b3a8-50c60e77ad80-kube-api-access-xb8rw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fpt9b\" (UID: \"f61de066-e6e8-4264-b3a8-50c60e77ad80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b" Jul 17 17:37:33.254087 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.253989 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgg7t\" (UniqueName: \"kubernetes.io/projected/73bf37df-8a00-4129-8ffc-957752068836-kube-api-access-kgg7t\") pod \"kuadrant-operator-controller-manager-86878c769f-swgch\" (UID: \"73bf37df-8a00-4129-8ffc-957752068836\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch" Jul 17 17:37:33.343502 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.343428 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/655171cf-3039-43ad-a635-3a1062d1f4f6-extensions-socket-volume\") pod \"655171cf-3039-43ad-a635-3a1062d1f4f6\" (UID: \"655171cf-3039-43ad-a635-3a1062d1f4f6\") " Jul 17 17:37:33.343725 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.343690 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655171cf-3039-43ad-a635-3a1062d1f4f6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "655171cf-3039-43ad-a635-3a1062d1f4f6" (UID: "655171cf-3039-43ad-a635-3a1062d1f4f6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:37:33.343792 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.343761 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skgwr\" (UniqueName: \"kubernetes.io/projected/655171cf-3039-43ad-a635-3a1062d1f4f6-kube-api-access-skgwr\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:37:33.357695 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.357663 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b" Jul 17 17:37:33.364907 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.364758 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch" Jul 17 17:37:33.445053 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.445014 2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/655171cf-3039-43ad-a635-3a1062d1f4f6-extensions-socket-volume\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:37:33.464665 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:33.464621 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="655171cf-3039-43ad-a635-3a1062d1f4f6" path="/var/lib/kubelet/pods/655171cf-3039-43ad-a635-3a1062d1f4f6/volumes" Jul 17 17:37:34.012217 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:34.012178 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bc9f9794f-btxkk"] Jul 17 17:37:34.143728 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:34.143685 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:34.147544 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:34.147524 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8sk6q" Jul 17 17:37:34.152397 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:37:34.152369 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-8sk6q\" is forbidden: User \"system:node:ip-10-0-136-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-43.ec2.internal' and this object" podUID="655171cf-3039-43ad-a635-3a1062d1f4f6" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8sk6q" Jul 17 17:37:34.240877 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:34.240827 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:34.355793 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:34.355699 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:34.497643 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:34.497605 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:35.580804 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:35.580760 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:35.680696 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:35.680649 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:35.964291 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:35.964197 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:36.099538 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:36.099507 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:38.568654 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:38.568608 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:38.625319 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:38.625238 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:43.975456 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:43.975414 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:44.137576 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:44.137539 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:44.138808 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:44.138791 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:44.142973 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:37:44.142561 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5586v\" is forbidden: User \"system:node:ip-10-0-136-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-43.ec2.internal' and this object" podUID="b535fef9-0683-4e72-adff-57e9c913ff65" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:44.144483 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:44.144462 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b"] Jul 17 17:37:44.146223 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:37:44.146195 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf61de066_e6e8_4264_b3a8_50c60e77ad80.slice/crio-67c6275cf682ca9a23f9dccbac7b7b21824a0f746b8f2639ccad088d0b7ea334 WatchSource:0}: Error finding container 67c6275cf682ca9a23f9dccbac7b7b21824a0f746b8f2639ccad088d0b7ea334: Status 404 returned error can't find the container with id 67c6275cf682ca9a23f9dccbac7b7b21824a0f746b8f2639ccad088d0b7ea334 Jul 17 17:37:44.166677 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:44.166655 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch"] Jul 17 17:37:44.178890 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:37:44.178840 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73bf37df_8a00_4129_8ffc_957752068836.slice/crio-b1e941e4865af51e8d73887b1832dcdefba9d9f31cf9dee2d4c0fff3ab2f9711 WatchSource:0}: Error finding container b1e941e4865af51e8d73887b1832dcdefba9d9f31cf9dee2d4c0fff3ab2f9711: Status 404 returned error can't find the container with id b1e941e4865af51e8d73887b1832dcdefba9d9f31cf9dee2d4c0fff3ab2f9711 Jul 17 17:37:44.198842 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:44.198810 2553 generic.go:356] "Generic (PLEG): container finished" podID="b535fef9-0683-4e72-adff-57e9c913ff65" containerID="3e9f0f072a98398e746b71ddc3edf7df2e52cb8ce387014d186c6043fa124976" exitCode=0 Jul 17 17:37:44.198958 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:44.198895 2553 scope.go:117] "RemoveContainer" containerID="3e9f0f072a98398e746b71ddc3edf7df2e52cb8ce387014d186c6043fa124976" Jul 17 17:37:44.198958 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:44.198918 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:44.200637 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:44.200614 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pxfjc" event={"ID":"2a4854fc-38d1-435f-aaad-181f75bb9cea","Type":"ContainerStarted","Data":"86af748360a65df3f9f443efbf78ca48b5a4b9f35277f11c100894a8438b4494"} Jul 17 17:37:44.201274 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:37:44.201244 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5586v\" is forbidden: User \"system:node:ip-10-0-136-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-43.ec2.internal' and this object" podUID="b535fef9-0683-4e72-adff-57e9c913ff65" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:44.202240 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:44.202195 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch" event={"ID":"73bf37df-8a00-4129-8ffc-957752068836","Type":"ContainerStarted","Data":"b1e941e4865af51e8d73887b1832dcdefba9d9f31cf9dee2d4c0fff3ab2f9711"} Jul 17 17:37:44.203500 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:37:44.203471 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5586v\" is forbidden: User \"system:node:ip-10-0-136-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-43.ec2.internal' and this object" podUID="b535fef9-0683-4e72-adff-57e9c913ff65" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:44.204027 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:44.204004 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b" event={"ID":"f61de066-e6e8-4264-b3a8-50c60e77ad80","Type":"ContainerStarted","Data":"67c6275cf682ca9a23f9dccbac7b7b21824a0f746b8f2639ccad088d0b7ea334"} Jul 17 17:37:44.208152 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:44.208135 2553 scope.go:117] "RemoveContainer" containerID="3e9f0f072a98398e746b71ddc3edf7df2e52cb8ce387014d186c6043fa124976" Jul 17 17:37:44.208392 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:37:44.208376 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9f0f072a98398e746b71ddc3edf7df2e52cb8ce387014d186c6043fa124976\": container with ID starting with 3e9f0f072a98398e746b71ddc3edf7df2e52cb8ce387014d186c6043fa124976 not found: ID does not exist" containerID="3e9f0f072a98398e746b71ddc3edf7df2e52cb8ce387014d186c6043fa124976" Jul 17 17:37:44.208458 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:44.208408 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e9f0f072a98398e746b71ddc3edf7df2e52cb8ce387014d186c6043fa124976"} err="failed to get container status \"3e9f0f072a98398e746b71ddc3edf7df2e52cb8ce387014d186c6043fa124976\": rpc error: code = NotFound desc = could not find container \"3e9f0f072a98398e746b71ddc3edf7df2e52cb8ce387014d186c6043fa124976\": container with ID starting with 3e9f0f072a98398e746b71ddc3edf7df2e52cb8ce387014d186c6043fa124976 not found: ID does not exist" Jul 17 17:37:44.219616 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:44.219568 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pxfjc" podStartSLOduration=0.74307245 podStartE2EDuration="22.21955235s" totalImagesPullingTime="21.4764799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:37:44.218732695 +0000 UTC m=+385.337560977" watchObservedRunningTime="2026-07-17 17:37:44.21955235 +0000 UTC m=+385.338380637" Jul 17 17:37:44.246309 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:44.246291 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b535fef9-0683-4e72-adff-57e9c913ff65-extensions-socket-volume\") pod \"b535fef9-0683-4e72-adff-57e9c913ff65\" (UID: \"b535fef9-0683-4e72-adff-57e9c913ff65\") " Jul 17 17:37:44.246407 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:44.246384 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vkdh\" (UniqueName: \"kubernetes.io/projected/b535fef9-0683-4e72-adff-57e9c913ff65-kube-api-access-4vkdh\") pod \"b535fef9-0683-4e72-adff-57e9c913ff65\" (UID: \"b535fef9-0683-4e72-adff-57e9c913ff65\") " Jul 17 17:37:44.246883 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:44.246829 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b535fef9-0683-4e72-adff-57e9c913ff65-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b535fef9-0683-4e72-adff-57e9c913ff65" (UID: "b535fef9-0683-4e72-adff-57e9c913ff65"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:37:44.248288 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:44.248272 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b535fef9-0683-4e72-adff-57e9c913ff65-kube-api-access-4vkdh" (OuterVolumeSpecName: "kube-api-access-4vkdh") pod "b535fef9-0683-4e72-adff-57e9c913ff65" (UID: "b535fef9-0683-4e72-adff-57e9c913ff65"). InnerVolumeSpecName "kube-api-access-4vkdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:37:44.347936 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:44.347906 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vkdh\" (UniqueName: \"kubernetes.io/projected/b535fef9-0683-4e72-adff-57e9c913ff65-kube-api-access-4vkdh\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:37:44.347936 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:44.347931 2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b535fef9-0683-4e72-adff-57e9c913ff65-extensions-socket-volume\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:37:44.509534 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:37:44.509504 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5586v\" is forbidden: User \"system:node:ip-10-0-136-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-43.ec2.internal' and this object" podUID="b535fef9-0683-4e72-adff-57e9c913ff65" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:45.210565 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:45.210530 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch" event={"ID":"73bf37df-8a00-4129-8ffc-957752068836","Type":"ContainerStarted","Data":"a1bc35c1ff782de443eaa007a2942391240dad6e944271c4b36f848928c51909"} Jul 17 17:37:45.211006 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:45.210659 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch" Jul 17 17:37:45.212166 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:45.212144 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b" event={"ID":"f61de066-e6e8-4264-b3a8-50c60e77ad80","Type":"ContainerStarted","Data":"1873211d89451e54ff1fc3b6e73f6416c217317cbf2d7cb69ca30f1de266a467"} Jul 17 17:37:45.213133 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:37:45.213110 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5586v\" is forbidden: User \"system:node:ip-10-0-136-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-43.ec2.internal' and this object" podUID="b535fef9-0683-4e72-adff-57e9c913ff65" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:45.233091 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:45.233046 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch" podStartSLOduration=12.233033431 podStartE2EDuration="12.233033431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:37:45.231929349 +0000 UTC m=+386.350757629" watchObservedRunningTime="2026-07-17 17:37:45.233033431 +0000 UTC m=+386.351861711" Jul 17 17:37:45.255182 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:45.255109 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b" podStartSLOduration=13.25509745 podStartE2EDuration="13.25509745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:37:45.253766219 +0000 UTC m=+386.372594524" watchObservedRunningTime="2026-07-17 17:37:45.25509745 +0000 UTC m=+386.373925731" Jul 17 17:37:45.255805 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:37:45.255772 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5586v\" is forbidden: User \"system:node:ip-10-0-136-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-43.ec2.internal' and this object" podUID="b535fef9-0683-4e72-adff-57e9c913ff65" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5586v" Jul 17 17:37:45.465462 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:45.465430 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b535fef9-0683-4e72-adff-57e9c913ff65" path="/var/lib/kubelet/pods/b535fef9-0683-4e72-adff-57e9c913ff65/volumes" Jul 17 17:37:46.216904 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:46.216872 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b" Jul 17 17:37:47.913236 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:47.913197 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:51.402070 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:51.402032 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:51.501366 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:51.501326 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:37:56.218468 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:56.218435 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch" Jul 17 17:37:56.296019 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:56.295989 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b"] Jul 17 17:37:56.296187 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:56.296168 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b" podUID="f61de066-e6e8-4264-b3a8-50c60e77ad80" containerName="manager" containerID="cri-o://1873211d89451e54ff1fc3b6e73f6416c217317cbf2d7cb69ca30f1de266a467" gracePeriod=10 Jul 17 17:37:56.297999 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:56.297975 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b" Jul 17 17:37:56.538346 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:56.538321 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b" Jul 17 17:37:56.657129 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:56.657103 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xb8rw\" (UniqueName: \"kubernetes.io/projected/f61de066-e6e8-4264-b3a8-50c60e77ad80-kube-api-access-xb8rw\") pod \"f61de066-e6e8-4264-b3a8-50c60e77ad80\" (UID: \"f61de066-e6e8-4264-b3a8-50c60e77ad80\") " Jul 17 17:37:56.657267 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:56.657150 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f61de066-e6e8-4264-b3a8-50c60e77ad80-extensions-socket-volume\") pod \"f61de066-e6e8-4264-b3a8-50c60e77ad80\" (UID: \"f61de066-e6e8-4264-b3a8-50c60e77ad80\") " Jul 17 17:37:56.657536 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:56.657515 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f61de066-e6e8-4264-b3a8-50c60e77ad80-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f61de066-e6e8-4264-b3a8-50c60e77ad80" (UID: "f61de066-e6e8-4264-b3a8-50c60e77ad80"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:37:56.658922 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:56.658900 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f61de066-e6e8-4264-b3a8-50c60e77ad80-kube-api-access-xb8rw" (OuterVolumeSpecName: "kube-api-access-xb8rw") pod "f61de066-e6e8-4264-b3a8-50c60e77ad80" (UID: "f61de066-e6e8-4264-b3a8-50c60e77ad80"). InnerVolumeSpecName "kube-api-access-xb8rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:37:56.758643 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:56.758617 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb8rw\" (UniqueName: \"kubernetes.io/projected/f61de066-e6e8-4264-b3a8-50c60e77ad80-kube-api-access-xb8rw\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:37:56.758643 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:56.758643 2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f61de066-e6e8-4264-b3a8-50c60e77ad80-extensions-socket-volume\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:37:57.269175 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:57.269140 2553 generic.go:356] "Generic (PLEG): container finished" podID="f61de066-e6e8-4264-b3a8-50c60e77ad80" containerID="1873211d89451e54ff1fc3b6e73f6416c217317cbf2d7cb69ca30f1de266a467" exitCode=0 Jul 17 17:37:57.269598 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:57.269185 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b" Jul 17 17:37:57.269598 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:57.269227 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b" event={"ID":"f61de066-e6e8-4264-b3a8-50c60e77ad80","Type":"ContainerDied","Data":"1873211d89451e54ff1fc3b6e73f6416c217317cbf2d7cb69ca30f1de266a467"} Jul 17 17:37:57.269598 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:57.269279 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b" event={"ID":"f61de066-e6e8-4264-b3a8-50c60e77ad80","Type":"ContainerDied","Data":"67c6275cf682ca9a23f9dccbac7b7b21824a0f746b8f2639ccad088d0b7ea334"} Jul 17 17:37:57.269598 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:57.269302 2553 scope.go:117] "RemoveContainer" containerID="1873211d89451e54ff1fc3b6e73f6416c217317cbf2d7cb69ca30f1de266a467" Jul 17 17:37:57.277699 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:57.277683 2553 scope.go:117] "RemoveContainer" containerID="1873211d89451e54ff1fc3b6e73f6416c217317cbf2d7cb69ca30f1de266a467" Jul 17 17:37:57.277986 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:37:57.277964 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1873211d89451e54ff1fc3b6e73f6416c217317cbf2d7cb69ca30f1de266a467\": container with ID starting with 1873211d89451e54ff1fc3b6e73f6416c217317cbf2d7cb69ca30f1de266a467 not found: ID does not exist" containerID="1873211d89451e54ff1fc3b6e73f6416c217317cbf2d7cb69ca30f1de266a467" Jul 17 17:37:57.278105 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:57.277994 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1873211d89451e54ff1fc3b6e73f6416c217317cbf2d7cb69ca30f1de266a467"} err="failed to get container status \"1873211d89451e54ff1fc3b6e73f6416c217317cbf2d7cb69ca30f1de266a467\": rpc error: code = NotFound desc = could not find container \"1873211d89451e54ff1fc3b6e73f6416c217317cbf2d7cb69ca30f1de266a467\": container with ID starting with 1873211d89451e54ff1fc3b6e73f6416c217317cbf2d7cb69ca30f1de266a467 not found: ID does not exist" Jul 17 17:37:57.290136 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:57.290113 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b"] Jul 17 17:37:57.295665 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:57.295648 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fpt9b"] Jul 17 17:37:57.463153 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:57.463122 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f61de066-e6e8-4264-b3a8-50c60e77ad80" path="/var/lib/kubelet/pods/f61de066-e6e8-4264-b3a8-50c60e77ad80/volumes" Jul 17 17:37:59.035132 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.035095 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bc9f9794f-btxkk" podUID="7db3deea-8080-44fe-b0c9-434a94a2d37f" containerName="console" containerID="cri-o://ed2037d9342cf9c304a1f3b7c96421510783994ceafaa05aa4676fe09304cbd8" gracePeriod=15 Jul 17 17:37:59.270667 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.270647 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:37:59.280470 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.280445 2553 generic.go:356] "Generic (PLEG): container finished" podID="7db3deea-8080-44fe-b0c9-434a94a2d37f" containerID="ed2037d9342cf9c304a1f3b7c96421510783994ceafaa05aa4676fe09304cbd8" exitCode=0 Jul 17 17:37:59.280572 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.280481 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc9f9794f-btxkk" Jul 17 17:37:59.280572 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.280526 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc9f9794f-btxkk" event={"ID":"7db3deea-8080-44fe-b0c9-434a94a2d37f","Type":"ContainerDied","Data":"ed2037d9342cf9c304a1f3b7c96421510783994ceafaa05aa4676fe09304cbd8"} Jul 17 17:37:59.280659 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.280606 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc9f9794f-btxkk" event={"ID":"7db3deea-8080-44fe-b0c9-434a94a2d37f","Type":"ContainerDied","Data":"ef609787d9461a69008e869e443e92a18a16f4b23c80b6871eceef33e2f068cd"} Jul 17 17:37:59.280659 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.280633 2553 scope.go:117] "RemoveContainer" containerID="ed2037d9342cf9c304a1f3b7c96421510783994ceafaa05aa4676fe09304cbd8" Jul 17 17:37:59.288240 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.288221 2553 scope.go:117] "RemoveContainer" containerID="ed2037d9342cf9c304a1f3b7c96421510783994ceafaa05aa4676fe09304cbd8" Jul 17 17:37:59.288489 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:37:59.288472 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed2037d9342cf9c304a1f3b7c96421510783994ceafaa05aa4676fe09304cbd8\": container with ID starting with ed2037d9342cf9c304a1f3b7c96421510783994ceafaa05aa4676fe09304cbd8 not found: ID does not exist" containerID="ed2037d9342cf9c304a1f3b7c96421510783994ceafaa05aa4676fe09304cbd8" Jul 17 17:37:59.288547 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.288498 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed2037d9342cf9c304a1f3b7c96421510783994ceafaa05aa4676fe09304cbd8"} err="failed to get container status \"ed2037d9342cf9c304a1f3b7c96421510783994ceafaa05aa4676fe09304cbd8\": rpc error: code = NotFound desc = could not find container \"ed2037d9342cf9c304a1f3b7c96421510783994ceafaa05aa4676fe09304cbd8\": container with ID starting with ed2037d9342cf9c304a1f3b7c96421510783994ceafaa05aa4676fe09304cbd8 not found: ID does not exist" Jul 17 17:37:59.379139 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.379113 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-oauth-serving-cert\") pod \"7db3deea-8080-44fe-b0c9-434a94a2d37f\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " Jul 17 17:37:59.379258 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.379143 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7db3deea-8080-44fe-b0c9-434a94a2d37f-console-oauth-config\") pod \"7db3deea-8080-44fe-b0c9-434a94a2d37f\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " Jul 17 17:37:59.379258 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.379159 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-console-config\") pod \"7db3deea-8080-44fe-b0c9-434a94a2d37f\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " Jul 17 17:37:59.379258 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.379189 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7db3deea-8080-44fe-b0c9-434a94a2d37f-console-serving-cert\") pod \"7db3deea-8080-44fe-b0c9-434a94a2d37f\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " Jul 17 17:37:59.379425 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.379307 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2gcp\" (UniqueName: \"kubernetes.io/projected/7db3deea-8080-44fe-b0c9-434a94a2d37f-kube-api-access-d2gcp\") pod \"7db3deea-8080-44fe-b0c9-434a94a2d37f\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " Jul 17 17:37:59.379425 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.379352 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-service-ca\") pod \"7db3deea-8080-44fe-b0c9-434a94a2d37f\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " Jul 17 17:37:59.379425 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.379408 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-trusted-ca-bundle\") pod \"7db3deea-8080-44fe-b0c9-434a94a2d37f\" (UID: \"7db3deea-8080-44fe-b0c9-434a94a2d37f\") " Jul 17 17:37:59.379571 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.379501 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7db3deea-8080-44fe-b0c9-434a94a2d37f" (UID: "7db3deea-8080-44fe-b0c9-434a94a2d37f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:37:59.379571 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.379557 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-console-config" (OuterVolumeSpecName: "console-config") pod "7db3deea-8080-44fe-b0c9-434a94a2d37f" (UID: "7db3deea-8080-44fe-b0c9-434a94a2d37f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:37:59.379796 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.379768 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-service-ca" (OuterVolumeSpecName: "service-ca") pod "7db3deea-8080-44fe-b0c9-434a94a2d37f" (UID: "7db3deea-8080-44fe-b0c9-434a94a2d37f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:37:59.379883 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.379837 2553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-oauth-serving-cert\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:37:59.379934 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.379881 2553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-console-config\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:37:59.379934 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.379890 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7db3deea-8080-44fe-b0c9-434a94a2d37f" (UID: "7db3deea-8080-44fe-b0c9-434a94a2d37f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:37:59.381325 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.381297 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7db3deea-8080-44fe-b0c9-434a94a2d37f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7db3deea-8080-44fe-b0c9-434a94a2d37f" (UID: "7db3deea-8080-44fe-b0c9-434a94a2d37f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:37:59.381325 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.381317 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7db3deea-8080-44fe-b0c9-434a94a2d37f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7db3deea-8080-44fe-b0c9-434a94a2d37f" (UID: "7db3deea-8080-44fe-b0c9-434a94a2d37f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:37:59.381458 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.381357 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db3deea-8080-44fe-b0c9-434a94a2d37f-kube-api-access-d2gcp" (OuterVolumeSpecName: "kube-api-access-d2gcp") pod "7db3deea-8080-44fe-b0c9-434a94a2d37f" (UID: "7db3deea-8080-44fe-b0c9-434a94a2d37f"). InnerVolumeSpecName "kube-api-access-d2gcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:37:59.480231 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.480211 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2gcp\" (UniqueName: \"kubernetes.io/projected/7db3deea-8080-44fe-b0c9-434a94a2d37f-kube-api-access-d2gcp\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:37:59.480231 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.480231 2553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-service-ca\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:37:59.480363 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.480241 2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7db3deea-8080-44fe-b0c9-434a94a2d37f-trusted-ca-bundle\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:37:59.480363 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.480249 2553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7db3deea-8080-44fe-b0c9-434a94a2d37f-console-oauth-config\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:37:59.480363 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.480258 2553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7db3deea-8080-44fe-b0c9-434a94a2d37f-console-serving-cert\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:37:59.603801 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.603737 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bc9f9794f-btxkk"] Jul 17 17:37:59.610454 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:37:59.610428 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bc9f9794f-btxkk"] Jul 17 17:38:01.463065 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:01.463036 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7db3deea-8080-44fe-b0c9-434a94a2d37f" path="/var/lib/kubelet/pods/7db3deea-8080-44fe-b0c9-434a94a2d37f/volumes" Jul 17 17:38:07.695864 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:07.695812 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:12.749683 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:12.749643 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:13.270652 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:13.270615 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:15.133890 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.133834 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:15.249465 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.249428 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:15.317103 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.317062 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:15.481039 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.481008 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:38:15.481366 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.481353 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7db3deea-8080-44fe-b0c9-434a94a2d37f" containerName="console" Jul 17 17:38:15.481420 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.481368 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7db3deea-8080-44fe-b0c9-434a94a2d37f" containerName="console" Jul 17 17:38:15.481420 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.481376 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f61de066-e6e8-4264-b3a8-50c60e77ad80" containerName="manager" Jul 17 17:38:15.481420 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.481381 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f61de066-e6e8-4264-b3a8-50c60e77ad80" containerName="manager" Jul 17 17:38:15.481521 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.481430 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f61de066-e6e8-4264-b3a8-50c60e77ad80" containerName="manager" Jul 17 17:38:15.481521 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.481439 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7db3deea-8080-44fe-b0c9-434a94a2d37f" containerName="console" Jul 17 17:38:15.507934 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.507905 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:38:15.508097 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.508006 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-ntgm8" Jul 17 17:38:15.512150 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.512131 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 17 17:38:15.515671 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.515648 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9fxq\" (UniqueName: \"kubernetes.io/projected/fc1ecae2-611f-440a-8044-fa0db5d0dc24-kube-api-access-h9fxq\") pod \"limitador-limitador-54d6d988d5-ntgm8\" (UID: \"fc1ecae2-611f-440a-8044-fa0db5d0dc24\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ntgm8" Jul 17 17:38:15.515759 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.515705 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:38:15.515759 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.515715 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc1ecae2-611f-440a-8044-fa0db5d0dc24-config-file\") pod \"limitador-limitador-54d6d988d5-ntgm8\" (UID: \"fc1ecae2-611f-440a-8044-fa0db5d0dc24\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ntgm8" Jul 17 17:38:15.616643 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.616614 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc1ecae2-611f-440a-8044-fa0db5d0dc24-config-file\") pod \"limitador-limitador-54d6d988d5-ntgm8\" (UID: \"fc1ecae2-611f-440a-8044-fa0db5d0dc24\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ntgm8" Jul 17 17:38:15.616799 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.616687 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9fxq\" (UniqueName: \"kubernetes.io/projected/fc1ecae2-611f-440a-8044-fa0db5d0dc24-kube-api-access-h9fxq\") pod \"limitador-limitador-54d6d988d5-ntgm8\" (UID: \"fc1ecae2-611f-440a-8044-fa0db5d0dc24\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ntgm8" Jul 17 17:38:15.617187 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.617168 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fc1ecae2-611f-440a-8044-fa0db5d0dc24-config-file\") pod \"limitador-limitador-54d6d988d5-ntgm8\" (UID: \"fc1ecae2-611f-440a-8044-fa0db5d0dc24\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ntgm8" Jul 17 17:38:15.624730 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.624704 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9fxq\" (UniqueName: \"kubernetes.io/projected/fc1ecae2-611f-440a-8044-fa0db5d0dc24-kube-api-access-h9fxq\") pod \"limitador-limitador-54d6d988d5-ntgm8\" (UID: \"fc1ecae2-611f-440a-8044-fa0db5d0dc24\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ntgm8" Jul 17 17:38:15.817776 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.817679 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-ntgm8" Jul 17 17:38:15.947213 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.947186 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:38:15.949525 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:38:15.949497 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1ecae2_611f_440a_8044_fa0db5d0dc24.slice/crio-b585793727bf4bd6feed93c7541da6a5cf750fd0b7d3c328ad5e3adace062c2b WatchSource:0}: Error finding container b585793727bf4bd6feed93c7541da6a5cf750fd0b7d3c328ad5e3adace062c2b: Status 404 returned error can't find the container with id b585793727bf4bd6feed93c7541da6a5cf750fd0b7d3c328ad5e3adace062c2b Jul 17 17:38:15.984314 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:15.984288 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:16.061075 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:16.061047 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:16.360479 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:16.360445 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-ntgm8" event={"ID":"fc1ecae2-611f-440a-8044-fa0db5d0dc24","Type":"ContainerStarted","Data":"b585793727bf4bd6feed93c7541da6a5cf750fd0b7d3c328ad5e3adace062c2b"} Jul 17 17:38:19.376659 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:19.376622 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-ntgm8" event={"ID":"fc1ecae2-611f-440a-8044-fa0db5d0dc24","Type":"ContainerStarted","Data":"e56a4e18afdd3a69ba21723569d8e254408213a170860d24f63da0fc688856ed"} Jul 17 17:38:19.377165 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:19.376722 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-ntgm8" Jul 17 17:38:19.396612 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:19.396555 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-ntgm8" podStartSLOduration=1.9579645399999999 podStartE2EDuration="4.396540857s" totalImagesPullingTime="2.438576317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:38:19.395984984 +0000 UTC m=+420.514813259" watchObservedRunningTime="2026-07-17 17:38:19.396540857 +0000 UTC m=+420.515369161" Jul 17 17:38:30.381679 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:30.381652 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-ntgm8" Jul 17 17:38:36.519242 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:36.519197 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:36.619735 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:36.619696 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:37.103071 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:37.103037 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:49.862967 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:49.862871 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:50.013686 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:50.013652 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:55.939953 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:55.939918 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:56.725108 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:56.725076 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:56.826811 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:56.826775 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:57.196748 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:57.196665 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:57.365626 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:57.365588 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:57.453557 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:57.453460 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:38:57.660675 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:38:57.660641 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:00.897939 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:39:00.897904 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:01.059928 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:39:01.059897 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:22.526524 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:39:22.526486 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:34.404932 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:39:34.404891 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:34.506197 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:39:34.506163 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:34.884633 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:39:34.884595 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:34.955454 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:39:34.955420 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:35.497687 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:39:35.497652 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:35.583969 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:39:35.583938 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:35.796520 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:39:35.796443 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:39:44.860956 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:39:44.860917 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:44.958978 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:39:44.958939 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:53.082495 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:39:53.082456 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:39:53.478412 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:39:53.478381 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:40:02.223708 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:40:02.223666 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:40:02.935261 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:40:02.935221 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:40:20.441352 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:40:20.441249 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:40:21.463783 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:40:21.463757 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:40:47.481707 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:40:47.481671 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:40:49.030615 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:40:49.030583 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:40:52.068800 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:40:52.068763 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:40:54.533061 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:40:54.533030 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:40:55.970519 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:40:55.970486 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:41:09.028670 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.028631 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt"] Jul 17 17:41:09.031907 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.031881 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:09.036988 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.036964 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 17 17:41:09.036988 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.036978 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kpxlh\"" Jul 17 17:41:09.036988 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.036987 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 17 17:41:09.037205 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.036997 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 17 17:41:09.043527 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.043508 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt"] Jul 17 17:41:09.067685 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.067657 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:41:09.093882 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.093837 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c1e3c1-87f1-4c3a-966b-2950163f1edd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt\" (UID: \"91c1e3c1-87f1-4c3a-966b-2950163f1edd\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:09.093967 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.093901 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg74k\" (UniqueName: \"kubernetes.io/projected/91c1e3c1-87f1-4c3a-966b-2950163f1edd-kube-api-access-zg74k\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt\" (UID: \"91c1e3c1-87f1-4c3a-966b-2950163f1edd\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:09.094012 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.093961 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91c1e3c1-87f1-4c3a-966b-2950163f1edd-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt\" (UID: \"91c1e3c1-87f1-4c3a-966b-2950163f1edd\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:09.094046 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.094015 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91c1e3c1-87f1-4c3a-966b-2950163f1edd-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt\" (UID: \"91c1e3c1-87f1-4c3a-966b-2950163f1edd\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:09.094082 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.094044 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91c1e3c1-87f1-4c3a-966b-2950163f1edd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt\" (UID: \"91c1e3c1-87f1-4c3a-966b-2950163f1edd\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:09.094082 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.094077 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91c1e3c1-87f1-4c3a-966b-2950163f1edd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt\" (UID: \"91c1e3c1-87f1-4c3a-966b-2950163f1edd\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:09.194668 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.194627 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91c1e3c1-87f1-4c3a-966b-2950163f1edd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt\" (UID: \"91c1e3c1-87f1-4c3a-966b-2950163f1edd\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:09.194872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.194697 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c1e3c1-87f1-4c3a-966b-2950163f1edd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt\" (UID: \"91c1e3c1-87f1-4c3a-966b-2950163f1edd\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:09.194872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.194724 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg74k\" (UniqueName: \"kubernetes.io/projected/91c1e3c1-87f1-4c3a-966b-2950163f1edd-kube-api-access-zg74k\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt\" (UID: \"91c1e3c1-87f1-4c3a-966b-2950163f1edd\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:09.194872 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.194754 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91c1e3c1-87f1-4c3a-966b-2950163f1edd-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt\" (UID: \"91c1e3c1-87f1-4c3a-966b-2950163f1edd\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:09.194992 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.194891 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91c1e3c1-87f1-4c3a-966b-2950163f1edd-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt\" (UID: \"91c1e3c1-87f1-4c3a-966b-2950163f1edd\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:09.194992 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.194938 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91c1e3c1-87f1-4c3a-966b-2950163f1edd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt\" (UID: \"91c1e3c1-87f1-4c3a-966b-2950163f1edd\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:09.195091 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.195059 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91c1e3c1-87f1-4c3a-966b-2950163f1edd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt\" (UID: \"91c1e3c1-87f1-4c3a-966b-2950163f1edd\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:09.195126 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.195093 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c1e3c1-87f1-4c3a-966b-2950163f1edd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt\" (UID: \"91c1e3c1-87f1-4c3a-966b-2950163f1edd\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:09.195166 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.195152 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91c1e3c1-87f1-4c3a-966b-2950163f1edd-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt\" (UID: \"91c1e3c1-87f1-4c3a-966b-2950163f1edd\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:09.197061 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.197033 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91c1e3c1-87f1-4c3a-966b-2950163f1edd-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt\" (UID: \"91c1e3c1-87f1-4c3a-966b-2950163f1edd\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:09.197264 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.197248 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91c1e3c1-87f1-4c3a-966b-2950163f1edd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt\" (UID: \"91c1e3c1-87f1-4c3a-966b-2950163f1edd\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:09.203005 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.202983 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg74k\" (UniqueName: \"kubernetes.io/projected/91c1e3c1-87f1-4c3a-966b-2950163f1edd-kube-api-access-zg74k\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt\" (UID: \"91c1e3c1-87f1-4c3a-966b-2950163f1edd\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:09.341407 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.341312 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:09.468002 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:09.467956 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt"] Jul 17 17:41:09.470704 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:41:09.470675 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91c1e3c1_87f1_4c3a_966b_2950163f1edd.slice/crio-00062faab23a827874280e886064daf60f8086e7d973260954be72ebd27bdfed WatchSource:0}: Error finding container 00062faab23a827874280e886064daf60f8086e7d973260954be72ebd27bdfed: Status 404 returned error can't find the container with id 00062faab23a827874280e886064daf60f8086e7d973260954be72ebd27bdfed Jul 17 17:41:10.128891 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:10.128831 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" event={"ID":"91c1e3c1-87f1-4c3a-966b-2950163f1edd","Type":"ContainerStarted","Data":"00062faab23a827874280e886064daf60f8086e7d973260954be72ebd27bdfed"} Jul 17 17:41:11.135762 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:11.135721 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" event={"ID":"91c1e3c1-87f1-4c3a-966b-2950163f1edd","Type":"ContainerStarted","Data":"9e3c2ce5599214b6259e5975270b02f7bff2e2ddc8d0ef7d3f4e74d9c7958b00"} Jul 17 17:41:11.154313 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:11.154252 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" podStartSLOduration=1.171615777 podStartE2EDuration="2.154237382s" totalImagesPullingTime="982.621605ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:41:11.153304085 +0000 UTC m=+592.272132378" watchObservedRunningTime="2026-07-17 17:41:11.154237382 +0000 UTC m=+592.273065664" Jul 17 17:41:12.028160 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:12.028124 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:41:19.342018 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:19.341986 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:19.342509 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:19.342056 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:19.354255 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:19.354233 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:41:20.188100 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:41:20.188071 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt" Jul 17 17:42:00.986276 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:42:00.986231 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:42:33.702367 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:42:33.702334 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:42:33.805510 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:42:33.805481 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:42:34.922830 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:42:34.922802 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:42:35.039572 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:42:35.039537 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:42:42.416525 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:42:42.416494 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:43:04.632677 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:43:04.632645 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:43:13.725031 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:43:13.724950 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:43:23.636358 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:43:23.636323 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:44:02.308858 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:44:02.308810 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:44:02.372935 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:44:02.372905 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:44:39.922793 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:44:39.922758 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:44:41.184671 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:44:41.184582 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:44:52.000838 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:44:52.000801 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:45:04.729354 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:45:04.729323 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:45:17.134136 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:45:17.134097 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:45:22.097400 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:45:22.097362 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:45:22.195745 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:45:22.195714 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:45:22.335644 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:45:22.335618 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:45:28.133490 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:45:28.133436 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:45:28.243449 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:45:28.243416 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:45:28.332601 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:45:28.332558 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:45:28.504901 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:45:28.504865 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:45:34.629349 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:45:34.629312 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:45:34.724875 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:45:34.724821 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:45:50.873536 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:45:50.873505 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:46:00.747572 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:46:00.747537 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:46:10.735033 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:46:10.734958 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:46:12.950624 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:46:12.950591 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:46:19.233518 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:46:19.233484 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:46:28.750537 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:46:28.750507 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:46:44.036656 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:46:44.036620 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:46:50.336891 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:46:50.336861 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:46:59.644554 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:46:59.644514 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:47:25.035592 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:47:25.035518 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:47:35.439078 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:47:35.439044 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:47:58.543154 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:47:58.543075 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:48:09.631322 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:48:09.631286 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:48:25.244732 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:48:25.244693 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:48:35.132916 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:48:35.132877 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:48:46.634781 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:48:46.634750 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:48:53.803606 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:48:53.803566 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:48:55.637696 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:48:55.637667 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:49:04.435471 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:04.435437 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:49:29.240543 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:29.240507 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:49:30.674098 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:30.674050 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch"] Jul 17 17:49:31.344382 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:31.344339 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch" podUID="73bf37df-8a00-4129-8ffc-957752068836" containerName="manager" containerID="cri-o://a1bc35c1ff782de443eaa007a2942391240dad6e944271c4b36f848928c51909" gracePeriod=10 Jul 17 17:49:34.180954 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:34.180931 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch" Jul 17 17:49:34.245094 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:34.245071 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73bf37df-8a00-4129-8ffc-957752068836-extensions-socket-volume\") pod \"73bf37df-8a00-4129-8ffc-957752068836\" (UID: \"73bf37df-8a00-4129-8ffc-957752068836\") " Jul 17 17:49:34.245233 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:34.245117 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgg7t\" (UniqueName: \"kubernetes.io/projected/73bf37df-8a00-4129-8ffc-957752068836-kube-api-access-kgg7t\") pod \"73bf37df-8a00-4129-8ffc-957752068836\" (UID: \"73bf37df-8a00-4129-8ffc-957752068836\") " Jul 17 17:49:34.245405 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:34.245381 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73bf37df-8a00-4129-8ffc-957752068836-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "73bf37df-8a00-4129-8ffc-957752068836" (UID: "73bf37df-8a00-4129-8ffc-957752068836"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:49:34.247024 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:34.246998 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73bf37df-8a00-4129-8ffc-957752068836-kube-api-access-kgg7t" (OuterVolumeSpecName: "kube-api-access-kgg7t") pod "73bf37df-8a00-4129-8ffc-957752068836" (UID: "73bf37df-8a00-4129-8ffc-957752068836"). InnerVolumeSpecName "kube-api-access-kgg7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:49:34.346148 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:34.346125 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgg7t\" (UniqueName: \"kubernetes.io/projected/73bf37df-8a00-4129-8ffc-957752068836-kube-api-access-kgg7t\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:49:34.346148 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:34.346145 2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73bf37df-8a00-4129-8ffc-957752068836-extensions-socket-volume\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 17:49:34.360091 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:34.360061 2553 generic.go:356] "Generic (PLEG): container finished" podID="73bf37df-8a00-4129-8ffc-957752068836" containerID="a1bc35c1ff782de443eaa007a2942391240dad6e944271c4b36f848928c51909" exitCode=0 Jul 17 17:49:34.360202 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:34.360108 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch" event={"ID":"73bf37df-8a00-4129-8ffc-957752068836","Type":"ContainerDied","Data":"a1bc35c1ff782de443eaa007a2942391240dad6e944271c4b36f848928c51909"} Jul 17 17:49:34.360202 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:34.360115 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch" Jul 17 17:49:34.360202 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:34.360154 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch" event={"ID":"73bf37df-8a00-4129-8ffc-957752068836","Type":"ContainerDied","Data":"b1e941e4865af51e8d73887b1832dcdefba9d9f31cf9dee2d4c0fff3ab2f9711"} Jul 17 17:49:34.360202 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:34.360179 2553 scope.go:117] "RemoveContainer" containerID="a1bc35c1ff782de443eaa007a2942391240dad6e944271c4b36f848928c51909" Jul 17 17:49:34.369999 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:34.369980 2553 scope.go:117] "RemoveContainer" containerID="a1bc35c1ff782de443eaa007a2942391240dad6e944271c4b36f848928c51909" Jul 17 17:49:34.370235 ip-10-0-136-43 kubenswrapper[2553]: E0717 17:49:34.370218 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1bc35c1ff782de443eaa007a2942391240dad6e944271c4b36f848928c51909\": container with ID starting with a1bc35c1ff782de443eaa007a2942391240dad6e944271c4b36f848928c51909 not found: ID does not exist" containerID="a1bc35c1ff782de443eaa007a2942391240dad6e944271c4b36f848928c51909" Jul 17 17:49:34.370292 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:34.370244 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1bc35c1ff782de443eaa007a2942391240dad6e944271c4b36f848928c51909"} err="failed to get container status \"a1bc35c1ff782de443eaa007a2942391240dad6e944271c4b36f848928c51909\": rpc error: code = NotFound desc = could not find container \"a1bc35c1ff782de443eaa007a2942391240dad6e944271c4b36f848928c51909\": container with ID starting with a1bc35c1ff782de443eaa007a2942391240dad6e944271c4b36f848928c51909 not found: ID does not exist" Jul 17 17:49:34.384537 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:34.384516 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch"] Jul 17 17:49:34.390321 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:34.390302 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-swgch"] Jul 17 17:49:35.463196 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:49:35.463167 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73bf37df-8a00-4129-8ffc-957752068836" path="/var/lib/kubelet/pods/73bf37df-8a00-4129-8ffc-957752068836/volumes" Jul 17 17:50:31.845099 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:31.845069 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76s4p"] Jul 17 17:50:31.845545 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:31.845440 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73bf37df-8a00-4129-8ffc-957752068836" containerName="manager" Jul 17 17:50:31.845545 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:31.845453 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="73bf37df-8a00-4129-8ffc-957752068836" containerName="manager" Jul 17 17:50:31.845545 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:31.845516 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="73bf37df-8a00-4129-8ffc-957752068836" containerName="manager" Jul 17 17:50:31.849571 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:31.849553 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76s4p" Jul 17 17:50:31.852528 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:31.852511 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2fmjr\"" Jul 17 17:50:31.862026 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:31.862001 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76s4p"] Jul 17 17:50:31.908956 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:31.908926 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd73c075-07c5-4827-a767-9101f18fab41-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-76s4p\" (UID: \"bd73c075-07c5-4827-a767-9101f18fab41\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76s4p" Jul 17 17:50:31.909068 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:31.908996 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4t5b\" (UniqueName: \"kubernetes.io/projected/bd73c075-07c5-4827-a767-9101f18fab41-kube-api-access-x4t5b\") pod \"kuadrant-operator-controller-manager-86878c769f-76s4p\" (UID: \"bd73c075-07c5-4827-a767-9101f18fab41\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76s4p" Jul 17 17:50:32.009507 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:32.009482 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4t5b\" (UniqueName: \"kubernetes.io/projected/bd73c075-07c5-4827-a767-9101f18fab41-kube-api-access-x4t5b\") pod \"kuadrant-operator-controller-manager-86878c769f-76s4p\" (UID: \"bd73c075-07c5-4827-a767-9101f18fab41\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76s4p" Jul 17 17:50:32.009636 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:32.009542 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd73c075-07c5-4827-a767-9101f18fab41-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-76s4p\" (UID: \"bd73c075-07c5-4827-a767-9101f18fab41\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76s4p" Jul 17 17:50:32.009844 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:32.009827 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd73c075-07c5-4827-a767-9101f18fab41-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-76s4p\" (UID: \"bd73c075-07c5-4827-a767-9101f18fab41\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76s4p" Jul 17 17:50:32.018964 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:32.018949 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4t5b\" (UniqueName: \"kubernetes.io/projected/bd73c075-07c5-4827-a767-9101f18fab41-kube-api-access-x4t5b\") pod \"kuadrant-operator-controller-manager-86878c769f-76s4p\" (UID: \"bd73c075-07c5-4827-a767-9101f18fab41\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76s4p" Jul 17 17:50:32.158621 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:32.158555 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76s4p" Jul 17 17:50:32.303273 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:32.303243 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76s4p"] Jul 17 17:50:32.304101 ip-10-0-136-43 kubenswrapper[2553]: W0717 17:50:32.304072 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd73c075_07c5_4827_a767_9101f18fab41.slice/crio-f2f6eaf5fb6f18a94afcfe1da563ba98f798c13278e8ff4fe6f7fccd42c19f23 WatchSource:0}: Error finding container f2f6eaf5fb6f18a94afcfe1da563ba98f798c13278e8ff4fe6f7fccd42c19f23: Status 404 returned error can't find the container with id f2f6eaf5fb6f18a94afcfe1da563ba98f798c13278e8ff4fe6f7fccd42c19f23 Jul 17 17:50:32.615391 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:32.615356 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76s4p" event={"ID":"bd73c075-07c5-4827-a767-9101f18fab41","Type":"ContainerStarted","Data":"821a9b9271a4b53cad3626d800004ae2a4e8e94c519ca77bb8692b55ebf6e803"} Jul 17 17:50:32.615559 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:32.615406 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76s4p" event={"ID":"bd73c075-07c5-4827-a767-9101f18fab41","Type":"ContainerStarted","Data":"f2f6eaf5fb6f18a94afcfe1da563ba98f798c13278e8ff4fe6f7fccd42c19f23"} Jul 17 17:50:32.615559 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:32.615446 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76s4p" Jul 17 17:50:32.637568 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:32.637515 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76s4p" podStartSLOduration=1.637496651 podStartE2EDuration="1.637496651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:50:32.636950614 +0000 UTC m=+1153.755778897" watchObservedRunningTime="2026-07-17 17:50:32.637496651 +0000 UTC m=+1153.756324932" Jul 17 17:50:33.328641 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:33.328609 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:43.622693 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:43.622618 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76s4p" Jul 17 17:50:54.631409 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:50:54.631375 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:51:00.488835 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:51:00.488800 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:06.340349 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:51:06.340313 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:51:14.843556 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:51:14.843528 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:51:25.935317 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:51:25.935285 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:51:36.840163 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:51:36.840130 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:51:48.029457 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:51:48.029426 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:51:56.540485 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:51:56.540451 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:52:06.849501 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:52:06.849470 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:52:19.439668 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:52:19.439640 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:52:19.466024 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:52:19.465988 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:52:20.569138 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:52:20.569101 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:52:26.542636 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:52:26.542600 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:52:35.432767 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:52:35.432731 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:52:44.038813 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:52:44.038783 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:53:02.680782 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:53:02.676874 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:53:12.141870 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:53:12.141814 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:53:23.437401 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:53:23.437367 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:53:31.665422 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:53:31.665390 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:53:40.537998 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:53:40.537964 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:53:51.036240 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:53:51.036209 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:54:00.238565 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:54:00.238535 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:54:08.833698 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:54:08.833665 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:54:18.232876 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:54:18.232829 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:54:24.274593 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:54:24.274540 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:24.356729 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:54:24.356689 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:29.738599 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:54:29.738557 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:54:39.237629 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:54:39.237588 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:54:52.150773 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:54:52.150736 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:55:01.035033 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:55:01.035002 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:55:10.345730 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:55:10.345695 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:55:23.132867 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:55:23.132820 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:55:24.798763 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:55:24.798733 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:55:32.440865 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:55:32.440809 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:55:45.540668 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:55:45.540632 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:55:58.236257 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:55:58.236224 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:56:10.341076 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:10.341040 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:56:21.434013 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:21.433978 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:56:24.221283 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:24.221253 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:31.808192 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:31.808151 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:35.133259 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:35.133219 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:56:40.072521 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:40.072488 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:45.312575 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:45.312476 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:48.479749 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:48.479710 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:48.577303 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:48.577269 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:48.955197 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:48.955114 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:49.143202 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:49.143172 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:56:54.851192 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:54.851152 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:54.925347 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:54.925318 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:55.067132 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:55.067100 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:55.154662 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:55.154580 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:55.250718 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:55.250688 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:55.397523 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:55.397490 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:55.730420 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:55.730376 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:55.992455 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:55.992367 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:56.094409 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:56.094378 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:58.133531 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:58.133496 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:56:58.447429 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:56:58.447397 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:01.517205 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:01.517176 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:01.590215 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:01.590184 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:05.070356 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:05.070332 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:07.392628 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:07.392598 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:07.493164 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:07.493125 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:07.686101 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:07.686020 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:07.796811 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:07.796781 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:07.899492 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:07.899455 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:08.000362 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:08.000334 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:08.180818 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:08.180778 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:08.300705 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:08.300593 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:08.351733 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:08.351701 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:57:08.692088 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:08.691967 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:08.817328 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:08.817287 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:08.917121 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:08.917075 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:11.528243 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:11.528192 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:13.775098 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:13.775066 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:13.884614 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:13.884576 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:15.307325 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:15.307297 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:19.735677 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:19.735645 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:19.837488 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:19.837440 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:19.961934 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:19.961896 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:20.137645 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:20.137562 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:20.223697 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:20.223661 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:20.295060 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:20.295032 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:20.521650 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:20.521615 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:21.606171 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:21.606145 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:23.567347 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:23.567317 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:23.734278 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:57:23.734240 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:58:52.575664 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:58:52.575628 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:52.629923 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:58:52.629891 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:29.757797 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:29.757761 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:29.872063 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:29.872031 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:32.534383 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:32.534342 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:59:35.523678 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:35.523647 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:35.624099 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:35.624058 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:35.793154 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:35.793127 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:35.900175 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:35.900139 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:35.977655 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:35.977621 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:36.047696 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:36.047613 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:36.158894 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:36.158827 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:36.289936 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:36.288879 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:36.390575 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:36.390465 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:36.512478 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:36.512441 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:36.612997 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:36.612967 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:40.136877 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:40.136826 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:59:40.523325 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:40.523296 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:41.536950 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:41.536917 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:41.619797 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:41.619769 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:43.236828 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:43.236783 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:47.703377 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:47.703343 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:47.806506 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:47.806474 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:48.217906 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:48.217872 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:48.313146 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:48.313114 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:48.412058 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:48.412024 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:49.529323 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:49.529286 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 17:59:49.900783 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:49.900687 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:55.787185 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:55.787151 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:55.893068 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:55.893032 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:55.990024 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:55.989992 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:56.086944 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:56.086831 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:56.533675 ip-10-0-136-43 kubenswrapper[2553]: I0717 17:59:56.533642 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:00:04.151302 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:00:04.151271 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:00:04.252281 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:00:04.252248 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:00:14.854039 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:00:14.854001 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:00:23.026780 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:00:23.026746 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:00:28.320114 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:00:28.320079 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:00:28.390039 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:00:28.390007 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:00:28.469613 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:00:28.469584 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:00:28.568411 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:00:28.568382 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:00:44.505985 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:00:44.505955 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:00:50.321624 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:00:50.321589 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:00:50.421179 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:00:50.421144 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:00.893693 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:00.893660 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:01.763192 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:01.763158 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:07.577217 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:07.577179 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:07.637154 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:07.637126 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:07.721484 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:07.721445 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:07.819254 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:07.819225 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:07.970668 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:07.970585 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:08.716978 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:08.716942 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:08.816666 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:08.816627 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:08.916887 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:08.916836 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:10.632276 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:10.632245 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:11.034962 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:11.034927 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:01:17.278060 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:17.278024 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:17.381951 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:17.381913 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:20.630883 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:20.630831 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:01:22.956689 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:22.956645 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:23.028258 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:23.028222 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:23.201383 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:23.201348 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:23.542038 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:23.542002 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:23.796603 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:23.796513 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:25.782590 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:25.782552 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:25.875942 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:25.875910 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:25.942690 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:25.942656 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:26.429839 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:26.429805 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:26.553700 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:26.553671 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:26.659796 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:26.659763 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:31.345266 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:31.345233 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:31.836671 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:31.836631 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:01:32.073733 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:32.073689 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:32.250340 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:32.250297 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:32.575239 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:32.575155 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:38.384155 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:38.384116 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:38.442622 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:38.442591 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:39.139358 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:39.139327 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:44.903033 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:44.902999 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:45.002920 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:45.002879 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:45.728526 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:45.728481 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:45.823817 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:45.823787 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:48.541726 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:48.541686 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:01:51.635168 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:51.635131 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:51.744958 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:51.744920 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:51.834048 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:51.834016 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:52.016457 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:52.016425 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:58.195650 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:58.195616 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:01:58.295730 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:01:58.295690 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:05.200535 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:05.200500 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:05.501784 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:05.501752 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:05.573576 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:05.573543 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:07.089640 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:07.089604 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:07.180752 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:07.180718 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:07.298998 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:07.298965 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:07.627407 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:07.627372 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:07.847343 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:07.847307 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:07.951384 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:07.951301 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:12.636159 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:12.636127 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:15.126083 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:15.126042 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:02:30.251753 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.251718 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj"] Jul 17 18:02:30.255174 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.255151 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:30.259188 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.259158 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7548b62e\"/\"e2e-lifecycle-model-7548b62e-kserve-self-signed-certs\"" Jul 17 18:02:30.259335 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.259198 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7548b62e\"/\"openshift-service-ca.crt\"" Jul 17 18:02:30.259335 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.259211 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7548b62e\"/\"kube-root-ca.crt\"" Jul 17 18:02:30.259335 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.259231 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7548b62e\"/\"default-dockercfg-dz62l\"" Jul 17 18:02:30.265117 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.265086 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj"] Jul 17 18:02:30.279517 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.279488 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:30.295148 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.295114 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-tmp-dir\") pod \"e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:30.295300 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.295155 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-model-cache\") pod \"e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:30.295300 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.295222 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-home\") pod \"e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:30.295300 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.295245 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-dshm\") pod \"e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:30.295404 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.295322 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcdmt\" (UniqueName: \"kubernetes.io/projected/4162d923-d5b1-45f5-bd01-378773eb9970-kube-api-access-mcdmt\") pod \"e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:30.295404 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.295369 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4162d923-d5b1-45f5-bd01-378773eb9970-tls-certs\") pod \"e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:30.395971 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.395937 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-home\") pod \"e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:30.395971 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.395973 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-dshm\") pod \"e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:30.396188 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.396006 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcdmt\" (UniqueName: \"kubernetes.io/projected/4162d923-d5b1-45f5-bd01-378773eb9970-kube-api-access-mcdmt\") pod \"e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:30.396188 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.396048 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4162d923-d5b1-45f5-bd01-378773eb9970-tls-certs\") pod \"e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:30.396188 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.396079 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-tmp-dir\") pod \"e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:30.396188 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.396108 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-model-cache\") pod \"e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:30.396511 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.396482 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-model-cache\") pod \"e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:30.396511 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.396497 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-home\") pod \"e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:30.396690 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.396527 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-tmp-dir\") pod \"e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:30.398273 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.398250 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-dshm\") pod \"e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:30.398543 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.398517 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4162d923-d5b1-45f5-bd01-378773eb9970-tls-certs\") pod \"e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:30.404782 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.404746 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcdmt\" (UniqueName: \"kubernetes.io/projected/4162d923-d5b1-45f5-bd01-378773eb9970-kube-api-access-mcdmt\") pod \"e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:30.565517 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.565421 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:30.717755 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.717730 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj"] Jul 17 18:02:30.719235 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:02:30.719209 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4162d923_d5b1_45f5_bd01_378773eb9970.slice/crio-ee0dcbbf6e6b8b9bcdcbeaf411459195567f98d80984ac35fb35853d88d2fb3a WatchSource:0}: Error finding container ee0dcbbf6e6b8b9bcdcbeaf411459195567f98d80984ac35fb35853d88d2fb3a: Status 404 returned error can't find the container with id ee0dcbbf6e6b8b9bcdcbeaf411459195567f98d80984ac35fb35853d88d2fb3a Jul 17 18:02:30.851623 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.851535 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" event={"ID":"4162d923-d5b1-45f5-bd01-378773eb9970","Type":"ContainerStarted","Data":"b6fd732cfb1fa981645601fddb81b9445ee5cbb998d61f65f0716d79bc6ab79f"} Jul 17 18:02:30.851623 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.851581 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" event={"ID":"4162d923-d5b1-45f5-bd01-378773eb9970","Type":"ContainerStarted","Data":"ee0dcbbf6e6b8b9bcdcbeaf411459195567f98d80984ac35fb35853d88d2fb3a"} Jul 17 18:02:30.877316 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:30.877251 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" podStartSLOduration=0.877229636 podStartE2EDuration="877.229636ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:02:30.876076734 +0000 UTC m=+1871.994905015" watchObservedRunningTime="2026-07-17 18:02:30.877229636 +0000 UTC m=+1871.996057917" Jul 17 18:02:33.897448 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:33.897410 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:02:40.565554 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:40.565521 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:40.566154 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:40.565569 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:40.577381 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:40.577357 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:40.907564 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:40.907483 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:02:47.714144 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:02:47.714105 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:03:39.795932 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:03:39.795877 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:03:39.858826 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:03:39.858780 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:05:04.998708 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:05:04.998661 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:05:05.101496 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:05:05.101453 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:05:56.627996 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:05:56.627925 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:05:58.635565 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:05:58.635530 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:06:01.552755 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:01.552716 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj"] Jul 17 18:06:01.611551 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:01.611519 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:16.572901 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.572811 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" podUID="4162d923-d5b1-45f5-bd01-378773eb9970" containerName="main" containerID="cri-o://b6fd732cfb1fa981645601fddb81b9445ee5cbb998d61f65f0716d79bc6ab79f" gracePeriod=45 Jul 17 18:06:16.823138 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.823074 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:06:16.866091 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.866052 2553 generic.go:356] "Generic (PLEG): container finished" podID="4162d923-d5b1-45f5-bd01-378773eb9970" containerID="b6fd732cfb1fa981645601fddb81b9445ee5cbb998d61f65f0716d79bc6ab79f" exitCode=0 Jul 17 18:06:16.866274 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.866094 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" Jul 17 18:06:16.866274 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.866101 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" event={"ID":"4162d923-d5b1-45f5-bd01-378773eb9970","Type":"ContainerDied","Data":"b6fd732cfb1fa981645601fddb81b9445ee5cbb998d61f65f0716d79bc6ab79f"} Jul 17 18:06:16.866274 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.866151 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj" event={"ID":"4162d923-d5b1-45f5-bd01-378773eb9970","Type":"ContainerDied","Data":"ee0dcbbf6e6b8b9bcdcbeaf411459195567f98d80984ac35fb35853d88d2fb3a"} Jul 17 18:06:16.866274 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.866173 2553 scope.go:117] "RemoveContainer" containerID="b6fd732cfb1fa981645601fddb81b9445ee5cbb998d61f65f0716d79bc6ab79f" Jul 17 18:06:16.875137 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.875114 2553 scope.go:117] "RemoveContainer" containerID="b6fd732cfb1fa981645601fddb81b9445ee5cbb998d61f65f0716d79bc6ab79f" Jul 17 18:06:16.875397 ip-10-0-136-43 kubenswrapper[2553]: E0717 18:06:16.875379 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6fd732cfb1fa981645601fddb81b9445ee5cbb998d61f65f0716d79bc6ab79f\": container with ID starting with b6fd732cfb1fa981645601fddb81b9445ee5cbb998d61f65f0716d79bc6ab79f not found: ID does not exist" containerID="b6fd732cfb1fa981645601fddb81b9445ee5cbb998d61f65f0716d79bc6ab79f" Jul 17 18:06:16.875467 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.875406 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6fd732cfb1fa981645601fddb81b9445ee5cbb998d61f65f0716d79bc6ab79f"} err="failed to get container status \"b6fd732cfb1fa981645601fddb81b9445ee5cbb998d61f65f0716d79bc6ab79f\": rpc error: code = NotFound desc = could not find container \"b6fd732cfb1fa981645601fddb81b9445ee5cbb998d61f65f0716d79bc6ab79f\": container with ID starting with b6fd732cfb1fa981645601fddb81b9445ee5cbb998d61f65f0716d79bc6ab79f not found: ID does not exist" Jul 17 18:06:16.970569 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.970533 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-model-cache\") pod \"4162d923-d5b1-45f5-bd01-378773eb9970\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " Jul 17 18:06:16.970569 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.970567 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-home\") pod \"4162d923-d5b1-45f5-bd01-378773eb9970\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " Jul 17 18:06:16.970792 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.970584 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-dshm\") pod \"4162d923-d5b1-45f5-bd01-378773eb9970\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " Jul 17 18:06:16.970792 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.970613 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-tmp-dir\") pod \"4162d923-d5b1-45f5-bd01-378773eb9970\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " Jul 17 18:06:16.970792 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.970646 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcdmt\" (UniqueName: \"kubernetes.io/projected/4162d923-d5b1-45f5-bd01-378773eb9970-kube-api-access-mcdmt\") pod \"4162d923-d5b1-45f5-bd01-378773eb9970\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " Jul 17 18:06:16.970792 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.970668 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4162d923-d5b1-45f5-bd01-378773eb9970-tls-certs\") pod \"4162d923-d5b1-45f5-bd01-378773eb9970\" (UID: \"4162d923-d5b1-45f5-bd01-378773eb9970\") " Jul 17 18:06:16.971086 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.970884 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-model-cache" (OuterVolumeSpecName: "model-cache") pod "4162d923-d5b1-45f5-bd01-378773eb9970" (UID: "4162d923-d5b1-45f5-bd01-378773eb9970"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:06:16.971086 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.970891 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-home" (OuterVolumeSpecName: "home") pod "4162d923-d5b1-45f5-bd01-378773eb9970" (UID: "4162d923-d5b1-45f5-bd01-378773eb9970"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:06:16.971086 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.970985 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4162d923-d5b1-45f5-bd01-378773eb9970" (UID: "4162d923-d5b1-45f5-bd01-378773eb9970"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:06:16.971210 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.971188 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-model-cache\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:06:16.971210 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.971203 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-home\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:06:16.971297 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.971234 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-tmp-dir\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:06:16.972985 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.972963 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4162d923-d5b1-45f5-bd01-378773eb9970-kube-api-access-mcdmt" (OuterVolumeSpecName: "kube-api-access-mcdmt") pod "4162d923-d5b1-45f5-bd01-378773eb9970" (UID: "4162d923-d5b1-45f5-bd01-378773eb9970"). InnerVolumeSpecName "kube-api-access-mcdmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:06:16.973218 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.973197 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-dshm" (OuterVolumeSpecName: "dshm") pod "4162d923-d5b1-45f5-bd01-378773eb9970" (UID: "4162d923-d5b1-45f5-bd01-378773eb9970"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:06:16.973306 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:16.973205 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4162d923-d5b1-45f5-bd01-378773eb9970-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4162d923-d5b1-45f5-bd01-378773eb9970" (UID: "4162d923-d5b1-45f5-bd01-378773eb9970"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:06:17.072040 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:17.072005 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4162d923-d5b1-45f5-bd01-378773eb9970-dshm\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:06:17.072040 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:17.072033 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcdmt\" (UniqueName: \"kubernetes.io/projected/4162d923-d5b1-45f5-bd01-378773eb9970-kube-api-access-mcdmt\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:06:17.072040 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:17.072044 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4162d923-d5b1-45f5-bd01-378773eb9970-tls-certs\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:06:17.189297 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:17.189270 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj"] Jul 17 18:06:17.193555 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:17.193519 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7548b62e/e2e-lifecycle-model-7548b62e-kserve-f568cd779-9fmmj"] Jul 17 18:06:17.464923 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:17.464824 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4162d923-d5b1-45f5-bd01-378773eb9970" path="/var/lib/kubelet/pods/4162d923-d5b1-45f5-bd01-378773eb9970/volumes" Jul 17 18:06:38.752473 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:38.752438 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:38.985696 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:38.985659 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:42.451123 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:42.451071 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:06:45.018407 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:45.018369 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:45.119015 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:45.118981 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:45.487241 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:45.487209 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:45.614824 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:45.614795 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:46.829278 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:46.829240 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:46.933597 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:46.933553 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:47.032027 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:47.031996 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:47.839874 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:47.839815 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:47.939919 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:47.939892 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:48.044496 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:48.044465 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:49.565774 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:49.565739 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:53.398717 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:53.398671 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:53.472174 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:53.472141 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:55.824128 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:55.824093 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:55.925304 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:55.925277 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:56.633050 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:56.633015 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:06:56.649433 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:06:56.649403 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:02.978548 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:02.978509 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:03.083826 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:03.083790 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:03.182745 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:03.182697 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:06.644354 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:06.644323 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:07:13.766643 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:13.766544 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:14.132231 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:14.132146 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:35.191796 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:35.191753 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:41.619197 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:41.619163 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:41.702289 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:41.702261 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:42.141495 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:42.141462 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:42.550529 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:42.550490 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:42.748325 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:42.748291 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:44.896440 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:44.896412 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:45.126826 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:45.126797 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:45.270643 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:45.270610 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:46.280803 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:46.280760 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:46.437327 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:46.437303 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:50.397348 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:50.397320 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:50.887387 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:50.887360 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:51.418553 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:51.418516 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:51.844714 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:51.844668 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:53.930324 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:53.930288 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:54.078123 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:54.078088 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:54.247792 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:54.247765 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:54.420125 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:54.420096 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:54.488426 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:54.488402 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:07:54.866008 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:54.865982 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:54.964791 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:54.964767 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:59.434035 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:07:59.434003 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:06.256945 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.256906 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:08:06.494528 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.494501 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd"] Jul 17 18:08:06.494879 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.494866 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4162d923-d5b1-45f5-bd01-378773eb9970" containerName="main" Jul 17 18:08:06.494931 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.494881 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4162d923-d5b1-45f5-bd01-378773eb9970" containerName="main" Jul 17 18:08:06.494931 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.494929 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4162d923-d5b1-45f5-bd01-378773eb9970" containerName="main" Jul 17 18:08:06.497968 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.497953 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:06.503897 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.503870 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-72cf7b\"/\"test-model-72cf7b-kserve-self-signed-certs\"" Jul 17 18:08:06.503897 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.503885 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-72cf7b\"/\"default-dockercfg-cpj28\"" Jul 17 18:08:06.504097 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.503911 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-72cf7b\"/\"kube-root-ca.crt\"" Jul 17 18:08:06.504097 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.503965 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-72cf7b\"/\"openshift-service-ca.crt\"" Jul 17 18:08:06.521102 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.521047 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd"] Jul 17 18:08:06.557020 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.556996 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:06.567912 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.567887 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsnsh\" (UniqueName: \"kubernetes.io/projected/105c9ae9-e224-496b-86a4-05f916821deb-kube-api-access-tsnsh\") pod \"test-model-72cf7b-kserve-56f554779f-dgmvd\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:06.568015 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.567927 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-dshm\") pod \"test-model-72cf7b-kserve-56f554779f-dgmvd\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:06.568015 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.567992 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-tmp-dir\") pod \"test-model-72cf7b-kserve-56f554779f-dgmvd\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:06.568094 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.568025 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-home\") pod \"test-model-72cf7b-kserve-56f554779f-dgmvd\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:06.568094 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.568052 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-model-cache\") pod \"test-model-72cf7b-kserve-56f554779f-dgmvd\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:06.568094 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.568070 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105c9ae9-e224-496b-86a4-05f916821deb-tls-certs\") pod \"test-model-72cf7b-kserve-56f554779f-dgmvd\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:06.668524 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.668499 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-dshm\") pod \"test-model-72cf7b-kserve-56f554779f-dgmvd\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:06.668644 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.668539 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-tmp-dir\") pod \"test-model-72cf7b-kserve-56f554779f-dgmvd\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:06.668698 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.668655 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-home\") pod \"test-model-72cf7b-kserve-56f554779f-dgmvd\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:06.668749 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.668705 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-model-cache\") pod \"test-model-72cf7b-kserve-56f554779f-dgmvd\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:06.668749 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.668725 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105c9ae9-e224-496b-86a4-05f916821deb-tls-certs\") pod \"test-model-72cf7b-kserve-56f554779f-dgmvd\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:06.668879 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.668765 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsnsh\" (UniqueName: \"kubernetes.io/projected/105c9ae9-e224-496b-86a4-05f916821deb-kube-api-access-tsnsh\") pod \"test-model-72cf7b-kserve-56f554779f-dgmvd\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:06.668972 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.668946 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-tmp-dir\") pod \"test-model-72cf7b-kserve-56f554779f-dgmvd\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:06.669088 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.669036 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-home\") pod \"test-model-72cf7b-kserve-56f554779f-dgmvd\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:06.669088 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.669049 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-model-cache\") pod \"test-model-72cf7b-kserve-56f554779f-dgmvd\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:06.670565 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.670548 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-dshm\") pod \"test-model-72cf7b-kserve-56f554779f-dgmvd\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:06.670985 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.670966 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105c9ae9-e224-496b-86a4-05f916821deb-tls-certs\") pod \"test-model-72cf7b-kserve-56f554779f-dgmvd\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:06.678788 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.678760 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsnsh\" (UniqueName: \"kubernetes.io/projected/105c9ae9-e224-496b-86a4-05f916821deb-kube-api-access-tsnsh\") pod \"test-model-72cf7b-kserve-56f554779f-dgmvd\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:06.806734 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.806679 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:06.938151 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:06.937998 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd"] Jul 17 18:08:06.940263 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:08:06.940234 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod105c9ae9_e224_496b_86a4_05f916821deb.slice/crio-ceeb3be61b07dcdf5ba0cad5e035e7b66a73617f3d8847fd5a94a2a6755074fa WatchSource:0}: Error finding container ceeb3be61b07dcdf5ba0cad5e035e7b66a73617f3d8847fd5a94a2a6755074fa: Status 404 returned error can't find the container with id ceeb3be61b07dcdf5ba0cad5e035e7b66a73617f3d8847fd5a94a2a6755074fa Jul 17 18:08:07.331186 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:07.331161 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:07.357652 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:07.357618 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" event={"ID":"105c9ae9-e224-496b-86a4-05f916821deb","Type":"ContainerStarted","Data":"8eebaa6ff98cafdbf3e59a2c0c4a74bba883eab9b01e08c6f6d26c81b1aaa958"} Jul 17 18:08:07.357792 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:07.357659 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" event={"ID":"105c9ae9-e224-496b-86a4-05f916821deb","Type":"ContainerStarted","Data":"ceeb3be61b07dcdf5ba0cad5e035e7b66a73617f3d8847fd5a94a2a6755074fa"} Jul 17 18:08:07.379006 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:07.378955 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" podStartSLOduration=1.378935683 podStartE2EDuration="1.378935683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:08:07.377562843 +0000 UTC m=+2208.496391125" watchObservedRunningTime="2026-07-17 18:08:07.378935683 +0000 UTC m=+2208.497763965" Jul 17 18:08:07.428883 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:07.428835 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:15.889265 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:15.889235 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:15.987931 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:15.987895 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:16.556597 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:16.556566 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:08:16.806829 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:16.806755 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:16.806829 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:16.806789 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:16.818646 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:16.818625 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:17.420357 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:17.420328 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:08:33.136030 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.135996 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk"] Jul 17 18:08:33.139376 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.139357 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:33.146983 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.146962 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3145cb\"/\"default-dockercfg-fhsxg\"" Jul 17 18:08:33.147101 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.147033 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3145cb\"/\"test-model-3145cb-kserve-self-signed-certs\"" Jul 17 18:08:33.147101 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.147052 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3145cb\"/\"openshift-service-ca.crt\"" Jul 17 18:08:33.147101 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.147081 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3145cb\"/\"kube-root-ca.crt\"" Jul 17 18:08:33.152750 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.152726 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk"] Jul 17 18:08:33.182909 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.182886 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:33.283291 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.283272 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb22a476-ad52-4bb8-84b6-31c9825cad59-tls-certs\") pod \"test-model-3145cb-kserve-5645d7bfcc-lbbrk\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:33.283390 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.283296 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-tmp-dir\") pod \"test-model-3145cb-kserve-5645d7bfcc-lbbrk\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:33.283390 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.283342 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-dshm\") pod \"test-model-3145cb-kserve-5645d7bfcc-lbbrk\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:33.283390 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.283378 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s9tn\" (UniqueName: \"kubernetes.io/projected/bb22a476-ad52-4bb8-84b6-31c9825cad59-kube-api-access-2s9tn\") pod \"test-model-3145cb-kserve-5645d7bfcc-lbbrk\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:33.283493 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.283424 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-model-cache\") pod \"test-model-3145cb-kserve-5645d7bfcc-lbbrk\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:33.283493 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.283457 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-home\") pod \"test-model-3145cb-kserve-5645d7bfcc-lbbrk\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:33.384126 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.384108 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb22a476-ad52-4bb8-84b6-31c9825cad59-tls-certs\") pod \"test-model-3145cb-kserve-5645d7bfcc-lbbrk\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:33.384224 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.384134 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-tmp-dir\") pod \"test-model-3145cb-kserve-5645d7bfcc-lbbrk\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:33.384224 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.384175 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-dshm\") pod \"test-model-3145cb-kserve-5645d7bfcc-lbbrk\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:33.384224 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.384200 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s9tn\" (UniqueName: \"kubernetes.io/projected/bb22a476-ad52-4bb8-84b6-31c9825cad59-kube-api-access-2s9tn\") pod \"test-model-3145cb-kserve-5645d7bfcc-lbbrk\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:33.384351 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.384249 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-model-cache\") pod \"test-model-3145cb-kserve-5645d7bfcc-lbbrk\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:33.384351 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.384294 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-home\") pod \"test-model-3145cb-kserve-5645d7bfcc-lbbrk\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:33.384560 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.384542 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-tmp-dir\") pod \"test-model-3145cb-kserve-5645d7bfcc-lbbrk\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:33.384626 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.384601 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-home\") pod \"test-model-3145cb-kserve-5645d7bfcc-lbbrk\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:33.384708 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.384667 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-model-cache\") pod \"test-model-3145cb-kserve-5645d7bfcc-lbbrk\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:33.386532 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.386478 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-dshm\") pod \"test-model-3145cb-kserve-5645d7bfcc-lbbrk\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:33.386671 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.386654 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb22a476-ad52-4bb8-84b6-31c9825cad59-tls-certs\") pod \"test-model-3145cb-kserve-5645d7bfcc-lbbrk\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:33.393875 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.393837 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s9tn\" (UniqueName: \"kubernetes.io/projected/bb22a476-ad52-4bb8-84b6-31c9825cad59-kube-api-access-2s9tn\") pod \"test-model-3145cb-kserve-5645d7bfcc-lbbrk\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:33.448255 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.448231 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:33.577989 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:33.577889 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk"] Jul 17 18:08:33.580262 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:08:33.580230 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb22a476_ad52_4bb8_84b6_31c9825cad59.slice/crio-6710251f4971f67f0b85be0447f6d05e46e6e958d46235e2462423b22ad4dfd0 WatchSource:0}: Error finding container 6710251f4971f67f0b85be0447f6d05e46e6e958d46235e2462423b22ad4dfd0: Status 404 returned error can't find the container with id 6710251f4971f67f0b85be0447f6d05e46e6e958d46235e2462423b22ad4dfd0 Jul 17 18:08:34.488508 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:34.488468 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" event={"ID":"bb22a476-ad52-4bb8-84b6-31c9825cad59","Type":"ContainerStarted","Data":"6bca25367ea5f33e3b5cceb9f451693f9f3a8a1d5c9f8dc6a5fdede832843db6"} Jul 17 18:08:34.488885 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:34.488522 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" event={"ID":"bb22a476-ad52-4bb8-84b6-31c9825cad59","Type":"ContainerStarted","Data":"6710251f4971f67f0b85be0447f6d05e46e6e958d46235e2462423b22ad4dfd0"} Jul 17 18:08:34.506244 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:34.506193 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" podStartSLOduration=1.506176025 podStartE2EDuration="1.506176025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:08:34.505548152 +0000 UTC m=+2235.624376432" watchObservedRunningTime="2026-07-17 18:08:34.506176025 +0000 UTC m=+2235.625004308" Jul 17 18:08:37.637871 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:37.637815 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:08:41.073372 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:41.073248 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:43.448528 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:43.448494 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:43.448934 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:43.448587 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:43.464547 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:43.464518 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:43.535397 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:43.535373 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:08:51.935320 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:51.935288 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:08:58.337254 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:58.337221 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd"] Jul 17 18:08:58.359159 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:58.359134 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:59.257887 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:59.257839 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk"] Jul 17 18:08:59.278724 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:59.278701 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:08:59.776051 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:08:59.776021 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:04.639313 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:04.639280 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:09:04.851329 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:04.851300 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:13.356923 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.356841 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" podUID="105c9ae9-e224-496b-86a4-05f916821deb" containerName="main" containerID="cri-o://8eebaa6ff98cafdbf3e59a2c0c4a74bba883eab9b01e08c6f6d26c81b1aaa958" gracePeriod=45 Jul 17 18:09:13.612469 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.612414 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:09:13.661534 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.661503 2553 generic.go:356] "Generic (PLEG): container finished" podID="105c9ae9-e224-496b-86a4-05f916821deb" containerID="8eebaa6ff98cafdbf3e59a2c0c4a74bba883eab9b01e08c6f6d26c81b1aaa958" exitCode=0 Jul 17 18:09:13.661668 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.661544 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" Jul 17 18:09:13.661668 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.661590 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" event={"ID":"105c9ae9-e224-496b-86a4-05f916821deb","Type":"ContainerDied","Data":"8eebaa6ff98cafdbf3e59a2c0c4a74bba883eab9b01e08c6f6d26c81b1aaa958"} Jul 17 18:09:13.661668 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.661633 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd" event={"ID":"105c9ae9-e224-496b-86a4-05f916821deb","Type":"ContainerDied","Data":"ceeb3be61b07dcdf5ba0cad5e035e7b66a73617f3d8847fd5a94a2a6755074fa"} Jul 17 18:09:13.661668 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.661654 2553 scope.go:117] "RemoveContainer" containerID="8eebaa6ff98cafdbf3e59a2c0c4a74bba883eab9b01e08c6f6d26c81b1aaa958" Jul 17 18:09:13.669831 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.669816 2553 scope.go:117] "RemoveContainer" containerID="8eebaa6ff98cafdbf3e59a2c0c4a74bba883eab9b01e08c6f6d26c81b1aaa958" Jul 17 18:09:13.670136 ip-10-0-136-43 kubenswrapper[2553]: E0717 18:09:13.670117 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eebaa6ff98cafdbf3e59a2c0c4a74bba883eab9b01e08c6f6d26c81b1aaa958\": container with ID starting with 8eebaa6ff98cafdbf3e59a2c0c4a74bba883eab9b01e08c6f6d26c81b1aaa958 not found: ID does not exist" containerID="8eebaa6ff98cafdbf3e59a2c0c4a74bba883eab9b01e08c6f6d26c81b1aaa958" Jul 17 18:09:13.670212 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.670142 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eebaa6ff98cafdbf3e59a2c0c4a74bba883eab9b01e08c6f6d26c81b1aaa958"} err="failed to get container status \"8eebaa6ff98cafdbf3e59a2c0c4a74bba883eab9b01e08c6f6d26c81b1aaa958\": rpc error: code = NotFound desc = could not find container \"8eebaa6ff98cafdbf3e59a2c0c4a74bba883eab9b01e08c6f6d26c81b1aaa958\": container with ID starting with 8eebaa6ff98cafdbf3e59a2c0c4a74bba883eab9b01e08c6f6d26c81b1aaa958 not found: ID does not exist" Jul 17 18:09:13.696572 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.696551 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-tmp-dir\") pod \"105c9ae9-e224-496b-86a4-05f916821deb\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " Jul 17 18:09:13.696660 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.696584 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-model-cache\") pod \"105c9ae9-e224-496b-86a4-05f916821deb\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " Jul 17 18:09:13.696660 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.696652 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsnsh\" (UniqueName: \"kubernetes.io/projected/105c9ae9-e224-496b-86a4-05f916821deb-kube-api-access-tsnsh\") pod \"105c9ae9-e224-496b-86a4-05f916821deb\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " Jul 17 18:09:13.696748 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.696673 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-home\") pod \"105c9ae9-e224-496b-86a4-05f916821deb\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " Jul 17 18:09:13.696748 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.696702 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-dshm\") pod \"105c9ae9-e224-496b-86a4-05f916821deb\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " Jul 17 18:09:13.696748 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.696730 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105c9ae9-e224-496b-86a4-05f916821deb-tls-certs\") pod \"105c9ae9-e224-496b-86a4-05f916821deb\" (UID: \"105c9ae9-e224-496b-86a4-05f916821deb\") " Jul 17 18:09:13.696906 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.696810 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "105c9ae9-e224-496b-86a4-05f916821deb" (UID: "105c9ae9-e224-496b-86a4-05f916821deb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:09:13.696906 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.696893 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-model-cache" (OuterVolumeSpecName: "model-cache") pod "105c9ae9-e224-496b-86a4-05f916821deb" (UID: "105c9ae9-e224-496b-86a4-05f916821deb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:09:13.697024 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.696996 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-home" (OuterVolumeSpecName: "home") pod "105c9ae9-e224-496b-86a4-05f916821deb" (UID: "105c9ae9-e224-496b-86a4-05f916821deb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:09:13.697081 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.697069 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-tmp-dir\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:09:13.697135 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.697082 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-model-cache\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:09:13.697135 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.697092 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-home\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:09:13.698770 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.698736 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-dshm" (OuterVolumeSpecName: "dshm") pod "105c9ae9-e224-496b-86a4-05f916821deb" (UID: "105c9ae9-e224-496b-86a4-05f916821deb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:09:13.698877 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.698768 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105c9ae9-e224-496b-86a4-05f916821deb-kube-api-access-tsnsh" (OuterVolumeSpecName: "kube-api-access-tsnsh") pod "105c9ae9-e224-496b-86a4-05f916821deb" (UID: "105c9ae9-e224-496b-86a4-05f916821deb"). InnerVolumeSpecName "kube-api-access-tsnsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:09:13.698877 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.698833 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/105c9ae9-e224-496b-86a4-05f916821deb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "105c9ae9-e224-496b-86a4-05f916821deb" (UID: "105c9ae9-e224-496b-86a4-05f916821deb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:09:13.798348 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.798319 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsnsh\" (UniqueName: \"kubernetes.io/projected/105c9ae9-e224-496b-86a4-05f916821deb-kube-api-access-tsnsh\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:09:13.798348 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.798340 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/105c9ae9-e224-496b-86a4-05f916821deb-dshm\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:09:13.798348 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.798349 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105c9ae9-e224-496b-86a4-05f916821deb-tls-certs\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:09:13.984858 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.984828 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd"] Jul 17 18:09:13.988773 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:13.988753 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-72cf7b/test-model-72cf7b-kserve-56f554779f-dgmvd"] Jul 17 18:09:14.277992 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.277950 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" podUID="bb22a476-ad52-4bb8-84b6-31c9825cad59" containerName="main" containerID="cri-o://6bca25367ea5f33e3b5cceb9f451693f9f3a8a1d5c9f8dc6a5fdede832843db6" gracePeriod=45 Jul 17 18:09:14.525287 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.525268 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:09:14.605771 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.605704 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-home\") pod \"bb22a476-ad52-4bb8-84b6-31c9825cad59\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " Jul 17 18:09:14.605771 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.605763 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s9tn\" (UniqueName: \"kubernetes.io/projected/bb22a476-ad52-4bb8-84b6-31c9825cad59-kube-api-access-2s9tn\") pod \"bb22a476-ad52-4bb8-84b6-31c9825cad59\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " Jul 17 18:09:14.605981 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.605799 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-dshm\") pod \"bb22a476-ad52-4bb8-84b6-31c9825cad59\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " Jul 17 18:09:14.605981 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.605877 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb22a476-ad52-4bb8-84b6-31c9825cad59-tls-certs\") pod \"bb22a476-ad52-4bb8-84b6-31c9825cad59\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " Jul 17 18:09:14.605981 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.605914 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-tmp-dir\") pod \"bb22a476-ad52-4bb8-84b6-31c9825cad59\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " Jul 17 18:09:14.605981 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.605940 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-model-cache\") pod \"bb22a476-ad52-4bb8-84b6-31c9825cad59\" (UID: \"bb22a476-ad52-4bb8-84b6-31c9825cad59\") " Jul 17 18:09:14.605981 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.605942 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-home" (OuterVolumeSpecName: "home") pod "bb22a476-ad52-4bb8-84b6-31c9825cad59" (UID: "bb22a476-ad52-4bb8-84b6-31c9825cad59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:09:14.606217 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.606170 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb22a476-ad52-4bb8-84b6-31c9825cad59" (UID: "bb22a476-ad52-4bb8-84b6-31c9825cad59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:09:14.606217 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.606191 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb22a476-ad52-4bb8-84b6-31c9825cad59" (UID: "bb22a476-ad52-4bb8-84b6-31c9825cad59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:09:14.606293 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.606215 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-home\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:09:14.608057 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.608037 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb22a476-ad52-4bb8-84b6-31c9825cad59-kube-api-access-2s9tn" (OuterVolumeSpecName: "kube-api-access-2s9tn") pod "bb22a476-ad52-4bb8-84b6-31c9825cad59" (UID: "bb22a476-ad52-4bb8-84b6-31c9825cad59"). InnerVolumeSpecName "kube-api-access-2s9tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:09:14.608304 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.608279 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb22a476-ad52-4bb8-84b6-31c9825cad59-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb22a476-ad52-4bb8-84b6-31c9825cad59" (UID: "bb22a476-ad52-4bb8-84b6-31c9825cad59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:09:14.608400 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.608300 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-dshm" (OuterVolumeSpecName: "dshm") pod "bb22a476-ad52-4bb8-84b6-31c9825cad59" (UID: "bb22a476-ad52-4bb8-84b6-31c9825cad59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:09:14.667629 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.667602 2553 generic.go:356] "Generic (PLEG): container finished" podID="bb22a476-ad52-4bb8-84b6-31c9825cad59" containerID="6bca25367ea5f33e3b5cceb9f451693f9f3a8a1d5c9f8dc6a5fdede832843db6" exitCode=0 Jul 17 18:09:14.667731 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.667635 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" event={"ID":"bb22a476-ad52-4bb8-84b6-31c9825cad59","Type":"ContainerDied","Data":"6bca25367ea5f33e3b5cceb9f451693f9f3a8a1d5c9f8dc6a5fdede832843db6"} Jul 17 18:09:14.667731 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.667686 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" event={"ID":"bb22a476-ad52-4bb8-84b6-31c9825cad59","Type":"ContainerDied","Data":"6710251f4971f67f0b85be0447f6d05e46e6e958d46235e2462423b22ad4dfd0"} Jul 17 18:09:14.667731 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.667711 2553 scope.go:117] "RemoveContainer" containerID="6bca25367ea5f33e3b5cceb9f451693f9f3a8a1d5c9f8dc6a5fdede832843db6" Jul 17 18:09:14.667906 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.667651 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk" Jul 17 18:09:14.677570 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.677549 2553 scope.go:117] "RemoveContainer" containerID="6bca25367ea5f33e3b5cceb9f451693f9f3a8a1d5c9f8dc6a5fdede832843db6" Jul 17 18:09:14.677888 ip-10-0-136-43 kubenswrapper[2553]: E0717 18:09:14.677843 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bca25367ea5f33e3b5cceb9f451693f9f3a8a1d5c9f8dc6a5fdede832843db6\": container with ID starting with 6bca25367ea5f33e3b5cceb9f451693f9f3a8a1d5c9f8dc6a5fdede832843db6 not found: ID does not exist" containerID="6bca25367ea5f33e3b5cceb9f451693f9f3a8a1d5c9f8dc6a5fdede832843db6" Jul 17 18:09:14.677976 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.677896 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bca25367ea5f33e3b5cceb9f451693f9f3a8a1d5c9f8dc6a5fdede832843db6"} err="failed to get container status \"6bca25367ea5f33e3b5cceb9f451693f9f3a8a1d5c9f8dc6a5fdede832843db6\": rpc error: code = NotFound desc = could not find container \"6bca25367ea5f33e3b5cceb9f451693f9f3a8a1d5c9f8dc6a5fdede832843db6\": container with ID starting with 6bca25367ea5f33e3b5cceb9f451693f9f3a8a1d5c9f8dc6a5fdede832843db6 not found: ID does not exist" Jul 17 18:09:14.690219 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.690199 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk"] Jul 17 18:09:14.693901 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.693881 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3145cb/test-model-3145cb-kserve-5645d7bfcc-lbbrk"] Jul 17 18:09:14.707554 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.707537 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s9tn\" (UniqueName: \"kubernetes.io/projected/bb22a476-ad52-4bb8-84b6-31c9825cad59-kube-api-access-2s9tn\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:09:14.707554 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.707553 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-dshm\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:09:14.707654 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.707563 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb22a476-ad52-4bb8-84b6-31c9825cad59-tls-certs\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:09:14.707654 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.707572 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-tmp-dir\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:09:14.707654 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:14.707579 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb22a476-ad52-4bb8-84b6-31c9825cad59-model-cache\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:09:15.463411 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:15.463380 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="105c9ae9-e224-496b-86a4-05f916821deb" path="/var/lib/kubelet/pods/105c9ae9-e224-496b-86a4-05f916821deb/volumes" Jul 17 18:09:15.463749 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:15.463736 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb22a476-ad52-4bb8-84b6-31c9825cad59" path="/var/lib/kubelet/pods/bb22a476-ad52-4bb8-84b6-31c9825cad59/volumes" Jul 17 18:09:43.471740 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:43.471712 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:43.912433 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:43.912401 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:44.067305 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:44.067273 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:44.594201 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:44.594169 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:47.076666 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:47.076633 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:09:52.084914 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:52.084879 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:52.201339 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:52.201314 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:52.490896 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:52.490869 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:52.741864 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:52.741780 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:53.142950 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:53.142871 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:53.674426 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:53.674379 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:53.955010 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:53.954919 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:55.646504 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:55.646451 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:55.813806 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:55.813779 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:56.037352 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:56.037326 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:56.192216 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:56.192167 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:56.826335 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:56.826299 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:09:57.003917 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:09:57.003887 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:01.143290 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:01.143254 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:01.364602 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:01.364567 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:10:01.646627 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:01.646599 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:02.161303 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:02.161273 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:02.460190 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:02.460119 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:04.863115 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:04.863087 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:05.121023 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:05.120932 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:05.382860 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:05.382724 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:05.552873 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:05.552816 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:06.244214 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:06.244177 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:06.353484 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:06.353442 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:07.725431 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:07.725398 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:07.879943 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:07.879865 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:09.299129 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:09.299099 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:09.477811 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:09.477784 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:10.414894 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:10.414841 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:12.386431 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:12.386396 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:10:16.983001 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:16.982969 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:18.225451 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:18.225416 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:18.377656 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:18.377608 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:19.302531 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:19.302502 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:26.329388 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:26.329357 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:27.564944 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:27.564911 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:27.720713 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:27.720675 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:32.859379 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:32.859340 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:32.987839 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:32.987785 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:33.264196 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:33.264165 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:33.499582 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:33.499554 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:34.086274 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:34.086238 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:36.673700 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:36.673662 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:10:42.602059 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:42.602026 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:49.100071 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:49.100040 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:49.196817 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:49.196790 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:49.502402 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:49.502364 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:49.723633 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:49.723607 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:50.116349 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:50.116320 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:50.485817 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:50.485787 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:50.724924 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:50.724891 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:52.833263 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:52.833232 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:52.928734 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:52.928683 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:53.012369 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:53.012342 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:53.348904 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:53.348873 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:53.672161 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:53.672086 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:53.770922 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:53.770889 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:54.139058 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:54.139030 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:10:58.350992 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:58.350963 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:58.723700 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:58.723667 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:59.071009 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:59.070936 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:59.230430 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:10:59.230403 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:00.820018 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:00.819988 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:00.966519 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:00.966484 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:01.127170 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:01.127096 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:01.848204 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:01.848174 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:01.955302 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:01.955277 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:06.382070 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:06.382035 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:09.135543 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:09.135514 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:11:14.559756 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:14.559716 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:14.663032 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:14.662999 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:22.944549 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:22.944510 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:23.047459 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:23.047427 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:40.354336 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.354298 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f"] Jul 17 18:11:40.354866 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.354650 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="105c9ae9-e224-496b-86a4-05f916821deb" containerName="main" Jul 17 18:11:40.354866 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.354662 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="105c9ae9-e224-496b-86a4-05f916821deb" containerName="main" Jul 17 18:11:40.354866 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.354675 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb22a476-ad52-4bb8-84b6-31c9825cad59" containerName="main" Jul 17 18:11:40.354866 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.354681 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb22a476-ad52-4bb8-84b6-31c9825cad59" containerName="main" Jul 17 18:11:40.354866 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.354762 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="105c9ae9-e224-496b-86a4-05f916821deb" containerName="main" Jul 17 18:11:40.354866 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.354775 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb22a476-ad52-4bb8-84b6-31c9825cad59" containerName="main" Jul 17 18:11:40.357958 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.357941 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:40.361777 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.361753 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0a98d9\"/\"default-dockercfg-r9nf2\"" Jul 17 18:11:40.361777 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.361754 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0a98d9\"/\"auth-test-model-0a98d9-kserve-self-signed-certs\"" Jul 17 18:11:40.361980 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.361821 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0a98d9\"/\"openshift-service-ca.crt\"" Jul 17 18:11:40.361980 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.361754 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0a98d9\"/\"kube-root-ca.crt\"" Jul 17 18:11:40.367813 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.367787 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f"] Jul 17 18:11:40.443149 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.443124 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-tmp-dir\") pod \"auth-test-model-0a98d9-kserve-85b749d57f-gjb5f\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:40.443270 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.443186 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-model-cache\") pod \"auth-test-model-0a98d9-kserve-85b749d57f-gjb5f\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:40.443270 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.443231 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-dshm\") pod \"auth-test-model-0a98d9-kserve-85b749d57f-gjb5f\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:40.443270 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.443245 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3360f3f8-ce81-4330-806d-3c74570133c7-tls-certs\") pod \"auth-test-model-0a98d9-kserve-85b749d57f-gjb5f\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:40.443270 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.443261 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb5hv\" (UniqueName: \"kubernetes.io/projected/3360f3f8-ce81-4330-806d-3c74570133c7-kube-api-access-wb5hv\") pod \"auth-test-model-0a98d9-kserve-85b749d57f-gjb5f\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:40.443403 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.443315 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-home\") pod \"auth-test-model-0a98d9-kserve-85b749d57f-gjb5f\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:40.457040 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.457012 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:40.544134 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.544108 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-dshm\") pod \"auth-test-model-0a98d9-kserve-85b749d57f-gjb5f\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:40.544264 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.544138 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3360f3f8-ce81-4330-806d-3c74570133c7-tls-certs\") pod \"auth-test-model-0a98d9-kserve-85b749d57f-gjb5f\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:40.544264 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.544161 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb5hv\" (UniqueName: \"kubernetes.io/projected/3360f3f8-ce81-4330-806d-3c74570133c7-kube-api-access-wb5hv\") pod \"auth-test-model-0a98d9-kserve-85b749d57f-gjb5f\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:40.544264 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.544200 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-home\") pod \"auth-test-model-0a98d9-kserve-85b749d57f-gjb5f\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:40.544264 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.544230 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-tmp-dir\") pod \"auth-test-model-0a98d9-kserve-85b749d57f-gjb5f\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:40.544481 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.544318 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-model-cache\") pod \"auth-test-model-0a98d9-kserve-85b749d57f-gjb5f\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:40.544575 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.544554 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-home\") pod \"auth-test-model-0a98d9-kserve-85b749d57f-gjb5f\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:40.544636 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.544582 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-tmp-dir\") pod \"auth-test-model-0a98d9-kserve-85b749d57f-gjb5f\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:40.544696 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.544646 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-model-cache\") pod \"auth-test-model-0a98d9-kserve-85b749d57f-gjb5f\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:40.546362 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.546346 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-dshm\") pod \"auth-test-model-0a98d9-kserve-85b749d57f-gjb5f\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:40.546526 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.546507 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3360f3f8-ce81-4330-806d-3c74570133c7-tls-certs\") pod \"auth-test-model-0a98d9-kserve-85b749d57f-gjb5f\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:40.553783 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.553752 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb5hv\" (UniqueName: \"kubernetes.io/projected/3360f3f8-ce81-4330-806d-3c74570133c7-kube-api-access-wb5hv\") pod \"auth-test-model-0a98d9-kserve-85b749d57f-gjb5f\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:40.667956 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.667878 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:40.797149 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:40.797124 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f"] Jul 17 18:11:40.798992 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:11:40.798968 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3360f3f8_ce81_4330_806d_3c74570133c7.slice/crio-344fdaaeb9137322bfebd492d48740103a4103abca7db76fc77b60390989cbb2 WatchSource:0}: Error finding container 344fdaaeb9137322bfebd492d48740103a4103abca7db76fc77b60390989cbb2: Status 404 returned error can't find the container with id 344fdaaeb9137322bfebd492d48740103a4103abca7db76fc77b60390989cbb2 Jul 17 18:11:41.335434 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:41.335396 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" event={"ID":"3360f3f8-ce81-4330-806d-3c74570133c7","Type":"ContainerStarted","Data":"1395bcb444c120f615004c2b409c62bee8aecbc90ebfbed9d6db516cb337bc12"} Jul 17 18:11:41.335588 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:41.335443 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" event={"ID":"3360f3f8-ce81-4330-806d-3c74570133c7","Type":"ContainerStarted","Data":"344fdaaeb9137322bfebd492d48740103a4103abca7db76fc77b60390989cbb2"} Jul 17 18:11:41.357642 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:41.357597 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" podStartSLOduration=1.3575814990000001 podStartE2EDuration="1.357581499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:11:41.356375818 +0000 UTC m=+2422.475204099" watchObservedRunningTime="2026-07-17 18:11:41.357581499 +0000 UTC m=+2422.476409780" Jul 17 18:11:43.830740 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:43.830705 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:11:50.668371 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:50.668335 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:50.668781 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:50.668401 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:50.683436 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:50.683413 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:11:51.392031 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:11:51.392005 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:12:01.752486 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.752448 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7"] Jul 17 18:12:01.760713 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.759279 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:01.762633 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.762605 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-37d0b7\"/\"auth-test-model-37d0b7-kserve-self-signed-certs\"" Jul 17 18:12:01.763882 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.763703 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-37d0b7\"/\"kube-root-ca.crt\"" Jul 17 18:12:01.763882 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.763752 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-37d0b7\"/\"openshift-service-ca.crt\"" Jul 17 18:12:01.763882 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.763753 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-37d0b7\"/\"default-dockercfg-tbg82\"" Jul 17 18:12:01.770469 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.770446 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7"] Jul 17 18:12:01.802684 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.802658 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:01.829995 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.829971 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-dshm\") pod \"auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:01.830099 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.829998 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5798ad6b-6a78-4c98-bb75-f093562f6b63-tls-certs\") pod \"auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:01.830099 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.830023 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-tmp-dir\") pod \"auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:01.830099 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.830091 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-model-cache\") pod \"auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:01.830202 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.830112 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nzqk\" (UniqueName: \"kubernetes.io/projected/5798ad6b-6a78-4c98-bb75-f093562f6b63-kube-api-access-6nzqk\") pod \"auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:01.830202 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.830155 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-home\") pod \"auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:01.931310 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.931290 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-tmp-dir\") pod \"auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:01.931402 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.931325 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-model-cache\") pod \"auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:01.931402 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.931351 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nzqk\" (UniqueName: \"kubernetes.io/projected/5798ad6b-6a78-4c98-bb75-f093562f6b63-kube-api-access-6nzqk\") pod \"auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:01.931402 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.931387 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-home\") pod \"auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:01.931504 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.931473 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-dshm\") pod \"auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:01.931504 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.931494 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5798ad6b-6a78-4c98-bb75-f093562f6b63-tls-certs\") pod \"auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:01.931724 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.931708 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-model-cache\") pod \"auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:01.931778 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.931759 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-tmp-dir\") pod \"auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:01.931814 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.931785 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-home\") pod \"auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:01.933555 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.933529 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-dshm\") pod \"auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:01.933841 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.933824 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5798ad6b-6a78-4c98-bb75-f093562f6b63-tls-certs\") pod \"auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:01.941356 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:01.941337 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nzqk\" (UniqueName: \"kubernetes.io/projected/5798ad6b-6a78-4c98-bb75-f093562f6b63-kube-api-access-6nzqk\") pod \"auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:02.070765 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:02.070707 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:02.197523 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:02.197496 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7"] Jul 17 18:12:02.199129 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:12:02.199104 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5798ad6b_6a78_4c98_bb75_f093562f6b63.slice/crio-5f75f84aabd7003a5abd35344dc9ab09dae196b6f0a77d06c98e21f8c122d68e WatchSource:0}: Error finding container 5f75f84aabd7003a5abd35344dc9ab09dae196b6f0a77d06c98e21f8c122d68e: Status 404 returned error can't find the container with id 5f75f84aabd7003a5abd35344dc9ab09dae196b6f0a77d06c98e21f8c122d68e Jul 17 18:12:02.438706 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:02.438612 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" event={"ID":"5798ad6b-6a78-4c98-bb75-f093562f6b63","Type":"ContainerStarted","Data":"592bc21c04c84837d1f2488ce6ba642872e0a79eef54ecb8ad504446d1238423"} Jul 17 18:12:02.438706 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:02.438654 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" event={"ID":"5798ad6b-6a78-4c98-bb75-f093562f6b63","Type":"ContainerStarted","Data":"5f75f84aabd7003a5abd35344dc9ab09dae196b6f0a77d06c98e21f8c122d68e"} Jul 17 18:12:02.460387 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:02.460339 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" podStartSLOduration=1.460322762 podStartE2EDuration="1.460322762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:12:02.458692935 +0000 UTC m=+2443.577521216" watchObservedRunningTime="2026-07-17 18:12:02.460322762 +0000 UTC m=+2443.579151021" Jul 17 18:12:05.430685 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:05.430642 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:12:12.071676 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:12.071619 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:12.071676 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:12.071683 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:12.084227 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:12.084201 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:12.493215 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:12.493189 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:12:18.560863 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:18.560815 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:12:21.735948 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:21.735915 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:12:38.237264 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:12:38.237231 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:13:00.041701 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:13:00.041663 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:13:03.502074 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:13:03.502045 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:04.732821 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:13:04.732779 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:19.833869 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:13:19.833762 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:13:36.945447 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:13:36.945413 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:13:45.780952 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:13:45.780915 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:13:56.740606 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:13:56.740568 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:14:14.139292 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:14.139260 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:14:18.923041 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:18.923000 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f"] Jul 17 18:14:18.943899 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:18.943869 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:19.762688 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:19.762653 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7"] Jul 17 18:14:19.782058 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:19.782034 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:24.360611 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.360579 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt"] Jul 17 18:14:24.363967 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.363952 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:24.366728 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.366710 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0a98d9\"/\"sub-test-model-0a98d9-kserve-self-signed-certs\"" Jul 17 18:14:24.378833 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.378807 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt"] Jul 17 18:14:24.387444 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.387397 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:24.450166 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.450139 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-dshm\") pod \"sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:24.450316 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.450184 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wqlw\" (UniqueName: \"kubernetes.io/projected/bf8013ff-a730-4a8b-97ac-30f814d00a27-kube-api-access-2wqlw\") pod \"sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:24.450316 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.450207 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-model-cache\") pod \"sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:24.450416 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.450319 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-tmp-dir\") pod \"sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:24.450416 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.450345 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-home\") pod \"sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:24.450416 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.450362 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf8013ff-a730-4a8b-97ac-30f814d00a27-tls-certs\") pod \"sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:24.551658 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.551626 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-tmp-dir\") pod \"sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:24.551658 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.551656 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-home\") pod \"sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:24.551936 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.551671 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf8013ff-a730-4a8b-97ac-30f814d00a27-tls-certs\") pod \"sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:24.551936 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.551723 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-dshm\") pod \"sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:24.551936 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.551756 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wqlw\" (UniqueName: \"kubernetes.io/projected/bf8013ff-a730-4a8b-97ac-30f814d00a27-kube-api-access-2wqlw\") pod \"sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:24.551936 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.551778 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-model-cache\") pod \"sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:24.552154 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.552124 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-home\") pod \"sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:24.552202 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.552165 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-tmp-dir\") pod \"sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:24.552246 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.552227 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-model-cache\") pod \"sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:24.553927 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.553908 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-dshm\") pod \"sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:24.554107 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.554089 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf8013ff-a730-4a8b-97ac-30f814d00a27-tls-certs\") pod \"sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:24.559944 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.559921 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wqlw\" (UniqueName: \"kubernetes.io/projected/bf8013ff-a730-4a8b-97ac-30f814d00a27-kube-api-access-2wqlw\") pod \"sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:24.673949 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.673864 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:24.813749 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:24.813721 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt"] Jul 17 18:14:24.818146 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:14:24.818014 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf8013ff_a730_4a8b_97ac_30f814d00a27.slice/crio-bc5a7a1bf92c4770c1335d0fdc33e9c5ede07e330abc77736232588206907c0b WatchSource:0}: Error finding container bc5a7a1bf92c4770c1335d0fdc33e9c5ede07e330abc77736232588206907c0b: Status 404 returned error can't find the container with id bc5a7a1bf92c4770c1335d0fdc33e9c5ede07e330abc77736232588206907c0b Jul 17 18:14:25.088983 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:25.088951 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" event={"ID":"bf8013ff-a730-4a8b-97ac-30f814d00a27","Type":"ContainerStarted","Data":"c8140df1e77ec302a5dedbd0b71b035a3c068d80cb0f37ce58d08fcb2a402e55"} Jul 17 18:14:25.088983 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:25.088992 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" event={"ID":"bf8013ff-a730-4a8b-97ac-30f814d00a27","Type":"ContainerStarted","Data":"bc5a7a1bf92c4770c1335d0fdc33e9c5ede07e330abc77736232588206907c0b"} Jul 17 18:14:25.108435 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:25.108384 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" podStartSLOduration=1.108368093 podStartE2EDuration="1.108368093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:14:25.107353885 +0000 UTC m=+2586.226182166" watchObservedRunningTime="2026-07-17 18:14:25.108368093 +0000 UTC m=+2586.227196388" Jul 17 18:14:33.942586 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:33.942532 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" podUID="3360f3f8-ce81-4330-806d-3c74570133c7" containerName="main" containerID="cri-o://1395bcb444c120f615004c2b409c62bee8aecbc90ebfbed9d6db516cb337bc12" gracePeriod=45 Jul 17 18:14:33.943551 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:33.943525 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:14:34.132073 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.132045 2553 generic.go:356] "Generic (PLEG): container finished" podID="3360f3f8-ce81-4330-806d-3c74570133c7" containerID="1395bcb444c120f615004c2b409c62bee8aecbc90ebfbed9d6db516cb337bc12" exitCode=0 Jul 17 18:14:34.132213 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.132125 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" event={"ID":"3360f3f8-ce81-4330-806d-3c74570133c7","Type":"ContainerDied","Data":"1395bcb444c120f615004c2b409c62bee8aecbc90ebfbed9d6db516cb337bc12"} Jul 17 18:14:34.182441 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.182415 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:14:34.341249 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.341214 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-model-cache\") pod \"3360f3f8-ce81-4330-806d-3c74570133c7\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " Jul 17 18:14:34.341449 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.341266 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-dshm\") pod \"3360f3f8-ce81-4330-806d-3c74570133c7\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " Jul 17 18:14:34.341449 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.341286 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3360f3f8-ce81-4330-806d-3c74570133c7-tls-certs\") pod \"3360f3f8-ce81-4330-806d-3c74570133c7\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " Jul 17 18:14:34.341449 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.341312 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-tmp-dir\") pod \"3360f3f8-ce81-4330-806d-3c74570133c7\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " Jul 17 18:14:34.341449 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.341352 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-home\") pod \"3360f3f8-ce81-4330-806d-3c74570133c7\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " Jul 17 18:14:34.341449 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.341375 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wb5hv\" (UniqueName: \"kubernetes.io/projected/3360f3f8-ce81-4330-806d-3c74570133c7-kube-api-access-wb5hv\") pod \"3360f3f8-ce81-4330-806d-3c74570133c7\" (UID: \"3360f3f8-ce81-4330-806d-3c74570133c7\") " Jul 17 18:14:34.341709 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.341571 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-model-cache" (OuterVolumeSpecName: "model-cache") pod "3360f3f8-ce81-4330-806d-3c74570133c7" (UID: "3360f3f8-ce81-4330-806d-3c74570133c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:14:34.341709 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.341581 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3360f3f8-ce81-4330-806d-3c74570133c7" (UID: "3360f3f8-ce81-4330-806d-3c74570133c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:14:34.341709 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.341606 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-home" (OuterVolumeSpecName: "home") pod "3360f3f8-ce81-4330-806d-3c74570133c7" (UID: "3360f3f8-ce81-4330-806d-3c74570133c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:14:34.341709 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.341689 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-model-cache\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:14:34.341709 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.341704 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-tmp-dir\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:14:34.341709 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.341712 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-home\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:14:34.343466 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.343440 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-dshm" (OuterVolumeSpecName: "dshm") pod "3360f3f8-ce81-4330-806d-3c74570133c7" (UID: "3360f3f8-ce81-4330-806d-3c74570133c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:14:34.343572 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.343511 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3360f3f8-ce81-4330-806d-3c74570133c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3360f3f8-ce81-4330-806d-3c74570133c7" (UID: "3360f3f8-ce81-4330-806d-3c74570133c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:14:34.343572 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.343535 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3360f3f8-ce81-4330-806d-3c74570133c7-kube-api-access-wb5hv" (OuterVolumeSpecName: "kube-api-access-wb5hv") pod "3360f3f8-ce81-4330-806d-3c74570133c7" (UID: "3360f3f8-ce81-4330-806d-3c74570133c7"). InnerVolumeSpecName "kube-api-access-wb5hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:14:34.443181 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.443150 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3360f3f8-ce81-4330-806d-3c74570133c7-dshm\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:14:34.443181 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.443176 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3360f3f8-ce81-4330-806d-3c74570133c7-tls-certs\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:14:34.443181 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.443187 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb5hv\" (UniqueName: \"kubernetes.io/projected/3360f3f8-ce81-4330-806d-3c74570133c7-kube-api-access-wb5hv\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:14:34.674038 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.673949 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:34.674038 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.673993 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:34.686294 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.686265 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:34.782272 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:34.782230 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" podUID="5798ad6b-6a78-4c98-bb75-f093562f6b63" containerName="main" containerID="cri-o://592bc21c04c84837d1f2488ce6ba642872e0a79eef54ecb8ad504446d1238423" gracePeriod=45 Jul 17 18:14:35.033895 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.033872 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:14:35.138226 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.138189 2553 generic.go:356] "Generic (PLEG): container finished" podID="5798ad6b-6a78-4c98-bb75-f093562f6b63" containerID="592bc21c04c84837d1f2488ce6ba642872e0a79eef54ecb8ad504446d1238423" exitCode=0 Jul 17 18:14:35.138377 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.138236 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" Jul 17 18:14:35.138377 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.138272 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" event={"ID":"5798ad6b-6a78-4c98-bb75-f093562f6b63","Type":"ContainerDied","Data":"592bc21c04c84837d1f2488ce6ba642872e0a79eef54ecb8ad504446d1238423"} Jul 17 18:14:35.138377 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.138311 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7" event={"ID":"5798ad6b-6a78-4c98-bb75-f093562f6b63","Type":"ContainerDied","Data":"5f75f84aabd7003a5abd35344dc9ab09dae196b6f0a77d06c98e21f8c122d68e"} Jul 17 18:14:35.138377 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.138333 2553 scope.go:117] "RemoveContainer" containerID="592bc21c04c84837d1f2488ce6ba642872e0a79eef54ecb8ad504446d1238423" Jul 17 18:14:35.139585 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.139570 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" Jul 17 18:14:35.139680 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.139619 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f" event={"ID":"3360f3f8-ce81-4330-806d-3c74570133c7","Type":"ContainerDied","Data":"344fdaaeb9137322bfebd492d48740103a4103abca7db76fc77b60390989cbb2"} Jul 17 18:14:35.147974 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.147956 2553 scope.go:117] "RemoveContainer" containerID="592bc21c04c84837d1f2488ce6ba642872e0a79eef54ecb8ad504446d1238423" Jul 17 18:14:35.148325 ip-10-0-136-43 kubenswrapper[2553]: E0717 18:14:35.148295 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592bc21c04c84837d1f2488ce6ba642872e0a79eef54ecb8ad504446d1238423\": container with ID starting with 592bc21c04c84837d1f2488ce6ba642872e0a79eef54ecb8ad504446d1238423 not found: ID does not exist" containerID="592bc21c04c84837d1f2488ce6ba642872e0a79eef54ecb8ad504446d1238423" Jul 17 18:14:35.148412 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.148332 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592bc21c04c84837d1f2488ce6ba642872e0a79eef54ecb8ad504446d1238423"} err="failed to get container status \"592bc21c04c84837d1f2488ce6ba642872e0a79eef54ecb8ad504446d1238423\": rpc error: code = NotFound desc = could not find container \"592bc21c04c84837d1f2488ce6ba642872e0a79eef54ecb8ad504446d1238423\": container with ID starting with 592bc21c04c84837d1f2488ce6ba642872e0a79eef54ecb8ad504446d1238423 not found: ID does not exist" Jul 17 18:14:35.148412 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.148354 2553 scope.go:117] "RemoveContainer" containerID="1395bcb444c120f615004c2b409c62bee8aecbc90ebfbed9d6db516cb337bc12" Jul 17 18:14:35.149333 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.149314 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-tmp-dir\") pod \"5798ad6b-6a78-4c98-bb75-f093562f6b63\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " Jul 17 18:14:35.149432 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.149362 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nzqk\" (UniqueName: \"kubernetes.io/projected/5798ad6b-6a78-4c98-bb75-f093562f6b63-kube-api-access-6nzqk\") pod \"5798ad6b-6a78-4c98-bb75-f093562f6b63\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " Jul 17 18:14:35.149432 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.149400 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-home\") pod \"5798ad6b-6a78-4c98-bb75-f093562f6b63\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " Jul 17 18:14:35.149535 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.149432 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5798ad6b-6a78-4c98-bb75-f093562f6b63-tls-certs\") pod \"5798ad6b-6a78-4c98-bb75-f093562f6b63\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " Jul 17 18:14:35.149535 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.149496 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-dshm\") pod \"5798ad6b-6a78-4c98-bb75-f093562f6b63\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " Jul 17 18:14:35.149698 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.149541 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-model-cache\") pod \"5798ad6b-6a78-4c98-bb75-f093562f6b63\" (UID: \"5798ad6b-6a78-4c98-bb75-f093562f6b63\") " Jul 17 18:14:35.149755 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.149692 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5798ad6b-6a78-4c98-bb75-f093562f6b63" (UID: "5798ad6b-6a78-4c98-bb75-f093562f6b63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:14:35.149901 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.149881 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-tmp-dir\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:14:35.150004 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.149918 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-model-cache" (OuterVolumeSpecName: "model-cache") pod "5798ad6b-6a78-4c98-bb75-f093562f6b63" (UID: "5798ad6b-6a78-4c98-bb75-f093562f6b63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:14:35.150004 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.149985 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-home" (OuterVolumeSpecName: "home") pod "5798ad6b-6a78-4c98-bb75-f093562f6b63" (UID: "5798ad6b-6a78-4c98-bb75-f093562f6b63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:14:35.151659 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.151635 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5798ad6b-6a78-4c98-bb75-f093562f6b63-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5798ad6b-6a78-4c98-bb75-f093562f6b63" (UID: "5798ad6b-6a78-4c98-bb75-f093562f6b63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:14:35.151815 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.151796 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5798ad6b-6a78-4c98-bb75-f093562f6b63-kube-api-access-6nzqk" (OuterVolumeSpecName: "kube-api-access-6nzqk") pod "5798ad6b-6a78-4c98-bb75-f093562f6b63" (UID: "5798ad6b-6a78-4c98-bb75-f093562f6b63"). InnerVolumeSpecName "kube-api-access-6nzqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:14:35.151918 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.151798 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-dshm" (OuterVolumeSpecName: "dshm") pod "5798ad6b-6a78-4c98-bb75-f093562f6b63" (UID: "5798ad6b-6a78-4c98-bb75-f093562f6b63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:14:35.151987 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.151941 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:14:35.162268 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.162248 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f"] Jul 17 18:14:35.169916 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.169898 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-0a98d9/auth-test-model-0a98d9-kserve-85b749d57f-gjb5f"] Jul 17 18:14:35.250803 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.250779 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-home\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:14:35.250803 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.250802 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5798ad6b-6a78-4c98-bb75-f093562f6b63-tls-certs\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:14:35.250803 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.250809 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-dshm\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:14:35.251099 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.250817 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5798ad6b-6a78-4c98-bb75-f093562f6b63-model-cache\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:14:35.251099 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.250824 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nzqk\" (UniqueName: \"kubernetes.io/projected/5798ad6b-6a78-4c98-bb75-f093562f6b63-kube-api-access-6nzqk\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:14:35.463862 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.463817 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3360f3f8-ce81-4330-806d-3c74570133c7" path="/var/lib/kubelet/pods/3360f3f8-ce81-4330-806d-3c74570133c7/volumes" Jul 17 18:14:35.464117 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.464101 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7"] Jul 17 18:14:35.465219 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:35.465199 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-37d0b7/auth-test-model-37d0b7-kserve-5b59b5b8ff-mx4m7"] Jul 17 18:14:37.463616 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:37.463582 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5798ad6b-6a78-4c98-bb75-f093562f6b63" path="/var/lib/kubelet/pods/5798ad6b-6a78-4c98-bb75-f093562f6b63/volumes" Jul 17 18:14:47.405944 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:47.405873 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:55.556658 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.556623 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4"] Jul 17 18:14:55.557116 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.557028 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3360f3f8-ce81-4330-806d-3c74570133c7" containerName="main" Jul 17 18:14:55.557116 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.557042 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3360f3f8-ce81-4330-806d-3c74570133c7" containerName="main" Jul 17 18:14:55.557116 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.557053 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5798ad6b-6a78-4c98-bb75-f093562f6b63" containerName="main" Jul 17 18:14:55.557116 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.557058 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5798ad6b-6a78-4c98-bb75-f093562f6b63" containerName="main" Jul 17 18:14:55.557116 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.557112 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5798ad6b-6a78-4c98-bb75-f093562f6b63" containerName="main" Jul 17 18:14:55.557279 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.557124 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3360f3f8-ce81-4330-806d-3c74570133c7" containerName="main" Jul 17 18:14:55.562109 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.562087 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:14:55.565999 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.565972 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-37d0b7\"/\"kube-root-ca.crt\"" Jul 17 18:14:55.565999 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.565985 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-37d0b7\"/\"sub-test-model-37d0b7-kserve-self-signed-certs\"" Jul 17 18:14:55.566206 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.565993 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-37d0b7\"/\"openshift-service-ca.crt\"" Jul 17 18:14:55.566206 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.566015 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-37d0b7\"/\"default-dockercfg-tbg82\"" Jul 17 18:14:55.571458 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.571436 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4"] Jul 17 18:14:55.585725 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.585688 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:55.620286 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.619877 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-dshm\") pod \"sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:14:55.620286 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.619936 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-tmp-dir\") pod \"sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:14:55.620286 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.619985 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9pwc\" (UniqueName: \"kubernetes.io/projected/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-kube-api-access-f9pwc\") pod \"sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:14:55.620286 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.620015 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-model-cache\") pod \"sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:14:55.620286 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.620058 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-tls-certs\") pod \"sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:14:55.620286 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.620126 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-home\") pod \"sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:14:55.721015 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.720978 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-dshm\") pod \"sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:14:55.721191 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.721025 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-tmp-dir\") pod \"sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:14:55.721191 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.721066 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9pwc\" (UniqueName: \"kubernetes.io/projected/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-kube-api-access-f9pwc\") pod \"sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:14:55.721191 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.721087 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-model-cache\") pod \"sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:14:55.721191 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.721122 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-tls-certs\") pod \"sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:14:55.721396 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.721192 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-home\") pod \"sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:14:55.721499 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.721471 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-tmp-dir\") pod \"sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:14:55.721569 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.721514 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-home\") pod \"sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:14:55.721569 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.721555 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-model-cache\") pod \"sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:14:55.723221 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.723201 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-dshm\") pod \"sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:14:55.723617 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.723598 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-tls-certs\") pod \"sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:14:55.730938 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.730913 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9pwc\" (UniqueName: \"kubernetes.io/projected/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-kube-api-access-f9pwc\") pod \"sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:14:55.871894 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:55.871754 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:14:56.000891 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:56.000862 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4"] Jul 17 18:14:56.002925 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:14:56.002896 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeceb1e87_7dc7_4bc7_91ce_9f3a93bbaeda.slice/crio-ca19036ebd5280628cdeb4bbad7c90b9c5414a4dc33b7eb2f765d16564f206d3 WatchSource:0}: Error finding container ca19036ebd5280628cdeb4bbad7c90b9c5414a4dc33b7eb2f765d16564f206d3: Status 404 returned error can't find the container with id ca19036ebd5280628cdeb4bbad7c90b9c5414a4dc33b7eb2f765d16564f206d3 Jul 17 18:14:56.238651 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:56.238617 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" event={"ID":"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda","Type":"ContainerStarted","Data":"b07eff975d6f4c2d07ee3df14d9136f29c48bcd962b79ad6bf19e487ac947c27"} Jul 17 18:14:56.238812 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:56.238666 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" event={"ID":"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda","Type":"ContainerStarted","Data":"ca19036ebd5280628cdeb4bbad7c90b9c5414a4dc33b7eb2f765d16564f206d3"} Jul 17 18:14:56.256300 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:14:56.256256 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" podStartSLOduration=1.256240316 podStartE2EDuration="1.256240316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:14:56.255372575 +0000 UTC m=+2617.374200858" watchObservedRunningTime="2026-07-17 18:14:56.256240316 +0000 UTC m=+2617.375068596" Jul 17 18:15:00.142837 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.142800 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0a98d9-29738535-hz757"] Jul 17 18:15:00.146641 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.146619 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3145cb-29738535-8pqsh"] Jul 17 18:15:00.146768 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.146740 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0a98d9-29738535-hz757" Jul 17 18:15:00.149478 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.149457 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dzkq8\"" Jul 17 18:15:00.149832 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.149816 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-24f8a1bf-29738535-zcpsl"] Jul 17 18:15:00.149936 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.149921 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3145cb-29738535-8pqsh" Jul 17 18:15:00.152810 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.152792 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c9e4ef-29738535-2d2fj"] Jul 17 18:15:00.152936 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.152919 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-24f8a1bf-29738535-zcpsl" Jul 17 18:15:00.160187 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.160166 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0a98d9-29738535-hz757" Jul 17 18:15:00.165680 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.165352 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4355f0b1-29738535-mnvgq"] Jul 17 18:15:00.165963 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.165923 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c9e4ef-29738535-2d2fj" Jul 17 18:15:00.171286 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.171262 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3145cb-29738535-8pqsh" Jul 17 18:15:00.172007 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.171983 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0a98d9-29738535-hz757"] Jul 17 18:15:00.172116 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.172041 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3145cb-29738535-8pqsh"] Jul 17 18:15:00.172184 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.172138 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4355f0b1-29738535-mnvgq" Jul 17 18:15:00.173805 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.173779 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c9e4ef-29738535-2d2fj"] Jul 17 18:15:00.179596 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.179571 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4355f0b1-29738535-mnvgq"] Jul 17 18:15:00.187141 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.186021 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7548b62e-29738535-vl76c"] Jul 17 18:15:00.197510 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.192591 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7548b62e-29738535-vl76c" Jul 17 18:15:00.197510 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.192777 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee229465-29738535-rhvxk"] Jul 17 18:15:00.201050 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.200365 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e63578-29738535-vvx6m"] Jul 17 18:15:00.201201 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.201148 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee229465-29738535-rhvxk" Jul 17 18:15:00.204949 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.204925 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738535-6pq9b"] Jul 17 18:15:00.205615 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.205596 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e63578-29738535-vvx6m" Jul 17 18:15:00.210918 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.210688 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-37d0b7-29738535-j8kdn"] Jul 17 18:15:00.211232 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.211216 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738535-6pq9b" Jul 17 18:15:00.214896 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.214780 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-24f8a1bf-29738535-zcpsl"] Jul 17 18:15:00.214896 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.214818 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee229465-29738535-rhvxk"] Jul 17 18:15:00.214896 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.214843 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7548b62e-29738535-vl76c"] Jul 17 18:15:00.215107 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.214929 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-37d0b7-29738535-j8kdn" Jul 17 18:15:00.217622 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.217394 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738535-6pq9b"] Jul 17 18:15:00.223946 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.223925 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72cf7b-29738535-l4c86"] Jul 17 18:15:00.226223 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.226195 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-24f8a1bf-29738535-zcpsl" Jul 17 18:15:00.229969 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.228960 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72cf7b-29738535-l4c86" Jul 17 18:15:00.229969 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.229866 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-37d0b7-29738535-j8kdn"] Jul 17 18:15:00.238017 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.236137 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d1f23e1-29738535-smxfk"] Jul 17 18:15:00.245625 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.245460 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-84841c77-29738535-qcq6w"] Jul 17 18:15:00.248767 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.248341 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d1f23e1-29738535-smxfk" Jul 17 18:15:00.250646 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.250266 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e63578-29738535-vvx6m"] Jul 17 18:15:00.250646 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.250355 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-84841c77-29738535-qcq6w" Jul 17 18:15:00.261686 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.261625 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72cf7b-29738535-l4c86"] Jul 17 18:15:00.265930 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.265904 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d1f23e1-29738535-smxfk"] Jul 17 18:15:00.269030 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.268207 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-84841c77-29738535-qcq6w"] Jul 17 18:15:00.274077 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.273818 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c9e4ef-29738535-2d2fj" Jul 17 18:15:00.293691 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.293466 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4355f0b1-29738535-mnvgq" Jul 17 18:15:00.307899 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.307702 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7548b62e-29738535-vl76c" Jul 17 18:15:00.325603 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.324587 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee229465-29738535-rhvxk" Jul 17 18:15:00.348658 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.347961 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e63578-29738535-vvx6m" Jul 17 18:15:00.362089 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.358462 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738535-6pq9b" Jul 17 18:15:00.376000 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.374648 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-37d0b7-29738535-j8kdn" Jul 17 18:15:00.405767 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.405092 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72cf7b-29738535-l4c86" Jul 17 18:15:00.448121 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.447938 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d1f23e1-29738535-smxfk" Jul 17 18:15:00.470413 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.470224 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-84841c77-29738535-qcq6w" Jul 17 18:15:00.617130 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.617047 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4355f0b1-29738535-mnvgq"] Jul 17 18:15:00.636827 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.632206 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:15:00.655968 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.655263 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0a98d9-29738535-hz757"] Jul 17 18:15:00.668569 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:15:00.665117 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776bc04a_aaf6_4276_9008_880027518b41.slice/crio-ee69496ac09a14cb1df0c9d4c111e85f235584b29008172980ca00cd4ca3d9cb WatchSource:0}: Error finding container ee69496ac09a14cb1df0c9d4c111e85f235584b29008172980ca00cd4ca3d9cb: Status 404 returned error can't find the container with id ee69496ac09a14cb1df0c9d4c111e85f235584b29008172980ca00cd4ca3d9cb Jul 17 18:15:00.668569 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.668484 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c9e4ef-29738535-2d2fj"] Jul 17 18:15:00.682802 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:15:00.682534 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f539ee_1d44_46e2_b2cc_20badad59414.slice/crio-3b2c029c0ba5fbb18bdc6f15f62f0f6a4061c51fd532ba25b3eb417b031183a9 WatchSource:0}: Error finding container 3b2c029c0ba5fbb18bdc6f15f62f0f6a4061c51fd532ba25b3eb417b031183a9: Status 404 returned error can't find the container with id 3b2c029c0ba5fbb18bdc6f15f62f0f6a4061c51fd532ba25b3eb417b031183a9 Jul 17 18:15:00.684382 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.683160 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7548b62e-29738535-vl76c"] Jul 17 18:15:00.694195 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:15:00.694134 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6617328d_6a6e_4b1d_a4c0_302bcbd2d106.slice/crio-c4e9c2f4c3256b9bd5fe37854d9e221ea288dddecd956813beb67753a1a80e8a WatchSource:0}: Error finding container c4e9c2f4c3256b9bd5fe37854d9e221ea288dddecd956813beb67753a1a80e8a: Status 404 returned error can't find the container with id c4e9c2f4c3256b9bd5fe37854d9e221ea288dddecd956813beb67753a1a80e8a Jul 17 18:15:00.696132 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.696060 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3145cb-29738535-8pqsh"] Jul 17 18:15:00.706809 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.706788 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-24f8a1bf-29738535-zcpsl"] Jul 17 18:15:00.710784 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:15:00.710759 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce6784f_2bd7_4306_8bad_fab1e6ed32da.slice/crio-f1e2b64bb2913b78b172447c410dea29b6b60b288a5f4103bad075b35574869d WatchSource:0}: Error finding container f1e2b64bb2913b78b172447c410dea29b6b60b288a5f4103bad075b35574869d: Status 404 returned error can't find the container with id f1e2b64bb2913b78b172447c410dea29b6b60b288a5f4103bad075b35574869d Jul 17 18:15:00.840546 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.840520 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:15:00.969079 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.969044 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee229465-29738535-rhvxk"] Jul 17 18:15:00.971388 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:15:00.971359 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29dabe77_16c2_4ead_8b5d_74c6e65b7a48.slice/crio-0a9cbdcb7265332d91fe3acc1f68bd809a3cd872e96af6e29b77bcea01b89efe WatchSource:0}: Error finding container 0a9cbdcb7265332d91fe3acc1f68bd809a3cd872e96af6e29b77bcea01b89efe: Status 404 returned error can't find the container with id 0a9cbdcb7265332d91fe3acc1f68bd809a3cd872e96af6e29b77bcea01b89efe Jul 17 18:15:00.978198 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.973916 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-37d0b7-29738535-j8kdn"] Jul 17 18:15:00.978198 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:00.977120 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e63578-29738535-vvx6m"] Jul 17 18:15:01.007979 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:01.007958 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-84841c77-29738535-qcq6w"] Jul 17 18:15:01.010227 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:15:01.010201 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf17809e6_2524_480d_88ec_dbfa620d7fce.slice/crio-2638aa324f8e4ba70e0a72fe1929fd2067f4da4f1fc6aafbef6eb99b0ac93729 WatchSource:0}: Error finding container 2638aa324f8e4ba70e0a72fe1929fd2067f4da4f1fc6aafbef6eb99b0ac93729: Status 404 returned error can't find the container with id 2638aa324f8e4ba70e0a72fe1929fd2067f4da4f1fc6aafbef6eb99b0ac93729 Jul 17 18:15:01.011611 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:01.011590 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738535-6pq9b"] Jul 17 18:15:01.014449 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:15:01.014427 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca604483_cc82_4b48_83c8_d346adc619ef.slice/crio-8c13c9b2cf5700d23be83e6d9b1a237db51c0f8aa2052273f6b50081b6c4df27 WatchSource:0}: Error finding container 8c13c9b2cf5700d23be83e6d9b1a237db51c0f8aa2052273f6b50081b6c4df27: Status 404 returned error can't find the container with id 8c13c9b2cf5700d23be83e6d9b1a237db51c0f8aa2052273f6b50081b6c4df27 Jul 17 18:15:01.040517 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:01.040346 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d1f23e1-29738535-smxfk"] Jul 17 18:15:01.042134 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:01.042114 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72cf7b-29738535-l4c86"] Jul 17 18:15:01.042591 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:15:01.042569 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df39780_6eeb_4d61_aa2d_57479841ad48.slice/crio-bf8e8253157d5b846c83afd63a865415467352d991a9e6c2b0c235313afa52ec WatchSource:0}: Error finding container bf8e8253157d5b846c83afd63a865415467352d991a9e6c2b0c235313afa52ec: Status 404 returned error can't find the container with id bf8e8253157d5b846c83afd63a865415467352d991a9e6c2b0c235313afa52ec Jul 17 18:15:01.043356 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:15:01.043334 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8efd1900_9837_4052_a280_e028c740cf21.slice/crio-3c03aec7b7a8f20ccbac935a0e5b72323da736612eac1f3a74aea2561e5ca8d9 WatchSource:0}: Error finding container 3c03aec7b7a8f20ccbac935a0e5b72323da736612eac1f3a74aea2561e5ca8d9: Status 404 returned error can't find the container with id 3c03aec7b7a8f20ccbac935a0e5b72323da736612eac1f3a74aea2561e5ca8d9 Jul 17 18:15:01.045159 ip-10-0-136-43 kubenswrapper[2553]: E0717 18:15:01.045138 2553 kuberuntime_manager.go:1459] "Unhandled Error" err="container cleanup start failed in pod maas-api-key-cleanup-e2e-inf-a-72cf7b-29738535-l4c86_odh-ai-gateway-infra(8efd1900-9837-4052-a280-e028c740cf21): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 17 18:15:01.045211 ip-10-0-136-43 kubenswrapper[2553]: E0717 18:15:01.045174 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with ErrImagePull: \"pull QPS exceeded\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72cf7b-29738535-l4c86" podUID="8efd1900-9837-4052-a280-e028c740cf21" Jul 17 18:15:01.272154 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:01.272117 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738535-6pq9b" event={"ID":"ca604483-cc82-4b48-83c8-d346adc619ef","Type":"ContainerStarted","Data":"8c13c9b2cf5700d23be83e6d9b1a237db51c0f8aa2052273f6b50081b6c4df27"} Jul 17 18:15:01.273442 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:01.273417 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e63578-29738535-vvx6m" event={"ID":"5148216b-3305-4cc4-8758-8fe645060236","Type":"ContainerStarted","Data":"555b293a68130f125a2f13fb6c1256850dc310500c40d401b125119b02294f35"} Jul 17 18:15:01.274732 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:01.274704 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3145cb-29738535-8pqsh" event={"ID":"207d5598-707a-499a-bf0f-e9a1258a6106","Type":"ContainerStarted","Data":"05e415d9789bbc719ccd7287e6a6630ad5cb9d69d5690b41a31e1a09bd067482"} Jul 17 18:15:01.275963 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:01.275939 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d1f23e1-29738535-smxfk" event={"ID":"6df39780-6eeb-4d61-aa2d-57479841ad48","Type":"ContainerStarted","Data":"bf8e8253157d5b846c83afd63a865415467352d991a9e6c2b0c235313afa52ec"} Jul 17 18:15:01.277246 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:01.277222 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72cf7b-29738535-l4c86" event={"ID":"8efd1900-9837-4052-a280-e028c740cf21","Type":"ContainerStarted","Data":"3c03aec7b7a8f20ccbac935a0e5b72323da736612eac1f3a74aea2561e5ca8d9"} Jul 17 18:15:01.278343 ip-10-0-136-43 kubenswrapper[2553]: E0717 18:15:01.278315 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/ubi9/ubi-minimal:9.7\\\": ErrImagePull: pull QPS exceeded\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72cf7b-29738535-l4c86" podUID="8efd1900-9837-4052-a280-e028c740cf21" Jul 17 18:15:01.278596 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:01.278573 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-37d0b7-29738535-j8kdn" event={"ID":"2014f2e6-cd15-44c2-b891-fe4be8b13f50","Type":"ContainerStarted","Data":"4caea0ea34a8f23429a663569c0d91a78fdfc9f3ec7bb7dc6b56dcbb41e2802c"} Jul 17 18:15:01.279786 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:01.279764 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee229465-29738535-rhvxk" event={"ID":"29dabe77-16c2-4ead-8b5d-74c6e65b7a48","Type":"ContainerStarted","Data":"0a9cbdcb7265332d91fe3acc1f68bd809a3cd872e96af6e29b77bcea01b89efe"} Jul 17 18:15:01.280826 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:01.280808 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-84841c77-29738535-qcq6w" event={"ID":"f17809e6-2524-480d-88ec-dbfa620d7fce","Type":"ContainerStarted","Data":"2638aa324f8e4ba70e0a72fe1929fd2067f4da4f1fc6aafbef6eb99b0ac93729"} Jul 17 18:15:01.281890 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:01.281871 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c9e4ef-29738535-2d2fj" event={"ID":"c1f539ee-1d44-46e2-b2cc-20badad59414","Type":"ContainerStarted","Data":"3b2c029c0ba5fbb18bdc6f15f62f0f6a4061c51fd532ba25b3eb417b031183a9"} Jul 17 18:15:01.282973 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:01.282952 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7548b62e-29738535-vl76c" event={"ID":"6617328d-6a6e-4b1d-a4c0-302bcbd2d106","Type":"ContainerStarted","Data":"c4e9c2f4c3256b9bd5fe37854d9e221ea288dddecd956813beb67753a1a80e8a"} Jul 17 18:15:01.284045 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:01.284027 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4355f0b1-29738535-mnvgq" event={"ID":"a7078029-1712-4365-af96-6f3da763995e","Type":"ContainerStarted","Data":"02e08bff378917ad8d07679f1a5f9128fc0abf7614b00e00e45398c960917e9a"} Jul 17 18:15:01.285181 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:01.285162 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-24f8a1bf-29738535-zcpsl" event={"ID":"bce6784f-2bd7-4306-8bad-fab1e6ed32da","Type":"ContainerStarted","Data":"f1e2b64bb2913b78b172447c410dea29b6b60b288a5f4103bad075b35574869d"} Jul 17 18:15:01.286343 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:01.286326 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0a98d9-29738535-hz757" event={"ID":"776bc04a-aaf6-4276-9008-880027518b41","Type":"ContainerStarted","Data":"ee69496ac09a14cb1df0c9d4c111e85f235584b29008172980ca00cd4ca3d9cb"} Jul 17 18:15:02.331354 ip-10-0-136-43 kubenswrapper[2553]: E0717 18:15:02.331318 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/ubi9/ubi-minimal:9.7\\\": ErrImagePull: pull QPS exceeded\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72cf7b-29738535-l4c86" podUID="8efd1900-9837-4052-a280-e028c740cf21" Jul 17 18:15:05.360799 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:05.360768 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3145cb-29738535-8pqsh" event={"ID":"207d5598-707a-499a-bf0f-e9a1258a6106","Type":"ContainerStarted","Data":"cd36fc50d7a4ff7d6a364e9d6d9a3d1ba99bebc9880317e88132782822381d4c"} Jul 17 18:15:05.362217 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:05.362195 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-37d0b7-29738535-j8kdn" event={"ID":"2014f2e6-cd15-44c2-b891-fe4be8b13f50","Type":"ContainerStarted","Data":"2011350f7f70dd42baa15422567d4c23b981884b7ccac12292cae272e87adb71"} Jul 17 18:15:05.363514 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:05.363494 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4355f0b1-29738535-mnvgq" event={"ID":"a7078029-1712-4365-af96-6f3da763995e","Type":"ContainerStarted","Data":"e10bcac4ff2c39d28769088bfe4dfc69777fc3c64b505067f630949bcb6477fe"} Jul 17 18:15:05.364928 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:05.364908 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-24f8a1bf-29738535-zcpsl" event={"ID":"bce6784f-2bd7-4306-8bad-fab1e6ed32da","Type":"ContainerStarted","Data":"3acbf9a077a4a93ac3838dc7ae61e626eae796fc5674b505e4b52c91923c4c46"} Jul 17 18:15:05.377474 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:05.377433 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3145cb-29738535-8pqsh" podStartSLOduration=1.091831382 podStartE2EDuration="5.377418947s" totalImagesPullingTime="4.285587565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:15:05.376955123 +0000 UTC m=+2626.495783404" watchObservedRunningTime="2026-07-17 18:15:05.377418947 +0000 UTC m=+2626.496247228" Jul 17 18:15:05.394573 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:05.394535 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-24f8a1bf-29738535-zcpsl" podStartSLOduration=1.219026393 podStartE2EDuration="5.394522507s" totalImagesPullingTime="4.175496114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:15:05.393613439 +0000 UTC m=+2626.512441721" watchObservedRunningTime="2026-07-17 18:15:05.394522507 +0000 UTC m=+2626.513350783" Jul 17 18:15:05.409953 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:05.409906 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-37d0b7-29738535-j8kdn" podStartSLOduration=1.461229376 podStartE2EDuration="5.409894476s" totalImagesPullingTime="3.9486651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:15:05.408269553 +0000 UTC m=+2626.527097833" watchObservedRunningTime="2026-07-17 18:15:05.409894476 +0000 UTC m=+2626.528722772" Jul 17 18:15:05.425967 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:05.425930 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4355f0b1-29738535-mnvgq" podStartSLOduration=1.16435446 podStartE2EDuration="5.425922161s" totalImagesPullingTime="4.261567701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:15:05.4249483 +0000 UTC m=+2626.543776583" watchObservedRunningTime="2026-07-17 18:15:05.425922161 +0000 UTC m=+2626.544750441" Jul 17 18:15:05.873719 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:05.871936 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:15:05.873719 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:05.872694 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:15:05.888543 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:05.888524 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:15:06.375564 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:06.375529 2553 generic.go:356] "Generic (PLEG): container finished" podID="2014f2e6-cd15-44c2-b891-fe4be8b13f50" containerID="2011350f7f70dd42baa15422567d4c23b981884b7ccac12292cae272e87adb71" exitCode=0 Jul 17 18:15:06.375935 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:06.375580 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-37d0b7-29738535-j8kdn" event={"ID":"2014f2e6-cd15-44c2-b891-fe4be8b13f50","Type":"ContainerDied","Data":"2011350f7f70dd42baa15422567d4c23b981884b7ccac12292cae272e87adb71"} Jul 17 18:15:06.377550 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:06.377390 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee229465-29738535-rhvxk" event={"ID":"29dabe77-16c2-4ead-8b5d-74c6e65b7a48","Type":"ContainerStarted","Data":"0b02d8880e6ed765f6e8667ab2d57fc3bbc9f2fa0af9c8e26e0fe9cfdb389331"} Jul 17 18:15:06.379306 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:06.379282 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-84841c77-29738535-qcq6w" event={"ID":"f17809e6-2524-480d-88ec-dbfa620d7fce","Type":"ContainerStarted","Data":"6ba2808229d8f5caad7b0b29a1eb48f6be6c3fb5112c7a4eef0ddf66edb4f4c0"} Jul 17 18:15:06.381112 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:06.381087 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c9e4ef-29738535-2d2fj" event={"ID":"c1f539ee-1d44-46e2-b2cc-20badad59414","Type":"ContainerStarted","Data":"43de2fb17672ad8b3cad346d89166fd273271ecb55a9c51e7ed6cc647a1a62fc"} Jul 17 18:15:06.382408 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:06.382388 2553 generic.go:356] "Generic (PLEG): container finished" podID="a7078029-1712-4365-af96-6f3da763995e" containerID="e10bcac4ff2c39d28769088bfe4dfc69777fc3c64b505067f630949bcb6477fe" exitCode=0 Jul 17 18:15:06.471061 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:06.382448 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4355f0b1-29738535-mnvgq" event={"ID":"a7078029-1712-4365-af96-6f3da763995e","Type":"ContainerDied","Data":"e10bcac4ff2c39d28769088bfe4dfc69777fc3c64b505067f630949bcb6477fe"} Jul 17 18:15:06.471061 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:06.383627 2553 generic.go:356] "Generic (PLEG): container finished" podID="bce6784f-2bd7-4306-8bad-fab1e6ed32da" containerID="3acbf9a077a4a93ac3838dc7ae61e626eae796fc5674b505e4b52c91923c4c46" exitCode=0 Jul 17 18:15:06.471061 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:06.383705 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-24f8a1bf-29738535-zcpsl" event={"ID":"bce6784f-2bd7-4306-8bad-fab1e6ed32da","Type":"ContainerDied","Data":"3acbf9a077a4a93ac3838dc7ae61e626eae796fc5674b505e4b52c91923c4c46"} Jul 17 18:15:06.471061 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:06.385139 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738535-6pq9b" event={"ID":"ca604483-cc82-4b48-83c8-d346adc619ef","Type":"ContainerStarted","Data":"33197452c7e4aac57ea04d2f37e95d50c2a1877a8ddbec5a82d7dd00eb8ecef6"} Jul 17 18:15:06.471061 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:06.386491 2553 generic.go:356] "Generic (PLEG): container finished" podID="207d5598-707a-499a-bf0f-e9a1258a6106" containerID="cd36fc50d7a4ff7d6a364e9d6d9a3d1ba99bebc9880317e88132782822381d4c" exitCode=0 Jul 17 18:15:06.471061 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:06.386558 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3145cb-29738535-8pqsh" event={"ID":"207d5598-707a-499a-bf0f-e9a1258a6106","Type":"ContainerDied","Data":"cd36fc50d7a4ff7d6a364e9d6d9a3d1ba99bebc9880317e88132782822381d4c"} Jul 17 18:15:06.471061 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:06.387814 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d1f23e1-29738535-smxfk" event={"ID":"6df39780-6eeb-4d61-aa2d-57479841ad48","Type":"ContainerStarted","Data":"6014a4b784010626752194506dcf926d32964d8abc43035befb6a5f7eaa65ac5"} Jul 17 18:15:06.471061 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:06.399768 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:15:06.471061 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:06.426922 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-84841c77-29738535-qcq6w" podStartSLOduration=1.950496995 podStartE2EDuration="6.426909031s" totalImagesPullingTime="4.476412036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:15:06.426023467 +0000 UTC m=+2627.544851748" watchObservedRunningTime="2026-07-17 18:15:06.426909031 +0000 UTC m=+2627.545737350" Jul 17 18:15:06.485160 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:06.479951 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee229465-29738535-rhvxk" podStartSLOduration=2.380020265 podStartE2EDuration="6.479931701s" totalImagesPullingTime="4.099911436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:15:06.476611966 +0000 UTC m=+2627.595440249" watchObservedRunningTime="2026-07-17 18:15:06.479931701 +0000 UTC m=+2627.598759984" Jul 17 18:15:06.497911 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:06.497782 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d1f23e1-29738535-smxfk" podStartSLOduration=2.257025592 podStartE2EDuration="6.49776425s" totalImagesPullingTime="4.240738658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:15:06.49632902 +0000 UTC m=+2627.615157299" watchObservedRunningTime="2026-07-17 18:15:06.49776425 +0000 UTC m=+2627.616592532" Jul 17 18:15:06.513795 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:06.513754 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c9e4ef-29738535-2d2fj" podStartSLOduration=1.327207893 podStartE2EDuration="6.513742979s" totalImagesPullingTime="5.186535086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:15:06.513346214 +0000 UTC m=+2627.632174496" watchObservedRunningTime="2026-07-17 18:15:06.513742979 +0000 UTC m=+2627.632571260" Jul 17 18:15:06.531690 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:06.531656 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738535-6pq9b" podStartSLOduration=1.858843263 podStartE2EDuration="6.531645842s" totalImagesPullingTime="4.672802579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:15:06.530863243 +0000 UTC m=+2627.649691521" watchObservedRunningTime="2026-07-17 18:15:06.531645842 +0000 UTC m=+2627.650474123" Jul 17 18:15:07.394801 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:07.394774 2553 generic.go:356] "Generic (PLEG): container finished" podID="ca604483-cc82-4b48-83c8-d346adc619ef" containerID="33197452c7e4aac57ea04d2f37e95d50c2a1877a8ddbec5a82d7dd00eb8ecef6" exitCode=0 Jul 17 18:15:07.395163 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:07.394874 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738535-6pq9b" event={"ID":"ca604483-cc82-4b48-83c8-d346adc619ef","Type":"ContainerDied","Data":"33197452c7e4aac57ea04d2f37e95d50c2a1877a8ddbec5a82d7dd00eb8ecef6"} Jul 17 18:15:07.396359 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:07.396338 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e63578-29738535-vvx6m" event={"ID":"5148216b-3305-4cc4-8758-8fe645060236","Type":"ContainerStarted","Data":"1401d359590817731c9284245924141933508405e143ca23110fabb898c1fb07"} Jul 17 18:15:07.397748 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:07.397725 2553 generic.go:356] "Generic (PLEG): container finished" podID="6df39780-6eeb-4d61-aa2d-57479841ad48" containerID="6014a4b784010626752194506dcf926d32964d8abc43035befb6a5f7eaa65ac5" exitCode=0 Jul 17 18:15:07.397838 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:07.397748 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d1f23e1-29738535-smxfk" event={"ID":"6df39780-6eeb-4d61-aa2d-57479841ad48","Type":"ContainerDied","Data":"6014a4b784010626752194506dcf926d32964d8abc43035befb6a5f7eaa65ac5"} Jul 17 18:15:07.399228 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:07.399209 2553 generic.go:356] "Generic (PLEG): container finished" podID="29dabe77-16c2-4ead-8b5d-74c6e65b7a48" containerID="0b02d8880e6ed765f6e8667ab2d57fc3bbc9f2fa0af9c8e26e0fe9cfdb389331" exitCode=0 Jul 17 18:15:07.399299 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:07.399272 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee229465-29738535-rhvxk" event={"ID":"29dabe77-16c2-4ead-8b5d-74c6e65b7a48","Type":"ContainerDied","Data":"0b02d8880e6ed765f6e8667ab2d57fc3bbc9f2fa0af9c8e26e0fe9cfdb389331"} Jul 17 18:15:07.400560 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:07.400539 2553 generic.go:356] "Generic (PLEG): container finished" podID="f17809e6-2524-480d-88ec-dbfa620d7fce" containerID="6ba2808229d8f5caad7b0b29a1eb48f6be6c3fb5112c7a4eef0ddf66edb4f4c0" exitCode=0 Jul 17 18:15:07.400640 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:07.400607 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-84841c77-29738535-qcq6w" event={"ID":"f17809e6-2524-480d-88ec-dbfa620d7fce","Type":"ContainerDied","Data":"6ba2808229d8f5caad7b0b29a1eb48f6be6c3fb5112c7a4eef0ddf66edb4f4c0"} Jul 17 18:15:07.401941 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:07.401924 2553 generic.go:356] "Generic (PLEG): container finished" podID="c1f539ee-1d44-46e2-b2cc-20badad59414" containerID="43de2fb17672ad8b3cad346d89166fd273271ecb55a9c51e7ed6cc647a1a62fc" exitCode=0 Jul 17 18:15:07.402015 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:07.401997 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c9e4ef-29738535-2d2fj" event={"ID":"c1f539ee-1d44-46e2-b2cc-20badad59414","Type":"ContainerDied","Data":"43de2fb17672ad8b3cad346d89166fd273271ecb55a9c51e7ed6cc647a1a62fc"} Jul 17 18:15:07.403305 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:07.403285 2553 generic.go:356] "Generic (PLEG): container finished" podID="6617328d-6a6e-4b1d-a4c0-302bcbd2d106" containerID="13b9521306a1af2c91e1438874002d28c5035855dca0ea9cf69802d6812e6272" exitCode=0 Jul 17 18:15:07.403392 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:07.403308 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7548b62e-29738535-vl76c" event={"ID":"6617328d-6a6e-4b1d-a4c0-302bcbd2d106","Type":"ContainerDied","Data":"13b9521306a1af2c91e1438874002d28c5035855dca0ea9cf69802d6812e6272"} Jul 17 18:15:07.404657 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:07.404617 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0a98d9-29738535-hz757" event={"ID":"776bc04a-aaf6-4276-9008-880027518b41","Type":"ContainerStarted","Data":"d0a235f87b3777cf280a7696a557a7179831f1818f02085e5f30425bbd67276f"} Jul 17 18:15:07.446654 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:07.446557 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0a98d9-29738535-hz757" podStartSLOduration=1.6235431550000001 podStartE2EDuration="7.446534616s" totalImagesPullingTime="5.822991461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:15:07.444560075 +0000 UTC m=+2628.563388369" watchObservedRunningTime="2026-07-17 18:15:07.446534616 +0000 UTC m=+2628.565362897" Jul 17 18:15:07.461509 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:07.461309 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e63578-29738535-vvx6m" podStartSLOduration=2.160150737 podStartE2EDuration="7.46128938s" totalImagesPullingTime="5.301138643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:15:07.460506389 +0000 UTC m=+2628.579334672" watchObservedRunningTime="2026-07-17 18:15:07.46128938 +0000 UTC m=+2628.580117665" Jul 17 18:15:07.565283 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:07.565254 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-24f8a1bf-29738535-zcpsl" Jul 17 18:15:07.635316 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:07.635251 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3145cb-29738535-8pqsh" Jul 17 18:15:07.638346 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:07.638328 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4355f0b1-29738535-mnvgq" Jul 17 18:15:07.641660 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:07.641641 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-37d0b7-29738535-j8kdn" Jul 17 18:15:08.411764 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.411546 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4355f0b1-29738535-mnvgq" event={"ID":"a7078029-1712-4365-af96-6f3da763995e","Type":"ContainerDied","Data":"02e08bff378917ad8d07679f1a5f9128fc0abf7614b00e00e45398c960917e9a"} Jul 17 18:15:08.411764 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.411601 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4355f0b1-29738535-mnvgq" Jul 17 18:15:08.411764 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.411615 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02e08bff378917ad8d07679f1a5f9128fc0abf7614b00e00e45398c960917e9a" Jul 17 18:15:08.413390 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.413309 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-24f8a1bf-29738535-zcpsl" event={"ID":"bce6784f-2bd7-4306-8bad-fab1e6ed32da","Type":"ContainerDied","Data":"f1e2b64bb2913b78b172447c410dea29b6b60b288a5f4103bad075b35574869d"} Jul 17 18:15:08.413390 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.413337 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-24f8a1bf-29738535-zcpsl" Jul 17 18:15:08.413390 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.413364 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1e2b64bb2913b78b172447c410dea29b6b60b288a5f4103bad075b35574869d" Jul 17 18:15:08.415091 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.415065 2553 generic.go:356] "Generic (PLEG): container finished" podID="776bc04a-aaf6-4276-9008-880027518b41" containerID="d0a235f87b3777cf280a7696a557a7179831f1818f02085e5f30425bbd67276f" exitCode=0 Jul 17 18:15:08.415091 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.415096 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0a98d9-29738535-hz757" event={"ID":"776bc04a-aaf6-4276-9008-880027518b41","Type":"ContainerDied","Data":"d0a235f87b3777cf280a7696a557a7179831f1818f02085e5f30425bbd67276f"} Jul 17 18:15:08.416749 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.416724 2553 generic.go:356] "Generic (PLEG): container finished" podID="5148216b-3305-4cc4-8758-8fe645060236" containerID="1401d359590817731c9284245924141933508405e143ca23110fabb898c1fb07" exitCode=0 Jul 17 18:15:08.416885 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.416816 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e63578-29738535-vvx6m" event={"ID":"5148216b-3305-4cc4-8758-8fe645060236","Type":"ContainerDied","Data":"1401d359590817731c9284245924141933508405e143ca23110fabb898c1fb07"} Jul 17 18:15:08.418094 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.418078 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3145cb-29738535-8pqsh" Jul 17 18:15:08.418195 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.418108 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3145cb-29738535-8pqsh" event={"ID":"207d5598-707a-499a-bf0f-e9a1258a6106","Type":"ContainerDied","Data":"05e415d9789bbc719ccd7287e6a6630ad5cb9d69d5690b41a31e1a09bd067482"} Jul 17 18:15:08.418195 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.418147 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05e415d9789bbc719ccd7287e6a6630ad5cb9d69d5690b41a31e1a09bd067482" Jul 17 18:15:08.419516 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.419492 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-37d0b7-29738535-j8kdn" Jul 17 18:15:08.419643 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.419543 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-37d0b7-29738535-j8kdn" event={"ID":"2014f2e6-cd15-44c2-b891-fe4be8b13f50","Type":"ContainerDied","Data":"4caea0ea34a8f23429a663569c0d91a78fdfc9f3ec7bb7dc6b56dcbb41e2802c"} Jul 17 18:15:08.419643 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.419599 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4caea0ea34a8f23429a663569c0d91a78fdfc9f3ec7bb7dc6b56dcbb41e2802c" Jul 17 18:15:08.597318 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.597297 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7548b62e-29738535-vl76c" Jul 17 18:15:08.633400 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.633371 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d1f23e1-29738535-smxfk" Jul 17 18:15:08.637573 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.637544 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee229465-29738535-rhvxk" Jul 17 18:15:08.705187 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.705162 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c9e4ef-29738535-2d2fj" Jul 17 18:15:08.711228 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.711209 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738535-6pq9b" Jul 17 18:15:08.714457 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:08.714438 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-84841c77-29738535-qcq6w" Jul 17 18:15:09.005294 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.002482 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:15:09.099030 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.099001 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:15:09.426989 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.426915 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee229465-29738535-rhvxk" Jul 17 18:15:09.426989 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.426915 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee229465-29738535-rhvxk" event={"ID":"29dabe77-16c2-4ead-8b5d-74c6e65b7a48","Type":"ContainerDied","Data":"0a9cbdcb7265332d91fe3acc1f68bd809a3cd872e96af6e29b77bcea01b89efe"} Jul 17 18:15:09.427430 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.427024 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a9cbdcb7265332d91fe3acc1f68bd809a3cd872e96af6e29b77bcea01b89efe" Jul 17 18:15:09.428206 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.428192 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-84841c77-29738535-qcq6w" Jul 17 18:15:09.428310 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.428192 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-84841c77-29738535-qcq6w" event={"ID":"f17809e6-2524-480d-88ec-dbfa620d7fce","Type":"ContainerDied","Data":"2638aa324f8e4ba70e0a72fe1929fd2067f4da4f1fc6aafbef6eb99b0ac93729"} Jul 17 18:15:09.428365 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.428309 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2638aa324f8e4ba70e0a72fe1929fd2067f4da4f1fc6aafbef6eb99b0ac93729" Jul 17 18:15:09.429480 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.429463 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c9e4ef-29738535-2d2fj" Jul 17 18:15:09.429574 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.429480 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c9e4ef-29738535-2d2fj" event={"ID":"c1f539ee-1d44-46e2-b2cc-20badad59414","Type":"ContainerDied","Data":"3b2c029c0ba5fbb18bdc6f15f62f0f6a4061c51fd532ba25b3eb417b031183a9"} Jul 17 18:15:09.429574 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.429519 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b2c029c0ba5fbb18bdc6f15f62f0f6a4061c51fd532ba25b3eb417b031183a9" Jul 17 18:15:09.430913 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.430885 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7548b62e-29738535-vl76c" event={"ID":"6617328d-6a6e-4b1d-a4c0-302bcbd2d106","Type":"ContainerDied","Data":"c4e9c2f4c3256b9bd5fe37854d9e221ea288dddecd956813beb67753a1a80e8a"} Jul 17 18:15:09.431004 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.430932 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4e9c2f4c3256b9bd5fe37854d9e221ea288dddecd956813beb67753a1a80e8a" Jul 17 18:15:09.431004 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.430896 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7548b62e-29738535-vl76c" Jul 17 18:15:09.432303 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.432282 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738535-6pq9b" Jul 17 18:15:09.432406 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.432317 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738535-6pq9b" event={"ID":"ca604483-cc82-4b48-83c8-d346adc619ef","Type":"ContainerDied","Data":"8c13c9b2cf5700d23be83e6d9b1a237db51c0f8aa2052273f6b50081b6c4df27"} Jul 17 18:15:09.432406 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.432356 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c13c9b2cf5700d23be83e6d9b1a237db51c0f8aa2052273f6b50081b6c4df27" Jul 17 18:15:09.433799 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.433770 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d1f23e1-29738535-smxfk" Jul 17 18:15:09.433904 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.433803 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d1f23e1-29738535-smxfk" event={"ID":"6df39780-6eeb-4d61-aa2d-57479841ad48","Type":"ContainerDied","Data":"bf8e8253157d5b846c83afd63a865415467352d991a9e6c2b0c235313afa52ec"} Jul 17 18:15:09.433904 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.433883 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf8e8253157d5b846c83afd63a865415467352d991a9e6c2b0c235313afa52ec" Jul 17 18:15:09.597863 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.597824 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e63578-29738535-vvx6m" Jul 17 18:15:09.600646 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:09.600629 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0a98d9-29738535-hz757" Jul 17 18:15:10.440758 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:10.440726 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0a98d9-29738535-hz757" Jul 17 18:15:10.441199 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:10.440756 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0a98d9-29738535-hz757" event={"ID":"776bc04a-aaf6-4276-9008-880027518b41","Type":"ContainerDied","Data":"ee69496ac09a14cb1df0c9d4c111e85f235584b29008172980ca00cd4ca3d9cb"} Jul 17 18:15:10.441199 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:10.440805 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee69496ac09a14cb1df0c9d4c111e85f235584b29008172980ca00cd4ca3d9cb" Jul 17 18:15:10.442083 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:10.442060 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e63578-29738535-vvx6m" event={"ID":"5148216b-3305-4cc4-8758-8fe645060236","Type":"ContainerDied","Data":"555b293a68130f125a2f13fb6c1256850dc310500c40d401b125119b02294f35"} Jul 17 18:15:10.442083 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:10.442082 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e63578-29738535-vvx6m" Jul 17 18:15:10.442083 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:10.442097 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="555b293a68130f125a2f13fb6c1256850dc310500c40d401b125119b02294f35" Jul 17 18:15:15.470549 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:15.470517 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72cf7b-29738535-l4c86" event={"ID":"8efd1900-9837-4052-a280-e028c740cf21","Type":"ContainerStarted","Data":"b1190c7c1658db3b89a0b7fe5e5aaebc55ace1ca292dfe3a1d8f97b8a13b6fe7"} Jul 17 18:15:15.492172 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:15.492118 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72cf7b-29738535-l4c86" podStartSLOduration=15.492098962 podStartE2EDuration="15.492098962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-17 18:15:15.491190272 +0000 UTC m=+2636.610018553" watchObservedRunningTime="2026-07-17 18:15:15.492098962 +0000 UTC m=+2636.610927244" Jul 17 18:15:16.478986 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:16.478954 2553 generic.go:356] "Generic (PLEG): container finished" podID="8efd1900-9837-4052-a280-e028c740cf21" containerID="b1190c7c1658db3b89a0b7fe5e5aaebc55ace1ca292dfe3a1d8f97b8a13b6fe7" exitCode=0 Jul 17 18:15:16.479411 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:16.479028 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72cf7b-29738535-l4c86" event={"ID":"8efd1900-9837-4052-a280-e028c740cf21","Type":"ContainerDied","Data":"b1190c7c1658db3b89a0b7fe5e5aaebc55ace1ca292dfe3a1d8f97b8a13b6fe7"} Jul 17 18:15:17.623720 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:17.623695 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72cf7b-29738535-l4c86" Jul 17 18:15:18.490399 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:18.490368 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72cf7b-29738535-l4c86" event={"ID":"8efd1900-9837-4052-a280-e028c740cf21","Type":"ContainerDied","Data":"3c03aec7b7a8f20ccbac935a0e5b72323da736612eac1f3a74aea2561e5ca8d9"} Jul 17 18:15:18.490542 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:18.490413 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c03aec7b7a8f20ccbac935a0e5b72323da736612eac1f3a74aea2561e5ca8d9" Jul 17 18:15:18.490542 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:18.490382 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72cf7b-29738535-l4c86" Jul 17 18:15:25.549642 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:25.549606 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:15:30.949327 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:30.949286 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt"] Jul 17 18:15:30.972720 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:30.972691 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:15:31.856003 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:31.855972 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4"] Jul 17 18:15:31.880841 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:31.880815 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:15:38.056071 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056040 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw"] Jul 17 18:15:38.056437 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056392 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2014f2e6-cd15-44c2-b891-fe4be8b13f50" containerName="cleanup" Jul 17 18:15:38.056437 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056404 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2014f2e6-cd15-44c2-b891-fe4be8b13f50" containerName="cleanup" Jul 17 18:15:38.056437 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056415 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6617328d-6a6e-4b1d-a4c0-302bcbd2d106" containerName="cleanup" Jul 17 18:15:38.056437 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056420 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6617328d-6a6e-4b1d-a4c0-302bcbd2d106" containerName="cleanup" Jul 17 18:15:38.056437 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056430 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca604483-cc82-4b48-83c8-d346adc619ef" containerName="cleanup" Jul 17 18:15:38.056437 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056435 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca604483-cc82-4b48-83c8-d346adc619ef" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056445 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="207d5598-707a-499a-bf0f-e9a1258a6106" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056451 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="207d5598-707a-499a-bf0f-e9a1258a6106" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056455 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29dabe77-16c2-4ead-8b5d-74c6e65b7a48" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056461 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="29dabe77-16c2-4ead-8b5d-74c6e65b7a48" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056468 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce6784f-2bd7-4306-8bad-fab1e6ed32da" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056473 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce6784f-2bd7-4306-8bad-fab1e6ed32da" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056481 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7078029-1712-4365-af96-6f3da763995e" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056487 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7078029-1712-4365-af96-6f3da763995e" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056493 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6df39780-6eeb-4d61-aa2d-57479841ad48" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056499 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6df39780-6eeb-4d61-aa2d-57479841ad48" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056507 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5148216b-3305-4cc4-8758-8fe645060236" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056512 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5148216b-3305-4cc4-8758-8fe645060236" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056519 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f17809e6-2524-480d-88ec-dbfa620d7fce" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056524 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f17809e6-2524-480d-88ec-dbfa620d7fce" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056529 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1f539ee-1d44-46e2-b2cc-20badad59414" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056535 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1f539ee-1d44-46e2-b2cc-20badad59414" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056541 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8efd1900-9837-4052-a280-e028c740cf21" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056546 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8efd1900-9837-4052-a280-e028c740cf21" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056553 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776bc04a-aaf6-4276-9008-880027518b41" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056558 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="776bc04a-aaf6-4276-9008-880027518b41" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056607 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2014f2e6-cd15-44c2-b891-fe4be8b13f50" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056613 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="776bc04a-aaf6-4276-9008-880027518b41" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056619 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="bce6784f-2bd7-4306-8bad-fab1e6ed32da" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056625 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8efd1900-9837-4052-a280-e028c740cf21" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056630 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca604483-cc82-4b48-83c8-d346adc619ef" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056635 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1f539ee-1d44-46e2-b2cc-20badad59414" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056641 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6617328d-6a6e-4b1d-a4c0-302bcbd2d106" containerName="cleanup" Jul 17 18:15:38.056634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056645 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="207d5598-707a-499a-bf0f-e9a1258a6106" containerName="cleanup" Jul 17 18:15:38.057445 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056650 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6df39780-6eeb-4d61-aa2d-57479841ad48" containerName="cleanup" Jul 17 18:15:38.057445 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056655 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f17809e6-2524-480d-88ec-dbfa620d7fce" containerName="cleanup" Jul 17 18:15:38.057445 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056660 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5148216b-3305-4cc4-8758-8fe645060236" containerName="cleanup" Jul 17 18:15:38.057445 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056666 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7078029-1712-4365-af96-6f3da763995e" containerName="cleanup" Jul 17 18:15:38.057445 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.056671 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="29dabe77-16c2-4ead-8b5d-74c6e65b7a48" containerName="cleanup" Jul 17 18:15:38.064716 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.064698 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:38.067381 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.067359 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0a98d9\"/\"rate-test-model-0a98d9-kserve-self-signed-certs\"" Jul 17 18:15:38.071279 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.071256 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw"] Jul 17 18:15:38.090400 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.090376 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:15:38.117675 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.117652 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8bh5\" (UniqueName: \"kubernetes.io/projected/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-kube-api-access-h8bh5\") pod \"rate-test-model-0a98d9-kserve-666874597f-h69vw\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:38.117772 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.117716 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-tmp-dir\") pod \"rate-test-model-0a98d9-kserve-666874597f-h69vw\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:38.117772 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.117740 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-tls-certs\") pod \"rate-test-model-0a98d9-kserve-666874597f-h69vw\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:38.117843 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.117804 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-dshm\") pod \"rate-test-model-0a98d9-kserve-666874597f-h69vw\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:38.117843 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.117834 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-model-cache\") pod \"rate-test-model-0a98d9-kserve-666874597f-h69vw\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:38.117981 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.117919 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-home\") pod \"rate-test-model-0a98d9-kserve-666874597f-h69vw\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:38.218728 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.218695 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-dshm\") pod \"rate-test-model-0a98d9-kserve-666874597f-h69vw\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:38.218912 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.218736 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-model-cache\") pod \"rate-test-model-0a98d9-kserve-666874597f-h69vw\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:38.218912 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.218808 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-home\") pod \"rate-test-model-0a98d9-kserve-666874597f-h69vw\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:38.218912 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.218836 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8bh5\" (UniqueName: \"kubernetes.io/projected/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-kube-api-access-h8bh5\") pod \"rate-test-model-0a98d9-kserve-666874597f-h69vw\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:38.218912 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.218902 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-tmp-dir\") pod \"rate-test-model-0a98d9-kserve-666874597f-h69vw\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:38.219135 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.218925 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-tls-certs\") pod \"rate-test-model-0a98d9-kserve-666874597f-h69vw\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:38.219269 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.219247 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-model-cache\") pod \"rate-test-model-0a98d9-kserve-666874597f-h69vw\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:38.219332 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.219275 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-home\") pod \"rate-test-model-0a98d9-kserve-666874597f-h69vw\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:38.219380 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.219344 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-tmp-dir\") pod \"rate-test-model-0a98d9-kserve-666874597f-h69vw\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:38.220928 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.220912 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-dshm\") pod \"rate-test-model-0a98d9-kserve-666874597f-h69vw\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:38.221333 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.221314 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-tls-certs\") pod \"rate-test-model-0a98d9-kserve-666874597f-h69vw\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:38.227206 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.227187 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8bh5\" (UniqueName: \"kubernetes.io/projected/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-kube-api-access-h8bh5\") pod \"rate-test-model-0a98d9-kserve-666874597f-h69vw\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:38.375561 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.375497 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:38.511139 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.511116 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw"] Jul 17 18:15:38.512654 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:15:38.512629 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6d0b0c3_202e_4033_8021_42bcb6ab4cdd.slice/crio-3184e18d8ec2e25b8fee5ff6836062fa78e133b36f2568708b7590d4937d819d WatchSource:0}: Error finding container 3184e18d8ec2e25b8fee5ff6836062fa78e133b36f2568708b7590d4937d819d: Status 404 returned error can't find the container with id 3184e18d8ec2e25b8fee5ff6836062fa78e133b36f2568708b7590d4937d819d Jul 17 18:15:38.591684 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.591651 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" event={"ID":"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd","Type":"ContainerStarted","Data":"6824deea9db961f7f1f473f04a0ab594480c16ce6630ebc367b34cbb22caa6d7"} Jul 17 18:15:38.591912 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.591884 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" event={"ID":"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd","Type":"ContainerStarted","Data":"3184e18d8ec2e25b8fee5ff6836062fa78e133b36f2568708b7590d4937d819d"} Jul 17 18:15:38.611151 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:38.611105 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" podStartSLOduration=0.611089036 podStartE2EDuration="611.089036ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:15:38.609236916 +0000 UTC m=+2659.728065208" watchObservedRunningTime="2026-07-17 18:15:38.611089036 +0000 UTC m=+2659.729917316" Jul 17 18:15:45.445889 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:45.445839 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:15:45.968323 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:45.968289 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" podUID="bf8013ff-a730-4a8b-97ac-30f814d00a27" containerName="main" containerID="cri-o://c8140df1e77ec302a5dedbd0b71b035a3c068d80cb0f37ce58d08fcb2a402e55" gracePeriod=45 Jul 17 18:15:46.218409 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.218351 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:15:46.289878 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.289833 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf8013ff-a730-4a8b-97ac-30f814d00a27-tls-certs\") pod \"bf8013ff-a730-4a8b-97ac-30f814d00a27\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " Jul 17 18:15:46.290015 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.289904 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-model-cache\") pod \"bf8013ff-a730-4a8b-97ac-30f814d00a27\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " Jul 17 18:15:46.290015 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.289996 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-dshm\") pod \"bf8013ff-a730-4a8b-97ac-30f814d00a27\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " Jul 17 18:15:46.290085 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.290023 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-tmp-dir\") pod \"bf8013ff-a730-4a8b-97ac-30f814d00a27\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " Jul 17 18:15:46.290085 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.290045 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-home\") pod \"bf8013ff-a730-4a8b-97ac-30f814d00a27\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " Jul 17 18:15:46.290176 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.290083 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wqlw\" (UniqueName: \"kubernetes.io/projected/bf8013ff-a730-4a8b-97ac-30f814d00a27-kube-api-access-2wqlw\") pod \"bf8013ff-a730-4a8b-97ac-30f814d00a27\" (UID: \"bf8013ff-a730-4a8b-97ac-30f814d00a27\") " Jul 17 18:15:46.290231 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.290172 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf8013ff-a730-4a8b-97ac-30f814d00a27" (UID: "bf8013ff-a730-4a8b-97ac-30f814d00a27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:15:46.290341 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.290301 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-home" (OuterVolumeSpecName: "home") pod "bf8013ff-a730-4a8b-97ac-30f814d00a27" (UID: "bf8013ff-a730-4a8b-97ac-30f814d00a27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:15:46.290341 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.290310 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf8013ff-a730-4a8b-97ac-30f814d00a27" (UID: "bf8013ff-a730-4a8b-97ac-30f814d00a27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:15:46.290524 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.290415 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-model-cache\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:15:46.290524 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.290428 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-tmp-dir\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:15:46.290524 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.290436 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-home\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:15:46.291956 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.291929 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-dshm" (OuterVolumeSpecName: "dshm") pod "bf8013ff-a730-4a8b-97ac-30f814d00a27" (UID: "bf8013ff-a730-4a8b-97ac-30f814d00a27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:15:46.292057 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.292035 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8013ff-a730-4a8b-97ac-30f814d00a27-kube-api-access-2wqlw" (OuterVolumeSpecName: "kube-api-access-2wqlw") pod "bf8013ff-a730-4a8b-97ac-30f814d00a27" (UID: "bf8013ff-a730-4a8b-97ac-30f814d00a27"). InnerVolumeSpecName "kube-api-access-2wqlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:15:46.292119 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.292102 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf8013ff-a730-4a8b-97ac-30f814d00a27-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf8013ff-a730-4a8b-97ac-30f814d00a27" (UID: "bf8013ff-a730-4a8b-97ac-30f814d00a27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:15:46.391633 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.391609 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf8013ff-a730-4a8b-97ac-30f814d00a27-tls-certs\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:15:46.391633 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.391633 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf8013ff-a730-4a8b-97ac-30f814d00a27-dshm\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:15:46.391785 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.391648 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wqlw\" (UniqueName: \"kubernetes.io/projected/bf8013ff-a730-4a8b-97ac-30f814d00a27-kube-api-access-2wqlw\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:15:46.635630 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.635589 2553 generic.go:356] "Generic (PLEG): container finished" podID="bf8013ff-a730-4a8b-97ac-30f814d00a27" containerID="c8140df1e77ec302a5dedbd0b71b035a3c068d80cb0f37ce58d08fcb2a402e55" exitCode=0 Jul 17 18:15:46.635630 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.635626 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" Jul 17 18:15:46.636125 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.635641 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" event={"ID":"bf8013ff-a730-4a8b-97ac-30f814d00a27","Type":"ContainerDied","Data":"c8140df1e77ec302a5dedbd0b71b035a3c068d80cb0f37ce58d08fcb2a402e55"} Jul 17 18:15:46.636125 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.635691 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt" event={"ID":"bf8013ff-a730-4a8b-97ac-30f814d00a27","Type":"ContainerDied","Data":"bc5a7a1bf92c4770c1335d0fdc33e9c5ede07e330abc77736232588206907c0b"} Jul 17 18:15:46.636125 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.635719 2553 scope.go:117] "RemoveContainer" containerID="c8140df1e77ec302a5dedbd0b71b035a3c068d80cb0f37ce58d08fcb2a402e55" Jul 17 18:15:46.644993 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.644976 2553 scope.go:117] "RemoveContainer" containerID="c8140df1e77ec302a5dedbd0b71b035a3c068d80cb0f37ce58d08fcb2a402e55" Jul 17 18:15:46.645240 ip-10-0-136-43 kubenswrapper[2553]: E0717 18:15:46.645217 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8140df1e77ec302a5dedbd0b71b035a3c068d80cb0f37ce58d08fcb2a402e55\": container with ID starting with c8140df1e77ec302a5dedbd0b71b035a3c068d80cb0f37ce58d08fcb2a402e55 not found: ID does not exist" containerID="c8140df1e77ec302a5dedbd0b71b035a3c068d80cb0f37ce58d08fcb2a402e55" Jul 17 18:15:46.645376 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.645245 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8140df1e77ec302a5dedbd0b71b035a3c068d80cb0f37ce58d08fcb2a402e55"} err="failed to get container status \"c8140df1e77ec302a5dedbd0b71b035a3c068d80cb0f37ce58d08fcb2a402e55\": rpc error: code = NotFound desc = could not find container \"c8140df1e77ec302a5dedbd0b71b035a3c068d80cb0f37ce58d08fcb2a402e55\": container with ID starting with c8140df1e77ec302a5dedbd0b71b035a3c068d80cb0f37ce58d08fcb2a402e55 not found: ID does not exist" Jul 17 18:15:46.658311 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.658289 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt"] Jul 17 18:15:46.664087 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.664064 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-0a98d9/sub-test-model-0a98d9-kserve-7965b6b6b9-f9klt"] Jul 17 18:15:46.875232 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:46.875190 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" podUID="eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda" containerName="main" containerID="cri-o://b07eff975d6f4c2d07ee3df14d9136f29c48bcd962b79ad6bf19e487ac947c27" gracePeriod=45 Jul 17 18:15:47.124452 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.124431 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:15:47.198163 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.198140 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-tls-certs\") pod \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " Jul 17 18:15:47.198329 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.198175 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-tmp-dir\") pod \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " Jul 17 18:15:47.198329 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.198198 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-dshm\") pod \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " Jul 17 18:15:47.198329 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.198269 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-home\") pod \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " Jul 17 18:15:47.198329 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.198298 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9pwc\" (UniqueName: \"kubernetes.io/projected/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-kube-api-access-f9pwc\") pod \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " Jul 17 18:15:47.198329 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.198325 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-model-cache\") pod \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\" (UID: \"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda\") " Jul 17 18:15:47.198586 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.198423 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda" (UID: "eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:15:47.198586 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.198560 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-home" (OuterVolumeSpecName: "home") pod "eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda" (UID: "eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:15:47.198697 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.198606 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-model-cache" (OuterVolumeSpecName: "model-cache") pod "eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda" (UID: "eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:15:47.198740 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.198700 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-tmp-dir\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:15:47.198740 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.198711 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-home\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:15:47.198740 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.198719 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-model-cache\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:15:47.200282 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.200259 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda" (UID: "eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:15:47.200282 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.200257 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-dshm" (OuterVolumeSpecName: "dshm") pod "eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda" (UID: "eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:15:47.200426 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.200285 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-kube-api-access-f9pwc" (OuterVolumeSpecName: "kube-api-access-f9pwc") pod "eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda" (UID: "eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda"). InnerVolumeSpecName "kube-api-access-f9pwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:15:47.299800 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.299723 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-dshm\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:15:47.299800 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.299756 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9pwc\" (UniqueName: \"kubernetes.io/projected/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-kube-api-access-f9pwc\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:15:47.299800 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.299772 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda-tls-certs\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:15:47.464089 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.464056 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf8013ff-a730-4a8b-97ac-30f814d00a27" path="/var/lib/kubelet/pods/bf8013ff-a730-4a8b-97ac-30f814d00a27/volumes" Jul 17 18:15:47.644165 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.644075 2553 generic.go:356] "Generic (PLEG): container finished" podID="eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda" containerID="b07eff975d6f4c2d07ee3df14d9136f29c48bcd962b79ad6bf19e487ac947c27" exitCode=0 Jul 17 18:15:47.644165 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.644116 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" Jul 17 18:15:47.644165 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.644113 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" event={"ID":"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda","Type":"ContainerDied","Data":"b07eff975d6f4c2d07ee3df14d9136f29c48bcd962b79ad6bf19e487ac947c27"} Jul 17 18:15:47.644165 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.644164 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4" event={"ID":"eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda","Type":"ContainerDied","Data":"ca19036ebd5280628cdeb4bbad7c90b9c5414a4dc33b7eb2f765d16564f206d3"} Jul 17 18:15:47.644747 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.644188 2553 scope.go:117] "RemoveContainer" containerID="b07eff975d6f4c2d07ee3df14d9136f29c48bcd962b79ad6bf19e487ac947c27" Jul 17 18:15:47.652608 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.652590 2553 scope.go:117] "RemoveContainer" containerID="b07eff975d6f4c2d07ee3df14d9136f29c48bcd962b79ad6bf19e487ac947c27" Jul 17 18:15:47.652877 ip-10-0-136-43 kubenswrapper[2553]: E0717 18:15:47.652834 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b07eff975d6f4c2d07ee3df14d9136f29c48bcd962b79ad6bf19e487ac947c27\": container with ID starting with b07eff975d6f4c2d07ee3df14d9136f29c48bcd962b79ad6bf19e487ac947c27 not found: ID does not exist" containerID="b07eff975d6f4c2d07ee3df14d9136f29c48bcd962b79ad6bf19e487ac947c27" Jul 17 18:15:47.652939 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.652885 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b07eff975d6f4c2d07ee3df14d9136f29c48bcd962b79ad6bf19e487ac947c27"} err="failed to get container status \"b07eff975d6f4c2d07ee3df14d9136f29c48bcd962b79ad6bf19e487ac947c27\": rpc error: code = NotFound desc = could not find container \"b07eff975d6f4c2d07ee3df14d9136f29c48bcd962b79ad6bf19e487ac947c27\": container with ID starting with b07eff975d6f4c2d07ee3df14d9136f29c48bcd962b79ad6bf19e487ac947c27 not found: ID does not exist" Jul 17 18:15:47.661478 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.661453 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4"] Jul 17 18:15:47.665334 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:47.665315 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-37d0b7/sub-test-model-37d0b7-kserve-6869dffbdf-dvvn4"] Jul 17 18:15:48.375924 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:48.375886 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:48.375924 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:48.375925 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:48.388031 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:48.388003 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:48.660424 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:48.660351 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:15:49.462798 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:15:49.462767 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda" path="/var/lib/kubelet/pods/eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda/volumes" Jul 17 18:16:09.969349 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:09.969308 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c"] Jul 17 18:16:09.970017 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:09.969820 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda" containerName="main" Jul 17 18:16:09.970017 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:09.969839 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda" containerName="main" Jul 17 18:16:09.970017 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:09.969889 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf8013ff-a730-4a8b-97ac-30f814d00a27" containerName="main" Jul 17 18:16:09.970017 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:09.969899 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf8013ff-a730-4a8b-97ac-30f814d00a27" containerName="main" Jul 17 18:16:09.970017 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:09.969991 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf8013ff-a730-4a8b-97ac-30f814d00a27" containerName="main" Jul 17 18:16:09.970017 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:09.970004 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="eceb1e87-7dc7-4bc7-91ce-9f3a93bbaeda" containerName="main" Jul 17 18:16:09.973168 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:09.973145 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:09.977764 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:09.977743 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-37d0b7\"/\"default-dockercfg-tbg82\"" Jul 17 18:16:09.977764 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:09.977756 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-37d0b7\"/\"openshift-service-ca.crt\"" Jul 17 18:16:09.977962 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:09.977765 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-37d0b7\"/\"kube-root-ca.crt\"" Jul 17 18:16:09.977962 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:09.977891 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-37d0b7\"/\"rate-test-model-37d0b7-kserve-self-signed-certs\"" Jul 17 18:16:09.983451 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:09.983430 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c"] Jul 17 18:16:10.087536 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.087504 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-dshm\") pod \"rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:10.087536 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.087536 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-model-cache\") pod \"rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:10.087723 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.087556 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-tmp-dir\") pod \"rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:10.087723 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.087688 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trljd\" (UniqueName: \"kubernetes.io/projected/ca31ff4a-2494-48f8-9901-c449741d59a3-kube-api-access-trljd\") pod \"rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:10.087803 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.087741 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca31ff4a-2494-48f8-9901-c449741d59a3-tls-certs\") pod \"rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:10.087803 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.087771 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-home\") pod \"rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:10.188100 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.188079 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-dshm\") pod \"rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:10.188220 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.188104 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-model-cache\") pod \"rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:10.188220 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.188120 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-tmp-dir\") pod \"rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:10.188220 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.188166 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trljd\" (UniqueName: \"kubernetes.io/projected/ca31ff4a-2494-48f8-9901-c449741d59a3-kube-api-access-trljd\") pod \"rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:10.188220 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.188197 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca31ff4a-2494-48f8-9901-c449741d59a3-tls-certs\") pod \"rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:10.188220 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.188212 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-home\") pod \"rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:10.188571 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.188552 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-model-cache\") pod \"rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:10.188658 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.188573 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-home\") pod \"rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:10.188658 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.188608 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-tmp-dir\") pod \"rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:10.190408 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.190384 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-dshm\") pod \"rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:10.190592 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.190578 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca31ff4a-2494-48f8-9901-c449741d59a3-tls-certs\") pod \"rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:10.196389 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.196371 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trljd\" (UniqueName: \"kubernetes.io/projected/ca31ff4a-2494-48f8-9901-c449741d59a3-kube-api-access-trljd\") pod \"rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:10.282644 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.282624 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:10.367220 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.367192 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:16:10.426761 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.426737 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c"] Jul 17 18:16:10.428292 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:16:10.428264 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca31ff4a_2494_48f8_9901_c449741d59a3.slice/crio-1f7d9728581a84af7c5a39fc3fada0dbedeb2728e7ce56346902bf43419cebe8 WatchSource:0}: Error finding container 1f7d9728581a84af7c5a39fc3fada0dbedeb2728e7ce56346902bf43419cebe8: Status 404 returned error can't find the container with id 1f7d9728581a84af7c5a39fc3fada0dbedeb2728e7ce56346902bf43419cebe8 Jul 17 18:16:10.766966 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.766932 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" event={"ID":"ca31ff4a-2494-48f8-9901-c449741d59a3","Type":"ContainerStarted","Data":"862675ca325b05caa60121bf400130acfb312fb185e34588aa3f2368b81451a2"} Jul 17 18:16:10.766966 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.766970 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" event={"ID":"ca31ff4a-2494-48f8-9901-c449741d59a3","Type":"ContainerStarted","Data":"1f7d9728581a84af7c5a39fc3fada0dbedeb2728e7ce56346902bf43419cebe8"} Jul 17 18:16:10.788734 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:10.788676 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" podStartSLOduration=1.788661029 podStartE2EDuration="1.788661029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:16:10.786166854 +0000 UTC m=+2691.904995132" watchObservedRunningTime="2026-07-17 18:16:10.788661029 +0000 UTC m=+2691.907489311" Jul 17 18:16:14.978861 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:14.978811 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:16:15.502096 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:15.502062 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:16:15.613944 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:15.613915 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:16:20.282945 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:20.282912 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:20.283338 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:20.283255 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:20.295086 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:20.295062 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:20.827434 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:20.827405 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:16:23.854508 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:23.854477 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:16:23.955785 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:23.955754 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:16:37.235901 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:37.235866 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:16:54.436965 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:54.436932 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:16:57.081399 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:57.081367 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:16:57.142261 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:16:57.142235 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:01.416096 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:01.416065 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw"] Jul 17 18:17:01.427922 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:01.427897 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:02.241571 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:02.241541 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c"] Jul 17 18:17:02.264390 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:02.264359 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:03.286599 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:03.286563 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7681-139859116052528"] Jul 17 18:17:03.289995 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:03.289978 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116052528" Jul 17 18:17:03.292872 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:03.292835 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q6v2p\"" Jul 17 18:17:03.300298 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:03.300277 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-139859116052528"] Jul 17 18:17:03.401059 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:03.401031 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5xpg\" (UniqueName: \"kubernetes.io/projected/bd6fe6b4-60f3-4ba9-b14e-029385ed69b3-kube-api-access-n5xpg\") pod \"test-curl-7681-139859116052528\" (UID: \"bd6fe6b4-60f3-4ba9-b14e-029385ed69b3\") " pod="opendatahub/test-curl-7681-139859116052528" Jul 17 18:17:03.501975 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:03.501948 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5xpg\" (UniqueName: \"kubernetes.io/projected/bd6fe6b4-60f3-4ba9-b14e-029385ed69b3-kube-api-access-n5xpg\") pod \"test-curl-7681-139859116052528\" (UID: \"bd6fe6b4-60f3-4ba9-b14e-029385ed69b3\") " pod="opendatahub/test-curl-7681-139859116052528" Jul 17 18:17:03.510659 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:03.510633 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5xpg\" (UniqueName: \"kubernetes.io/projected/bd6fe6b4-60f3-4ba9-b14e-029385ed69b3-kube-api-access-n5xpg\") pod \"test-curl-7681-139859116052528\" (UID: \"bd6fe6b4-60f3-4ba9-b14e-029385ed69b3\") " pod="opendatahub/test-curl-7681-139859116052528" Jul 17 18:17:03.599328 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:03.599274 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116052528" Jul 17 18:17:03.725753 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:03.725729 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-139859116052528"] Jul 17 18:17:03.727504 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:17:03.727480 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd6fe6b4_60f3_4ba9_b14e_029385ed69b3.slice/crio-7525dc49878eabd88be0813d5432f3d2fa07fe3435175ed8cca9f2f2090448cd WatchSource:0}: Error finding container 7525dc49878eabd88be0813d5432f3d2fa07fe3435175ed8cca9f2f2090448cd: Status 404 returned error can't find the container with id 7525dc49878eabd88be0813d5432f3d2fa07fe3435175ed8cca9f2f2090448cd Jul 17 18:17:04.040907 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:04.040870 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-139859116052528" event={"ID":"bd6fe6b4-60f3-4ba9-b14e-029385ed69b3","Type":"ContainerStarted","Data":"7525dc49878eabd88be0813d5432f3d2fa07fe3435175ed8cca9f2f2090448cd"} Jul 17 18:17:05.047888 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:05.047860 2553 generic.go:356] "Generic (PLEG): container finished" podID="bd6fe6b4-60f3-4ba9-b14e-029385ed69b3" containerID="0850a11409952f589bab574fd2dda0b53cfa69b4d04552f347120daceedb0cd8" exitCode=0 Jul 17 18:17:05.048230 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:05.047928 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-139859116052528" event={"ID":"bd6fe6b4-60f3-4ba9-b14e-029385ed69b3","Type":"ContainerDied","Data":"0850a11409952f589bab574fd2dda0b53cfa69b4d04552f347120daceedb0cd8"} Jul 17 18:17:06.179951 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:06.179925 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116052528" Jul 17 18:17:06.201439 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:06.201418 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7681-139859116052528_bd6fe6b4-60f3-4ba9-b14e-029385ed69b3/test-curl-7681-139859116052528/0.log" Jul 17 18:17:06.225443 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:06.225417 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7681-139859116052528"] Jul 17 18:17:06.229868 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:06.229725 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7681-139859116052528"] Jul 17 18:17:06.328304 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:06.328239 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5xpg\" (UniqueName: \"kubernetes.io/projected/bd6fe6b4-60f3-4ba9-b14e-029385ed69b3-kube-api-access-n5xpg\") pod \"bd6fe6b4-60f3-4ba9-b14e-029385ed69b3\" (UID: \"bd6fe6b4-60f3-4ba9-b14e-029385ed69b3\") " Jul 17 18:17:06.330297 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:06.330274 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6fe6b4-60f3-4ba9-b14e-029385ed69b3-kube-api-access-n5xpg" (OuterVolumeSpecName: "kube-api-access-n5xpg") pod "bd6fe6b4-60f3-4ba9-b14e-029385ed69b3" (UID: "bd6fe6b4-60f3-4ba9-b14e-029385ed69b3"). InnerVolumeSpecName "kube-api-access-n5xpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:17:06.362729 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:06.362704 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7681-139859116930608"] Jul 17 18:17:06.363076 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:06.363062 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd6fe6b4-60f3-4ba9-b14e-029385ed69b3" containerName="test-curl-7681-139859116052528" Jul 17 18:17:06.363142 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:06.363077 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd6fe6b4-60f3-4ba9-b14e-029385ed69b3" containerName="test-curl-7681-139859116052528" Jul 17 18:17:06.363142 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:06.363136 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd6fe6b4-60f3-4ba9-b14e-029385ed69b3" containerName="test-curl-7681-139859116052528" Jul 17 18:17:06.366017 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:06.366003 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116930608" Jul 17 18:17:06.371460 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:06.371438 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-139859116930608"] Jul 17 18:17:06.429658 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:06.429634 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6xv5\" (UniqueName: \"kubernetes.io/projected/6ce889ad-8fda-452f-9e43-aaa61cf88bc3-kube-api-access-q6xv5\") pod \"test-curl-7681-139859116930608\" (UID: \"6ce889ad-8fda-452f-9e43-aaa61cf88bc3\") " pod="opendatahub/test-curl-7681-139859116930608" Jul 17 18:17:06.429747 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:06.429676 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5xpg\" (UniqueName: \"kubernetes.io/projected/bd6fe6b4-60f3-4ba9-b14e-029385ed69b3-kube-api-access-n5xpg\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:17:06.530192 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:06.530169 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6xv5\" (UniqueName: \"kubernetes.io/projected/6ce889ad-8fda-452f-9e43-aaa61cf88bc3-kube-api-access-q6xv5\") pod \"test-curl-7681-139859116930608\" (UID: \"6ce889ad-8fda-452f-9e43-aaa61cf88bc3\") " pod="opendatahub/test-curl-7681-139859116930608" Jul 17 18:17:06.538317 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:06.538296 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6xv5\" (UniqueName: \"kubernetes.io/projected/6ce889ad-8fda-452f-9e43-aaa61cf88bc3-kube-api-access-q6xv5\") pod \"test-curl-7681-139859116930608\" (UID: \"6ce889ad-8fda-452f-9e43-aaa61cf88bc3\") " pod="opendatahub/test-curl-7681-139859116930608" Jul 17 18:17:06.675794 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:06.675725 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116930608" Jul 17 18:17:06.803366 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:06.803339 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-139859116930608"] Jul 17 18:17:06.806180 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:17:06.806155 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ce889ad_8fda_452f_9e43_aaa61cf88bc3.slice/crio-1849da358cc8739bb0acba4ca9755b4ac591088bd18b9299be8af9ca3e30353c WatchSource:0}: Error finding container 1849da358cc8739bb0acba4ca9755b4ac591088bd18b9299be8af9ca3e30353c: Status 404 returned error can't find the container with id 1849da358cc8739bb0acba4ca9755b4ac591088bd18b9299be8af9ca3e30353c Jul 17 18:17:07.058906 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:07.058876 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-139859116930608" event={"ID":"6ce889ad-8fda-452f-9e43-aaa61cf88bc3","Type":"ContainerStarted","Data":"1849da358cc8739bb0acba4ca9755b4ac591088bd18b9299be8af9ca3e30353c"} Jul 17 18:17:07.060118 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:07.060102 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116052528" Jul 17 18:17:07.060118 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:07.060110 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7525dc49878eabd88be0813d5432f3d2fa07fe3435175ed8cca9f2f2090448cd" Jul 17 18:17:07.070412 ip-10-0-136-43 kubenswrapper[2553]: E0717 18:17:07.070388 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-139859116052528\" is forbidden: User \"system:node:ip-10-0-136-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-43.ec2.internal' and this object" podUID="bd6fe6b4-60f3-4ba9-b14e-029385ed69b3" pod="opendatahub/test-curl-7681-139859116052528" Jul 17 18:17:07.463681 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:07.463615 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd6fe6b4-60f3-4ba9-b14e-029385ed69b3" path="/var/lib/kubelet/pods/bd6fe6b4-60f3-4ba9-b14e-029385ed69b3/volumes" Jul 17 18:17:08.066646 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:08.066618 2553 generic.go:356] "Generic (PLEG): container finished" podID="6ce889ad-8fda-452f-9e43-aaa61cf88bc3" containerID="0af3f107a7bc48be0329f32c6ce200b29e9ba43eda58236d12e99f9c36da52a8" exitCode=0 Jul 17 18:17:08.066787 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:08.066708 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-139859116930608" event={"ID":"6ce889ad-8fda-452f-9e43-aaa61cf88bc3","Type":"ContainerDied","Data":"0af3f107a7bc48be0329f32c6ce200b29e9ba43eda58236d12e99f9c36da52a8"} Jul 17 18:17:09.198571 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:09.198546 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116930608" Jul 17 18:17:09.221520 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:09.221497 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7681-139859116930608_6ce889ad-8fda-452f-9e43-aaa61cf88bc3/test-curl-7681-139859116930608/0.log" Jul 17 18:17:09.248735 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:09.248709 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7681-139859116930608"] Jul 17 18:17:09.252634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:09.252612 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7681-139859116930608"] Jul 17 18:17:09.351483 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:09.351424 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6xv5\" (UniqueName: \"kubernetes.io/projected/6ce889ad-8fda-452f-9e43-aaa61cf88bc3-kube-api-access-q6xv5\") pod \"6ce889ad-8fda-452f-9e43-aaa61cf88bc3\" (UID: \"6ce889ad-8fda-452f-9e43-aaa61cf88bc3\") " Jul 17 18:17:09.353340 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:09.353320 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce889ad-8fda-452f-9e43-aaa61cf88bc3-kube-api-access-q6xv5" (OuterVolumeSpecName: "kube-api-access-q6xv5") pod "6ce889ad-8fda-452f-9e43-aaa61cf88bc3" (UID: "6ce889ad-8fda-452f-9e43-aaa61cf88bc3"). InnerVolumeSpecName "kube-api-access-q6xv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:17:09.432919 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:09.432895 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7681-139859116010496"] Jul 17 18:17:09.433243 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:09.433230 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce889ad-8fda-452f-9e43-aaa61cf88bc3" containerName="test-curl-7681-139859116930608" Jul 17 18:17:09.433281 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:09.433245 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce889ad-8fda-452f-9e43-aaa61cf88bc3" containerName="test-curl-7681-139859116930608" Jul 17 18:17:09.433311 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:09.433307 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ce889ad-8fda-452f-9e43-aaa61cf88bc3" containerName="test-curl-7681-139859116930608" Jul 17 18:17:09.436172 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:09.436158 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116010496" Jul 17 18:17:09.442520 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:09.442502 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-139859116010496"] Jul 17 18:17:09.452432 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:09.452412 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6xv5\" (UniqueName: \"kubernetes.io/projected/6ce889ad-8fda-452f-9e43-aaa61cf88bc3-kube-api-access-q6xv5\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:17:09.463077 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:09.463056 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ce889ad-8fda-452f-9e43-aaa61cf88bc3" path="/var/lib/kubelet/pods/6ce889ad-8fda-452f-9e43-aaa61cf88bc3/volumes" Jul 17 18:17:09.553925 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:09.553620 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp49b\" (UniqueName: \"kubernetes.io/projected/63cf7969-7e26-46d3-b853-b53ee2cd0fde-kube-api-access-cp49b\") pod \"test-curl-7681-139859116010496\" (UID: \"63cf7969-7e26-46d3-b853-b53ee2cd0fde\") " pod="opendatahub/test-curl-7681-139859116010496" Jul 17 18:17:09.654501 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:09.654444 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp49b\" (UniqueName: \"kubernetes.io/projected/63cf7969-7e26-46d3-b853-b53ee2cd0fde-kube-api-access-cp49b\") pod \"test-curl-7681-139859116010496\" (UID: \"63cf7969-7e26-46d3-b853-b53ee2cd0fde\") " pod="opendatahub/test-curl-7681-139859116010496" Jul 17 18:17:09.663216 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:09.663190 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp49b\" (UniqueName: \"kubernetes.io/projected/63cf7969-7e26-46d3-b853-b53ee2cd0fde-kube-api-access-cp49b\") pod \"test-curl-7681-139859116010496\" (UID: \"63cf7969-7e26-46d3-b853-b53ee2cd0fde\") " pod="opendatahub/test-curl-7681-139859116010496" Jul 17 18:17:09.745902 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:09.745878 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116010496" Jul 17 18:17:09.873182 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:09.873158 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-139859116010496"] Jul 17 18:17:09.875296 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:17:09.875267 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63cf7969_7e26_46d3_b853_b53ee2cd0fde.slice/crio-4408c9f69ec34ed6c75d94412f5ddfee295371e8010b5bdf87d3308c796ff8ac WatchSource:0}: Error finding container 4408c9f69ec34ed6c75d94412f5ddfee295371e8010b5bdf87d3308c796ff8ac: Status 404 returned error can't find the container with id 4408c9f69ec34ed6c75d94412f5ddfee295371e8010b5bdf87d3308c796ff8ac Jul 17 18:17:09.933011 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:09.932988 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:17:10.077443 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:10.077411 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-139859116010496" event={"ID":"63cf7969-7e26-46d3-b853-b53ee2cd0fde","Type":"ContainerStarted","Data":"4408c9f69ec34ed6c75d94412f5ddfee295371e8010b5bdf87d3308c796ff8ac"} Jul 17 18:17:10.078651 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:10.078629 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116930608" Jul 17 18:17:10.078651 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:10.078649 2553 scope.go:117] "RemoveContainer" containerID="0af3f107a7bc48be0329f32c6ce200b29e9ba43eda58236d12e99f9c36da52a8" Jul 17 18:17:11.085237 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:11.085200 2553 generic.go:356] "Generic (PLEG): container finished" podID="63cf7969-7e26-46d3-b853-b53ee2cd0fde" containerID="7f1b0a6468953231a54e3d0a03e708278e5a3f7a866d8f768354e95858204dac" exitCode=0 Jul 17 18:17:11.085661 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:11.085283 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-139859116010496" event={"ID":"63cf7969-7e26-46d3-b853-b53ee2cd0fde","Type":"ContainerDied","Data":"7f1b0a6468953231a54e3d0a03e708278e5a3f7a866d8f768354e95858204dac"} Jul 17 18:17:12.225662 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:12.225639 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116010496" Jul 17 18:17:12.246242 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:12.246196 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7681-139859116010496_63cf7969-7e26-46d3-b853-b53ee2cd0fde/test-curl-7681-139859116010496/0.log" Jul 17 18:17:12.271433 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:12.271405 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7681-139859116010496"] Jul 17 18:17:12.274513 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:12.274490 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7681-139859116010496"] Jul 17 18:17:12.377725 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:12.377654 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cp49b\" (UniqueName: \"kubernetes.io/projected/63cf7969-7e26-46d3-b853-b53ee2cd0fde-kube-api-access-cp49b\") pod \"63cf7969-7e26-46d3-b853-b53ee2cd0fde\" (UID: \"63cf7969-7e26-46d3-b853-b53ee2cd0fde\") " Jul 17 18:17:12.379482 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:12.379455 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63cf7969-7e26-46d3-b853-b53ee2cd0fde-kube-api-access-cp49b" (OuterVolumeSpecName: "kube-api-access-cp49b") pod "63cf7969-7e26-46d3-b853-b53ee2cd0fde" (UID: "63cf7969-7e26-46d3-b853-b53ee2cd0fde"). InnerVolumeSpecName "kube-api-access-cp49b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:17:12.394430 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:12.394399 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7681-139859116010496"] Jul 17 18:17:12.394773 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:12.394761 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63cf7969-7e26-46d3-b853-b53ee2cd0fde" containerName="test-curl-7681-139859116010496" Jul 17 18:17:12.394773 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:12.394774 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="63cf7969-7e26-46d3-b853-b53ee2cd0fde" containerName="test-curl-7681-139859116010496" Jul 17 18:17:12.394870 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:12.394824 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="63cf7969-7e26-46d3-b853-b53ee2cd0fde" containerName="test-curl-7681-139859116010496" Jul 17 18:17:12.397774 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:12.397758 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116010496" Jul 17 18:17:12.404510 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:12.404485 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-139859116010496"] Jul 17 18:17:12.479118 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:12.479096 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkqx9\" (UniqueName: \"kubernetes.io/projected/826cbbc4-fbeb-4faa-b4bf-87762ee8f65c-kube-api-access-zkqx9\") pod \"test-curl-7681-139859116010496\" (UID: \"826cbbc4-fbeb-4faa-b4bf-87762ee8f65c\") " pod="opendatahub/test-curl-7681-139859116010496" Jul 17 18:17:12.479223 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:12.479201 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp49b\" (UniqueName: \"kubernetes.io/projected/63cf7969-7e26-46d3-b853-b53ee2cd0fde-kube-api-access-cp49b\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:17:12.579582 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:12.579555 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkqx9\" (UniqueName: \"kubernetes.io/projected/826cbbc4-fbeb-4faa-b4bf-87762ee8f65c-kube-api-access-zkqx9\") pod \"test-curl-7681-139859116010496\" (UID: \"826cbbc4-fbeb-4faa-b4bf-87762ee8f65c\") " pod="opendatahub/test-curl-7681-139859116010496" Jul 17 18:17:12.588614 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:12.588592 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkqx9\" (UniqueName: \"kubernetes.io/projected/826cbbc4-fbeb-4faa-b4bf-87762ee8f65c-kube-api-access-zkqx9\") pod \"test-curl-7681-139859116010496\" (UID: \"826cbbc4-fbeb-4faa-b4bf-87762ee8f65c\") " pod="opendatahub/test-curl-7681-139859116010496" Jul 17 18:17:12.708096 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:12.708036 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116010496" Jul 17 18:17:12.833009 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:12.832984 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-139859116010496"] Jul 17 18:17:12.835066 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:17:12.835042 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod826cbbc4_fbeb_4faa_b4bf_87762ee8f65c.slice/crio-20ed9321005bf70f9e3c3f3983a8b348feb173d646c65c4149f52c3d4636ba91 WatchSource:0}: Error finding container 20ed9321005bf70f9e3c3f3983a8b348feb173d646c65c4149f52c3d4636ba91: Status 404 returned error can't find the container with id 20ed9321005bf70f9e3c3f3983a8b348feb173d646c65c4149f52c3d4636ba91 Jul 17 18:17:13.096693 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:13.096665 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116010496" Jul 17 18:17:13.096824 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:13.096685 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4408c9f69ec34ed6c75d94412f5ddfee295371e8010b5bdf87d3308c796ff8ac" Jul 17 18:17:13.097895 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:13.097874 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-139859116010496" event={"ID":"826cbbc4-fbeb-4faa-b4bf-87762ee8f65c","Type":"ContainerStarted","Data":"20ed9321005bf70f9e3c3f3983a8b348feb173d646c65c4149f52c3d4636ba91"} Jul 17 18:17:13.110275 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:13.110248 2553 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7681-139859116010496" oldPodUID="63cf7969-7e26-46d3-b853-b53ee2cd0fde" podUID="826cbbc4-fbeb-4faa-b4bf-87762ee8f65c" Jul 17 18:17:13.462984 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:13.462904 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63cf7969-7e26-46d3-b853-b53ee2cd0fde" path="/var/lib/kubelet/pods/63cf7969-7e26-46d3-b853-b53ee2cd0fde/volumes" Jul 17 18:17:14.104793 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:14.104763 2553 generic.go:356] "Generic (PLEG): container finished" podID="826cbbc4-fbeb-4faa-b4bf-87762ee8f65c" containerID="77931561f3dbb31ec517e922a9084268c1b0ee845182e42ac0c81e8bdb3d99d7" exitCode=0 Jul 17 18:17:14.104968 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:14.104861 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-139859116010496" event={"ID":"826cbbc4-fbeb-4faa-b4bf-87762ee8f65c","Type":"ContainerDied","Data":"77931561f3dbb31ec517e922a9084268c1b0ee845182e42ac0c81e8bdb3d99d7"} Jul 17 18:17:15.233658 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:15.233637 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116010496" Jul 17 18:17:15.253184 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:15.253164 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7681-139859116010496_826cbbc4-fbeb-4faa-b4bf-87762ee8f65c/test-curl-7681-139859116010496/0.log" Jul 17 18:17:15.277826 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:15.277796 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7681-139859116010496"] Jul 17 18:17:15.281271 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:15.281249 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7681-139859116010496"] Jul 17 18:17:15.400453 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:15.400400 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkqx9\" (UniqueName: \"kubernetes.io/projected/826cbbc4-fbeb-4faa-b4bf-87762ee8f65c-kube-api-access-zkqx9\") pod \"826cbbc4-fbeb-4faa-b4bf-87762ee8f65c\" (UID: \"826cbbc4-fbeb-4faa-b4bf-87762ee8f65c\") " Jul 17 18:17:15.402342 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:15.402318 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826cbbc4-fbeb-4faa-b4bf-87762ee8f65c-kube-api-access-zkqx9" (OuterVolumeSpecName: "kube-api-access-zkqx9") pod "826cbbc4-fbeb-4faa-b4bf-87762ee8f65c" (UID: "826cbbc4-fbeb-4faa-b4bf-87762ee8f65c"). InnerVolumeSpecName "kube-api-access-zkqx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:17:15.408232 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:15.408211 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7681-139859116010640"] Jul 17 18:17:15.408803 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:15.408786 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="826cbbc4-fbeb-4faa-b4bf-87762ee8f65c" containerName="test-curl-7681-139859116010496" Jul 17 18:17:15.408841 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:15.408808 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="826cbbc4-fbeb-4faa-b4bf-87762ee8f65c" containerName="test-curl-7681-139859116010496" Jul 17 18:17:15.408926 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:15.408913 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="826cbbc4-fbeb-4faa-b4bf-87762ee8f65c" containerName="test-curl-7681-139859116010496" Jul 17 18:17:15.412232 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:15.412216 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116010640" Jul 17 18:17:15.420325 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:15.420295 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-139859116010640"] Jul 17 18:17:15.463055 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:15.463029 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="826cbbc4-fbeb-4faa-b4bf-87762ee8f65c" path="/var/lib/kubelet/pods/826cbbc4-fbeb-4faa-b4bf-87762ee8f65c/volumes" Jul 17 18:17:15.501709 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:15.501684 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlxxz\" (UniqueName: \"kubernetes.io/projected/54ca0cff-9dff-4f6f-9a85-83875cf2bc4e-kube-api-access-dlxxz\") pod \"test-curl-7681-139859116010640\" (UID: \"54ca0cff-9dff-4f6f-9a85-83875cf2bc4e\") " pod="opendatahub/test-curl-7681-139859116010640" Jul 17 18:17:15.501808 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:15.501793 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkqx9\" (UniqueName: \"kubernetes.io/projected/826cbbc4-fbeb-4faa-b4bf-87762ee8f65c-kube-api-access-zkqx9\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:17:15.602837 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:15.602817 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlxxz\" (UniqueName: \"kubernetes.io/projected/54ca0cff-9dff-4f6f-9a85-83875cf2bc4e-kube-api-access-dlxxz\") pod \"test-curl-7681-139859116010640\" (UID: \"54ca0cff-9dff-4f6f-9a85-83875cf2bc4e\") " pod="opendatahub/test-curl-7681-139859116010640" Jul 17 18:17:15.611070 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:15.611050 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlxxz\" (UniqueName: \"kubernetes.io/projected/54ca0cff-9dff-4f6f-9a85-83875cf2bc4e-kube-api-access-dlxxz\") pod \"test-curl-7681-139859116010640\" (UID: \"54ca0cff-9dff-4f6f-9a85-83875cf2bc4e\") " pod="opendatahub/test-curl-7681-139859116010640" Jul 17 18:17:15.721707 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:15.721640 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116010640" Jul 17 18:17:15.844128 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:15.844103 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-139859116010640"] Jul 17 18:17:15.845284 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:17:15.845257 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54ca0cff_9dff_4f6f_9a85_83875cf2bc4e.slice/crio-e82adaf07fa0bd53afd2955c57001c176aa712e9e984ca14fa9f4e5e32b61c09 WatchSource:0}: Error finding container e82adaf07fa0bd53afd2955c57001c176aa712e9e984ca14fa9f4e5e32b61c09: Status 404 returned error can't find the container with id e82adaf07fa0bd53afd2955c57001c176aa712e9e984ca14fa9f4e5e32b61c09 Jul 17 18:17:16.115534 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.115514 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116010496" Jul 17 18:17:16.115657 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.115573 2553 scope.go:117] "RemoveContainer" containerID="77931561f3dbb31ec517e922a9084268c1b0ee845182e42ac0c81e8bdb3d99d7" Jul 17 18:17:16.116861 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.116822 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-139859116010640" event={"ID":"54ca0cff-9dff-4f6f-9a85-83875cf2bc4e","Type":"ContainerStarted","Data":"e82adaf07fa0bd53afd2955c57001c176aa712e9e984ca14fa9f4e5e32b61c09"} Jul 17 18:17:16.435008 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.434916 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" podUID="c6d0b0c3-202e-4033-8021-42bcb6ab4cdd" containerName="main" containerID="cri-o://6824deea9db961f7f1f473f04a0ab594480c16ce6630ebc367b34cbb22caa6d7" gracePeriod=45 Jul 17 18:17:16.692313 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.692257 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:17:16.812660 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.812636 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-tls-certs\") pod \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " Jul 17 18:17:16.812812 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.812683 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-dshm\") pod \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " Jul 17 18:17:16.812812 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.812729 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-tmp-dir\") pod \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " Jul 17 18:17:16.812812 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.812757 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-home\") pod \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " Jul 17 18:17:16.812812 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.812786 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8bh5\" (UniqueName: \"kubernetes.io/projected/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-kube-api-access-h8bh5\") pod \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " Jul 17 18:17:16.813016 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.812891 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-model-cache\") pod \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\" (UID: \"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd\") " Jul 17 18:17:16.813016 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.812944 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6d0b0c3-202e-4033-8021-42bcb6ab4cdd" (UID: "c6d0b0c3-202e-4033-8021-42bcb6ab4cdd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:17:16.813156 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.813028 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-home" (OuterVolumeSpecName: "home") pod "c6d0b0c3-202e-4033-8021-42bcb6ab4cdd" (UID: "c6d0b0c3-202e-4033-8021-42bcb6ab4cdd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:17:16.813210 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.813179 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6d0b0c3-202e-4033-8021-42bcb6ab4cdd" (UID: "c6d0b0c3-202e-4033-8021-42bcb6ab4cdd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:17:16.813303 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.813285 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-tmp-dir\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:17:16.813360 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.813308 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-home\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:17:16.813360 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.813322 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-model-cache\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:17:16.814735 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.814711 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-dshm" (OuterVolumeSpecName: "dshm") pod "c6d0b0c3-202e-4033-8021-42bcb6ab4cdd" (UID: "c6d0b0c3-202e-4033-8021-42bcb6ab4cdd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:17:16.814824 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.814714 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6d0b0c3-202e-4033-8021-42bcb6ab4cdd" (UID: "c6d0b0c3-202e-4033-8021-42bcb6ab4cdd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:17:16.814824 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.814766 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-kube-api-access-h8bh5" (OuterVolumeSpecName: "kube-api-access-h8bh5") pod "c6d0b0c3-202e-4033-8021-42bcb6ab4cdd" (UID: "c6d0b0c3-202e-4033-8021-42bcb6ab4cdd"). InnerVolumeSpecName "kube-api-access-h8bh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:17:16.914578 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.914550 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8bh5\" (UniqueName: \"kubernetes.io/projected/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-kube-api-access-h8bh5\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:17:16.914578 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.914575 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-tls-certs\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:17:16.914723 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:16.914588 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd-dshm\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:17:17.124816 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.124794 2553 generic.go:356] "Generic (PLEG): container finished" podID="c6d0b0c3-202e-4033-8021-42bcb6ab4cdd" containerID="6824deea9db961f7f1f473f04a0ab594480c16ce6630ebc367b34cbb22caa6d7" exitCode=0 Jul 17 18:17:17.124959 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.124863 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" Jul 17 18:17:17.124959 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.124889 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" event={"ID":"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd","Type":"ContainerDied","Data":"6824deea9db961f7f1f473f04a0ab594480c16ce6630ebc367b34cbb22caa6d7"} Jul 17 18:17:17.124959 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.124932 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw" event={"ID":"c6d0b0c3-202e-4033-8021-42bcb6ab4cdd","Type":"ContainerDied","Data":"3184e18d8ec2e25b8fee5ff6836062fa78e133b36f2568708b7590d4937d819d"} Jul 17 18:17:17.124959 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.124956 2553 scope.go:117] "RemoveContainer" containerID="6824deea9db961f7f1f473f04a0ab594480c16ce6630ebc367b34cbb22caa6d7" Jul 17 18:17:17.126544 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.126523 2553 generic.go:356] "Generic (PLEG): container finished" podID="54ca0cff-9dff-4f6f-9a85-83875cf2bc4e" containerID="9647f375a063fbd14176c44d2c17beb6ce08a05bafd125e49d0a489e1d9d9fad" exitCode=0 Jul 17 18:17:17.126655 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.126593 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-139859116010640" event={"ID":"54ca0cff-9dff-4f6f-9a85-83875cf2bc4e","Type":"ContainerDied","Data":"9647f375a063fbd14176c44d2c17beb6ce08a05bafd125e49d0a489e1d9d9fad"} Jul 17 18:17:17.133801 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.133786 2553 scope.go:117] "RemoveContainer" containerID="6824deea9db961f7f1f473f04a0ab594480c16ce6630ebc367b34cbb22caa6d7" Jul 17 18:17:17.134059 ip-10-0-136-43 kubenswrapper[2553]: E0717 18:17:17.134039 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6824deea9db961f7f1f473f04a0ab594480c16ce6630ebc367b34cbb22caa6d7\": container with ID starting with 6824deea9db961f7f1f473f04a0ab594480c16ce6630ebc367b34cbb22caa6d7 not found: ID does not exist" containerID="6824deea9db961f7f1f473f04a0ab594480c16ce6630ebc367b34cbb22caa6d7" Jul 17 18:17:17.134147 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.134063 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6824deea9db961f7f1f473f04a0ab594480c16ce6630ebc367b34cbb22caa6d7"} err="failed to get container status \"6824deea9db961f7f1f473f04a0ab594480c16ce6630ebc367b34cbb22caa6d7\": rpc error: code = NotFound desc = could not find container \"6824deea9db961f7f1f473f04a0ab594480c16ce6630ebc367b34cbb22caa6d7\": container with ID starting with 6824deea9db961f7f1f473f04a0ab594480c16ce6630ebc367b34cbb22caa6d7 not found: ID does not exist" Jul 17 18:17:17.156262 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.156241 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw"] Jul 17 18:17:17.162598 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.162581 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-0a98d9/rate-test-model-0a98d9-kserve-666874597f-h69vw"] Jul 17 18:17:17.260461 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.260438 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" podUID="ca31ff4a-2494-48f8-9901-c449741d59a3" containerName="main" containerID="cri-o://862675ca325b05caa60121bf400130acfb312fb185e34588aa3f2368b81451a2" gracePeriod=45 Jul 17 18:17:17.464674 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.464639 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6d0b0c3-202e-4033-8021-42bcb6ab4cdd" path="/var/lib/kubelet/pods/c6d0b0c3-202e-4033-8021-42bcb6ab4cdd/volumes" Jul 17 18:17:17.507407 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.507380 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:17:17.619997 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.619969 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-model-cache\") pod \"ca31ff4a-2494-48f8-9901-c449741d59a3\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " Jul 17 18:17:17.620155 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.620034 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-dshm\") pod \"ca31ff4a-2494-48f8-9901-c449741d59a3\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " Jul 17 18:17:17.620155 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.620052 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-tmp-dir\") pod \"ca31ff4a-2494-48f8-9901-c449741d59a3\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " Jul 17 18:17:17.620155 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.620077 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-home\") pod \"ca31ff4a-2494-48f8-9901-c449741d59a3\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " Jul 17 18:17:17.620155 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.620117 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trljd\" (UniqueName: \"kubernetes.io/projected/ca31ff4a-2494-48f8-9901-c449741d59a3-kube-api-access-trljd\") pod \"ca31ff4a-2494-48f8-9901-c449741d59a3\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " Jul 17 18:17:17.620373 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.620179 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca31ff4a-2494-48f8-9901-c449741d59a3-tls-certs\") pod \"ca31ff4a-2494-48f8-9901-c449741d59a3\" (UID: \"ca31ff4a-2494-48f8-9901-c449741d59a3\") " Jul 17 18:17:17.620373 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.620239 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca31ff4a-2494-48f8-9901-c449741d59a3" (UID: "ca31ff4a-2494-48f8-9901-c449741d59a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:17:17.620373 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.620317 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca31ff4a-2494-48f8-9901-c449741d59a3" (UID: "ca31ff4a-2494-48f8-9901-c449741d59a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:17:17.620373 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.620333 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-home" (OuterVolumeSpecName: "home") pod "ca31ff4a-2494-48f8-9901-c449741d59a3" (UID: "ca31ff4a-2494-48f8-9901-c449741d59a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:17:17.620511 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.620494 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-tmp-dir\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:17:17.620511 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.620508 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-home\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:17:17.620615 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.620517 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-model-cache\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:17:17.622185 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.622160 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-dshm" (OuterVolumeSpecName: "dshm") pod "ca31ff4a-2494-48f8-9901-c449741d59a3" (UID: "ca31ff4a-2494-48f8-9901-c449741d59a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:17:17.622312 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.622293 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca31ff4a-2494-48f8-9901-c449741d59a3-kube-api-access-trljd" (OuterVolumeSpecName: "kube-api-access-trljd") pod "ca31ff4a-2494-48f8-9901-c449741d59a3" (UID: "ca31ff4a-2494-48f8-9901-c449741d59a3"). InnerVolumeSpecName "kube-api-access-trljd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:17:17.622391 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.622366 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca31ff4a-2494-48f8-9901-c449741d59a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca31ff4a-2494-48f8-9901-c449741d59a3" (UID: "ca31ff4a-2494-48f8-9901-c449741d59a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:17:17.720969 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.720912 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trljd\" (UniqueName: \"kubernetes.io/projected/ca31ff4a-2494-48f8-9901-c449741d59a3-kube-api-access-trljd\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:17:17.720969 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.720936 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca31ff4a-2494-48f8-9901-c449741d59a3-tls-certs\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:17:17.720969 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:17.720947 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca31ff4a-2494-48f8-9901-c449741d59a3-dshm\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:17:18.135539 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.135509 2553 generic.go:356] "Generic (PLEG): container finished" podID="ca31ff4a-2494-48f8-9901-c449741d59a3" containerID="862675ca325b05caa60121bf400130acfb312fb185e34588aa3f2368b81451a2" exitCode=0 Jul 17 18:17:18.135707 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.135545 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" Jul 17 18:17:18.135707 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.135593 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" event={"ID":"ca31ff4a-2494-48f8-9901-c449741d59a3","Type":"ContainerDied","Data":"862675ca325b05caa60121bf400130acfb312fb185e34588aa3f2368b81451a2"} Jul 17 18:17:18.135707 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.135640 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c" event={"ID":"ca31ff4a-2494-48f8-9901-c449741d59a3","Type":"ContainerDied","Data":"1f7d9728581a84af7c5a39fc3fada0dbedeb2728e7ce56346902bf43419cebe8"} Jul 17 18:17:18.135707 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.135662 2553 scope.go:117] "RemoveContainer" containerID="862675ca325b05caa60121bf400130acfb312fb185e34588aa3f2368b81451a2" Jul 17 18:17:18.146391 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.146370 2553 scope.go:117] "RemoveContainer" containerID="862675ca325b05caa60121bf400130acfb312fb185e34588aa3f2368b81451a2" Jul 17 18:17:18.146657 ip-10-0-136-43 kubenswrapper[2553]: E0717 18:17:18.146637 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"862675ca325b05caa60121bf400130acfb312fb185e34588aa3f2368b81451a2\": container with ID starting with 862675ca325b05caa60121bf400130acfb312fb185e34588aa3f2368b81451a2 not found: ID does not exist" containerID="862675ca325b05caa60121bf400130acfb312fb185e34588aa3f2368b81451a2" Jul 17 18:17:18.146730 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.146668 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862675ca325b05caa60121bf400130acfb312fb185e34588aa3f2368b81451a2"} err="failed to get container status \"862675ca325b05caa60121bf400130acfb312fb185e34588aa3f2368b81451a2\": rpc error: code = NotFound desc = could not find container \"862675ca325b05caa60121bf400130acfb312fb185e34588aa3f2368b81451a2\": container with ID starting with 862675ca325b05caa60121bf400130acfb312fb185e34588aa3f2368b81451a2 not found: ID does not exist" Jul 17 18:17:18.158833 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.158812 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c"] Jul 17 18:17:18.164440 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.164417 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-37d0b7/rate-test-model-37d0b7-kserve-5866dfc77f-c9f4c"] Jul 17 18:17:18.274980 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.274955 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116010640" Jul 17 18:17:18.302889 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.302842 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7681-139859116010640_54ca0cff-9dff-4f6f-9a85-83875cf2bc4e/test-curl-7681-139859116010640/0.log" Jul 17 18:17:18.348044 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.348023 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7681-139859116010640"] Jul 17 18:17:18.351485 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.351463 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7681-139859116010640"] Jul 17 18:17:18.426887 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.426810 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlxxz\" (UniqueName: \"kubernetes.io/projected/54ca0cff-9dff-4f6f-9a85-83875cf2bc4e-kube-api-access-dlxxz\") pod \"54ca0cff-9dff-4f6f-9a85-83875cf2bc4e\" (UID: \"54ca0cff-9dff-4f6f-9a85-83875cf2bc4e\") " Jul 17 18:17:18.428714 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.428696 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ca0cff-9dff-4f6f-9a85-83875cf2bc4e-kube-api-access-dlxxz" (OuterVolumeSpecName: "kube-api-access-dlxxz") pod "54ca0cff-9dff-4f6f-9a85-83875cf2bc4e" (UID: "54ca0cff-9dff-4f6f-9a85-83875cf2bc4e"). InnerVolumeSpecName "kube-api-access-dlxxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:17:18.520401 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.520375 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7681-139859116010640"] Jul 17 18:17:18.520751 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.520737 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ca0cff-9dff-4f6f-9a85-83875cf2bc4e" containerName="test-curl-7681-139859116010640" Jul 17 18:17:18.520799 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.520753 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ca0cff-9dff-4f6f-9a85-83875cf2bc4e" containerName="test-curl-7681-139859116010640" Jul 17 18:17:18.520799 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.520781 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6d0b0c3-202e-4033-8021-42bcb6ab4cdd" containerName="main" Jul 17 18:17:18.520799 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.520787 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6d0b0c3-202e-4033-8021-42bcb6ab4cdd" containerName="main" Jul 17 18:17:18.520799 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.520794 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca31ff4a-2494-48f8-9901-c449741d59a3" containerName="main" Jul 17 18:17:18.520799 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.520799 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca31ff4a-2494-48f8-9901-c449741d59a3" containerName="main" Jul 17 18:17:18.520984 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.520870 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6d0b0c3-202e-4033-8021-42bcb6ab4cdd" containerName="main" Jul 17 18:17:18.520984 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.520880 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ca0cff-9dff-4f6f-9a85-83875cf2bc4e" containerName="test-curl-7681-139859116010640" Jul 17 18:17:18.520984 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.520889 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca31ff4a-2494-48f8-9901-c449741d59a3" containerName="main" Jul 17 18:17:18.523875 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.523841 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116010640" Jul 17 18:17:18.527493 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.527466 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlxxz\" (UniqueName: \"kubernetes.io/projected/54ca0cff-9dff-4f6f-9a85-83875cf2bc4e-kube-api-access-dlxxz\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:17:18.528564 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.528543 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-139859116010640"] Jul 17 18:17:18.628571 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.628500 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2588q\" (UniqueName: \"kubernetes.io/projected/f73b384a-2af5-4e34-a390-db83acdd778f-kube-api-access-2588q\") pod \"test-curl-7681-139859116010640\" (UID: \"f73b384a-2af5-4e34-a390-db83acdd778f\") " pod="opendatahub/test-curl-7681-139859116010640" Jul 17 18:17:18.729110 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.729089 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2588q\" (UniqueName: \"kubernetes.io/projected/f73b384a-2af5-4e34-a390-db83acdd778f-kube-api-access-2588q\") pod \"test-curl-7681-139859116010640\" (UID: \"f73b384a-2af5-4e34-a390-db83acdd778f\") " pod="opendatahub/test-curl-7681-139859116010640" Jul 17 18:17:18.737560 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.737535 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2588q\" (UniqueName: \"kubernetes.io/projected/f73b384a-2af5-4e34-a390-db83acdd778f-kube-api-access-2588q\") pod \"test-curl-7681-139859116010640\" (UID: \"f73b384a-2af5-4e34-a390-db83acdd778f\") " pod="opendatahub/test-curl-7681-139859116010640" Jul 17 18:17:18.833632 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.833612 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116010640" Jul 17 18:17:18.963369 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:18.963345 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-139859116010640"] Jul 17 18:17:18.965239 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:17:18.965207 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf73b384a_2af5_4e34_a390_db83acdd778f.slice/crio-edeaae9eab66b36859079d17dc5daeb3cc7f382dc4404826e01cc60200fe75c5 WatchSource:0}: Error finding container edeaae9eab66b36859079d17dc5daeb3cc7f382dc4404826e01cc60200fe75c5: Status 404 returned error can't find the container with id edeaae9eab66b36859079d17dc5daeb3cc7f382dc4404826e01cc60200fe75c5 Jul 17 18:17:19.143712 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:19.143686 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116010640" Jul 17 18:17:19.143863 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:19.143730 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e82adaf07fa0bd53afd2955c57001c176aa712e9e984ca14fa9f4e5e32b61c09" Jul 17 18:17:19.151549 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:19.151516 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-139859116010640" event={"ID":"f73b384a-2af5-4e34-a390-db83acdd778f","Type":"ContainerStarted","Data":"edeaae9eab66b36859079d17dc5daeb3cc7f382dc4404826e01cc60200fe75c5"} Jul 17 18:17:19.161581 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:19.161558 2553 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7681-139859116010640" oldPodUID="54ca0cff-9dff-4f6f-9a85-83875cf2bc4e" podUID="f73b384a-2af5-4e34-a390-db83acdd778f" Jul 17 18:17:19.463649 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:19.463622 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54ca0cff-9dff-4f6f-9a85-83875cf2bc4e" path="/var/lib/kubelet/pods/54ca0cff-9dff-4f6f-9a85-83875cf2bc4e/volumes" Jul 17 18:17:19.464008 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:19.463994 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca31ff4a-2494-48f8-9901-c449741d59a3" path="/var/lib/kubelet/pods/ca31ff4a-2494-48f8-9901-c449741d59a3/volumes" Jul 17 18:17:20.157498 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:20.157461 2553 generic.go:356] "Generic (PLEG): container finished" podID="f73b384a-2af5-4e34-a390-db83acdd778f" containerID="7253b22a946f18045ca002847d4cb30a561157a6deb56a289d60cbd9f535faee" exitCode=0 Jul 17 18:17:20.157872 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:20.157541 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-139859116010640" event={"ID":"f73b384a-2af5-4e34-a390-db83acdd778f","Type":"ContainerDied","Data":"7253b22a946f18045ca002847d4cb30a561157a6deb56a289d60cbd9f535faee"} Jul 17 18:17:21.295642 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:21.295618 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116010640" Jul 17 18:17:21.319140 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:21.319115 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7681-139859116010640_f73b384a-2af5-4e34-a390-db83acdd778f/test-curl-7681-139859116010640/0.log" Jul 17 18:17:21.351001 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:21.350974 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7681-139859116010640"] Jul 17 18:17:21.357527 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:21.357509 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7681-139859116010640"] Jul 17 18:17:21.449509 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:21.449458 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2588q\" (UniqueName: \"kubernetes.io/projected/f73b384a-2af5-4e34-a390-db83acdd778f-kube-api-access-2588q\") pod \"f73b384a-2af5-4e34-a390-db83acdd778f\" (UID: \"f73b384a-2af5-4e34-a390-db83acdd778f\") " Jul 17 18:17:21.451331 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:21.451299 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73b384a-2af5-4e34-a390-db83acdd778f-kube-api-access-2588q" (OuterVolumeSpecName: "kube-api-access-2588q") pod "f73b384a-2af5-4e34-a390-db83acdd778f" (UID: "f73b384a-2af5-4e34-a390-db83acdd778f"). InnerVolumeSpecName "kube-api-access-2588q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:17:21.463203 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:21.463181 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f73b384a-2af5-4e34-a390-db83acdd778f" path="/var/lib/kubelet/pods/f73b384a-2af5-4e34-a390-db83acdd778f/volumes" Jul 17 18:17:21.550595 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:21.550576 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2588q\" (UniqueName: \"kubernetes.io/projected/f73b384a-2af5-4e34-a390-db83acdd778f-kube-api-access-2588q\") on node \"ip-10-0-136-43.ec2.internal\" DevicePath \"\"" Jul 17 18:17:21.935237 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:21.935183 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:22.169278 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:22.169257 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-139859116010640" Jul 17 18:17:22.169390 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:22.169309 2553 scope.go:117] "RemoveContainer" containerID="7253b22a946f18045ca002847d4cb30a561157a6deb56a289d60cbd9f535faee" Jul 17 18:17:27.904011 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:27.903981 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:28.004795 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:28.004769 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:28.219812 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:28.219732 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:28.444641 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:28.444597 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:28.938316 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:28.938279 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:31.637994 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:31.637962 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:17:35.427989 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:35.427949 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:35.498467 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:35.498441 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:35.740561 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:35.740514 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:35.957538 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:35.957505 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:42.967572 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:42.967493 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f5f54cb78-bf48t_65d884c4-b8c9-44bd-8e18-833eb3f38bd6/copy-manifests/0.log" Jul 17 18:17:42.974350 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:42.974326 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f5f54cb78-bf48t_65d884c4-b8c9-44bd-8e18-833eb3f38bd6/manager/0.log" Jul 17 18:17:45.234424 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:45.234393 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ntgm8"] Jul 17 18:17:45.405366 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:45.405339 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738535-6pq9b_ca604483-cc82-4b48-83c8-d346adc619ef/cleanup/0.log" Jul 17 18:17:45.645815 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:45.645733 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-ait-1d1f23e1-29738535-smxfk_6df39780-6eeb-4d61-aa2d-57479841ad48/cleanup/0.log" Jul 17 18:17:45.884925 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:45.884896 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-ait-ee229465-29738535-rhvxk_29dabe77-16c2-4ead-8b5d-74c6e65b7a48/cleanup/0.log" Jul 17 18:17:45.999561 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:45.999541 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-api-a-e63578-29738535-vvx6m_5148216b-3305-4cc4-8758-8fe645060236/cleanup/0.log" Jul 17 18:17:46.121493 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:46.121472 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-api-b-c9e4ef-29738535-2d2fj_c1f539ee-1d44-46e2-b2cc-20badad59414/cleanup/0.log" Jul 17 18:17:46.361982 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:46.361916 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-derive-84841c77-29738535-qcq6w_f17809e6-2524-480d-88ec-dbfa620d7fce/cleanup/0.log" Jul 17 18:17:46.477402 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:46.477380 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-inf-a-72cf7b-29738535-l4c86_8efd1900-9837-4052-a280-e028c740cf21/cleanup/0.log" Jul 17 18:17:46.594359 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:46.594340 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-inf-b-3145cb-29738535-8pqsh_207d5598-707a-499a-bf0f-e9a1258a6106/cleanup/0.log" Jul 17 18:17:46.852696 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:46.852659 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-migrate-24f8a1bf-29738535-zcpsl_bce6784f-2bd7-4306-8bad-fab1e6ed32da/cleanup/0.log" Jul 17 18:17:46.969739 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:46.969716 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-mt-4355f0b1-29738535-mnvgq_a7078029-1712-4365-af96-6f3da763995e/cleanup/0.log" Jul 17 18:17:47.086034 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:47.086008 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-mt-7548b62e-29738535-vl76c_6617328d-6a6e-4b1d-a4c0-302bcbd2d106/cleanup/0.log" Jul 17 18:17:47.204806 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:47.204734 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-a-0a98d9-29738535-hz757_776bc04a-aaf6-4276-9008-880027518b41/cleanup/0.log" Jul 17 18:17:47.332436 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:47.332407 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-b-37d0b7-29738535-j8kdn_2014f2e6-cd15-44c2-b891-fe4be8b13f50/cleanup/0.log" Jul 17 18:17:49.083546 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:49.083515 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-mzljg_e032c1ee-dd72-4343-bafb-1a5ff6c0f538/manager/0.log" Jul 17 18:17:49.202208 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:49.202183 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-pxfjc_2a4854fc-38d1-435f-aaad-181f75bb9cea/kuadrant-console-plugin/0.log" Jul 17 18:17:49.467479 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:49.467407 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-76s4p_bd73c075-07c5-4827-a767-9101f18fab41/manager/0.log" Jul 17 18:17:49.587521 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:49.587495 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-ntgm8_fc1ecae2-611f-440a-8044-fa0db5d0dc24/limitador/0.log" Jul 17 18:17:50.344134 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:50.344105 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-56969c4fd6-2tqjl_f0abe360-abcf-4529-82d4-f37d8caf0141/kube-auth-proxy/0.log" Jul 17 18:17:51.960628 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:51.960596 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-lknrt_91c1e3c1-87f1-4c3a-966b-2950163f1edd/main/0.log" Jul 17 18:17:57.726125 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:57.726093 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ncz5z/must-gather-fwfnt"] Jul 17 18:17:57.726503 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:57.726427 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f73b384a-2af5-4e34-a390-db83acdd778f" containerName="test-curl-7681-139859116010640" Jul 17 18:17:57.726503 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:57.726439 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f73b384a-2af5-4e34-a390-db83acdd778f" containerName="test-curl-7681-139859116010640" Jul 17 18:17:57.726581 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:57.726506 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f73b384a-2af5-4e34-a390-db83acdd778f" containerName="test-curl-7681-139859116010640" Jul 17 18:17:57.728805 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:57.728783 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ncz5z/must-gather-fwfnt" Jul 17 18:17:57.732162 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:57.732142 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ncz5z\"/\"default-dockercfg-9zsnx\"" Jul 17 18:17:57.732765 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:57.732746 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ncz5z\"/\"kube-root-ca.crt\"" Jul 17 18:17:57.732839 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:57.732814 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ncz5z\"/\"openshift-service-ca.crt\"" Jul 17 18:17:57.741005 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:57.740984 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ncz5z/must-gather-fwfnt"] Jul 17 18:17:57.743284 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:57.743264 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:57.834952 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:57.834931 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs29q\" (UniqueName: \"kubernetes.io/projected/43034b68-db2f-4588-944e-d8662eddc759-kube-api-access-xs29q\") pod \"must-gather-fwfnt\" (UID: \"43034b68-db2f-4588-944e-d8662eddc759\") " pod="openshift-must-gather-ncz5z/must-gather-fwfnt" Jul 17 18:17:57.835078 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:57.835016 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/43034b68-db2f-4588-944e-d8662eddc759-must-gather-output\") pod \"must-gather-fwfnt\" (UID: \"43034b68-db2f-4588-944e-d8662eddc759\") " pod="openshift-must-gather-ncz5z/must-gather-fwfnt" Jul 17 18:17:57.842960 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:57.842939 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:57.935504 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:57.935477 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/43034b68-db2f-4588-944e-d8662eddc759-must-gather-output\") pod \"must-gather-fwfnt\" (UID: \"43034b68-db2f-4588-944e-d8662eddc759\") " pod="openshift-must-gather-ncz5z/must-gather-fwfnt" Jul 17 18:17:57.935649 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:57.935558 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs29q\" (UniqueName: \"kubernetes.io/projected/43034b68-db2f-4588-944e-d8662eddc759-kube-api-access-xs29q\") pod \"must-gather-fwfnt\" (UID: \"43034b68-db2f-4588-944e-d8662eddc759\") " pod="openshift-must-gather-ncz5z/must-gather-fwfnt" Jul 17 18:17:57.935793 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:57.935773 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/43034b68-db2f-4588-944e-d8662eddc759-must-gather-output\") pod \"must-gather-fwfnt\" (UID: \"43034b68-db2f-4588-944e-d8662eddc759\") " pod="openshift-must-gather-ncz5z/must-gather-fwfnt" Jul 17 18:17:57.945855 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:57.945828 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs29q\" (UniqueName: \"kubernetes.io/projected/43034b68-db2f-4588-944e-d8662eddc759-kube-api-access-xs29q\") pod \"must-gather-fwfnt\" (UID: \"43034b68-db2f-4588-944e-d8662eddc759\") " pod="openshift-must-gather-ncz5z/must-gather-fwfnt" Jul 17 18:17:58.037936 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:58.037869 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ncz5z/must-gather-fwfnt" Jul 17 18:17:58.179060 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:58.179037 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ncz5z/must-gather-fwfnt"] Jul 17 18:17:58.181158 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:17:58.181135 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43034b68_db2f_4588_944e_d8662eddc759.slice/crio-58270bc24f5eb6eebcbc566356dc19c4ad541cb3d38d2c93934880f502206cf1 WatchSource:0}: Error finding container 58270bc24f5eb6eebcbc566356dc19c4ad541cb3d38d2c93934880f502206cf1: Status 404 returned error can't find the container with id 58270bc24f5eb6eebcbc566356dc19c4ad541cb3d38d2c93934880f502206cf1 Jul 17 18:17:58.358126 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:58.358057 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ncz5z/must-gather-fwfnt" event={"ID":"43034b68-db2f-4588-944e-d8662eddc759","Type":"ContainerStarted","Data":"58270bc24f5eb6eebcbc566356dc19c4ad541cb3d38d2c93934880f502206cf1"} Jul 17 18:17:59.365673 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:59.365633 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ncz5z/must-gather-fwfnt" event={"ID":"43034b68-db2f-4588-944e-d8662eddc759","Type":"ContainerStarted","Data":"f7e97b4ba7a25b2428045446930cc798891ca91f6b40fc0879e5a9cddd8915a6"} Jul 17 18:17:59.366036 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:59.365685 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ncz5z/must-gather-fwfnt" event={"ID":"43034b68-db2f-4588-944e-d8662eddc759","Type":"ContainerStarted","Data":"80c0735131a2bc72f7cdb97c512f6864abf59517c05b9208a796e328e702209c"} Jul 17 18:17:59.382332 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:17:59.382274 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ncz5z/must-gather-fwfnt" podStartSLOduration=1.62709656 podStartE2EDuration="2.382253308s" totalImagesPullingTime="755.156748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:17:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:17:59.381725015 +0000 UTC m=+2800.500553295" watchObservedRunningTime="2026-07-17 18:17:59.382253308 +0000 UTC m=+2800.501081590" Jul 17 18:18:00.459587 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:00.459559 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-czb2s_2758fa5d-c686-4179-af0b-bc2b26fcf0eb/global-pull-secret-syncer/0.log" Jul 17 18:18:00.635568 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:00.635532 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7vkng_5177b409-481a-4b8f-ae82-c1e28eebd561/konnectivity-agent/0.log" Jul 17 18:18:00.789354 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:00.789322 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-43.ec2.internal_33e1e8a8a145842c049c2ce22b537c78/haproxy/0.log" Jul 17 18:18:05.799378 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:05.799349 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-mzljg_e032c1ee-dd72-4343-bafb-1a5ff6c0f538/manager/0.log" Jul 17 18:18:05.828384 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:05.828330 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-pxfjc_2a4854fc-38d1-435f-aaad-181f75bb9cea/kuadrant-console-plugin/0.log" Jul 17 18:18:06.063407 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:06.063324 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-76s4p_bd73c075-07c5-4827-a767-9101f18fab41/manager/0.log" Jul 17 18:18:06.086123 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:06.086096 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-ntgm8_fc1ecae2-611f-440a-8044-fa0db5d0dc24/limitador/0.log" Jul 17 18:18:08.022182 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.022129 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-7nnmr_084294fe-58d0-4e65-8405-f234886b8e9f/kube-state-metrics/0.log" Jul 17 18:18:08.040355 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.040321 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-7nnmr_084294fe-58d0-4e65-8405-f234886b8e9f/kube-rbac-proxy-main/0.log" Jul 17 18:18:08.064488 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.064456 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-7nnmr_084294fe-58d0-4e65-8405-f234886b8e9f/kube-rbac-proxy-self/0.log" Jul 17 18:18:08.091172 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.091110 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6f8f95d894-jn6s5_93703e27-00df-46bb-a91b-c09de578a2f7/metrics-server/0.log" Jul 17 18:18:08.132542 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.132513 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-6ft8n_59604b04-8411-4c2b-98db-db26c020c9fa/monitoring-plugin/0.log" Jul 17 18:18:08.306701 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.306652 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xrsqr_5e65b113-df6f-45da-a128-5fa48d45f304/node-exporter/0.log" Jul 17 18:18:08.330596 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.330562 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xrsqr_5e65b113-df6f-45da-a128-5fa48d45f304/kube-rbac-proxy/0.log" Jul 17 18:18:08.351440 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.351410 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xrsqr_5e65b113-df6f-45da-a128-5fa48d45f304/init-textfile/0.log" Jul 17 18:18:08.453559 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.453521 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-kwgnz_7864aa4a-1e4e-449c-ab69-ac18ec25a4d6/kube-rbac-proxy-main/0.log" Jul 17 18:18:08.476576 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.476457 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-kwgnz_7864aa4a-1e4e-449c-ab69-ac18ec25a4d6/kube-rbac-proxy-self/0.log" Jul 17 18:18:08.496267 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.496236 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-kwgnz_7864aa4a-1e4e-449c-ab69-ac18ec25a4d6/openshift-state-metrics/0.log" Jul 17 18:18:08.529325 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.529287 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc697c1e-495a-4c6a-9daf-8c8b6670e4d7/prometheus/0.log" Jul 17 18:18:08.552480 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.552450 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc697c1e-495a-4c6a-9daf-8c8b6670e4d7/config-reloader/0.log" Jul 17 18:18:08.573211 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.573190 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc697c1e-495a-4c6a-9daf-8c8b6670e4d7/thanos-sidecar/0.log" Jul 17 18:18:08.594654 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.594627 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc697c1e-495a-4c6a-9daf-8c8b6670e4d7/kube-rbac-proxy-web/0.log" Jul 17 18:18:08.614672 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.614646 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc697c1e-495a-4c6a-9daf-8c8b6670e4d7/kube-rbac-proxy/0.log" Jul 17 18:18:08.640080 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.640027 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc697c1e-495a-4c6a-9daf-8c8b6670e4d7/kube-rbac-proxy-thanos/0.log" Jul 17 18:18:08.663358 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.663297 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc697c1e-495a-4c6a-9daf-8c8b6670e4d7/init-config-reloader/0.log" Jul 17 18:18:08.687986 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.687961 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-cb46q_73e0afbd-63ea-4c9a-ae7b-3db773f2605c/prometheus-operator/0.log" Jul 17 18:18:08.705056 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.705026 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-cb46q_73e0afbd-63ea-4c9a-ae7b-3db773f2605c/kube-rbac-proxy/0.log" Jul 17 18:18:08.773013 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.772924 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-66ffb784d-rg7z6_f70096da-1921-4a36-b12b-082bcdf6d55d/telemeter-client/0.log" Jul 17 18:18:08.806649 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.806625 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-66ffb784d-rg7z6_f70096da-1921-4a36-b12b-082bcdf6d55d/reload/0.log" Jul 17 18:18:08.826655 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.826628 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-66ffb784d-rg7z6_f70096da-1921-4a36-b12b-082bcdf6d55d/kube-rbac-proxy/0.log" Jul 17 18:18:08.853354 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.853324 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bf948c56c-shknk_faa4f800-8ebb-4d34-9920-41545b5a3b4b/thanos-query/0.log" Jul 17 18:18:08.878119 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.878086 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bf948c56c-shknk_faa4f800-8ebb-4d34-9920-41545b5a3b4b/kube-rbac-proxy-web/0.log" Jul 17 18:18:08.897617 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.897560 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bf948c56c-shknk_faa4f800-8ebb-4d34-9920-41545b5a3b4b/kube-rbac-proxy/0.log" Jul 17 18:18:08.918371 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.918344 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bf948c56c-shknk_faa4f800-8ebb-4d34-9920-41545b5a3b4b/prom-label-proxy/0.log" Jul 17 18:18:08.937817 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.937794 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bf948c56c-shknk_faa4f800-8ebb-4d34-9920-41545b5a3b4b/kube-rbac-proxy-rules/0.log" Jul 17 18:18:08.959721 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:08.959698 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bf948c56c-shknk_faa4f800-8ebb-4d34-9920-41545b5a3b4b/kube-rbac-proxy-metrics/0.log" Jul 17 18:18:09.898468 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:09.898434 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l"] Jul 17 18:18:09.903524 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:09.903502 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" Jul 17 18:18:09.924790 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:09.924759 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l"] Jul 17 18:18:09.981113 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:09.981077 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mkdk\" (UniqueName: \"kubernetes.io/projected/2f6d2b75-dba7-474f-9a5a-274885e13f65-kube-api-access-7mkdk\") pod \"perf-node-gather-daemonset-wtt6l\" (UID: \"2f6d2b75-dba7-474f-9a5a-274885e13f65\") " pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" Jul 17 18:18:09.981286 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:09.981133 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2f6d2b75-dba7-474f-9a5a-274885e13f65-podres\") pod \"perf-node-gather-daemonset-wtt6l\" (UID: \"2f6d2b75-dba7-474f-9a5a-274885e13f65\") " pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" Jul 17 18:18:09.981286 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:09.981177 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f6d2b75-dba7-474f-9a5a-274885e13f65-sys\") pod \"perf-node-gather-daemonset-wtt6l\" (UID: \"2f6d2b75-dba7-474f-9a5a-274885e13f65\") " pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" Jul 17 18:18:09.981286 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:09.981199 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2f6d2b75-dba7-474f-9a5a-274885e13f65-proc\") pod \"perf-node-gather-daemonset-wtt6l\" (UID: \"2f6d2b75-dba7-474f-9a5a-274885e13f65\") " pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" Jul 17 18:18:09.981404 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:09.981361 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f6d2b75-dba7-474f-9a5a-274885e13f65-lib-modules\") pod \"perf-node-gather-daemonset-wtt6l\" (UID: \"2f6d2b75-dba7-474f-9a5a-274885e13f65\") " pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" Jul 17 18:18:10.081993 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:10.081949 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mkdk\" (UniqueName: \"kubernetes.io/projected/2f6d2b75-dba7-474f-9a5a-274885e13f65-kube-api-access-7mkdk\") pod \"perf-node-gather-daemonset-wtt6l\" (UID: \"2f6d2b75-dba7-474f-9a5a-274885e13f65\") " pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" Jul 17 18:18:10.082255 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:10.082236 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2f6d2b75-dba7-474f-9a5a-274885e13f65-podres\") pod \"perf-node-gather-daemonset-wtt6l\" (UID: \"2f6d2b75-dba7-474f-9a5a-274885e13f65\") " pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" Jul 17 18:18:10.082398 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:10.082384 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f6d2b75-dba7-474f-9a5a-274885e13f65-sys\") pod \"perf-node-gather-daemonset-wtt6l\" (UID: \"2f6d2b75-dba7-474f-9a5a-274885e13f65\") " pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" Jul 17 18:18:10.082584 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:10.082563 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2f6d2b75-dba7-474f-9a5a-274885e13f65-proc\") pod \"perf-node-gather-daemonset-wtt6l\" (UID: \"2f6d2b75-dba7-474f-9a5a-274885e13f65\") " pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" Jul 17 18:18:10.082820 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:10.082792 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f6d2b75-dba7-474f-9a5a-274885e13f65-lib-modules\") pod \"perf-node-gather-daemonset-wtt6l\" (UID: \"2f6d2b75-dba7-474f-9a5a-274885e13f65\") " pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" Jul 17 18:18:10.083066 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:10.083047 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f6d2b75-dba7-474f-9a5a-274885e13f65-lib-modules\") pod \"perf-node-gather-daemonset-wtt6l\" (UID: \"2f6d2b75-dba7-474f-9a5a-274885e13f65\") " pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" Jul 17 18:18:10.083276 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:10.083252 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f6d2b75-dba7-474f-9a5a-274885e13f65-sys\") pod \"perf-node-gather-daemonset-wtt6l\" (UID: \"2f6d2b75-dba7-474f-9a5a-274885e13f65\") " pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" Jul 17 18:18:10.083380 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:10.083363 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2f6d2b75-dba7-474f-9a5a-274885e13f65-podres\") pod \"perf-node-gather-daemonset-wtt6l\" (UID: \"2f6d2b75-dba7-474f-9a5a-274885e13f65\") " pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" Jul 17 18:18:10.083432 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:10.083405 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2f6d2b75-dba7-474f-9a5a-274885e13f65-proc\") pod \"perf-node-gather-daemonset-wtt6l\" (UID: \"2f6d2b75-dba7-474f-9a5a-274885e13f65\") " pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" Jul 17 18:18:10.095026 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:10.094970 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mkdk\" (UniqueName: \"kubernetes.io/projected/2f6d2b75-dba7-474f-9a5a-274885e13f65-kube-api-access-7mkdk\") pod \"perf-node-gather-daemonset-wtt6l\" (UID: \"2f6d2b75-dba7-474f-9a5a-274885e13f65\") " pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" Jul 17 18:18:10.239951 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:10.239910 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" Jul 17 18:18:10.427581 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:10.427528 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l"] Jul 17 18:18:10.430862 ip-10-0-136-43 kubenswrapper[2553]: W0717 18:18:10.430809 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2f6d2b75_dba7_474f_9a5a_274885e13f65.slice/crio-2d4f7941693db66b43c3ad5fbd93e8abea0e795f64ceb1c4592fbdf4fdf42bb3 WatchSource:0}: Error finding container 2d4f7941693db66b43c3ad5fbd93e8abea0e795f64ceb1c4592fbdf4fdf42bb3: Status 404 returned error can't find the container with id 2d4f7941693db66b43c3ad5fbd93e8abea0e795f64ceb1c4592fbdf4fdf42bb3 Jul 17 18:18:10.460835 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:10.460593 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" event={"ID":"2f6d2b75-dba7-474f-9a5a-274885e13f65","Type":"ContainerStarted","Data":"2d4f7941693db66b43c3ad5fbd93e8abea0e795f64ceb1c4592fbdf4fdf42bb3"} Jul 17 18:18:11.470701 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:11.470665 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" event={"ID":"2f6d2b75-dba7-474f-9a5a-274885e13f65","Type":"ContainerStarted","Data":"d3840e6ec103b006c0cdf981c666be978efa77b2db7f7417d9ad805898f7c85f"} Jul 17 18:18:11.471186 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:11.470732 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" Jul 17 18:18:11.491238 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:11.491184 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" podStartSLOduration=2.491167918 podStartE2EDuration="2.491167918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:18:11.488505943 +0000 UTC m=+2812.607334225" watchObservedRunningTime="2026-07-17 18:18:11.491167918 +0000 UTC m=+2812.609996198" Jul 17 18:18:12.749090 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:12.749058 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-phcbk_66ba0e1c-5d64-45ba-8db5-69fbd6715a14/dns/0.log" Jul 17 18:18:12.769292 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:12.769271 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-phcbk_66ba0e1c-5d64-45ba-8db5-69fbd6715a14/kube-rbac-proxy/0.log" Jul 17 18:18:12.836587 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:12.836559 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7w7jr_bd6be5c2-6170-4336-9626-0e499a629268/dns-node-resolver/0.log" Jul 17 18:18:13.397096 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:13.397067 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-56bb5f997f-5jxmw_5f7a3c2f-74e0-4ae2-982a-5abf88bf41af/registry/0.log" Jul 17 18:18:13.467391 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:13.467365 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xx4rh_8f7f37cc-005b-44d5-a728-90cdb84b5d22/node-ca/0.log" Jul 17 18:18:14.448622 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:14.448600 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-56969c4fd6-2tqjl_f0abe360-abcf-4529-82d4-f37d8caf0141/kube-auth-proxy/0.log" Jul 17 18:18:15.251526 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:15.251493 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8jchv_8d6c365e-40fc-4cee-ad65-e984c0686bfe/serve-healthcheck-canary/0.log" Jul 17 18:18:15.917777 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:15.917748 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mhq7k_5657e768-43ad-48c2-b520-adf83fbab4d4/kube-rbac-proxy/0.log" Jul 17 18:18:15.938669 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:15.938643 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mhq7k_5657e768-43ad-48c2-b520-adf83fbab4d4/exporter/0.log" Jul 17 18:18:15.958481 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:15.958459 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mhq7k_5657e768-43ad-48c2-b520-adf83fbab4d4/extractor/0.log" Jul 17 18:18:17.492962 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:17.492493 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ncz5z/perf-node-gather-daemonset-wtt6l" Jul 17 18:18:18.630898 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:18.630864 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f5f54cb78-bf48t_65d884c4-b8c9-44bd-8e18-833eb3f38bd6/manager/0.log" Jul 17 18:18:18.654031 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:18.653881 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f5f54cb78-bf48t_65d884c4-b8c9-44bd-8e18-833eb3f38bd6/copy-manifests/0.log" Jul 17 18:18:20.081334 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:20.081304 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-76cbd9fc7d-lnhf4_cb94d097-c65f-482f-ab4f-cbb930048f6c/manager/0.log" Jul 17 18:18:20.405216 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:20.405133 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-56969c4fd6-2tqjl_f0abe360-abcf-4529-82d4-f37d8caf0141/kube-auth-proxy/0.log" Jul 17 18:18:21.813634 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:21.813609 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-56969c4fd6-2tqjl_f0abe360-abcf-4529-82d4-f37d8caf0141/kube-auth-proxy/0.log" Jul 17 18:18:26.497519 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:26.497491 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-krv7b_2fb88f09-39c9-4e4b-91c4-7c3d21b071ba/kube-multus-additional-cni-plugins/0.log" Jul 17 18:18:26.518339 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:26.518313 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-krv7b_2fb88f09-39c9-4e4b-91c4-7c3d21b071ba/egress-router-binary-copy/0.log" Jul 17 18:18:26.540049 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:26.540027 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-krv7b_2fb88f09-39c9-4e4b-91c4-7c3d21b071ba/cni-plugins/0.log" Jul 17 18:18:26.564120 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:26.564097 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-krv7b_2fb88f09-39c9-4e4b-91c4-7c3d21b071ba/bond-cni-plugin/0.log" Jul 17 18:18:26.588200 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:26.588179 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-krv7b_2fb88f09-39c9-4e4b-91c4-7c3d21b071ba/routeoverride-cni/0.log" Jul 17 18:18:26.609031 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:26.609011 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-krv7b_2fb88f09-39c9-4e4b-91c4-7c3d21b071ba/whereabouts-cni-bincopy/0.log" Jul 17 18:18:26.633508 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:26.633486 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-krv7b_2fb88f09-39c9-4e4b-91c4-7c3d21b071ba/whereabouts-cni/0.log" Jul 17 18:18:26.669237 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:26.669217 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-v5sbn_b54d03ca-d903-45ac-b3e5-c4c70d5574cf/kube-multus/0.log" Jul 17 18:18:26.691667 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:26.691648 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fvs2m_8a107e73-1e71-4f81-9751-d19706139fad/network-metrics-daemon/0.log" Jul 17 18:18:26.712394 ip-10-0-136-43 kubenswrapper[2553]: I0717 18:18:26.712376 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fvs2m_8a107e73-1e71-4f81-9751-d19706139fad/kube-rbac-proxy/0.log"